Converting Class objects into Data to serialise as .dat file

Thanks tera, that does help, I had not thought about storing the ids of the objects in that way, and the scenario you mention reminds me of an API I used years ago, I think I will give that a try :slight_smile:

I have setup ids already on each object and use it for comparisons and look up in some scenarios.

You are correct that all this data gets converted into triangle mesh and then then gets converted into a vertex struct buffer and vertex index struct buffer, and passed off to the GPU. I still have a lot of work with regards to some of the GPU integration, but one step at a time :)

This is how it is currently looking

1 Like

Hi Simon,
you could try a sub-library of my project, able to archive and de-archive your data in binary format. It's more low-level than Codable but it's also orders of magnitude faster.

Most system types are already implemented, and it's very easy to add implementations of others anyway.

Your code becomes:

import Foundation
import simd

class Node: BinaryIOType {
	public var position : simd_float3 = [0, 0, 0]
	public var name 	: String = "Node"
	public var vertices : Array<simd_float3> = []
	
	init() {}
	
	//	Just implement these two methods of the BinaryIOType protocol:
	func write(to writer: inout BinaryWriter) throws {
		try position.write(to: &writer)
		try name.write(to: &writer)
		try vertices.write(to: &writer)
	}
	// read each variable in the same order it was written
	required init( from reader: inout BinaryReader ) throws {
		self.position	= try simd_float3(from: &reader)
		self.name		= try String(from: &reader)
		self.vertices	= try Array<simd_float3>(from: &reader)
	}
}


/// Default instance
var newClassNode = Node()
/// Modified instances
var moved_newClassNode = Node()

moved_newClassNode.name		= "Updated_Node_Position"
moved_newClassNode.position = [1, 1, 1]
moved_newClassNode.vertices = [ [0, 0, 0], [1, 1, 1], [2, 2, 2], [3, 3, 3], [4, 4, 4],
								[5, 5, 5], [6, 6, 6], [7, 7, 7], [8, 8, 8], [9, 9, 9], ]

// You can even use Data here, but [UInt8] is faster:
var data_test	= try newClassNode.binaryData() as [UInt8]
var data_test2	= try moved_newClassNode.binaryData() as [UInt8]

print( "newClassNode data: \(data_test)\n" )
print( "moved_newClassNode data: \(data_test2)\n" )

// etc....

// Of course you can recreate the value from the data:

var decoded_newClassNode 		= try Node(binaryData: data_test)
var decoded_moved_newClassNode 	= try Node(binaryData: data_test2)


extension Node: Equatable {
	static func == (lhs: Node, rhs: Node) -> Bool {
		return lhs.position == rhs.position && lhs.name == rhs.name && lhs.vertices == rhs.vertices
	}
}

print("-- EQUALITY CHECK: ----------------------------")
print("\t• decoded newClassNode equality check       = \( newClassNode == decoded_newClassNode)")
print("\t• decoded moved_newClassNode equality check = \( moved_newClassNode == decoded_moved_newClassNode)")

This is the output:

newClassNode data: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 78, 111, 100, 101, 0, 0, 0, 0, 0, 0, 0, 0, 0]

moved_newClassNode data: [0, 0, 128, 63, 0, 0, 128, 63, 0, 0, 128, 63, 85, 112, 100, 97, 116, 101, 100, 95, 78, 111, 100, 101, 95, 80, 111, 115, 105, 116, 105, 111, 110, 0, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 128, 63, 0, 0, 128, 63, 0, 0, 128, 63, 0, 0, 0, 64, 0, 0, 0, 64, 0, 0, 0, 64, 0, 0, 64, 64, 0, 0, 64, 64, 0, 0, 64, 64, 0, 0, 128, 64, 0, 0, 128, 64, 0, 0, 128, 64, 0, 0, 160, 64, 0, 0, 160, 64, 0, 0, 160, 64, 0, 0, 192, 64, 0, 0, 192, 64, 0, 0, 192, 64, 0, 0, 224, 64, 0, 0, 224, 64, 0, 0, 224, 64, 0, 0, 0, 65, 0, 0, 0, 65, 0, 0, 0, 65, 0, 0, 16, 65, 0, 0, 16, 65, 0, 0, 16, 65]

-- EQUALITY CHECK: ----------------------------
	• decoded newClassNode equality check       = true
	• decoded moved_newClassNode equality check = true

If you want to give it a try, go here, select the branch IdentitySupportForValueTypes, download the files contained in GraphCodable/Sources/GraphCodable/BinaryIO/ and add they to your project.

I concur with that observation. In some of my tests Codable based serialisation was the slowest, Mirror based serialisation was about an order of magnitude quicker than Codable and manual serialisation was an order of magnitude quicker than Mirror, making it two orders of magnitude faster than Codable.

1 Like

Two orders of magnitude:

•••••• LargeTest TEST ••••••••••••••••••••••••••
BinaryIO           7.75ms (σ/avg =  9.1%) Data size = 3603100
Codable (JSON)   730.87ms (σ/avg =  1.0%) Data size = 1662739
Codable (PLIST)  481.92ms (σ/avg =  1.0%) Data size = 1246961
•••••• LargeTest2 TEST ••••••••••••••••••••••••••
BinaryIO          14.41ms (σ/avg = 14.3%) Data size = 876808
Codable (JSON)   483.32ms (σ/avg =  0.5%) Data size = 1465132
Codable (PLIST)  491.60ms (σ/avg =  0.9%) Data size = 2411269

Thanks Loooop,

I will take a look into it :slight_smile:

I debugged my test and found one of the major contributor to slowness in the Codable based serialiser:

class MyEncoder: Encoder {
    ...
    func encode(_ value: Float) {
        ...
    }
    func encode<T: Encodable>(_ val: T) {
        try! val.encode(to: self)
    }
    ....

    private struct KeyedContainer<Key: CodingKey>: KeyedEncodingContainerProtocol {
        var myEncoder: MyEncoder
        
        func encode<T: Encodable>(_ value: T, forKey key: Key) {
            myEncoder.encode(value)
        }
        ...
    }
    ...
}

Stepping through the line "myEncoder.encode(value)" where the value passed was Float, the resulting "encode" method being called was not the specialised "encode(_ value: Float)" but generic "encode<T: Encodable>(_ val: T)" (why it is this way is discussed here). I put the workaround and interestingly that one change sped the serialised 10 times, so now my codable based serialised is just 10x slower than the manual serialiser.

1 Like

I found this great talk on a data structure that I think would work for my required needs, not sure if any of you have seen it.

This would be great to have in the Swift library, as an optional array type

Besides how value semantic rocks he's talking about a flavour of B-Tree (a data structure we know and love since 70's). Indeed a ready-made array-API-like structure like this could be useful to have in the standard library.

Another thing I'd recommend you to review is BSP trees.

1 Like

Thanks for the BSPTree suggestion, I have implemented a KD-tree for scene and object interaction, but have not looked into tying that into a way of temporarily storing undo structure. Something to look into

Update on my side: I did some testing using index look up table for all the components (vertex, edges, face) this allowed me to remove the referencing in my arrays, and allows for the arrays to be converted into Data types very easily. Can confirm that this setup works with the XOR Byte setup that I mentioned early on.

Going to look into the immer repo mentioned in the youtube video, looks like someone has added a Swift Package to it, so will give it a try.