I have a simple struct (nested struct of primitive types/enum) and want to load from/save to large file in chunks, sometimes in time-critical region. Right now I'm exploring my options.
I could use Codable, but JSON doesn't seem to support random access (you have to parse the entire file). I don't want to load the entire file into memory. I could separate each chunk and encode/decode them independently, but then I don't know the upper limit of the encoded data size.
I could implement BinaryEncoder/Decoder but it is a lot of work for a single-use code.
I can make use of the new SE-0245 to load data from outside storage. The only problem is that I don't know how to make sure that the memory layout is fixed across version (post ABI stability is fine, I want to use SE-0245 after all). I suppose I can use @frozen, but I'm not too sure how it works.
I think 3. looks promising. I suppose my question is Is there a way to fix a struct memory layout?. I don't need to know the layout, only that it doesn't change moving forward.
Like Karl says, the only truly reliable (public) way to guarantee the layout of structs in Swift is for it to be a C struct. You could guarantee the exported/imported layout via UnsafeMutableRaw[Buffer]Pointer and manually import/export the properties, which would be like your option #2. But that is quite a bit of work.
Do you mean to save/load as array of primitive types (Int32/Bool), then rearrange them into struct later? That’d be a bit of work, but should still be a lot less than #2.
Yes, this would necessitate some more complex logic if you have nesting or non-primitive types, but if you're only working with primitives, it should be relatively straightforward to export them via storeBytes.
I'll leave someone else to chime in on @frozen, as I've never used it and am not familiar with its semantics. Although reading SE-0260, I wouldn't really rely on it actually fixing the layout.
This attribute promises that stored instance properties within the struct will not be added,removed, or reordered , and that an enum will never add,remove, or reorder its cases (note removing, and sometimes reordering, cases can already be source breaking, not just ABI breaking). The compiler will use this for optimization purposes when compiling clients of the type. The precise set of allowed changes is defined below. The fields are not guaranteed to be laid out in declaration order. The compiler may choose to reorder fields, for example to minimize padding while satisfying alignment requirements.
and particularly
Even concrete instantiations may not have a known size or alignment. A frozen struct with a field that's a *non-*frozen, has a size that may not be known until run time.
and
The fields are not guaranteed to be laid out in declaration order. The compiler may choose to reorder fields, for example to minimize padding while satisfying alignment requirements.
Yeah I did read that. Though it seems to me that so long as I don’t edit the struct and mark them as @frozen all the way through, compiler won’t reoptimize the current layout. We could wait for someone who knows for sure to chime in, perhaps @jrose would know?
In the mean time, I’ll try your suggestions (both @Karl c-struct and your primitive array one).
Can confirm that Swift structs do not have a guaranteed layout.
It is part of the ABI at this point that if you (1) compile with library evolution support, and (2) mark the struct as @frozen, then the struct must have a known layout, and that layout happens to be "the C layout but without tail padding". But we could change that in a future version of the compiler even for frozen structs as long as existing frameworks continue to be interpreted the old way.
If you decide to go down the C struct path, keep in mind that the layout of C structures is also not guaranteed to be stable across architectures. Even if you restrict yourself to ‘sane’ architectures (no 12-bit bytes!) you still have to worry about both alignment and endianness. C has tools to help with the former, the latter means you’ll still need to ‘visit’ every primitive value in your hierarchy of nested structs.