I have compiled a program represented by a custom binary format. It lives in memory as an UnsafeRawBufferPointer, but I would like to save it as a file.myformat on disk. I'll also need a way to obtain a read-only view of the file's contents. In short:
1. UnsafeRawBufferPointer -> file.myformat
2. file.myformat -> UnsafeRawBufferPointer
What is the recommended way to go to and from a memory-mapped file in Swift?
Memory-mapped files are not supported on all platforms that Swift supports. Would you be able to clarify if you're looking for a fully cross-platform solution? Otherwise which subset of platforms would you need?
Thanks for the clarification! Iâm targeting iOS and macOS, so full cross-platform support isnât necessary at this stage. If there are any platform-specific considerations or best practices for these platforms, Iâd appreciate hearing about them.
On Apple platforms you have to be very careful with memory mapping. If the targeted file is stored on a volume that can âgo awayâ â for example, on an external drive or a network volume â then memory mapping isnât safe because accessing a file thatâs gone away triggers a machine exception [1], which isnât something you can reasonably handle [2].
Still, if you want to go down this path there are two standard APIs for mapping a file:
mmap â See the mmap man page.
Data.init(contentsOf:options:) â Using either .mappedIfSafe or .alwaysMapped option.
You go the other way using your file system API of your choice: write, fwrite, System framework, FileHandle, and so on.
IMO memory mapping is overused. I see two primary motivations here:
Unix Loreâą is that memory mapping is the only way to do no-copy reads and writes. Thatâs never been true on Apple platforms, and hasnât be true on most Unix-y platforms for decades.
Folks think itâll be convenient because they can define a language structure that maps to their data structure and then theyâre just reading and writing fields rather than doing I/O. That rarely works out as well as you might hope [3].
And memory mapping has a lot of drawbacks:
The big one is the safety issue Iâve discussed above.
If the file is large, you have to worry about address space issues on iOS.
And on 32-bit platforms [4].
If youâre streaming through a large file you end up running all your I/O through the buffer cache, which is much less efficient than doing no-copy reads and writes.
If youâre accessing the file at random, those accesses put pressure on the VM system which cause other pages to get evicted, which might result in your I/O being fast but the rest of the system suffering.
Which isnât to say that this approach is always wrong, just that folks ofter try to use it when itâs not appropriate.
Share and Enjoy
Quinn âThe Eskimo!â @ DTS @ Apple
[1] Using the terminology defined here.
[2] I talk about that problem a lot in Implementing Your Own Crash Reporter.
[3] This technique has serious problems:
It only works if you defined the language structure in C, because Swift doesnât give you a way to define the layout of its structures.
Even then, Câs structure layout in more of a shared hallucination than something defined by the standard.
And you still have to deal with alignment issues.
And byte ordering.
[4] In the Apple ecosystem there is one remaining 32-bit platform, watchOS.
Agree with all you say in general, but there are some other approaches for [3] too - e.g. we use Flatbuffers as the on-disk and on-the-wire representation for our cached data (and uses LMDB for the mmap access to it) - for our use case that is working pretty nicely (but then, we also want to do sparse access of fields for e.g. filtering and wanted a zero-copy format that allows direct field acccess without parsing the full message). That can all be done in Swift, but is admittedly a fairly special usage.
Hi, Quinn! Thanks for pointing me in the right direction. I appreciate the time you took to respond. Your knowledgeable and candid perspective is invaluable.
Unix Loreâą is that memory mapping is the only way to do no-copy reads and writes. Thatâs never been true on Apple platforms, and hasnât been true on most Unix-y platforms for decades.
My use case only requires read access and a pointer to the fileâs contents. Is there a more modern or platform-preferred alternative for zero-copy file access? As far as I can tell, Iâll avoid the memory-mapping hazards you pointed out. Thanks again for your insights!
[...] we use Flatbuffers as the on-disk and on-the-wire representation for our cached data (and use LMDB for the mmap access to it) [...] we also want to do sparse access of fields [...] and [...] zero-copy [...].
While my use case differs from Hassilaâs, Iâm also working with a zero-copy format and sparse access patterns. On the surface, memory mapping appears to be the right tool for the job. That said, Iâd be interested in reading any alternative suggestions you or others might have!
The alternative is to allocate a buffer and read data into that. Or allocate a pool of buffers and read chunks of data into those.
In terms of transferring data between a file and a buffer, the mechanism to do that varies by platform. On Apple platforms you use fcntl to set F_NOCACHE, but this is only effective under specific circumstances. A good rule of thumb is that everything should be page aligned, that is, the buffer, the transfer length, and the offset into the file.
As to whether memory mapping is the right tool for the job, there are specific show stoppers to watch out for:
If the answer to any of those is âYesâ, then Iâd stay away from memory mapping. If not, you then get to evaluate secondary criteria, profile a prototype, and so on.
Share and Enjoy
Quinn âThe Eskimo!â @ DTS @ Apple
[1] I recently discovered MAP_RESILIENT_MEDIA, which will cause the mapping to return zeroes rather than trigger a machine exception if the file goes away. You could imagine designing a file format that took advantage of that option but, sheesh, thatâs gonna be full of pitfalls.
Oh, and this was added in macOS 10.11, just about a decade ago, so a) thereâs no worries about back deployment, and b) havenât learnt about it only recently, I clearly missed a memo (-;
[2] If youâre address space constrained you could, in theory, set up one or more windows on the file and move them around. I suspect that would really hurt performance, but I must admit to having never tried it.
Good to know there's a limit. Checked just now - it's around 5GB on my iPhone.
This looks quite unsafe option to me, e.g. I'm copying a file and instead of (safely) crashing the app continues copying zeroes, then I think all is good and done and tell the other guy on the remote end to delete the original file 
it's around 5GB on my iPhone.
Cool.
But for context, I regularly saw folks bump into this limit at around the 500 MB mark, even on 64-bit devices. Thatâs because page mapping tables have a cost, and so each address space was artificially limited to reduce that cost for folks who didnât need large amounts of address space. The com.apple.developer.kernel.extended-virtual-addressing entitlement allowed you to raise that limit.
This looks quite unsafe option to me
Well, it certainly can be. I can only imagine this option being useful if the file format were designed with it in mind.
I'm copying a file
If youâre using memory mapping to copy files, youâre doing it wrong. Specifically, youâre failing at the âAre you streaming through the file from end to end?â test.
But, yeah, your general point is correct. Using this option without due care and attention could easily result in data loss.
Share and Enjoy
Quinn âThe Eskimo!â @ DTS @ Apple