High performance serialization format seriously lacking

Hi, i've recently been involved in a project requiring a pretty good level of performance regarding serialization (to & from local storage), and found myself stuck between a few walls trying to use swift :

  • use a DB mapping each property of my data to a column (with or without an ORM like core data). This is nice, but becomes very cumbersome very quickly if the number of different properties or the complexity of your models becomes too large.

  • serialize my struct & classe to a "data" blob, and persist it on a file (either in a BLOB DB column, or directly in a file).

The second option lead me to try various serialization formats, from json to binary plist, to flatbuffer, to protobufs, and they seem to fall into two categories:
1- Map pretty conveniently with the Swift types & protocols (codable) and are convenient to use, but are way too slow by at least 2 orders of magnitude (taking seconds instead of tens of milliseconds on other platforms), like json & property lists. OR
2- are fast, but requires a lot of plumbing code to map to the wide range of swift types (eg: enums), with each mapping layer adding the risk of a mistake or a performance penalty. (flatbuffers & protobufs)

Are there any plan to add a high performance serialization tool in the stdlib that would map pretty decently with the rest of the library ?

2 Likes

This is just very likely to be out of scope for the standard library.

Moreover, I suspect there won't be any reasonable solution, as any stdlib implementations ought to be very generic, which quite often contrasts with the goal of being performant. The tradeoff you mention is very hard to avoid, so really your best option is to implement your serialization logic yourself if you really care so much about performance.

i hoped there was plan to maybe make the jsonencoder / decoder faster ? I've read about around the net, and it seems like swift's implementation is notoriously slow (it is too slow in my case, but i haven't compared it with other platforms personally)

Those are, again, not a part of the standard library, but Foundation, which, among other things, concerns itself quite a bit with Obj-C types, which inherently introduces more overhead than an encoder written for native Swift types would do.

FWIW, the topic has been discussed at least here and there are some (1, 2) recent PRs concerning JSON. You might want to look into the discussions and see if there's something that will be helpful, but you shouldn't hesitate to use a better implementation if you find one, since there's really nothing "standard" to the one provided by Foundation I believe.

Oh, my bad. Thanks a lot for the pointers, i completely missed the fact that it was coming from Foundation.

but requires a lot of plumbing code to map to the wide range of swift types (eg: enums), with each mapping layer

Could you expand what required plumbing code you mean? FlatBuffers would give very cheap ser&des and supports swift since a year back or so.

that's the one i'm actually using for now, but only for simple structs. Once you start going into enum with associated values, generic enums, etc, it obviously starts to fall a bit short (and i'm not blaming them).

They're officially designed to be a serialization format, they haven't (yet) provided an object-oriented (or protocol-oriented) API, even if the swift project is doing its best to try and match swift's type system.

Eg : if you design your types in a fsd file first, then use them, it's fine. But trying to map an arbitrarely complex swift structure to a flatbuffer buffer is clearly more involved than just declaring your struct Codable and do let data = FlatBufferEncoder().encode(codable)

I see. I usually would view arbitrary object graph serialization != high performance as mentioned above... but then, everyone has different frame of reference on performance matters.

Terms of Service

Privacy Policy

Cookie Policy