How to performantly store non trivial static data

Using Swift
1

I have a file with about 200 lines of some arbitrary data thrown into a dictionary. Now it’s time to increase the size of this file and I imagine it may get close to (or exceed) 10k lines of code. What are some performant ways of storing this data. I’d like to avoid using SQL in this case since I don’t imagine that the data here will change much.

2

Have you tried JSON and was it too slow?

3

JSON pays the cost of Swift's worst-in-class decoding performance and the lack of a lazy decoder, which means the entire blob will be in memory at once. So unless you find a custom solution that can operate lazily and without Codable, the performance will be pretty bad all around.

@bostshakur You need to be more specific about what performance you're trying to optimize (compile time, runtime, memory usage, random access, etc.) and how big each of those 10k items might be. But really, a SQLite DB is probably the simplest way to get performant access to that much static data, at least until Swift can precompile such data in reasonable time.

1 Like
4

Have you tried JSON and was it too slow?

@tera Not yet. I wanted to weigh the pros and cons of each approach before committing.

which means the entire blob will be in memory at once.

@Jon_Shier I guess SQL would be the only way to avoid this. I was trying to avoid optimizing too early but your comments makes me think this isn’t premature optimization at all.

You need to be more specific about what performance you're trying to optimize (compile time, runtime, memory usage, random access, etc.)

My biggest concern here is the in-memory footprint.

and how big each of those 10k items might be.

Once I have the data model fully fleshed out I’ll return and post here.

Thanks both for your input!

5

If you haven't measured your current implementation then it's premature.

1 Like
6

Starting with JSON is easy enough that you can evaluate it and move on without much cost. Implement your data model, see how long it takes to decode (you'll likely want to ensure it's decoded off the main actor, otherwise your UI will hang a bit), and how much memory it takes.

1 Like