If you can encode and then decode a move-only value, does that mean it was copyable all along? 
6 Likes
I could imagine an implementation where you're able do a uniqueness check at decode time so that you can fail decoding in a case where the 'original' resource is still alive. So you can model a resource that is 'resurrectable' but still not copyable!
(See also: good ol' implementing -[UIView copy] by way of NSKeyedArchiver 
)
Oh, sure, it's an old trick. The difference here is that we're not dealing with reference types (where you can always add another reference) but unique values.
I guess: are there real-world examples of types declared ~Copyable that can be meaningfully encoded/decoded? ~Escapable? Sure, e.g. RawSpan or Span where Element: Codable. But ~Copyable?
I can imagine wanting to encode a ~Copyable value to create a snapshot that is then stored/transmitted. Decoding less so.
Indeed, although having an Encodable "snapshot" type is also an option in that case:
extension MyMoveOnlyType {
struct Snapshot: Copyable, Codable {
...
}
borrowing func snapshot() -> Snapshot {
...
}
}
This could be useful for structs that are noncopyable only because they define a deinit. For example, a struct that is backed by an allocated UnsafePointer which it deallocates in a deinit. I’d imagine this could come up when bridging C libraries that have create_foo and destroy_foo methods.
1 Like
No, it's only the decoder I would need to be ~Escapable & ~Copyable.
My SQLite library implements the connection pointer as a non-copyable struct. When querying, SQLite returns prepared statement pointers you can step through to lazily fetch matching rows, and my library models those as non-copyable and non-escapable with a borrowing lifetime on the connection.
I would love to be able to create a SQLiteRowDecoder type that takes a prepared statement and allows regular structs to initialize themselves from it. This would require the decoder to be both ~Copyable and ~Escapable (with a borrowing lifetime on the prepared statement). The way we handle this today is using a visitor pattern, so it seems like a perfect match for the API you have already proposed here.
This would not be ideal for my use case, because it would require the snapshot value to opportunistically step through the entire prepared statement and copy all rows. Even if I were to create a new decoder per row, it would still require copying all columns (which would be fine in most regular cases where a row corresponds 1:1 to a struct, but I'm using SQLite in a relatively unorthodox way where lazy decoding is key).
1 Like
Got it. Yes, the intention is to allow decoders (and encoders) themselves to be ~Escapable & ~Copyable.
2 Likes
There's an argument to be made that random number generators should have been ~Copyable (or at least @noImplicitCopy), and many of those can be saved as and restored from seeds.
But at that point are we serializing the generator or it's seed?
I use a ~Copyable wrapper around Strings that are sensitive to leaking or logging: tokens, passwords, keys. These values are stored in my KeyStore server and vended in JSON blocks via API. By making them non-Copyable (and handling CustomMirror), I can guarantee at compile time that my code never accidentally provides these sensitive values to anything other than the final consumer.
To make this trick work, I had to implement a (private) DecodableNonCopyable framework that performs a lil magic via macros and one careful pointer use. In short, it’s a hack around the current lack of support for just decoding ~Copyable types. As pointed out by other posters, Encoding ~Copyable raises semantic (existential?) questions. But Decoding doesn’t and is really useful.
1 Like
I made a post in "Using Swift" which is related to this thread:
Correction: the encoder is 40% faster ;)
In short, the Encoder type is the configuration. Due to this, the compiler obtains more optimisation opportunities. The implementation uses conforming types to Encodable as usual, its strategies are reusable and composable, it's clean and maintainable code - and has no hacks, no unsafe code.
This pattern can be applied to decoders as well.
Quick Insights
Core Architecture
- StaticJSONEncoder with compile-time configuration via generic parameters
- Protocol-based strategies using associated types (no runtime dispatch)
- Reusable state and sink design for zero-allocation hot loops
- JSONDataSink optimized for UTF-8 and minimal copies
Performance Achievements *)
Foundation: 359ms, 5469M instructions, 2000K allocations
StaticEncoder: 212ms, 3165M instructions, 1200K allocations
With reuse **) : 192ms, 2816M instructions, 800K allocations
→ 41% faster (47% with reuse)
→ 42% fewer instructions
→ 40% fewer allocations
*) 100_000 encoding passes for a simple composite (nested) struct
**) Sink, which can be cleared, and Encoder State - which will always be cleared.
The state is still a class - unfortunately, it can't be avoided that easily.
1 Like
Hello again Swift Community!
Here we are, almost a full year after this thread began. I’ve spent a lot of time since then working to make this into a reality, and I’m happy to announce that I have something concrete for you to try out!
Please take a look at GitHub - swiftlang/swift-foundation at experimental/new-codable · GitHub ! You’ll find therein the core components of this design exposed in a new target temporarily called NewCodable:
CommonEncodableandCommonDecodableprotocols. These are the format-agnostic protocols that are the closest relatives toEncodableandDecodable, but are designed in such a way that avoids many of the performance limitations of the latter.CommonEncoderandCommonDecoderprotocols. Similarly, these are format-agnostic protocols for encoders and decoders in the same vein asEncoderandDecoder, but with a better design.JSONEncodableandJSONDecodableprotocols. These are shaped similarly to theirCommoncounterparts, but with modifications that are specific to JSON, including certain performance optimizations.JSONDecoderProtocol(naming discrepancy acknowledged, but I can’t name itJSONDecoder!) with concrete implementationsJSONParserDecoderandJSONPrimitiveDecoder.JSONDirectEncoder(no protocol here yet, despite room for an alternativeJSONElementEncoderto mirrorJSONElementDecoder. Will discuss elsewhere.)- “Top-level”
NewJSONEncoderandNewJSONDecoder(names TBD, since I similarly can’t name themJSONEncoderandJSONDecoder!) JSONElementand other supporting protocols and types.
Please note that swift-foundation is acting as a convenient and temporary holding place for this project. We’ll evolve it on this branch for a while as we iterate on it and refine it. We’ll determine what the final locations for the various pieces will be as we get closer to its productization.
This is still very much a work in progress! Please excuse our dust! You’ll find plenty of TODOs scattered throughout, somewhat haphazard inlining of functions, and probably plenty of bugs and issues. I have ported over many of the tests from the existing JSON Codable-based unit tests in swift-foundation, but a few of those are still failing for various reasons.
Despite all of that, I hope some of you get an opportunity to take a look and try adopting the protocols use the new encoders/decoders with some of your own types. Let us know about any pain points or friction you’re experiencing with the API.
There are some benchmarks included (still of bespoke infrastructure) which are quite promising. When decoding twitter.json, I personally observe that JSONDecodable and JSONParserDecoder provide a throughput improvement of approximately 6x over Decodable and JSONDecoder! I hope that with community involvement we’ll be able to improve both overall quality and performance to even higher levels.
To try using the “New Codable”, change your swift-foundation dependency to
.package(
url: "https://github.com/apple/swift-foundation.git",
branch: "experimental/new-codable"
)
and import NewCodable. Swift 6.3 is required.
To run the tests, execute the following in a swift-foundation checkout of this branch:
swift test --filter NewCodableTests
To run the benchmarks:
swift test --configuration release --filter NewCodableBenchmarks
Please take note of this rough roadmap that lies ahead for this project:
- Finalize the
Common*protocols - Finalize the
JSON*APIs - Support Embedded Swift
- Add support for XML and Binary Property List
- Design and implement macros for
@JSONCodable,@PropertyListCodable,@CommonCodable, and even@Codable. - Publish a library for owners of other format-specific protocols to facilitate development of their own
@<Format>Codablemacros that use common patterns established by the above macros.
If you see an issue or have a question, please speak up! I’ll be happy to try to address any concerns. Feel free to create an Issue on the swift-foundation GitHub tagged with the new-codable label. If you want to fix something, feel free to create a Pull Request against this experimental/new-codable branch, again tagged with the new-codable label.
To help drive community involvement towards finalizing milestones #1 and #2, I will be starting threads that discuss different aspects of the prototype’s API with explanations of why that design was chosen, challenges we’re still facing, and alternatives we’ve considered. I hope to get your focused feedback in those discussions.
As a footnote, while the inspiration for the origins of this project was heavily rooted in Rust serde, I found that the general shape of the musli crate was a better fit for adapting to Swift. (It’s quite a bit more reliant on closures with inout arguments rather than consumed and reborrowed mutable references. I’ll start a discussion about that as well.)
37 Likes
It may become a requirement that JSON encoding/decoding be possible without linking to Foundation. Have you spent any time investigating moving the new API out of Foundation?
Edit:
I can't read today.
3 Likes
Looks great, @kperryua - I'm especially liking the new JSONPrimitive 
1 Like
I am excited about this!
Forgive me if I’m missing something obvious. It doesn’t look like FoundationEssentials or any of this package’s products are actually vending NewCodable to make it accessible to a consuming project. Is that an oversight?
(I’ve reposted this topic here: "New Codable" prototype available for feedback . Please direct feedback about the prototype there.)
Whoops! I fixed this on the branch just now. Thanks so much for catching that!
1 Like