Hi @kperryua just a friendly ping a few weeks later, so it doesn't fall out of discussion completely.
Sorry if this has already been mentioned, but one major limitation of Codable is the inability to deal with an arbitrary blob of coded content. Essentially, if you don't know the structure of something it is very difficult to "forward" that structure verbatim. The only viable solution I've found currently is to attempt decoding each value as an Int; if that fails, as a Float; if that fails as a String; etc. Not only is this ugly, but it has significant performance issues. Use cases affected by this are:
- Storing any kind of structured data which may be extended in the future (i.e. adding properties to a struct in an API). In this scenario, you'd want to store the content verbatim so that if you update your program to later handle the new properties, they already exist in your data store.
- Forwarding structured data from one API to another, as is occasionally needed when integrating different APIs or SDKs.
- There is even a performance opportunity here. For example, the JSON decoder can represent the "opaque blob of JSON" type as Data which it guarantees is escaped correctly and all brackets are balanced, and re-encoding can just be "write this data to the encoder verbatim"
3 Likes
This is indeed a big problem. However, Data is only what you'd want if you're okay paying the lazy decoding cost when you actually need to access the value. (That tradeoff depends on the performance of actual decoding.) Any serialization solution should definitely handle that case, but it's only part of the solution. In cases where it's okay materialize a fully decoded type, we need a representation, and that representation almost always depends on the format we're actually decoding. For JSON that's a pretty trivial type given there are only five primitive JSON types, and you can do that in Codable today, it's just really slow.
So, as part of any approach which allows for per-format specialization, a default Any-like type should be supported (I'm torn on whether it should be required or whether there's a reasonable fallback like Data). That is, as part of making a new JSONDecoder, I should also provide a type like AnyJSON, which gives me the materialized primitives I can use to extract values and provide other API that typical users would want. For JSON this could include stringy key path lookup and type casting or other tools.
1 Like
@hassila Apologies—I appreciate the reminder. I have several comments, including yours, on my todo this that require me to do some research, think deeply about, and respond to. Delayed, but not forgotten! 
1 Like
The OP tangentially refers to JSONPrimitive (AKA JSONValue, AnyJSON, etc.) as a convenient bridge for enabling Codable support on a format-specialized encoder/decoder via generic protocol, but I recognize it absolutely has other uses like the ones you lay out. This additional deserved a more explicit callout.
I also like the idea of "Opaque blob of JSON", though it feels like that deserves to be a separate type all on its own. It's totally doable though since we're not constrained by format-agnostic protocols which make this very difficult. (This type would be trivially JSONCodable, but not CommonCodable, which doesn't even make sense generically.)
2 Likes
Agreed with @Geordie_J.
This does sound like something that could be configurable via macro, most likely as a future direction. I think there are some concerns and tradeoffs to consider with this approach as well. For example, even though the experience isn't terribly awesome right now, if you annotate a struct with @JSONCodable that has a property that itself isn't JSONCodable, the compiler will produce an error in the generated code that that type doesn't conform and you either need to make it conform, or provide an adaptor via macro. I'm not certain that could be achieved at compile time with a generic RTTI-based implementation, as I expect the first time we really have a chance to perform a type-check is dynamically after the runtime type is discovered. Macros themselves don't really have the ability to check conformances on properties, since all they have to work with is the AST.
1 Like
@hassila Thanks for your patience
Yes, but the aim of both Codable and this next-generation version of it is still primarily focused on the ergonomics of using Swift native types (as opposed to zero-copy referencing data structures) and facilitating the translation between those types and serialized representations.
TBH, I don't have any real-world experience with Flatbuffers, but in researching the Swift implementation for it, I discovered that there are some fundamental ergonomic limitations in the interfaces of the generated no-copy types. For instance (presumably because Swift.Array can't be backed by an alternative "no-copy" backing store, as well as inherent optionality of Flatbuffer values) an array of strings ends up with an interface like
public var hasStrings: Bool { get }
public var stringsCount: Int32 { get }
public func strings(at index: Int32) -> String?
instead of the more Swift-ergonomic [String] used by Codable types. I understand that for clients that have specific needs for no-copy behaviors, these kinds of ergonomic compromises are acceptable, but that's not the compromise I think we want to make in the core of this proposal.
That said, your description of a project that generates flatbuffer schema FROM native Swift type definitions and presumably from that the no-copy representation access code sounds very interesting. It could be possible for a Flatbuffers package to provide their own macros that follow some of the conventions used by this project, from which .fbs schema is generated as well as the "glue" code that turns the no-copy structures into native Swift structures…
While this is definitely a worthy problem to tackle, I think it's firmly outside the scope of this design. Or at least, I think a better, more optimized, and more ergonomic solution is possible in a different project. Codable (both old and new) allows clients to define arbitrary high-level data structure based on Swift-native times within the framework (native data types and "syntactical" structure) of the serialization format being targeted. The kind of binary parsing you're referring to is a different beast. There is no serialization format being targeted (apart from raw binary data), nor is the set of native Swift types sufficient to describe the kinds of "primitives" often required during decoding (e.g. a 3-bit integer followed by one pad byte or whatever).
1 Like
Missed this message. This does sound incredibly useful, but I think it's a feature request that is completely orthogonal to this particular proposal. By that I mean it seems like this is something one could implement on either the existing JSONDecoder or a future one based on these new concepts.
Hi @tevelee, I appreciate your comments and ideas.
Can you define these concerns in more detail?
Your alternative syntax suggestion of @Codable(.json) presents some problems, because the expectation is that each specialized format macro will want to generate its own code centered around the concrete types used in its associated protocol.
For instance, the JSON macro in the BlogPost example from the OP generates code with references to JSON decoding specific types and functions on them, like (emphasis added):
static func decode(from decoder: inout **JSONDecoder2**) throws -> Person {
try decoder.**decodeWithStructHint**(visitor: Visitor())
}
and
func visit(decoder: inout **JSONDecoder2.StructDecoder**) throws -> BlogPost {
This isn't code that can be generically defined in terms of inout some DecoderProtocol or generically generated by a singular @Codable() macro without once again locking us in to a singular "data format" for every single serialization format. For instance, a fully featured JSON decoder ought to have the ability to let the client indicate it wants to decode a "number" with arbitrary precision (this of course, just produces a String, but it's NOT a JSON string structurally, due to the lack of quotes). However, not every serialization format even has the native concept of arbitrary precision number—binary property list for example. Hence the need for format-specific types and macros.
(It would be REALLY cool to make @Codable(.json) syntax actually work by making .json resolve to some macro-expansion-time code linked in from the owning back that gets run by the shared @Codable macro to inform, augment, or override this general macro might function. One of my concerns and problems I hope to address in the future is the difficulty and burden of defining a format-specialized macro. But unfortunately this dream functionality doesn't exist for macros today.)
It's also possible I'm misapprehending your ideas entirely, so feel free to set me straight.
I can't tell if your CodingFormat / ConversionStrategy ideas hinge off the above idea or not, but I do like the idea of an extensible means of defining conversions between types.
I have been following the thread, but I don’t remember much discussion of anything similar to ‘KeyCodingStrategies’.
The example uses snake case for a key by adding an annotation on the individual property.
I would hope that it will be possible to annotate a key coding strategy for an entire type.
2 Likes
Mild off-topic/self-promotion hidden below.
Summary
I haven't added anything new in a while and the approach could certainly be overhauled with macros, but I built DeepCodable a few years ago specifically for that use case.
2 Likes
Just thinking loud here: I wonder if extending types is the most flexible approach, because what if you aggregate the blog post from different sources that all emit JSON in a slightly different way. Right now, this means, that you have to create new types for each source and then you need to convert between them. Could this be detangled?
In the last few days, I had the pleasure to work with RegexBuilder and I am wondering if we could use result builders in a creative way for this use case as well. In the end result builders are a great way to describe entity hierarchies.
struct BlogPostStructure: JSONStructure <BlogPost> {
var body: some JSONStructure {
JSONString(\.title)
JSONString(\.subtitle)
JSONString(\.publishDate, "date_published").formatted(\.iso8601)
JSONString(\.body)
JSONArray<JSONString>(\.tags, default: [])
}
}
let blogPost = BlogPostStructure().deserialize(data)
let data = BlogPostStructure().serialize()
6 Likes
I think in this scenario a common pattern will be to have JSONDecodable struct variants for each of the concrete JSON variants you expect to find from your sources, and a separate struct that is your canonical application model type, with initializers that take each of the JSONDecodable structs.
This is a really fascinating idea though. It should still allow the us to implement the decoding via a visitor pattern, but it potentially allows us to drastically reduce code size, by virtue of turning code into data. 
I'm going to prototype this a bit to get a better feeling for its pros and cons.
One immediate concern/question I have is how the BlogPost value is actually constructed with this KeyPath-based design. Does it require the type to have a parameterless init() and var bindings for all of its decodable properties? Is that acceptable? That was an option for the macro-based approach as well, but my initial impression was that A) people like lets wherever possible, and B) initializing a struct value to a "default" state might be undefined for a given case and also a potential (minor) performance penalty. My understanding is that writing via KeyPaths is not optimally efficient either.
I tinkered around with this a little as promised, and I do think it could work. BUT, I don't think we can justify dropping the macro-based approach for this entirely. A straightforward implementation would, I believe, necessitate some amount of type erasure and/or existentials, which means that this approach cannot be used for Embedded Swift—which is an important thing to support. A parameterpack-based implementation might be possible, but I have my doubts that the compiler could produce sufficiently optimized code for that implementation in its current state.
The upshot is that I believe it should be easy to make these "Structure" types conform to the same Visitor protocols that the macro expansion uses. That means a client could easily choose to replace the macro-based Visitor implementation with one implemented by a "Structure" type if it fits their use case more optimally (e.g. they require minimal code size and are willing to trade some performance hits for that). Indeed, it might even be possible (for simple cases at least) for a client provide a fully RTTI-based Visitor implementation, which is relevant to some discussion earlier in this thread.
Thus, I think it's of primary importance to make sure we get the protocols right first to ensure they remain flexible, powerful, and efficient. The ergonomics of writing conformances to those protocols, whether by macro or whatever can be treated as a separable task.
2 Likes
Hey Kevin,
Thanks for following up and clarifying.
That is true, although I think that specific ergonomic issue could be improved by the codegen for Swift, by instead having a public var strings: FlatbuffersRandomAccessCollection<String> as the generated interface, which would be closer (but not quite) (FlatbuffersRandomAccessCollection would implement RandomAccessCollection).
That is what we do right now (outside from the flatbuffers project), using Sourcery codegen and annotations - we did in fact port it to use Swift macros, but never merged that PR due to complexity of the code compared to Sourcery and compile time issues with macros - I primarily wanted to open this discussion as we'd obviously be interested in leveraging a standard part of the language (if it was rich enough to allow us to model this), instead of doing codegen ourselves.
I would argue that "raw binary data" also is a serialisation format if it is well defined, it is just that the keys are implicit by the position of the data in the payload. If we take the 3-byte integer followed by one pad byte example, it would be problematic, unless there are hooks for custom serialization/deserialization as part of the API, in which case you could just use your own "Int24P" (int 24 bit with padding) type for that - custom serialization/deserialization of your own types would work for that, no?
2 Likes
I agree, and cool and thank you for considering and putting the time into exploring this.
To me this also seems to be the right model. In my experience pretty often it is needed to differentiate between absent and null, e.g. use default value for nullbut throw an error for absent, because absent is an error according to API documentation.
I also met cases where Optional was used in backend API for varying reasons.
Imagine a case where only true is valid, false, null or absent are treated equally. The rationale is to make size of of transferred data smaller, because true value in reality will come in 2% of objects. When there is plenty of such flags, adding false value and the name of the field itself to json for 98% of objects is overhead and intensiously increases network traffic.
An example are fields like:
wasAlreadyBought,hasSpecialPrice,isForAdultsOnly,shouldBeHighlighted,shouldBePromoted,isLimited,lowAmount– almost always falseisActivealmost always true, because they typically should not appear in search results. False values can rarely appear when databases are not synchronized yet.
All these fields are about 50% of the whole object. Having an ability to use something like Patch for describing encoding / decoding behavior instead of providing full imp. of init(from decoder: any Decoder) will be very handy.
1 Like
Hi Kevin, thanks for putting thought into this! I've been thinking about serialization in Swift for a long time and wanted to write down some thoughts. For example, I've worked extensively on the serialization layer at Airbnb and my side project, SwiftClaude has a reimplementation of Codable via macro (@ToolInput) which also supports generating JSON Schemas for the coded type. I'll use JSON here to avoid writing <format> or something like that, but this applies for all forms of serialization.
While I think adding a multi-format (as opposed to cross-format) architecture is a good step, I'm personally skeptical that we can even come up with a single @JSONCodable with the right set of tradeoffs for everyone. That's not to say we shouldn't have a @JSONCodable that works well for most cases, but rather that we should have an even lower-level API which eschews user-defined Swift types altogether and can operate on the JSON (or whatever format) directly (and interoperates nicely with JSONCodable). Practically, this would mean making JSONDecoder or something like it public (in fact, at a previous company we actually just copy-pasted the JSONDecoder source into our app to be able to work with the JSON directly). On top of something like this we could build support for more esoteric features like OpaqueJSON or streaming support.
I specifically say adding a multi-format architecture since I think Codable still has a role to play as "I want these types to be serializable but don't care how". The way I see it there is Codable for generic serialization, JSONCodable for Swift-type-based-but-customizable-JSON-serialization, and JSONDecoder for low-level-because-JSONCodable-doesn't-fit-my use-case serialization. To elaborate on a specific use case, SwiftClaude's "Codable" analog (ToolInput) also generates a JSONSchema for the serializable type. This is desirable for my use case, but would probably be too limiting for a general purpose JSON serialization mechanism, and I don't think this scenario is unique.
One final thought is that a large portion of the mechanics of Codable, JSONCodable, and SwiftClaude's ToolInput is simply mapping structs to and from tuple-like representations (enums too, though those are a bit more complicated). It would be great if we can make something like this generic and built into the language. SwiftClaude uses macros, and it seems that the version of JSONCodable you propose does too. This works, but there are a number of edge cases that need to be handled correctly (like declarations of multiple properties i.e. let a, b: Int). It would be great if we could just have a general-purpose mechanism to implement this kind of thing. This would obviously be a bigger lift but could help the broader ecosystem. Something like:
- Tuples being able to conform to protocols (this is also a general problem for serialization currently, since you can't serialize a tuple).
- For
structs that have opted in, being able to initialize from and destructure to a tuple of its properties (with some affordance for getting the names of those properties as coding keys), and auto-conforming to any protocols their tuple representation conforms to. - Some similar solution for
enums.
5 Likes
Excellent OP @kperryua!
First off, I agee that the extra boilerplate used by your hypothetical Visitor pattern won't be an issue. Almost all init(from:) and encode(to:) are generated by the compiler so no-one would even notice.
However I don't think JSON and PropertyList are the best choices to base a Codable replacement on. Fundamentally I think they are almost identical as they can be described by an intermediate set of primitives, NSJSONSerialization and NSPropertyListSerialization are a testament to that fact. I'd argue a better basis might be Protobuf which requires a schema in order to decode the structure.
Additionally, wrt intermediate primitives, I'd like us to consider splitting types from formating. As I see it Codable is primarily the task of "en-typing" data into good Swift structures (as opposed to Any everywhere). However, ever since the beginning JSONEncoder mixed both types and data transformations rather than building on top of NSJSONSerialization (or something like it with a bit more information).
For example I dug out and brushed off some old code that encodes to primitives rather than Data directly and can be used in combination with NSJSONSerialization to do things JSONEncoder on its own can not.
My repo is at GitHub - jjrscott/PrimitiveCoder and here's a basic Date handling example:
let encoder = PrimitiveEncoder()
encoder.primitiveTypes = [Date.self]
let result = try encoder.encode(Date.now)
// produces .single(2025-04-19 18:51:24 +0000)
1 Like
Thanks to @kperryua for prompting me to take another look at this. I've got a PR (Better `description` for `DecodingError` by ZevEisenberg · Pull Request #80941 · swiftlang/swift · GitHub) that does everything I knew how to improve just by conforming DecodingError to CustomStringConvertible, and ideas for more things to improve if I can find a safe way to change the public interface of DecodingError. But either way, I'd love to be involved in the error reporting portion of whatever succeeds Codable!
3 Likes