I believe one workaround here would be for the JSONCodable macro to construct a BlogPostBuilder type behind the scenes. The BlogPostBuilder then returns a legit BlogPost from its values.
This may be splitting hairs but, present-day, this scenario will actually throw an error and not crash (assuming no try!). The CodingKey's init?(stringValue: String) will fail with an unknown string. This cascades to the KeyedDecodingContainer used to decode the enum failing because it will contain zero valid keys. Or, if the enum is already RawRepresentable, the init?(rawValue: RawValue) will fail and also trigger a thrown error.
But point taken that this is a common case that needs consideration. Should UnknownCaseRepresentable itself be something that the standard library provides for ALL uses of these kinds of enums? I don't think it'd be entirely unheard to need this functionality outside of decoding.
I think that's what the Visitor is? It collects all the values from the decoder, storing them in (optional) local variables. Then it constructs the BlogPost from them (or throws an error if not all the required properties are present).
1 Like
Also, I think this doesn't play well with Swift initialization rules either. When a let property is initialized with a default value like this, another initializer cannot provide an alternative value—because it's already set before the initializer runs, and a let can only be set once. (var works though).
It's also potentially unfriendly to macros. If a client writes this perfectly valid Swift:
struct Foo {
var foo = "bar"
}
… it becomes difficult (if not impossible in the general case) for the macro to determine the type of the property, which it needs in order to tell the decoder what type to expect for foo.
Yes of course. I guess what I was thinking of would be applicable to var properties with default values. Do I have it right this time that the default constructor in this case uses the default value parameters in the initializer allowing them to be omitted?
So what I'm thinking of is a struct like this:
@JSONCodable
struct BlogPost {
let title: String
let subtitle: String?
@CodingKey("date_published") @CodingFormat(.iso8601)
let publishDate: Date
let body: String
var tags: [String] = ["foo"] // instead of @CodingDefault(["foo"])
}
(I see only after hitting the button that @kperryua was thinking along the same lines)
I think the visitor function could be done with a potential combinatorial explosion at the end that might be fine for a macro:
struct Visitor: JSONDecodingStructVisitor {
typealias DecodedValue = BlogPost
func visit(decoder: inout JSONDecoder2.StructDecoder) throws -> BlogPost {
var title_tmp: String?
var subtitle_tmp: String?
var publishDate_tmp: Date?
var body_tmp: String?
var tags_tmp: [String]
while let field = try decoder.nextField(CodingFields.self) {
switch field {
case .title: title_tmp = try decoder.decodeValue(String.self)
case .subtitle: subtitle_tmp = try decoder.decodeValue(String.self)
case .publishDate:
let formatted = try decoder.decodeValue(String.self)
publishDate_tmp = try Date.ISO8601FormatStyle().parse(formatted)
case .body: body_tmp = try decoder.decodeValue(String.self)
case .tags: tags_tmp = try decoder.decodeValue([String].self)
case .unknown: try decoder.skipValue()
}
guard let title = title_tmp else { throw <missing required field error> }
let subtitle = subtitle_tmp
guard let publishDate = publishDate_tmp else { throw <missing required field error> }
guard let body = body_tmp else { throw <missing required field error> }
if let tags = tags_tmp {
return BlogPost(title: title, subtitle: subtitle, publishDate: publishDate, body: body, tags: tags)
} else {
return BlogPost(title: title, subtitle: subtitle, publishDate: publishDate, body: body)
}
}
}
Or instead setting the optional properties after initialization if there's no reason to avoid that, turning the combinations in a sequence of conditions:
var result = BlogPost(title: title, subtitle: subtitle, publishDate: publishDate, body: body)
if let tags = tags_tmp {
result.tags = tags
}
return result
Yes, this would be possible, modulo the implied property type problem I mentioned in my earlier reply. I'm slightly on the fence about whether it ought to be done though. Some people prefer let properties, for which the only solution is something like @CodingDefault(…), so that still needs to exist. Would it be confusing to have two different ways of achieving the same effect?
1 Like
I think this also cleanly solves the problem of having type definitions annotated with all these serialization specific macros. To repeat one of your points, it also solves the problem of being stuck when a type you need from another module doesn’t have the annotations you need.
1 Like
A few years ago, I believe I had started to look into this, hoping to expose some fields that I needed in order to improve my utility for formatting the existing error messages. But I believe I got bogged down trying to figure out which things were the right things in the right repo, and how I should be changing them. Perhaps I’ll take another crack at it, and post to a new thread if I run into trouble.
This is exciting! One request I have is for a @CodableIgnored macro. For example:
@JSONCodable
struct User {
@CodableIgnored
var someRuntimeSpecificField: Bool
// Other fields that will be encoded/decoded.
var lastLogin: Date
var id: String
}
Maybe this is an obvious detail, but I wanted to bring it up just in case.
4 Likes
Big +1 from me for native CBOR support. CBOR is used in mdoc / mdl (mobile driving licence) standards.
2 Likes
I would love to hear your thoughts on what “in due time” signifies 
. What are we missing? What are the difficulties?
I presume the initial design does not allow “borrowing” something like a string ref (UTF8Span maybe?) from the input buffer (assuming no escaping).
Are we sure we can address this use-case in the future? We might paint ourselves into a corner by the API that does not account for that possibility.
On the other hand, taking into consideration ~Escapable might create an inconvenient API for people who don’t care about avoiding copies/allocations (I presume most don’t, and probably shouldn’t).
To be fair, ~Escapable is not fully baked, and prototypical exploration is sure to be limited
[1] One reason I see deserializers make their core api async is to heap-allocate stack frames to avoid running out of stack on obsessively recursive (most likely malicious) payloads.
It’s trivial to craft a meg worth of [[[[[…]]]]] (or equivalent) and watch a service get DOS’d.
Especially since in the swift’s model you cannot recover a process from a panic.
One way to address this would be to provide context (akin to A user would then do so in their own implementation of the “decodable” protocol.userInfo?) which can be used to limit the recursion depth.
Scratch that. Thread/Task locals exist.
[1] I’ve heard/encountered this approach from Amos (fasterthanlime) on his serde alternative. merde/merde_core/src/metastack.rs at main · bearcove/merde · GitHub
3 Likes
It's great to see that we are starting to think about serialisation again. After gaining the experience with Codable over the years and understanding it's strengths and shortfalls I definitely think this is a topic worth visiting.
This may have been mentioned already, but something I would love to see is built in lossy deserialisation of lists with failure handling for logging. People have built extensions for this in Codable but having something built in would be great to see. This is valuable for lists of data which you know is going to change over time but you don't know how it's going to change as those decisions will be made in the future.
I've also had to fall back to using JSONSerialisation a few times over the years as it gives much greater flexibility to deal with payloads which you may not necessarily know the structure of up front, or where you may want to dig deep in to a JSON structure without having to define a bunch of coding keys or objects up front etc. Having the ability to pull data through some kind of path macro would be great to see.
3 Likes
Agreed. I'm using code based on GitHub - gonzalezreal/DefaultCodable: A convenient way to handle default values with Swift Codable types which comes up with a set of property wrappers that make Codable handle more cases. Also, some glue logic to deal better with optionals based on https://forums.swift.org/t/using-property-wrappers-with-codable/.
A new system should, at the minimum, provide all that builtin.
Yep, that's on the list—if this is decodable though, it needs some mechanism to provide a default value. (Probably via some decodable-member-wise initializer)
1 Like
Unfortunately, I have only a passing knowledge of CBOR, and zero knowledge of COSE. It sounds like support for this in a CBOR-specific package would be a great project. Not on my roadmap at this stage unfortunately, but let's make sure the patterns this design lays out remains compatible with this format! If you can lay out any specific details that set it apart from JSON / PropertyList / etc., that would be helpful!
Unfortunately, this is not presently an area of my personal expertise, so I'm hoping to either get up to speed, or that the experts in this area can chime in on what the design needs to extend support.
Being able to "borrow" a UTF8Span or RawSpan from a decoder is something that I hope to be able to support. I definitely don't want us to paint ourselves into an API corner that prevents this. And yes, I hope this is an area of "progressive disclosure" that doesn't get in the way in the way of "standard" API interactions.
As questioned above, is this new infrastructure only for json/propertylist/etc? It seems a bit unfortunate to restrict serdes core features to those only?
As questioned above, is this new infrastructure only for json/propertylist/etc? It seems a bit unfortunate to restrict serdes core features to those only?
No! It's intended to be open to as many formats as possible. Remember, we're promoting the use of format-specialized protocols, with the format-agnostic one intended for limited use on types that can only describe their encodable structure abstractly. Apart from supporting the format-agnostic protocol conformers, a hypothetical @CBORCodable macro and corresponding protocol is only expected to mirror the usage patterns and structure of JSONCodable, PropertyListCodable, etc. It will ultimately have its own entirely flexible interface. There's also expected to be some macro attribute overlap.
There are a couple of CBOR libraries that provide Codable support but the dream would be native support.
Unsurprisingly Apple wallet / pass kit uses CBOR.
The interesting thing about it is it doesn’t necessarily require a schema because each data item describes its type using tags that prefix the data.