"New Codable" prototype available for feedback

Evolution Discussion
,
21

I think it's very nice and clean approach, and it's a defaulted (non required) parameter for types that have init(). YMMV.

I see. For that the following pseudocode should do:

jsonField.forEach { field in
    self[field] = decoder.decode(field: field)
}

where the assignment is a no-op if there's no field "field.name" in T.

As for the field type, in broad strokes, the field is like a key path (think string). Not sure about the situation with its value though, could it be generic or not.

Your other concerns are valid.

Experience tells it's hard to change the foundation layer without disturbing what's on top, such endeavours typically lead to problems with backward compatibility or waste of efforts, or much bigger scope or a combination of those. I think we saw precedents in Swift evolution where we deliberately postponed certain things to be done after other for them to be done "in order", which in this case I believe should be NewMirror -> NewCodable. I might be wrong though, or it's a matter of opinion, so please don't feel discouraged.

22

I went ahead and implemented (with some LLM help) a couple of useful JSON macros at GitHub - t089/swift-json-macros at new-codable Ā· GitHub based on this work. Not fully tested and might be broken, but all the features I wanted were possible to do quite straightforwardly.

1 Like
23

Thatā€™s awesome @t089 , thank you for doing that!

This actually coincided with this PR that @tevelee posted for inclusion directly in the swift-foundation branch. Iā€™d be happy to have a combination of efforts here to get us the best result. :grinning_face:

(Note for all consumers: these macros donā€™t necessarily represent the full extent and design of the final macros, but I think these are an excellent jumping off point for that work!)

24

Could you point me to the current limitations?
Edit: Found some info here.
However... it's somewhat out of date, for example this compiles fine on embedded:

func mayFail() throws {}

and so is this:

protocol P: AnyObject { }
func casting(object: AnyObject) {
  if let p = object as? P {
  }
}

Is it known if the remaining limitations are temporary or permanent?

I didn't follow this so it was news to me that embedded supports existential at all ā€“ thought they are fundamentally impossible unwise ā€“ just checked: indeed it does work on the nightly build, before it was failing with - error: cannot use a value of protocol type 'any P' in embedded Swift.

25

I think this is more up-to-date here, but Iā€™m reaching out to make sure itā€™s accurate. This testclearly shows existentials of *non-class-bound *protocols working.

1 Like
26

I mean supporting multiple keys for the same property to allow reuse across highly similar but different JSON data. Of course, it will also be encoded multiple times during the encoding process.

That is true. However, this merely provides a tool; whether and how to use it depends entirely on the developer.

1 Like
27

Or "past" Swift :sweat_smile: (We lost this superpower of marking individual properties with protocols.) Here's how it would be done with the help of Obj-C:

Some machinery 
import Foundation
import ObjectiveC

// part of machinery

struct PropertyInfo {
    let name: String
    let objcClass: AnyClass?
    let protocols: [Protocol]
    let typeEncoding: String

    func conforms(to proto: Protocol) -> Bool {
        if let cls = objcClass {
            return class_conformsToProtocol(cls, proto)
        }
        return false
    }

    func hasProtocol(_ proto: Protocol) -> Bool {
        let target = NSStringFromProtocol(proto)
        return protocols.contains { NSStringFromProtocol($0) == target }
    }
}

func properties(of cls: AnyClass) -> [PropertyInfo] {
    var count: UInt32 = 0
    guard let properties = class_copyPropertyList(cls, &count) else { return [] }
    defer { free(properties) }

    return (0..<Int(count)).map { I in
        let prop = properties[I]
        let name = String(cString: property_getName(prop))
        var objcClass: AnyClass? = nil
        var protocols: [Protocol] = []
        var typeEncoding = "?"

        if let attrs = property_getAttributes(prop) {
            let attrString = String(cString: attrs)
            let parts = attrString.split(separator: ",")

            if let first = parts.first, first.hasPrefix("T") {
                typeEncoding = String(first.dropFirst())
            }

            if let start = attrString.range(of: "T@\""),
               let end = attrString[start.upperBound...].firstIndex(of: "\"") {
                var rawName = String(attrString[start.upperBound..<end])

                if let openAngle = rawName.firstIndex(of: "<"),
                   let closeAngle = rawName.firstIndex(of: ">") {
                    let protoString = rawName[rawName.index(after: openAngle)..<closeAngle]
                    protocols = protoString.split(separator: ",").compactMap {
                        objc_getProtocol(String($0))
                    }
                    rawName = String(rawName[..<openAngle])
                }

                objcClass = NSClassFromString(rawName)
            }
        }

        return PropertyInfo(name: name, objcClass: objcClass, protocols: protocols, typeEncoding: typeEncoding)
    }
}

func snakeCased(_ input: String) -> String {
    var result = ""
    let chars = Array(input)
    for (i, char) in chars.enumerated() {
        if char.isUppercase {
            let nextIsLower = i + 1 < chars.count && chars[i + 1].isLowercase
            let prevIsLower = i > 0 && chars[i - 1].isLowercase
            if i > 0 && (prevIsLower || nextIsLower) {
                result += "_"
            }
            result += char.lowercased()
        } else {
            result += String(char)
        }
    }
    return result
}

This function dumps class fields, snake-casing where needed:

// part of machinery
func dumpClassSnakeCasingWhereNeeded(_ cls: AnyClass) {
    print(cls, "{")
    for prop in properties(of: cls) {
        let protos = prop.protocols.map(NSStringFromProtocol)
        let key = prop.hasProtocol(SnakeCased.self) ? snakeCased(prop.name) : prop.name
        print("   ", key)
    }
    print("}")
}
Test example 
// ---------- BridgingHeader.h
#include "objc.h"

// ---------- objc.m
#include "objc.h"

@implementation User @end

// ---------- objc.h
#include <Foundation/Foundation.h>

@protocol SnakeCased @end


// ---------- objc.h continued
@interface User: NSObject
@property NSString<SnakeCased>* userName;
@property NSString* normalText;
@end

Testing in Swift:

dumpClassSnakeCasingWhereNeeded(User.self)

outputs:

User {
    user_name
    normalText
}

Looks like this obj-c feature of attaching protocol markers to individual properties was not useless after all!

28

Those protocols each have 10 or 12 requirements for fixed-width integers. (CommonDecoder is missing Int128 and UInt128 requirements.) Did you consider using generic requirements instead?

mutating func encode(
  _ value: some FixedWidthInteger
) throws(CodingError.Encoding)

mutating func decode<T: FixedWidthInteger>(
  _ type: T.Type
) throws(CodingError.Decoding) -> T

For example:

  • XML property list and JSON would encode/decode any decimal description.

  • CBOR "preferred serialization" encodes the shortest form (e.g. Int64.zero is one byte).

29

(U)Int128 support in the protocols will come across the entire API surfaceā€”it just wasnā€™t a high priority at the moment.

I did consider `FixedWidthInteger`, but my understanding of why the old Codable API did this still exists today: Swift can produce much more efficient specialized implementations of each type with separate overloads. Of course, a concrete implementation can @specialize, but thatā€™s still not quite the same as dispatching directly to the desired implementation. Perhaps some aggressive inlining could provide reasonable solutions here, but for now I felt it best to stick with the established pattern.

(I would reach for serde or musli as further support for this approach, but of course Rust doesnā€™t have a standard trait that is equivalent of FixedWidthInteger. kotlinx.serialization also does it this way, but I have even less familiarity with that whole ecosystem.)

1 Like
30

Hello all!

Thanks for your initial interest in this project. Your feedback is critical for us bringing this to a successful conclusion.

As promised, Iā€™ve started a series of discussions about the API surface that explains and justifies various design decisions. There are also a number of open topics that need further discussion and decision making in order to reach a satisfying conclusion.

I currently have several different discussions identified and organized into GitHub issues on the swift-foundation repo. Iā€™ll flesh each one out with concrete information. The first one is here, covering all the core decoding APIs:

Please join the discussion here and especially weigh in on the open questions/discussions at the bottom of the document.

1 Like
31

I also want to encourage folks to chime in on any outstanding implementation issues, or even offer PR contributions.

You're all welcome to comment on any outstanding PRs as well.

1 Like
32

The next discussion topic has started here, covering the "field" APIs:

1 Like
33

In Swift this would require type erasing boxes and dynamic casting, which are much more expensive in Swift than Kotlin, and is also fundamentally incompatible with Embedded Swift.

Iā€™ve seen ideas bounce against this wall a few times now and Iā€™m wondering what improvements could be made to the compiler that enable a flexible API design we can live with for years to come (e.g.: in the most recent case, a design that can include integer and string based field keys without an expensive performance penalty for Embedded Swift)?

34

@kiel, I share some of the the sentiment, believe me. I'm all for finding ways to evolve the language to expand the capabilities and expressiveness of APIs. Unfortunately, I'm not a great candidate for postulating about what changes could be reasonably made to the language to accommodate what we'd like to achieve. I'd love to get more attention from the LSG on the problems we're facing to see if they have any opinions about what the language could provide.

Swift's design definitely has some characteristics to it that impede expressiveness when performance is the primary goal, including the higher costs of working with dynamic types compared to other languages. Our commitment to Embedded Swift for this project support only intensifies our constraints. That said, I think Swift also gains a lot of advantage over other languages because of its design decisions. So I think we have to strike a careful balance between trying to expand the language's capabilities and finding the best API that fits what the language provides. It can be OK that the "best" Swift serialization API won't match some aspects of the "best" Rust or Kotlin serialization API.

It is frustrating though when the definition of "best Swift API" itself can vary so much depending on the preferences or requirements of the client. The "best" == "most capable" non-Embedded Swift API might be impossible for Embedded Swift. The "best" == "highest throughput" Swift API might be the hardest to use or have the worst code size. And so on for various definitions of "best". At the end of the day, it's impossible to please everybody, so when there aren't reasonable compromises, we are often forced to choose our priorities and move forward.

I've personally spent the better part of the last two years trying to tread this path as successfully as possible. But as I said, I'm not the best person to consider what else the language could reasonably provide, so by all means, I'll welcome any and all thoughts on the matter. :smile:

6 Likes
35

The next discussion topic has started here, covering the error reporting strategy

2 Likes
36

I've shifted gears lately a little bit on this project to focus more on the macro capabilities, building off what @t089 initially contributed (thanks again!).

One of the big things we've known would be needed from the beginning was a way to customize how a particular field is encoded and decoded. This is very often where developers are forced to abandon Codable code synthesis in favor of custom implementations (though clever use of property wrappers works sometimes). This is not the experience we want for the new Codable design. We want the macro functionality to reduce the need for custom implementations to a vanishingly small number of cases.

Please check out this PR to see what I'm proposing for macro and API design to enable these transformations. I hope you all will be as excited as I am about these advancements!

7 Likes
37

These advancements really are very exciting, and should definitely reduce the need for custom implementations. Well done to you and all contributors!

1 Like
38

One of the issues with macros is the inability to pass variables into it. So let's say I have a global date format style set, there's no way for me to use that with all my macros. Which makes switching or testing them difficult. E.g.

let formatStyleToUse = .iso8601

@JSONCodable @CommonCodable
struct Event {
  @CodableBy(.dateFormat(formatStyleToUse))
  let timestamp: Date
  let name: String
}

Depending on where the variable is declared, it's no in scope when the macro is evaluated so can't be used. Are there any workarounds for this?

(As an aside, you can't use AssertMacroExpansion in Swift Testing because it will happily tell you that the test passes, even if it fails - see Add support for Swift Testing in SwiftSyntaxMacrosTestsSupport by 0xTim Ā· Pull Request #3192 Ā· swiftlang/swift-syntax Ā· GitHub)

2 Likes
39

FWIW, this builds and runs just fine for me:

let globalDateFormat = Date.ISO8601FormatStyle()

@JSONCodable
struct CodableByArrayOfDates: Equatable {
    @CodableBy([.dateFormat(globalDateFormat)])
    let timestamps: [Date]
}

The macro does need to syntax and type check the contents at macro expansion time, but in this case, it can find the global and everything seems to work fine.

If you were to instead write .dateFormat(someVariableThatsDefinedInSomeOtherContext), then sure that won't work. But that's not inconsistent with any other aspect of Swift. (It feels like that would require some version of Kotlin's context() to work.)

Yep. But this is actually a custom function (vs *a*ssertMacroExpansion) that does Issue.record on errors to properly emit them to swift-testing.

40

For a variable from outer scope, you can probably just access it.
For code created by the outer macro (this is a little cursed, but it works in many contexts), you can escape its text somehow (e.g base64), pass the escaped text as an argument to the inner macro, and unescape it within.

I'm sorry.

2 Likes