[Proposal] Foundation Swift Encoders


(Itai Ferber) #1

Hi everyone,
This is a companion proposal to the Foundation Swift Archival & Serialization API <https://github.com/apple/swift-evolution/pull/639>. This introduces new encoders and decoders to be used as part of this system.
The proposal is available online <https://github.com/apple/swift-evolution/pull/640> and inlined below.

— Itai

Swift Encoders
Proposal: SE-NNNN <https://github.com/apple/swift-evolution/pull/640>
Author(s): Itai Ferber <https://github.com/itaiferber>, Michael LeHew <https://github.com/mlehew>, Tony Parker <https://github.com/parkera>
Review Manager: TBD
Status: Awaiting review
Associated PRs:
#8124 <https://github.com/apple/swift/pull/8124>
Introduction
As part of the proposal for a Swift archival and serialization API (SE-NNNN), we are also proposing new API for specific new encoders and decoders, as well as introducing support for new Codable types in NSKeyedArchiver and NSKeyedUnarchiver.

This proposal composes the latter two stages laid out in SE-NNNN.

Motivation
With the base API discussed in SE-NNNN, we want to provide new encoders for consumers of this API, as well as provide a consistent story for bridging this new API with our existing NSCoding implementations. We would like to offer a base level of support that users can depend on, and set a pattern that third parties can follow in implementing and extending their own encoders.

Proposed solution
We will:

Add two new encoders and decoders to support encoding Swift value trees in JSON and property list formats
Add support for passing Codable Swift values to NSKeyedArchiver and NSKeyedUnarchiver, and add Codable conformance to our Swift value types
Detailed design
New Encoders and Decoders

JSON

One of the key motivations for the introduction of this API was to allow safer interaction between Swift values and their JSON representations. For values which are Codable, users can encode to and decode from JSON with JSONEncoder and JSONDecoder:

open class JSONEncoder {
    // MARK: Top-Level Encoding

    /// Encodes the given top-level value and returns its JSON representation.
    ///
    /// - parameter value: The value to encode.
    /// - returns: A new `Data` value containing the encoded JSON data.
    /// - throws: `CocoaError.coderInvalidValue` if a non-comforming floating-point value is encountered during archiving, and the encoding strategy is `.throw`.
    /// - throws: An error if any value throws an error during encoding.
    open func encode<Value : Codable>(_ value: Value) throws -> Data

    // MARK: Customization

    /// The formatting of the output JSON data.
    public enum OutputFormatting {
        /// Produce JSON compacted by removing whitespace. This is the default formatting.
        case compact

        /// Produce human-readable JSON with indented output.
        case prettyPrinted
    }

    /// The strategy to use for encoding `Date` values.
    public enum DateEncodingStrategy {
        /// Defer to `Date` for choosing an encoding. This is the default strategy.
        case deferredToDate

        /// Encode the `Date` as a UNIX timestamp (as a JSON number).
        case secondsSince1970

        /// Encode the `Date` as UNIX millisecond timestamp (as a JSON number).
        case millisecondsSince1970

        /// Encode the `Date` as an ISO-8601-formatted string (in RFC 3339 format).
        @available(OSX 10.12, iOS 10.0, watchOS 3.0, tvOS 10.0, *)
        case iso8601

        /// Encode the `Date` as a string formatted by the given formatter.
        case formatted(DateFormatter)

        /// Encode the `Date` as a custom value encoded by the given closure.
        ///
        /// If the closure fails to encode a value into the given encoder, the encoder will encode an empty `.default` container in its place.
        case custom((_ value: Date, _ encoder: Encoder) throws -> Void)
    }

    /// The strategy to use for encoding `Data` values.
    public enum DataEncodingStrategy {
        /// Encoded the `Data` as a Base64-encoded string. This is the default strategy.
        case base64

        /// Encode the `Data` as a custom value encoded by the given closure.
        ///
        /// If the closure fails to encode a value into the given encoder, the encoder will encode an empty `.default` container in its place.
        case custom((_ value: Data, _ encoder: Encoder) throws -> Void)
    }

    /// The strategy to use for non-JSON-conforming floating-point values (IEEE 754 infinity and NaN).
    public enum NonConformingFloatEncodingStrategy {
        /// Throw upon encountering non-conforming values. This is the default strategy.
        case `throw`

        /// Encode the values using the given representation strings.
        case convertToString(positiveInfinity: String, negativeInfinity: String, nan: String)
    }

    /// The output format to produce. Defaults to `.compact`.
    open var outputFormatting: OutputFormatting

    /// The strategy to use in encoding dates. Defaults to `.deferredToDate`.
    open var dateEncodingStrategy: DateEncodingStrategy

    /// The strategy to use in encoding binary data. Defaults to `.base64`.
    open var dataEncodingStrategy: DataEncodingStrategy

    /// The strategy to use in encoding non-conforming numbers. Defaults to `.throw`.
    open var nonConformingFloatEncodingStrategy: NonConformingFloatEncodingStrategy
}

open class JSONDecoder {
    // MARK: Top-Level Decoding

    /// Decodes a top-level value of the given type from the given JSON representation.
    ///
    /// - parameter type: The type of the value to decode.
    /// - parameter data: The data to decode from.
    /// - returns: A value of the requested type.
    /// - throws: `CocoaError.coderReadCorrupt` if values requested from the payload are corrupted, or if the given data is not valid JSON.
    /// - throws: An error if any value throws an error during decoding.
    open func decode<Value : Codable>(_ type: Value.Type, from data: Data) throws -> Value

    // MARK: Customization

    /// The strategy to use for decoding `Date` values.
    public enum DateDecodingStrategy {
        /// Defer to `Date` for decoding. This is the default strategy.
        case deferredToDate

        /// Decode the `Date` as a UNIX timestamp from a JSON number.
        case secondsSince1970

        /// Decode the `Date` as UNIX millisecond timestamp from a JSON number.
        case millisecondsSince1970

        /// Decode the `Date` as an ISO-8601-formatted string (in RFC 3339 format).
        @available(OSX 10.12, iOS 10.0, watchOS 3.0, tvOS 10.0, *)
        case iso8601

        /// Decode the `Date` as a string parsed by the given formatter.
        case formatted(DateFormatter)

        /// Decode the `Date` as a custom value decoded by the given closure.
        case custom((_ decoder: Decoder) throws -> Date)
    }

    /// The strategy to use for decoding `Data` values.
    public enum DataDecodingStrategy {
        /// Decode the `Data` from a Base64-encoded string. This is the default strategy.
        case base64

        /// Decode the `Data` as a custom value decoded by the given closure.
        case custom((_ decoder: Decoder) throws -> Data)
    }

    /// The strategy to use for non-JSON-conforming floating-point values (IEEE 754 infinity and NaN).
    public enum NonConformingFloatDecodingStrategy {
        /// Throw upon encountering non-conforming values. This is the default strategy.
        case `throw`

        /// Decode the values from the given representation strings.
        case convertFromString(positiveInfinity: String, negativeInfinity: String, nan: String)
    }

    /// The strategy to use in decoding dates. Defaults to `.deferredToDate`.
    open var dateDecodingStrategy: DateDecodingStrategy

    /// The strategy to use in decoding binary data. Defaults to `.base64`.
    open var dataDecodingStrategy: DataDecodingStrategy

    /// The strategy to use in decoding non-conforming numbers. Defaults to `.throw`.
    open var nonConformingFloatDecodingStrategy: NonConformingFloatDecodingStrategy
}
Usage:

var encoder = JSONEncoder()
encoder.dateEncodingStrategy = .iso8601
encoder.dataEncodingStrategy = .custom(myBase85Encoder)

// Since JSON does not natively allow for infinite or NaN values, we can customize strategies for encoding these non-conforming values.
encoder.nonConformingFloatEncodingStrategy = .convertToString(positiveInfinity: "INF", negativeInfinity: "-INF", nan: "NaN")

// MyValue conforms to Codable
let topLevel = MyValue(...)

let payload: Data
do {
    payload = try encoder.encode(topLevel)
} catch {
    // Some value threw while encoding.
}

// ...

var decoder = JSONDecoder()
decoder.dateDecodingStrategy = .iso8601
decoder.dataDecodingStrategy = .custom(myBase85Decoder)

// Look for and match these values when decoding `Double`s or `Float`s.
decoder.nonConformingFloatDecodingStrategy = .convertFromString(positiveInfinity: "INF", negativeInfinity: "-INF", nan: "NaN")

let topLevel: MyValue
do {
    topLevel = try decoder.decode(MyValue.self, from: payload)
} catch {
    // Data was corrupted, or some value threw while decoding.
}
It should be noted here that JSONEncoder and JSONDecoder do not themselves conform to Encoder and Decoder; instead, they contain private nested types which do conform to Encoder and Decoder, which are passed to values' encode(to:)and init(from:). This is because JSONEncoder and JSONDecoder must present a different top-level API than they would at intermediate levels.

Property List

We also intend to support the property list format, with PropertyListEncoder and PropertyListDecoder:

open class PropertyListEncoder {
    // MARK: Top-Level Encoding

    /// Encodes the given top-level value and returns its property list representation.
    ///
    /// - parameter value: The value to encode.
    /// - returns: A new `Data` value containing the encoded property list data.
    /// - throws: An error if any value throws an error during encoding.
    open func encode<Value : Codable>(_ value: Value) throws -> Data

    // MARK: Customization

    /// The output format to write the property list data in. Defaults to `.binary`.
    open var outputFormat: PropertyListSerialization.PropertyListFormat
}

open class PropertyListDecoder {
    // MARK: Top-Level Decoding

    /// Decodes a top-level value of the given type from the given property list representation.
    ///
    /// - parameter type: The type of the value to decode.
    /// - parameter data: The data to decode from.
    /// - returns: A value of the requested type.
    /// - throws: `CocoaError.coderReadCorrupt` if values requested from the payload are corrupted, or if the given data is not a valid property list.
    /// - throws: An error if any value throws an error during decoding.
    open func decode<Value : Codable>(_ type: Value.Type, from data: Data) throws -> Value

    /// Decodes a top-level value of the given type from the given property list representation.
    ///
    /// - parameter type: The type of the value to decode.
    /// - parameter data: The data to decode from.
    /// - parameter format: The parsed property list format.
    /// - returns: A value of the requested type along with the detected format of the property list.
    /// - throws: `CocoaError.coderReadCorrupt` if values requested from the payload are corrupted, or if the given data is not a valid property list.
    /// - throws: An error if any value throws an error during decoding.
    open func decode<Value : Codable>(_ type: Value.Type, from data: Data, format: inout PropertyListSerialization.PropertyListFormat) throws -> Value
}
Usage:

let encoder = PropertyListEncoder()
let topLevel = MyValue(...)
let payload: Data
do {
    payload = try encoder.encode(topLevel)
} catch {
    // Some value threw while encoding.
}

// ...

let decoder = PropertyListDecoder()
let topLevel: MyValue
do {
    topLevel = try decoder.decode(MyValue.self, from: payload)
} catch {
    // Data was corrupted, or some value threw while decoding.
}
Like with JSON, PropertyListEncoder and PropertyListDecoder also provide private nested types which conform to Encoder and Decoder for performing the archival.

Foundation-Provided Errors

Along with providing the above encoders and decoders, we would like to promote the use of a common set of error codes and messages across all new encoders and decoders. A common vocabulary of expected errors allows end-users to write code agnostic about the specific encoder/decoder implementation they are working with, whether first-party or third-party:

extension CocoaError.Code {
    /// Thrown when a value incompatible with the output format is encoded.
    public static var coderInvalidValue: CocoaError.Code

    /// Thrown when a value of a given type is requested but the encountered value is of an incompatible type.
    public static var coderTypeMismatch: CocoaError.Code

    /// Thrown when read data is corrupted or otherwise invalid for the format. This value already exists today.
    public static var coderReadCorrupt: CocoaError.Code

    /// Thrown when a requested key or value is unexpectedly null or missing. This value already exists today.
    public static var coderValueNotFound: CocoaError.Code
}

// These reexpose the values above.
extension CocoaError {
    public static var coderInvalidValue: CocoaError.Code

    public static var coderTypeMismatch: CocoaError.Code
}
The localized description strings associated with the two new error codes are:

.coderInvalidValue: "The data is not valid for encoding in this format."
.coderTypeMismatch: "The data couldn't be read because it isn't in the correct format." (Precedent from NSCoderReadCorruptError.)
All of these errors will include the coding key path that led to the failure in the error's userInfo dictionary under NSCodingKeyContextErrorKey, along with a non-localized, developer-facing failure reason under NSDebugDescriptionErrorKey.

NSKeyedArchiver & NSKeyedUnarchiver Changes

Although our primary objectives for this new API revolve around Swift, we would like to make it easy for current consumers to make the transition to Codable where appropriate. As part of this, we would like to bridge compatibility between new Codabletypes (or newly-Codable-adopting types) and existing NSCoding types.

To do this, we want to introduce changes to NSKeyedArchiver and NSKeyedUnarchiver in Swift that allow archival of Codable types intermixed with NSCoding types:

// These are provided in the Swift overlay, and included in swift-corelibs-foundation.
extension NSKeyedArchiver {
    public func encodeCodable(_ codable: Codable?, forKey key: String) { ... }
}

extension NSKeyedUnarchiver {
    public func decodeCodable<T : Codable>(_ type: T.Type, forKey key: String) -> T? { ... }
}
NOTE: Since these changes are being made in extensions in the Swift overlay, it is not yet possible for these methods to be overridden. These can therefore not be added to NSCoder, since NSKeyedArchiver and NSKeyedUnarchiver would not be able to provide concrete implementations. In order to call these methods, it is necessary to downcast from an NSCoder to NSKeyedArchiver/NSKeyedUnarchiver directly. Since subclasses of NSKeyedArchiver and NSKeyedUnarchiver in Swift will inherit these implementations without being able to override them (which is wrong), we will NSRequiresConcreteImplementation() dynamically in subclasses.
The addition of these methods allows the introduction of Codable types into existing NSCoding structures, allowing for a transition to Codable types where appropriate.

Refining encode(_:forKey:)

Along with these extensions, we would like to refine the import of -[NSCoder encodeObject:forKey:], which is currently imported into Swift as encode(_: Any?, forKey: String). This method currently accepts Objective-C and Swift objects conforming to NSCoding (non-conforming objects produce a runtime error), as well as bridgeable Swift types (Data, String, Array, etc.); we would like to extend it to support new Swift Codable types, which would otherwise produce a runtime error upon call.

-[NSCoder encodeObject:forKey:] will be given a new Swift name of encodeObject(_:forKey:), and we will provide a replacement encode(_: Any?, forKey: String) in the overlay which will funnel out to either encodeCodable(_:forKey:) or encodeObject(_:forKey:) as appropriate. This should maintain source compatibility for end users already calling encode(_:forKey:), as well as behavior compatibility for subclassers of NSCoderand NSKeyedArchiver who may be providing their own encode(_:forKey:).

Semantics of Codable Types in Archives

There are a few things to note about including Codable values in NSKeyedArchiverarchives:

Bridgeable Foundation types will always bridge before encoding. This is to facilitate writing Foundation types in a compatible format from both Objective-C and Swift
On decode, these types will decode either as their Objective-C or Swift version, depending on user need (decodeObject(forKey:) will decode as an Objective-C object; decodeCodable(_:forKey:) as a Swift value)
User types, which are not bridgeable, do not write out a $class and can only be decoded in Swift. In the future, we may add API to allow Swift types to provide an Objective-C class to decode as, effectively allowing for user bridging across archival
Foundation Types Adopting Codable

The following Foundation Swift types will be adopting Codable, and will encode as their bridged types when encoded through NSKeyedArchiver, as mentioned above:

AffineTransform
Calendar
CharacterSet
Date
DateComponents
DateInterval
Decimal
IndexPath
IndexSet
Locale
Measurement
Notification
PersonNameComponents
TimeZone
URL
URLComponents
URLRequest
UUID
Along with these, the Array, Dictionary, and Set types will gain Codableconformance (as part of the Conditional Conformance feature), and encode through NSKeyedArchiver as NSArray, NSDictionary, and NSSet respectively.

Source compatibility
The majority of this proposal is additive. The changes to NSKeyedArchiver are intended to be non-source-breaking changes, and non-behavior-breaking changes for subclasses in Objective-C and Swift.

Effect on ABI stability
The addition of this API will not be an ABI-breaking change. However, this will add limitations for changes in future versions of Swift, as parts of the API will have to remain unchanged between versions of Swift (barring some additions, discussed below).

Effect on API resilience
Much like new API added to the standard library, once added, some changes to this API will be ABI- and source-breaking changes. Changes to the new encoder and decoder classes provided above will be restricted as described in the library evolution document <https://github.com/apple/swift/blob/master/docs/LibraryEvolution.rst> in the Swift repository; in particular, the removal of methods or nested types or changes to argument types will break client behavior. Additionally, additions to provided options enums will be a source-breaking change for users performing an exhaustive switch over their cases; removal of cases will be ABI-breaking.

Alternatives considered
None. This is a companion to the Swift Archival and Serialization API.


(Will Stanton) #2

Hello,

+1

This proposal seems helpful in standardizing how JSON objects can be written, and I commonly encode+decode JSON. The standard library JSON and PLIST encoders of Python are a strength, and Swift should be able to handle both formats just as easily. Still reading 'Swift Archival & Serialization’, but I believe both proposals will improve the safety and saneness of serializing/deserialization.

For the JSON coder, how does `deferredToDate` work? Would both the writer and reader have to agree to use `deferredToDate`?
Might it be better to force clients to pick a ‘real’ strategy? Why not default to one of the formats, perhaps ISO-8601?

(Not too important but also curious how much of a slowdown there will be when Xcode/SourceKit tries to autocomplete ‘enc’ or ‘dec’ for the Swift Archival & Serialization proposal?)

Regards,
Will Stanton

···

On Mar 15, 2017, at 6:43 PM, Itai Ferber via swift-evolution <swift-evolution@swift.org> wrote:

Hi everyone,
This is a companion proposal to the Foundation Swift Archival & Serialization API. This introduces new encoders and decoders to be used as part of this system.
The proposal is available online and inlined below.

— Itai

Swift Encoders
  • Proposal: SE-NNNN
  • Author(s): Itai Ferber, Michael LeHew, Tony Parker
  • Review Manager: TBD
  • Status: Awaiting review
  • Associated PRs:
    • #8124
Introduction
As part of the proposal for a Swift archival and serialization API (SE-NNNN), we are also proposing new API for specific new encoders and decoders, as well as introducing support for new Codable types in NSKeyedArchiver and NSKeyedUnarchiver.

This proposal composes the latter two stages laid out in SE-NNNN.

Motivation
With the base API discussed in SE-NNNN, we want to provide new encoders for consumers of this API, as well as provide a consistent story for bridging this new API with our existing NSCoding implementations. We would like to offer a base level of support that users can depend on, and set a pattern that third parties can follow in implementing and extending their own encoders.

Proposed solution
We will:

  • Add two new encoders and decoders to support encoding Swift value trees in JSON and property list formats
  • Add support for passing Codable Swift values to NSKeyedArchiver and NSKeyedUnarchiver, and add Codable conformance to our Swift value types
Detailed design
New Encoders and Decoders

JSON

One of the key motivations for the introduction of this API was to allow safer interaction between Swift values and their JSON representations. For values which are Codable, users can encode to and decode from JSON with JSONEncoder and JSONDecoder:

open class JSONEncoder {

// MARK: Top-Level Encoding

/// Encodes the given top-level value and returns its JSON representation.

///

/// - parameter value: The value to encode.

/// - returns: A new `Data` value containing the encoded JSON data.

/// - throws: `CocoaError.coderInvalidValue` if a non-comforming floating-point value is encountered during archiving, and the encoding strategy is `.throw`.

/// - throws: An error if any value throws an error during encoding.

    open
func encode<Value : Codable>(_ value: Value) throws -> Data

// MARK: Customization

/// The formatting of the output JSON data.

public enum OutputFormatting {

/// Produce JSON compacted by removing whitespace. This is the default formatting.

case
compact

/// Produce human-readable JSON with indented output.

case
prettyPrinted
    
}

/// The strategy to use for encoding `Date` values.

public enum DateEncodingStrategy {

/// Defer to `Date` for choosing an encoding. This is the default strategy.

case
deferredToDate

/// Encode the `Date` as a UNIX timestamp (as a JSON number).

case
secondsSince1970

/// Encode the `Date` as UNIX millisecond timestamp (as a JSON number).

case
millisecondsSince1970

/// Encode the `Date` as an ISO-8601-formatted string (in RFC 3339 format).

        @
available(OSX 10.12, iOS 10.0, watchOS 3.0, tvOS 10.0, *)

case
iso8601

/// Encode the `Date` as a string formatted by the given formatter.

case formatted(DateFormatter)

/// Encode the `Date` as a custom value encoded by the given closure.

///

/// If the closure fails to encode a value into the given encoder, the encoder will encode an empty `.default` container in its place.

case custom((_ value: Date, _ encoder: Encoder) throws -> Void)

}

/// The strategy to use for encoding `Data` values.

public enum DataEncodingStrategy {

/// Encoded the `Data` as a Base64-encoded string. This is the default strategy.

case
base64

/// Encode the `Data` as a custom value encoded by the given closure.

///

/// If the closure fails to encode a value into the given encoder, the encoder will encode an empty `.default` container in its place.

case custom((_ value: Data, _ encoder: Encoder) throws -> Void)

}

/// The strategy to use for non-JSON-conforming floating-point values (IEEE 754 infinity and NaN).

public enum NonConformingFloatEncodingStrategy {

/// Throw upon encountering non-conforming values. This is the default strategy.

case `throw
`

/// Encode the values using the given representation strings.

case convertToString(positiveInfinity: String, negativeInfinity: String, nan: String)

}

/// The output format to produce. Defaults to `.compact`.

    open
var outputFormatting: OutputFormatting

/// The strategy to use in encoding dates. Defaults to `.deferredToDate`.

    open
var dateEncodingStrategy: DateEncodingStrategy

/// The strategy to use in encoding binary data. Defaults to `.base64`.

    open
var dataEncodingStrategy: DataEncodingStrategy

/// The strategy to use in encoding non-conforming numbers. Defaults to `.throw`.

    open
var nonConformingFloatEncodingStrategy: NonConformingFloatEncodingStrategy
}

open
class JSONDecoder {

// MARK: Top-Level Decoding

/// Decodes a top-level value of the given type from the given JSON representation.

///

/// - parameter type: The type of the value to decode.

/// - parameter data: The data to decode from.

/// - returns: A value of the requested type.

/// - throws: `CocoaError.coderReadCorrupt` if values requested from the payload are corrupted, or if the given data is not valid JSON.

/// - throws: An error if any value throws an error during decoding.

    open
func decode<Value : Codable>(_ type: Value.Type, from data: Data) throws -> Value

// MARK: Customization

/// The strategy to use for decoding `Date` values.

public enum DateDecodingStrategy {

/// Defer to `Date` for decoding. This is the default strategy.

case
deferredToDate

/// Decode the `Date` as a UNIX timestamp from a JSON number.

case
secondsSince1970

/// Decode the `Date` as UNIX millisecond timestamp from a JSON number.

case
millisecondsSince1970

/// Decode the `Date` as an ISO-8601-formatted string (in RFC 3339 format).

        @
available(OSX 10.12, iOS 10.0, watchOS 3.0, tvOS 10.0, *)

case
iso8601

/// Decode the `Date` as a string parsed by the given formatter.

case formatted(DateFormatter)

/// Decode the `Date` as a custom value decoded by the given closure.

case custom((_ decoder: Decoder) throws -> Date)

}

/// The strategy to use for decoding `Data` values.

public enum DataDecodingStrategy {

/// Decode the `Data` from a Base64-encoded string. This is the default strategy.

case
base64

/// Decode the `Data` as a custom value decoded by the given closure.

case custom((_ decoder: Decoder) throws -> Data)

}

/// The strategy to use for non-JSON-conforming floating-point values (IEEE 754 infinity and NaN).

public enum NonConformingFloatDecodingStrategy {

/// Throw upon encountering non-conforming values. This is the default strategy.

case `throw
`

/// Decode the values from the given representation strings.

case convertFromString(positiveInfinity: String, negativeInfinity: String, nan: String)

}

/// The strategy to use in decoding dates. Defaults to `.deferredToDate`.

    open
var dateDecodingStrategy: DateDecodingStrategy

/// The strategy to use in decoding binary data. Defaults to `.base64`.

    open
var dataDecodingStrategy: DataDecodingStrategy

/// The strategy to use in decoding non-conforming numbers. Defaults to `.throw`.

    open
var nonConformingFloatDecodingStrategy: NonConformingFloatDecodingStrategy
}
Usage:

var encoder = JSONEncoder()

encoder
.dateEncodingStrategy = .
iso8601
encoder
.dataEncodingStrategy = .custom(myBase85Encoder)

// Since JSON does not natively allow for infinite or NaN values, we can customize strategies for encoding these non-conforming values.

encoder
.nonConformingFloatEncodingStrategy = .convertToString(positiveInfinity: "INF", negativeInfinity: "-INF", nan: "NaN")

// MyValue conforms to Codable
let topLevel = MyValue(...)

let payload: Data
do {

    payload
= try encoder.encode(topLevel)
} catch {

// Some value threw while encoding.
}

// ...

var decoder = JSONDecoder()

decoder
.dateDecodingStrategy = .
iso8601
decoder
.dataDecodingStrategy = .custom(myBase85Decoder)

// Look for and match these values when decoding `Double`s or `Float`s.

decoder
.nonConformingFloatDecodingStrategy = .convertFromString(positiveInfinity: "INF", negativeInfinity: "-INF", nan: "NaN")

let topLevel: MyValue
do {

    topLevel
= try decoder.decode(MyValue.self, from: payload)
} catch {

// Data was corrupted, or some value threw while decoding.
}
It should be noted here that JSONEncoder and JSONDecoder do not themselves conform to Encoder and Decoder; instead, they contain private nested types which do conform to Encoder and Decoder, which are passed to values' encode(to:)and init(from:). This is because JSONEncoder and JSONDecoder must present a different top-level API than they would at intermediate levels.

Property List

We also intend to support the property list format, with PropertyListEncoder and PropertyListDecoder:

open class PropertyListEncoder {

// MARK: Top-Level Encoding

/// Encodes the given top-level value and returns its property list representation.

///

/// - parameter value: The value to encode.

/// - returns: A new `Data` value containing the encoded property list data.

/// - throws: An error if any value throws an error during encoding.

    open
func encode<Value : Codable>(_ value: Value) throws -> Data

// MARK: Customization

/// The output format to write the property list data in. Defaults to `.binary`.

    open
var outputFormat: PropertyListSerialization.PropertyListFormat
}

open
class PropertyListDecoder {

// MARK: Top-Level Decoding

/// Decodes a top-level value of the given type from the given property list representation.

///

/// - parameter type: The type of the value to decode.

/// - parameter data: The data to decode from.

/// - returns: A value of the requested type.

/// - throws: `CocoaError.coderReadCorrupt` if values requested from the payload are corrupted, or if the given data is not a valid property list.

/// - throws: An error if any value throws an error during decoding.

    open
func decode<Value : Codable>(_ type: Value.Type, from data: Data) throws -> Value

/// Decodes a top-level value of the given type from the given property list representation.

///

/// - parameter type: The type of the value to decode.

/// - parameter data: The data to decode from.

/// - parameter format: The parsed property list format.

/// - returns: A value of the requested type along with the detected format of the property list.

/// - throws: `CocoaError.coderReadCorrupt` if values requested from the payload are corrupted, or if the given data is not a valid property list.

/// - throws: An error if any value throws an error during decoding.

    open
func decode<Value : Codable>(_ type: Value.Type, from data: Data, format: inout PropertyListSerialization.PropertyListFormat) throws -> Value
}
Usage:

let encoder = PropertyListEncoder()
let topLevel = MyValue(...)
let payload: Data
do {

    payload
= try encoder.encode(topLevel)
} catch {

// Some value threw while encoding.
}

// ...

let decoder = PropertyListDecoder()
let topLevel: MyValue
do {

    topLevel
= try decoder.decode(MyValue.self, from: payload)
} catch {

// Data was corrupted, or some value threw while decoding.
}
Like with JSON, PropertyListEncoder and PropertyListDecoder also provide private nested types which conform to Encoder and Decoder for performing the archival.

Foundation-Provided Errors

Along with providing the above encoders and decoders, we would like to promote the use of a common set of error codes and messages across all new encoders and decoders. A common vocabulary of expected errors allows end-users to write code agnostic about the specific encoder/decoder implementation they are working with, whether first-party or third-party:

extension CocoaError.Code {

/// Thrown when a value incompatible with the output format is encoded.

public static var coderInvalidValue: CocoaError.Code

/// Thrown when a value of a given type is requested but the encountered value is of an incompatible type.

public static var coderTypeMismatch: CocoaError.Code

/// Thrown when read data is corrupted or otherwise invalid for the format. This value already exists today.

public static var coderReadCorrupt: CocoaError.Code

/// Thrown when a requested key or value is unexpectedly null or missing. This value already exists today.

public static var coderValueNotFound: CocoaError.Code
}

// These reexpose the values above.
extension CocoaError {

public static var coderInvalidValue: CocoaError.Code

public static var coderTypeMismatch: CocoaError.Code
}
The localized description strings associated with the two new error codes are:

  • .coderInvalidValue: "The data is not valid for encoding in this format."
  • .coderTypeMismatch: "The data couldn't be read because it isn't in the correct format." (Precedent from NSCoderReadCorruptError.)
All of these errors will include the coding key path that led to the failure in the error's userInfo dictionary under NSCodingKeyContextErrorKey, along with a non-localized, developer-facing failure reason under NSDebugDescriptionErrorKey.

NSKeyedArchiver & NSKeyedUnarchiver Changes

Although our primary objectives for this new API revolve around Swift, we would like to make it easy for current consumers to make the transition to Codable where appropriate. As part of this, we would like to bridge compatibility between new Codabletypes (or newly-Codable-adopting types) and existing NSCoding types.

To do this, we want to introduce changes to NSKeyedArchiver and NSKeyedUnarchiver in Swift that allow archival of Codable types intermixed with NSCoding types:

// These are provided in the Swift overlay, and included in swift-corelibs-foundation.
extension NSKeyedArchiver {

public func encodeCodable(_ codable: Codable?, forKey key: String) { ... }
}

extension NSKeyedUnarchiver {

public func decodeCodable<T : Codable>(_ type: T.Type, forKey key: String) -> T? { ... }
}
NOTE: Since these changes are being made in extensions in the Swift overlay, it is not yet possible for these methods to be overridden. These can therefore not be added to NSCoder, since NSKeyedArchiver and NSKeyedUnarchiver would not be able to provide concrete implementations. In order to call these methods, it is necessary to downcast from an NSCoder to NSKeyedArchiver/NSKeyedUnarchiver directly. Since subclasses of NSKeyedArchiver and NSKeyedUnarchiver in Swift will inherit these implementations without being able to override them (which is wrong), we will NSRequiresConcreteImplementation() dynamically in subclasses.
The addition of these methods allows the introduction of Codable types into existing NSCoding structures, allowing for a transition to Codable types where appropriate.

Refining encode(_:forKey:)

Along with these extensions, we would like to refine the import of -[NSCoder encodeObject:forKey:], which is currently imported into Swift as encode(_: Any?, forKey: String). This method currently accepts Objective-C and Swift objects conforming to NSCoding (non-conforming objects produce a runtime error), as well as bridgeable Swift types (Data, String, Array, etc.); we would like to extend it to support new Swift Codable types, which would otherwise produce a runtime error upon call.

-[NSCoder encodeObject:forKey:] will be given a new Swift name of encodeObject(_:forKey:), and we will provide a replacement encode(_: Any?, forKey: String) in the overlay which will funnel out to either encodeCodable(_:forKey:) or encodeObject(_:forKey:) as appropriate. This should maintain source compatibility for end users already calling encode(_:forKey:), as well as behavior compatibility for subclassers of NSCoderand NSKeyedArchiver who may be providing their own encode(_:forKey:).

Semantics of Codable Types in Archives

There are a few things to note about including Codable values in NSKeyedArchiverarchives:

  • Bridgeable Foundation types will always bridge before encoding. This is to facilitate writing Foundation types in a compatible format from both Objective-C and Swift
    • On decode, these types will decode either as their Objective-C or Swift version, depending on user need (decodeObject(forKey:) will decode as an Objective-C object; decodeCodable(_:forKey:) as a Swift value)
  • User types, which are not bridgeable, do not write out a $class and can only be decoded in Swift. In the future, we may add API to allow Swift types to provide an Objective-C class to decode as, effectively allowing for user bridging across archival
Foundation Types Adopting Codable

The following Foundation Swift types will be adopting Codable, and will encode as their bridged types when encoded through NSKeyedArchiver, as mentioned above:

  • AffineTransform
  • Calendar
  • CharacterSet
  • Date
  • DateComponents
  • DateInterval
  • Decimal
  • IndexPath
  • IndexSet
  • Locale
  • Measurement
  • Notification
  • PersonNameComponents
  • TimeZone
  • URL
  • URLComponents
  • URLRequest
  • UUID
Along with these, the Array, Dictionary, and Set types will gain Codableconformance (as part of the Conditional Conformance feature), and encode through NSKeyedArchiver as NSArray, NSDictionary, and NSSet respectively.

Source compatibility
The majority of this proposal is additive. The changes to NSKeyedArchiver are intended to be non-source-breaking changes, and non-behavior-breaking changes for subclasses in Objective-C and Swift.

Effect on ABI stability
The addition of this API will not be an ABI-breaking change. However, this will add limitations for changes in future versions of Swift, as parts of the API will have to remain unchanged between versions of Swift (barring some additions, discussed below).

Effect on API resilience
Much like new API added to the standard library, once added, some changes to this API will be ABI- and source-breaking changes. Changes to the new encoder and decoder classes provided above will be restricted as described in the library evolution document in the Swift repository; in particular, the removal of methods or nested types or changes to argument types will break client behavior. Additionally, additions to provided options enums will be a source-breaking change for users performing an exhaustive switch over their cases; removal of cases will be ABI-breaking.

Alternatives considered
None. This is a companion to the Swift Archival and Serialization API.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Brent Royal-Gordon) #3

Hi everyone,
This is a companion proposal to the Foundation Swift Archival & Serialization API. This introduces new encoders and decoders to be used as part of this system.
The proposal is available online and inlined below.

Executive summary: I like where you're going with this, but I'm worried about flexibility.

I'm not going to quote every bit of the JSON section because Apple Mail seems to destroy the formatting when I reply, but: I think you've identified several of the most important customization points (Date, Data, and illegal Floats). However, I think:

* People may want to map illegal Floats to legal floating-point values (say, `greatestFiniteMagnitude`, `-greatestFiniteMagnitude`, and `0`) or map them to `null`s. They may also want different behavior for different things: imagine `(positiveInfinity: Double.greatestFiniteMagnitude, negativeInfinity: -Double.greatestFiniteMagnitude, nan: .throw)`.

* Large integers are another big concern that you don't address. Because JSON only supports doubles, APIs that use 64-bit IDs often need them to be passed as strings, frequently with a different key ("id_str" instead of "id").

* For that matter, style and capitalization are a problem. JSON style varies, but it *tends* to be snake_case, where Cocoa favors camelCase. You can address this at the CodingKey level by manually specifying string equivalents of all the coding keys, but that's kind of a pain, and it affects all of your code and all of your serializations.

I'm sorely tempted to suggest that we give the JSON encoder and decoder a delegate:

  public protocol JSONCodingDelegate {
    /// Returns the string name to be used when encoding or decoding the given CodingKey as JSON.
    ///
    /// - Returns: The string to use, or `nil` for the default.
    func jsonName(for key: CodingKey, at keyPath: [CodingKey], in encoderOrDecoder: AnyObject) throws -> String?

    // These are used when encoding/decoding any of the integer types.
    func jsonValue(from integer: Int64, at keyPath: [CodingKey], in encoder: JSONEncoder) throws -> JSONValue?
    func integer(from jsonValue: JSONValue, at keyPath: [CodingKey], in decoder: JSONDecoder) throws -> Int64?
    
    // These are used when encoding/decoding any of the floating-point types.
    func jsonValue(from number: Double, at keyPath: [CodingKey], in encoder: JSONEncoder) throws -> JSONValue?
    func number(from jsonValue: JSONValue, at keyPath: [CodingKey], in decoder: JSONDecoder) throws -> Double?
    
    // These are used when encoding/decoding Date.
    func jsonValue(from date: Date, at keyPath: [CodingKey], in encoder: JSONEncoder) throws -> JSONValue?
    func date(from jsonValue: JSONValue, at keyPath: [CodingKey], in decoder: JSONDecoder) throws -> Date?
    
    // These are used when encoding/decoding Data.
    func jsonValue(from data: Data, at keyPath: [CodingKey], in encoder: JSONEncoder) throws -> JSONValue?
    func data(from jsonValue: JSONValue, at keyPath: [CodingKey], in decoder: JSONDecoder) throws -> Data?
    
    func jsonValue(from double: Double, at keyPath: [CodingKey], in encoder: JSONEncoder) throws -> JSONValue?
    func integer(from jsonValue: JSONValue, at keyPath: [CodingKey], in decoder: JSONDecoder) throws -> Double?
  }
  public enum JSONValue {
    case string(String)
    case number(Double)
    case bool(Bool)
    case object([String: JSONValue])
    case array([JSONValue])
    case null
  }

Or, perhaps, that a more general form of this delegate be available on all encoders and decoders. But that may be overkill, and even if it *is* a good idea, it's one we can add later.

Property List

We also intend to support the property list format, with PropertyListEncoder and PropertyListDecoder:

No complaints here.

Foundation-Provided Errors

Along with providing the above encoders and decoders, we would like to promote the use of a common set of error codes and messages across all new encoders and decoders. A common vocabulary of expected errors allows end-users to write code agnostic about the specific encoder/decoder implementation they are working with, whether first-party or third-party:

extension CocoaError.Code {
    /// Thrown when a value incompatible with the output format is encoded.
    public static var coderInvalidValue: CocoaError.Code

    /// Thrown when a value of a given type is requested but the encountered value is of an incompatible type.
    public static var coderTypeMismatch: CocoaError.Code

    /// Thrown when read data is corrupted or otherwise invalid for the format. This value already exists today.
    public static var coderReadCorrupt: CocoaError.Code

    /// Thrown when a requested key or value is unexpectedly null or missing. This value already exists today.
    public static var coderValueNotFound: CocoaError.Code
}

[snip]

All of these errors will include the coding key path that led to the failure in the error's userInfo dictionary under NSCodingKeyContextErrorKey, along with a non-localized, developer-facing failure reason under NSDebugDescriptionErrorKey.

Now comes the part where I whine like a four-year-old:

"Do we haaaaaaaave to use the `userInfo` dictionary, papa?"

An enum with an associated value would be a much more natural way to express these errors and the data that comes with them. Failing that, at least give us some convenience properties. The untyped bag of stuff in the `userInfo` dictionary fills developers who spend all their time in Swift with fear and loathing.

Actually, if you wanted to help us out with the "untyped bag of stuff" problem in general, I for one wouldn't say "no":

  public struct TypedKey<Key: Hashable, Value> {
    public var key: Key
    public init(key: Key, valueType: Value.Type) {
      self.key = key
    }
  }
  extension Dictionary where Value == Any {
    public subscript<CastedValue>(typedKey: TypedKey<Key, CastedValue>) -> CastedValue? {
      get {
        return self[typedKey.key] as? CastedValue
      }
      set {
        self[typedKey.key] = newValue
      }
    }
  }

NSKeyedArchiver & NSKeyedUnarchiver Changes

Although our primary objectives for this new API revolve around Swift, we would like to make it easy for current consumers to make the transition to Codable where appropriate. As part of this, we would like to bridge compatibility between new Codabletypes (or newly-Codable-adopting types) and existing NSCoding types.

To do this, we want to introduce changes to NSKeyedArchiver and NSKeyedUnarchiver in Swift that allow archival of Codable types intermixed with NSCoding types:

// These are provided in the Swift overlay, and included in swift-corelibs-foundation.
extension NSKeyedArchiver {
    public func encodeCodable(_ codable: Codable?, forKey key: String) { ... }
}

extension NSKeyedUnarchiver {
    public func decodeCodable<T : Codable>(_ type: T.Type, forKey key: String) -> T? { ... }
}

NOTE: Since these changes are being made in extensions in the Swift overlay, it is not yet possible for these methods to be overridden. These can therefore not be added to NSCoder, since NSKeyedArchiver and NSKeyedUnarchiver would not be able to provide concrete implementations. In order to call these methods, it is necessary to downcast from an NSCoder to NSKeyedArchiver/NSKeyedUnarchiver directly. Since subclasses of NSKeyedArchiver and NSKeyedUnarchiver in Swift will inherit these implementations without being able to override them (which is wrong), we will NSRequiresConcreteImplementation() dynamically in subclasses.

The addition of these methods allows the introduction of Codable types into existing NSCoding structures, allowing for a transition to Codable types where appropriate.

I wonder about this.

Could `NSCoding` be imported in Swift as refining `Codable`? Then we could all just forget `NSCoding` exists, other than that certain types are less likely to properly handle being put into a JSONEncoder/Decoder. (Which, to tell the truth, is probably inevitable here; the Encoder and Decoder types look like they're probably too loosely defined to truly guarantee that you can mix-and-match coders and types without occasional problems.)

Semantics of Codable Types in Archives

There are a few things to note about including Codable values in NSKeyedArchiverarchives:

  • Bridgeable Foundation types will always bridge before encoding. This is to facilitate writing Foundation types in a compatible format from both Objective-C and Swift
    • On decode, these types will decode either as their Objective-C or Swift version, depending on user need (decodeObject(forKey:) will decode as an Objective-C object; decodeCodable(_:forKey:) as a Swift value)

This sounds sensible.

  • User types, which are not bridgeable, do not write out a $class and can only be decoded in Swift. In the future, we may add API to allow Swift types to provide an Objective-C class to decode as, effectively allowing for user bridging across archival

Even pure Swift class types? I guess that's probably necessary since even our private ability to look up classes at runtime doesn't cover things like generics, but...ugh.

Along with these, the Array, Dictionary, and Set types will gain Codableconformance (as part of the Conditional Conformance feature), and encode through NSKeyedArchiver as NSArray, NSDictionary, and NSSet respectively.

You might need to be careful here—we'll need to make sure that data structures of Swift types bridge properly. I suppose that means `_SwiftValue` will need to support `NSCoding` after all...

···

On Mar 15, 2017, at 3:43 PM, Itai Ferber via swift-evolution <swift-evolution@swift.org> wrote:

--
Brent Royal-Gordon
Architechies


(Zachary Waldowski) #4

Just a little one that came up with when thinking over the companion proposal… see inline.

···

On Mar 15, 2017, at 6:43 PM, Itai Ferber via swift-evolution <swift-evolution@swift.org> wrote:
Foundation-Provided Errors

Along with providing the above encoders and decoders, we would like to promote the use of a common set of error codes and messages across all new encoders and decoders. A common vocabulary of expected errors allows end-users to write code agnostic about the specific encoder/decoder implementation they are working with, whether first-party or third-party:

extension CocoaError.Code {
    public static var coderInvalidValue: CocoaError.Code
    public static var coderTypeMismatch: CocoaError.Code
    public static var coderReadCorrupt: CocoaError.Code
    public static var coderValueNotFound: CocoaError.Code
}

I find all the reasons compelling, but I don’t with the overall choice. A common vocabulary, side table info, and localization can be achieved with a dedicated error type for coders. I understand ABI (sorta) concerns are involved here, that older Foundations will continue to return the same NSError codes. However, it's never felt appropriate to create my own error instances with NSCocoaErrorDomain, and this feels very similar to that. What are our options here?

Zach Waldowski
zach@waldowski.me


(Ben Rimmington) #5

<https://github.com/apple/swift-evolution/pull/640>

Could the `JSONEncoder.OutputFormatting.prettyPrinted` case include the **stable ordering** of keys?

-- Ben


(Matthew Johnson) #6

Thank you again for bringing these great proposals forward!

I only have a couple of questions about this proposal.

I noticed that the types in this proposal don’t conform to Encoder and Decoder. Is the plan to have them to provide private conforming types to Codable types they are asked to encode or decode?

Why are the strategy and format properties readwrite instead of configured at initialization time? Is the intent that the encoder / decoder can be re-used with a different configuration in a subsequent call to encode or decode?

Finally, I agree with Brent’s comments regarding errors. I would prefer to see Foundation move away from NSError in favor of domain-specific error types. That said, the comment that this is a broader discussion for Foundation and not something to change in this proposal is reasonable. I hope Foundation will consider changing this in the future.

Matthew


(Itai Ferber) #7

Hi everyone,

With feedback from swift-evolution and additional internal review, we've pushed updates to this proposal, and to the Swift Archival & Serialization <https://github.com/apple/swift-evolution/pull/639> proposal.
Changes to here mostly mirror the ones made to Swift Archival & Serialization, but you can see a specific diff of what's changed here <https://github.com/apple/swift-evolution/pull/640/commits/add1faad9e8005de91909bdc29f66984f4869898>. Full content below.

We'll be looking to start the official review process very soon, so we're interested in any additional feedback.

Thanks!

— Itai

Swift Encoders
Proposal: SE-NNNN
Author(s): Itai Ferber <https://github.com/itaiferber>, Michael LeHew <https://github.com/mlehew>, Tony Parker <https://github.com/parkera>
Review Manager: TBD
Status: Awaiting review
Associated PRs:
#8124 <https://github.com/apple/swift/pull/8124>
Introduction
As part of the proposal for a Swift archival and serialization API (SE-NNNN <https://github.com/itaiferber/swift-evolution/blob/swift-archival-serialization/proposals/XXXX-swift-archival-serialization.md>), we are also proposing new API for specific new encoders and decoders, as well as introducing support for new Codable types in NSKeyedArchiver and NSKeyedUnarchiver.

This proposal composes the latter two stages laid out in SE-NNNN <https://github.com/itaiferber/swift-evolution/blob/swift-archival-serialization/proposals/XXXX-swift-archival-serialization.md>.

Motivation
With the base API discussed in SE-NNNN <https://github.com/itaiferber/swift-evolution/blob/swift-archival-serialization/proposals/XXXX-swift-archival-serialization.md>, we want to provide new encoders for consumers of this API, as well as provide a consistent story for bridging this new API with our existing NSCoding implementations. We would like to offer a base level of support that users can depend on, and set a pattern that third parties can follow in implementing and extending their own encoders.

Proposed solution
We will:

Add two new encoders and decoders to support encoding Swift value trees in JSON and property list formats
Add support for passing Codable Swift values to NSKeyedArchiver and NSKeyedUnarchiver, and add Codableconformance to our Swift value types
Detailed design
New Encoders and Decoders

JSON

One of the key motivations for the introduction of this API was to allow safer interaction between Swift values and their JSON representations. For values which are Codable, users can encode to and decode from JSON with JSONEncoder and JSONDecoder:

open class JSONEncoder {
    // MARK: Top-Level Encoding

    /// Encodes the given top-level value and returns its JSON representation.
    ///
    /// - parameter value: The value to encode.
    /// - returns: A new `Data` value containing the encoded JSON data.
    /// - throws: `CocoaError.coderInvalidValue` if a non-comforming floating-point value is encountered during archiving, and the encoding strategy is `.throw`.
    /// - throws: An error if any value throws an error during encoding.
    open func encode<T : Encodable>(_ value: T) throws -> Data

    // MARK: Customization

    /// The formatting of the output JSON data.
    public enum OutputFormatting {
        /// Produce JSON compacted by removing whitespace. This is the default formatting.
        case compact

        /// Produce human-readable JSON with indented output.
        case prettyPrinted
    }

    /// The strategy to use for encoding `Date` values.
    public enum DateEncodingStrategy {
        /// Defer to `Date` for choosing an encoding. This is the default strategy.
        case deferredToDate

        /// Encode the `Date` as a UNIX timestamp (as a JSON number).
        case secondsSince1970

        /// Encode the `Date` as UNIX millisecond timestamp (as a JSON number).
        case millisecondsSince1970

        /// Encode the `Date` as an ISO-8601-formatted string (in RFC 3339 format).
        @available(OSX 10.12, iOS 10.0, watchOS 3.0, tvOS 10.0, *)
        case iso8601

        /// Encode the `Date` as a string formatted by the given formatter.
        case formatted(DateFormatter)

        /// Encode the `Date` as a custom value encoded by the given closure.
        ///
        /// If the closure fails to encode a value into the given encoder, the encoder will encode an empty `.default` container in its place.
        case custom((_ value: Date, _ encoder: Encoder) throws -> Void)
    }

    /// The strategy to use for encoding `Data` values.
    public enum DataEncodingStrategy {
        /// Encoded the `Data` as a Base64-encoded string. This is the default strategy.
        case base64

        /// Encode the `Data` as a custom value encoded by the given closure.
        ///
        /// If the closure fails to encode a value into the given encoder, the encoder will encode an empty `.default` container in its place.
        case custom((_ value: Data, _ encoder: Encoder) throws -> Void)
    }

    /// The strategy to use for non-JSON-conforming floating-point values (IEEE 754 infinity and NaN).
    public enum NonConformingFloatEncodingStrategy {
        /// Throw upon encountering non-conforming values. This is the default strategy.
        case `throw`

        /// Encode the values using the given representation strings.
        case convertToString(positiveInfinity: String, negativeInfinity: String, nan: String)
    }

    /// The output format to produce. Defaults to `.compact`.
    open var outputFormatting: OutputFormatting

    /// The strategy to use in encoding dates. Defaults to `.deferredToDate`.
    open var dateEncodingStrategy: DateEncodingStrategy

    /// The strategy to use in encoding binary data. Defaults to `.base64`.
    open var dataEncodingStrategy: DataEncodingStrategy

    /// The strategy to use in encoding non-conforming numbers. Defaults to `.throw`.
    open var nonConformingFloatEncodingStrategy: NonConformingFloatEncodingStrategy

    /// Contextual information to expose during encoding.
    open var userInfo: [CodingUserInfoKey : Any]
}

open class JSONDecoder {
    // MARK: Top-Level Decoding

    /// Decodes a top-level value of the given type from the given JSON representation.
    ///
    /// - parameter type: The type of the value to decode.
    /// - parameter data: The data to decode from.
    /// - returns: A value of the requested type.
    /// - throws: `CocoaError.coderReadCorrupt` if values requested from the payload are corrupted, or if the given data is not valid JSON.
    /// - throws: An error if any value throws an error during decoding.
    open func decode<T : Decodable>(_ type: T.Type, from data: Data) throws -> Value

    // MARK: Customization

    /// The strategy to use for decoding `Date` values.
    public enum DateDecodingStrategy {
        /// Defer to `Date` for decoding. This is the default strategy.
        case deferredToDate

        /// Decode the `Date` as a UNIX timestamp from a JSON number.
        case secondsSince1970

        /// Decode the `Date` as UNIX millisecond timestamp from a JSON number.
        case millisecondsSince1970

        /// Decode the `Date` as an ISO-8601-formatted string (in RFC 3339 format).
        @available(OSX 10.12, iOS 10.0, watchOS 3.0, tvOS 10.0, *)
        case iso8601

        /// Decode the `Date` as a string parsed by the given formatter.
        case formatted(DateFormatter)

        /// Decode the `Date` as a custom value decoded by the given closure.
        case custom((_ decoder: Decoder) throws -> Date)
    }

    /// The strategy to use for decoding `Data` values.
    public enum DataDecodingStrategy {
        /// Decode the `Data` from a Base64-encoded string. This is the default strategy.
        case base64

        /// Decode the `Data` as a custom value decoded by the given closure.
        case custom((_ decoder: Decoder) throws -> Data)
    }

    /// The strategy to use for non-JSON-conforming floating-point values (IEEE 754 infinity and NaN).
    public enum NonConformingFloatDecodingStrategy {
        /// Throw upon encountering non-conforming values. This is the default strategy.
        case `throw`

        /// Decode the values from the given representation strings.
        case convertFromString(positiveInfinity: String, negativeInfinity: String, nan: String)
    }

    /// The strategy to use in decoding dates. Defaults to `.deferredToDate`.
    open var dateDecodingStrategy: DateDecodingStrategy

    /// The strategy to use in decoding binary data. Defaults to `.base64`.
    open var dataDecodingStrategy: DataDecodingStrategy

    /// The strategy to use in decoding non-conforming numbers. Defaults to `.throw`.
    open var nonConformingFloatDecodingStrategy: NonConformingFloatDecodingStrategy

    /// Contextual information to expose during decoding.
    open var userInfo: [CodingUserInfoKey : Any]
}
Usage:

var encoder = JSONEncoder()
encoder.dateEncodingStrategy = .iso8601
encoder.dataEncodingStrategy = .custom(myBase85Encoder)

// Since JSON does not natively allow for infinite or NaN values, we can customize strategies for encoding these non-conforming values.
encoder.nonConformingFloatEncodingStrategy = .convertToString(positiveInfinity: "INF", negativeInfinity: "-INF", nan: "NaN")

// MyValue conforms to Codable
let topLevel = MyValue(...)

let payload: Data
do {
    payload = try encoder.encode(topLevel)
} catch {
    // Some value threw while encoding.
}

// ...

var decoder = JSONDecoder()
decoder.dateDecodingStrategy = .iso8601
decoder.dataDecodingStrategy = .custom(myBase85Decoder)

// Look for and match these values when decoding `Double`s or `Float`s.
decoder.nonConformingFloatDecodingStrategy = .convertFromString(positiveInfinity: "INF", negativeInfinity: "-INF", nan: "NaN")

let topLevel: MyValue
do {
    topLevel = try decoder.decode(MyValue.self, from: payload)
} catch {
    // Data was corrupted, or some value threw while decoding.
}
It should be noted here that JSONEncoder and JSONDecoder do not themselves conform to Encoder and Decoder; instead, they contain private nested types which do conform to Encoder and Decoder, which are passed to values' encode(to:) and init(from:). This is because JSONEncoder and JSONDecoder must present a different top-level API than they would at intermediate levels.

Property List

We also intend to support the property list format, with PropertyListEncoder and PropertyListDecoder:

open class PropertyListEncoder {
    // MARK: Top-Level Encoding

    /// Encodes the given top-level value and returns its property list representation.
    ///
    /// - parameter value: The value to encode.
    /// - returns: A new `Data` value containing the encoded property list data.
    /// - throws: An error if any value throws an error during encoding.
    open func encode<T : Encodable>(_ value: T) throws -> Data

    // MARK: Customization

    /// The output format to write the property list data in. Defaults to `.binary`.
    open var outputFormat: PropertyListSerialization.PropertyListFormat

    /// Contextual information to expose during encoding.
    open var userInfo: [CodingUserInfoKey : Any]
}

open class PropertyListDecoder {
    // MARK: Top-Level Decoding

    /// Decodes a top-level value of the given type from the given property list representation.
    ///
    /// - parameter type: The type of the value to decode.
    /// - parameter data: The data to decode from.
    /// - returns: A value of the requested type.
    /// - throws: `CocoaError.coderReadCorrupt` if values requested from the payload are corrupted, or if the given data is not a valid property list.
    /// - throws: An error if any value throws an error during decoding.
    open func decode<T : Decodable>(_ type: T.Type, from data: Data) throws -> Value

    /// Decodes a top-level value of the given type from the given property list representation.
    ///
    /// - parameter type: The type of the value to decode.
    /// - parameter data: The data to decode from.
    /// - parameter format: The parsed property list format.
    /// - returns: A value of the requested type along with the detected format of the property list.
    /// - throws: `CocoaError.coderReadCorrupt` if values requested from the payload are corrupted, or if the given data is not a valid property list.
    /// - throws: An error if any value throws an error during decoding.
    open func decode<T : Decodable>(_ type: Value.Type, from data: Data, format: inout PropertyListSerialization.PropertyListFormat) throws -> Value

    // MARK: Customization

    /// Contextual information to expose during decoding.
    open var userInfo: [CodingUserInfoKey : Any]
}
Usage:

let encoder = PropertyListEncoder()
let topLevel = MyValue(...)
let payload: Data
do {
    payload = try encoder.encode(topLevel)
} catch {
    // Some value threw while encoding.
}

// ...

let decoder = PropertyListDecoder()
let topLevel: MyValue
do {
    topLevel = try decoder.decode(MyValue.self, from: payload)
} catch {
    // Data was corrupted, or some value threw while decoding.
}
Like with JSON, PropertyListEncoder and PropertyListDecoder also provide private nested types which conform to Encoder and Decoder for performing the archival.

Foundation-Provided Errors

Along with providing the above encoders and decoders, we would like to promote the use of a common set of error codes and messages across all new encoders and decoders. A common vocabulary of expected errors allows end-users to write code agnostic about the specific encoder/decoder implementation they are working with, whether first-party or third-party:

extension CocoaError.Code {
    /// Thrown when a value incompatible with the output format is encoded.
    public static var coderInvalidValue: CocoaError.Code

    /// Thrown when a value of a given type is requested but the encountered value is of an incompatible type.
    public static var coderTypeMismatch: CocoaError.Code

    /// Thrown when read data is corrupted or otherwise invalid for the format. This value already exists today.
    public static var coderReadCorrupt: CocoaError.Code

    /// Thrown when a requested key or value is unexpectedly null or missing. This value already exists today.
    public static var coderValueNotFound: CocoaError.Code
}

// These reexpose the values above.
extension CocoaError {
    public static var coderInvalidValue: CocoaError.Code

    public static var coderTypeMismatch: CocoaError.Code
}
The localized description strings associated with the two new error codes are:

.coderInvalidValue: "The data is not valid for encoding in this format."
.coderTypeMismatch: "The data couldn't be read because it isn't in the correct format." (Precedent from NSCoderReadCorruptError.)
All of these errors will include the coding key path that led to the failure in the error's userInfo dictionary under NSCodingPathErrorKey, along with a non-localized, developer-facing failure reason under NSDebugDescriptionErrorKey.

NSKeyedArchiver & NSKeyedUnarchiver Changes

Although our primary objectives for this new API revolve around Swift, we would like to make it easy for current consumers to make the transition to Codable where appropriate. As part of this, we would like to bridge compatibility between new Codable types (or newly-Codable-adopting types) and existing NSCoding types.

To do this, we want to introduce changes to NSKeyedArchiver and NSKeyedUnarchiver in Swift that allow archival of Codable types intermixed with NSCoding types:

// These are provided in the Swift overlay, and included in swift-corelibs-foundation.
extension NSKeyedArchiver {
    public func encodeCodable(_ codable: Encodable?, forKey key: String) { ... }
}

extension NSKeyedUnarchiver {
    public func decodeCodable<T : Decodable>(_ type: T.Type, forKey key: String) -> T? { ... }
}
NOTE: Since these changes are being made in extensions in the Swift overlay, it is not yet possible for these methods to be overridden. These can therefore not be added to NSCoder, since NSKeyedArchiver and NSKeyedUnarchiver would not be able to provide concrete implementations. In order to call these methods, it is necessary to downcast from an NSCoder to NSKeyedArchiver/NSKeyedUnarchiver directly. Since subclasses of NSKeyedArchiver and NSKeyedUnarchiverin Swift will inherit these implementations without being able to override them (which is wrong), we will NSRequiresConcreteImplementation() dynamically in subclasses.
The addition of these methods allows the introduction of Codable types into existing NSCoding structures, allowing for a transition to Codable types where appropriate.

Refining encode(_:forKey:)

Along with these extensions, we would like to refine the import of -[NSCoder encodeObject:forKey:], which is currently imported into Swift as encode(_: Any?, forKey: String). This method currently accepts Objective-C and Swift objects conforming to NSCoding (non-conforming objects produce a runtime error), as well as bridgeable Swift types (Data, String, Array, etc.); we would like to extend it to support new Swift Codable types, which would otherwise produce a runtime error upon call.

-[NSCoder encodeObject:forKey:] will be given a new Swift name of encodeObject(_:forKey:), and we will provide a replacement encode(_: Any?, forKey: String) in the overlay which will funnel out to either encodeCodable(_:forKey:)or encodeObject(_:forKey:) as appropriate. This should maintain source compatibility for end users already calling encode(_:forKey:), as well as behavior compatibility for subclassers of NSCoder and NSKeyedArchiver who may be providing their own encode(_:forKey:).

Semantics of Codable Types in Archives

There are a few things to note about including Codable values in NSKeyedArchiver archives:

Bridgeable Foundation types will always bridge before encoding. This is to facilitate writing Foundation types in a compatible format from both Objective-C and Swift
On decode, these types will decode either as their Objective-C or Swift version, depending on user need (decodeObject(forKey:) will decode as an Objective-C object; decodeCodable(_:forKey:) as a Swift value)
User types, which are not bridgeable, do not write out a $class and can only be decoded in Swift. In the future, we may add API to allow Swift types to provide an Objective-C class to decode as, effectively allowing for user bridging across archival
Foundation Types Adopting Codable

The following Foundation Swift types will be adopting Codable, and will encode as their bridged types when encoded through NSKeyedArchiver, as mentioned above:

AffineTransform
Calendar
CharacterSet
Date
DateComponents
DateInterval
Decimal
IndexPath
IndexSet
Locale
Measurement
Notification
PersonNameComponents
TimeZone
URL
URLComponents
URLRequest
UUID
Along with these, the Array, Dictionary, and Set types will gain Codable conformance (as part of the Conditional Conformance feature), and encode through NSKeyedArchiver as NSArray, NSDictionary, and NSSet respectively.

Source compatibility
The majority of this proposal is additive. The changes to NSKeyedArchiver are intended to be non-source-breaking changes, and non-behavior-breaking changes for subclasses in Objective-C and Swift.

Effect on ABI stability
The addition of this API will not be an ABI-breaking change. However, this will add limitations for changes in future versions of Swift, as parts of the API will have to remain unchanged between versions of Swift (barring some additions, discussed below).

Effect on API resilience
Much like new API added to the standard library, once added, some changes to this API will be ABI- and source-breaking changes. Changes to the new encoder and decoder classes provided above will be restricted as described in the library evolution document <https://github.com/apple/swift/blob/master/docs/LibraryEvolution.rst> in the Swift repository; in particular, the removal of methods or nested types or changes to argument types will break client behavior. Additionally, additions to provided options enums will be a source-breaking change for users performing an exhaustive switch over their cases; removal of cases will be ABI-breaking.


(Itai Ferber) #8

Hi Will,

Thanks for your comments!
`deferredToDate` simply uses the default implementation that `Date` provides — since it is not a primitive type like `Int` or `String` and conforms to `Codable` itself, it will have an implementation of `init(from:)` and `encode(to:)`. It will have an implementation that makes sense for `Date` in general, but since a big use-case for JSON lies in talking to external servers which you don't control, allowing for custom date formatting is important.

To that end, although ISO 8601 may make sense for some applications as the default, it is less efficient to format, encode, decode, and parse than, say, writing out a UNIX timestamp as a `Double` (or similar). Thus, the default is to allow `Date` to pick a representation that best suits itself, and if you need customization, you have the option to use it.

Since `Date` makes a concrete decision about how to encode, both sides will need to use `deferredToDate` for compatibility, in the same way that they would have to agree about ISO 8601, or any of the other options.

HTH!

— Itai

P.S. About Xcode autocompletion slowdown, I don't know to be honest, but I can't imagine it would be significant. Only certain types have `enc...` or `dec...` and even then, the list of methods isn't _that_ long.

···

On 15 Mar 2017, at 18:44, Will Stanton wrote:

Hello,

+1

This proposal seems helpful in standardizing how JSON objects can be written, and I commonly encode+decode JSON. The standard library JSON and PLIST encoders of Python are a strength, and Swift should be able to handle both formats just as easily. Still reading 'Swift Archival & Serialization’, but I believe both proposals will improve the safety and saneness of serializing/deserialization.

For the JSON coder, how does `deferredToDate` work? Would both the writer and reader have to agree to use `deferredToDate`?
Might it be better to force clients to pick a ‘real’ strategy? Why not default to one of the formats, perhaps ISO-8601?

(Not too important but also curious how much of a slowdown there will be when Xcode/SourceKit tries to autocomplete ‘enc’ or ‘dec’ for the Swift Archival & Serialization proposal?)

Regards,
Will Stanton

On Mar 15, 2017, at 6:43 PM, Itai Ferber via swift-evolution >> <swift-evolution@swift.org> wrote:

Hi everyone,
This is a companion proposal to the Foundation Swift Archival & Serialization API. This introduces new encoders and decoders to be used as part of this system.
The proposal is available online and inlined below.

— Itai

Swift Encoders
  • Proposal: SE-NNNN
  • Author(s): Itai Ferber, Michael LeHew, Tony Parker
  • Review Manager: TBD
  • Status: Awaiting review
  • Associated PRs:
    • #8124
Introduction
As part of the proposal for a Swift archival and serialization API (SE-NNNN), we are also proposing new API for specific new encoders and decoders, as well as introducing support for new Codable types in NSKeyedArchiver and NSKeyedUnarchiver.

This proposal composes the latter two stages laid out in SE-NNNN.

Motivation
With the base API discussed in SE-NNNN, we want to provide new encoders for consumers of this API, as well as provide a consistent story for bridging this new API with our existing NSCoding implementations. We would like to offer a base level of support that users can depend on, and set a pattern that third parties can follow in implementing and extending their own encoders.

Proposed solution
We will:

  • Add two new encoders and decoders to support encoding Swift value trees in JSON and property list formats
  • Add support for passing Codable Swift values to NSKeyedArchiver and NSKeyedUnarchiver, and add Codable conformance to our Swift value types
Detailed design
New Encoders and Decoders

JSON

One of the key motivations for the introduction of this API was to allow safer interaction between Swift values and their JSON representations. For values which are Codable, users can encode to and decode from JSON with JSONEncoder and JSONDecoder:

open class JSONEncoder {

// MARK: Top-Level Encoding

/// Encodes the given top-level value and returns its JSON representation.

///

/// - parameter value: The value to encode.

/// - returns: A new `Data` value containing the encoded JSON data.

/// - throws: `CocoaError.coderInvalidValue` if a non-comforming floating-point value is encountered during archiving, and the encoding strategy is `.throw`.

/// - throws: An error if any value throws an error during encoding.

    open
func encode<Value : Codable>(_ value: Value) throws -> Data

// MARK: Customization

/// The formatting of the output JSON data.

public enum OutputFormatting {

/// Produce JSON compacted by removing whitespace. This is the default formatting.

case
compact

/// Produce human-readable JSON with indented output.

case
prettyPrinted

}

/// The strategy to use for encoding `Date` values.

public enum DateEncodingStrategy {

/// Defer to `Date` for choosing an encoding. This is the default strategy.

case
deferredToDate

/// Encode the `Date` as a UNIX timestamp (as a JSON number).

case
secondsSince1970

/// Encode the `Date` as UNIX millisecond timestamp (as a JSON number).

case
millisecondsSince1970

/// Encode the `Date` as an ISO-8601-formatted string (in RFC 3339 format).

        @
available(OSX 10.12, iOS 10.0, watchOS 3.0, tvOS 10.0, *)

case
iso8601

/// Encode the `Date` as a string formatted by the given formatter.

case formatted(DateFormatter)

/// Encode the `Date` as a custom value encoded by the given closure.

///

/// If the closure fails to encode a value into the given encoder, the encoder will encode an empty `.default` container in its place.

case custom((_ value: Date, _ encoder: Encoder) throws -> Void)

}

/// The strategy to use for encoding `Data` values.

public enum DataEncodingStrategy {

/// Encoded the `Data` as a Base64-encoded string. This is the default strategy.

case
base64

/// Encode the `Data` as a custom value encoded by the given closure.

///

/// If the closure fails to encode a value into the given encoder, the encoder will encode an empty `.default` container in its place.

case custom((_ value: Data, _ encoder: Encoder) throws -> Void)

}

/// The strategy to use for non-JSON-conforming floating-point values (IEEE 754 infinity and NaN).

public enum NonConformingFloatEncodingStrategy {

/// Throw upon encountering non-conforming values. This is the default strategy.

case `throw
`

/// Encode the values using the given representation strings.

case convertToString(positiveInfinity: String, negativeInfinity: String, nan: String)

}

/// The output format to produce. Defaults to `.compact`.

    open
var outputFormatting: OutputFormatting

/// The strategy to use in encoding dates. Defaults to `.deferredToDate`.

    open
var dateEncodingStrategy: DateEncodingStrategy

/// The strategy to use in encoding binary data. Defaults to `.base64`.

    open
var dataEncodingStrategy: DataEncodingStrategy

/// The strategy to use in encoding non-conforming numbers. Defaults to `.throw`.

    open
var nonConformingFloatEncodingStrategy: NonConformingFloatEncodingStrategy
}

open
class JSONDecoder {

// MARK: Top-Level Decoding

/// Decodes a top-level value of the given type from the given JSON representation.

///

/// - parameter type: The type of the value to decode.

/// - parameter data: The data to decode from.

/// - returns: A value of the requested type.

/// - throws: `CocoaError.coderReadCorrupt` if values requested from the payload are corrupted, or if the given data is not valid JSON.

/// - throws: An error if any value throws an error during decoding.

    open
func decode<Value : Codable>(_ type: Value.Type, from data: Data) throws -> Value

// MARK: Customization

/// The strategy to use for decoding `Date` values.

public enum DateDecodingStrategy {

/// Defer to `Date` for decoding. This is the default strategy.

case
deferredToDate

/// Decode the `Date` as a UNIX timestamp from a JSON number.

case
secondsSince1970

/// Decode the `Date` as UNIX millisecond timestamp from a JSON number.

case
millisecondsSince1970

/// Decode the `Date` as an ISO-8601-formatted string (in RFC 3339 format).

        @
available(OSX 10.12, iOS 10.0, watchOS 3.0, tvOS 10.0, *)

case
iso8601

/// Decode the `Date` as a string parsed by the given formatter.

case formatted(DateFormatter)

/// Decode the `Date` as a custom value decoded by the given closure.

case custom((_ decoder: Decoder) throws -> Date)

}

/// The strategy to use for decoding `Data` values.

public enum DataDecodingStrategy {

/// Decode the `Data` from a Base64-encoded string. This is the default strategy.

case
base64

/// Decode the `Data` as a custom value decoded by the given closure.

case custom((_ decoder: Decoder) throws -> Data)

}

/// The strategy to use for non-JSON-conforming floating-point values (IEEE 754 infinity and NaN).

public enum NonConformingFloatDecodingStrategy {

/// Throw upon encountering non-conforming values. This is the default strategy.

case `throw
`

/// Decode the values from the given representation strings.

case convertFromString(positiveInfinity: String, negativeInfinity: String, nan: String)

}

/// The strategy to use in decoding dates. Defaults to `.deferredToDate`.

    open
var dateDecodingStrategy: DateDecodingStrategy

/// The strategy to use in decoding binary data. Defaults to `.base64`.

    open
var dataDecodingStrategy: DataDecodingStrategy

/// The strategy to use in decoding non-conforming numbers. Defaults to `.throw`.

    open
var nonConformingFloatDecodingStrategy: NonConformingFloatDecodingStrategy
}
Usage:

var encoder = JSONEncoder()

encoder
.dateEncodingStrategy = .
iso8601
encoder
.dataEncodingStrategy = .custom(myBase85Encoder)

// Since JSON does not natively allow for infinite or NaN values, we can customize strategies for encoding these non-conforming values.

encoder
.nonConformingFloatEncodingStrategy = .convertToString(positiveInfinity: "INF", negativeInfinity: "-INF", nan: "NaN")

// MyValue conforms to Codable
let topLevel = MyValue(...)

let payload: Data
do {

    payload
= try encoder.encode(topLevel)
} catch {

// Some value threw while encoding.
}

// ...

var decoder = JSONDecoder()

decoder
.dateDecodingStrategy = .
iso8601
decoder
.dataDecodingStrategy = .custom(myBase85Decoder)

// Look for and match these values when decoding `Double`s or `Float`s.

decoder
.nonConformingFloatDecodingStrategy = .convertFromString(positiveInfinity: "INF", negativeInfinity: "-INF", nan: "NaN")

let topLevel: MyValue
do {

    topLevel
= try decoder.decode(MyValue.self, from: payload)
} catch {

// Data was corrupted, or some value threw while decoding.
}
It should be noted here that JSONEncoder and JSONDecoder do not themselves conform to Encoder and Decoder; instead, they contain private nested types which do conform to Encoder and Decoder, which are passed to values' encode(to:)and init(from:). This is because JSONEncoder and JSONDecoder must present a different top-level API than they would at intermediate levels.

Property List

We also intend to support the property list format, with PropertyListEncoder and PropertyListDecoder:

open class PropertyListEncoder {

// MARK: Top-Level Encoding

/// Encodes the given top-level value and returns its property list representation.

///

/// - parameter value: The value to encode.

/// - returns: A new `Data` value containing the encoded property list data.

/// - throws: An error if any value throws an error during encoding.

    open
func encode<Value : Codable>(_ value: Value) throws -> Data

// MARK: Customization

/// The output format to write the property list data in. Defaults to `.binary`.

    open
var outputFormat: PropertyListSerialization.PropertyListFormat
}

open
class PropertyListDecoder {

// MARK: Top-Level Decoding

/// Decodes a top-level value of the given type from the given property list representation.

///

/// - parameter type: The type of the value to decode.

/// - parameter data: The data to decode from.

/// - returns: A value of the requested type.

/// - throws: `CocoaError.coderReadCorrupt` if values requested from the payload are corrupted, or if the given data is not a valid property list.

/// - throws: An error if any value throws an error during decoding.

    open
func decode<Value : Codable>(_ type: Value.Type, from data: Data) throws -> Value

/// Decodes a top-level value of the given type from the given property list representation.

///

/// - parameter type: The type of the value to decode.

/// - parameter data: The data to decode from.

/// - parameter format: The parsed property list format.

/// - returns: A value of the requested type along with the detected format of the property list.

/// - throws: `CocoaError.coderReadCorrupt` if values requested from the payload are corrupted, or if the given data is not a valid property list.

/// - throws: An error if any value throws an error during decoding.

    open
func decode<Value : Codable>(_ type: Value.Type, from data: Data, format: inout PropertyListSerialization.PropertyListFormat) throws -> Value
}
Usage:

let encoder = PropertyListEncoder()
let topLevel = MyValue(...)
let payload: Data
do {

    payload
= try encoder.encode(topLevel)
} catch {

// Some value threw while encoding.
}

// ...

let decoder = PropertyListDecoder()
let topLevel: MyValue
do {

    topLevel
= try decoder.decode(MyValue.self, from: payload)
} catch {

// Data was corrupted, or some value threw while decoding.
}
Like with JSON, PropertyListEncoder and PropertyListDecoder also provide private nested types which conform to Encoder and Decoder for performing the archival.

Foundation-Provided Errors

Along with providing the above encoders and decoders, we would like to promote the use of a common set of error codes and messages across all new encoders and decoders. A common vocabulary of expected errors allows end-users to write code agnostic about the specific encoder/decoder implementation they are working with, whether first-party or third-party:

extension CocoaError.Code {

/// Thrown when a value incompatible with the output format is encoded.

public static var coderInvalidValue: CocoaError.Code

/// Thrown when a value of a given type is requested but the encountered value is of an incompatible type.

public static var coderTypeMismatch: CocoaError.Code

/// Thrown when read data is corrupted or otherwise invalid for the format. This value already exists today.

public static var coderReadCorrupt: CocoaError.Code

/// Thrown when a requested key or value is unexpectedly null or missing. This value already exists today.

public static var coderValueNotFound: CocoaError.Code
}

// These reexpose the values above.
extension CocoaError {

public static var coderInvalidValue: CocoaError.Code

public static var coderTypeMismatch: CocoaError.Code
}
The localized description strings associated with the two new error codes are:

  • .coderInvalidValue: "The data is not valid for encoding in this format."
  • .coderTypeMismatch: "The data couldn't be read because it isn't in the correct format." (Precedent from NSCoderReadCorruptError.)
All of these errors will include the coding key path that led to the failure in the error's userInfo dictionary under NSCodingKeyContextErrorKey, along with a non-localized, developer-facing failure reason under NSDebugDescriptionErrorKey.

NSKeyedArchiver & NSKeyedUnarchiver Changes

Although our primary objectives for this new API revolve around Swift, we would like to make it easy for current consumers to make the transition to Codable where appropriate. As part of this, we would like to bridge compatibility between new Codabletypes (or newly-Codable-adopting types) and existing NSCoding types.

To do this, we want to introduce changes to NSKeyedArchiver and NSKeyedUnarchiver in Swift that allow archival of Codable types intermixed with NSCoding types:

// These are provided in the Swift overlay, and included in swift-corelibs-foundation.
extension NSKeyedArchiver {

public func encodeCodable(_ codable: Codable?, forKey key: String) { ... }
}

extension NSKeyedUnarchiver {

public func decodeCodable<T : Codable>(_ type: T.Type, forKey key: String) -> T? { ... }
}
NOTE: Since these changes are being made in extensions in the Swift overlay, it is not yet possible for these methods to be overridden. These can therefore not be added to NSCoder, since NSKeyedArchiver and NSKeyedUnarchiver would not be able to provide concrete implementations. In order to call these methods, it is necessary to downcast from an NSCoder to NSKeyedArchiver/NSKeyedUnarchiver directly. Since subclasses of NSKeyedArchiver and NSKeyedUnarchiver in Swift will inherit these implementations without being able to override them (which is wrong), we will NSRequiresConcreteImplementation() dynamically in subclasses.
The addition of these methods allows the introduction of Codable types into existing NSCoding structures, allowing for a transition to Codable types where appropriate.

Refining encode(_:forKey:)

Along with these extensions, we would like to refine the import of -[NSCoder encodeObject:forKey:], which is currently imported into Swift as encode(_: Any?, forKey: String). This method currently accepts Objective-C and Swift objects conforming to NSCoding (non-conforming objects produce a runtime error), as well as bridgeable Swift types (Data, String, Array, etc.); we would like to extend it to support new Swift Codable types, which would otherwise produce a runtime error upon call.

-[NSCoder encodeObject:forKey:] will be given a new Swift name of encodeObject(_:forKey:), and we will provide a replacement encode(_: Any?, forKey: String) in the overlay which will funnel out to either encodeCodable(_:forKey:) or encodeObject(_:forKey:) as appropriate. This should maintain source compatibility for end users already calling encode(_:forKey:), as well as behavior compatibility for subclassers of NSCoderand NSKeyedArchiver who may be providing their own encode(_:forKey:).

Semantics of Codable Types in Archives

There are a few things to note about including Codable values in NSKeyedArchiverarchives:

  • Bridgeable Foundation types will always bridge before encoding. This is to facilitate writing Foundation types in a compatible format from both Objective-C and Swift
    • On decode, these types will decode either as their Objective-C or Swift version, depending on user need (decodeObject(forKey:) will decode as an Objective-C object; decodeCodable(_:forKey:) as a Swift value)
  • User types, which are not bridgeable, do not write out a $class and can only be decoded in Swift. In the future, we may add API to allow Swift types to provide an Objective-C class to decode as, effectively allowing for user bridging across archival
Foundation Types Adopting Codable

The following Foundation Swift types will be adopting Codable, and will encode as their bridged types when encoded through NSKeyedArchiver, as mentioned above:

  • AffineTransform
  • Calendar
  • CharacterSet
  • Date
  • DateComponents
  • DateInterval
  • Decimal
  • IndexPath
  • IndexSet
  • Locale
  • Measurement
  • Notification
  • PersonNameComponents
  • TimeZone
  • URL
  • URLComponents
  • URLRequest
  • UUID
Along with these, the Array, Dictionary, and Set types will gain Codableconformance (as part of the Conditional Conformance feature), and encode through NSKeyedArchiver as NSArray, NSDictionary, and NSSet respectively.

Source compatibility
The majority of this proposal is additive. The changes to NSKeyedArchiver are intended to be non-source-breaking changes, and non-behavior-breaking changes for subclasses in Objective-C and Swift.

Effect on ABI stability
The addition of this API will not be an ABI-breaking change. However, this will add limitations for changes in future versions of Swift, as parts of the API will have to remain unchanged between versions of Swift (barring some additions, discussed below).

Effect on API resilience
Much like new API added to the standard library, once added, some changes to this API will be ABI- and source-breaking changes. Changes to the new encoder and decoder classes provided above will be restricted as described in the library evolution document in the Swift repository; in particular, the removal of methods or nested types or changes to argument types will break client behavior. Additionally, additions to provided options enums will be a source-breaking change for users performing an exhaustive switch over their cases; removal of cases will be ABI-breaking.

Alternatives considered
None. This is a companion to the Swift Archival and Serialization API.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Will Stanton) #9

Hello Itai,

Thanks for your response and its explanations!

Agreed that comprehension of multiple formats is important since there are a couple common ways of encoding JSON dates!

Still, ISO 8601 appears pretty often (though I don’t have data on that, Stack Overflow says RFC 7493 I-JSON prefers ISO 8601; https://tools.ietf.org/html/rfc7493#section-4.3), and as other servers might make/handle a lot of JSON produced to/from the API, I think it would be disadvantageous to default to `deferredToDate` (if `deferredToDate` doesn't use the ISO 8601 format).

As you mention, writers/readers have to agree on their format - my 2¢ is that ISO 8601 would be more common, and so a better default, than a Unix or reference date timestamp.

Regards,
Will Stanton

-—Gracias for the prediction :slight_smile:

···

On Mar 15, 2017, at 9:53 PM, Itai Ferber <iferber@apple.com> wrote:

Hi Will,

Thanks for your comments!
deferredToDate simply uses the default implementation that Date provides — since it is not a primitive type like Int or String and conforms to Codable itself, it will have an implementation of init(from:) and encode(to:). It will have an implementation that makes sense for Date in general, but since a big use-case for JSON lies in talking to external servers which you don't control, allowing for custom date formatting is important.

To that end, although ISO 8601 may make sense for some applications as the default, it is less efficient to format, encode, decode, and parse than, say, writing out a UNIX timestamp as a Double (or similar). Thus, the default is to allow Date to pick a representation that best suits itself, and if you need customization, you have the option to use it.

Since Date makes a concrete decision about how to encode, both sides will need to use deferredToDate for compatibility, in the same way that they would have to agree about ISO 8601, or any of the other options.

HTH!

— Itai

P.S. About Xcode autocompletion slowdown, I don't know to be honest, but I can't imagine it would be significant. Only certain types have enc... or dec... and even then, the list of methods isn't that long.


(Itai Ferber) #10

Foundation’s common currency type for errors is unequivocally `NSError`, and we’re not looking to change that as part of this proposal.
Reusing errors from the Cocoa domain has always been a reasonable choice for 3rd-party developers, and we’d like to keep doing this.

We are looking to improve the experience of throwing such errors (with some trivial extensions) by allowing you to do something like `throw CocoaError.error(.coderInvalidValue)`. (Of course, don’t quote me on the exact syntax, as this is totally subject to change.)
This is pretty simple, and nets you free user-facing localization.

···

On 15 Mar 2017, at 22:46, Zach Waldowski wrote:

Just a little one that came up with when thinking over the companion proposal… see inline.

On Mar 15, 2017, at 6:43 PM, Itai Ferber via swift-evolution >> <swift-evolution@swift.org> wrote:
Foundation-Provided Errors

Along with providing the above encoders and decoders, we would like to promote the use of a common set of error codes and messages across all new encoders and decoders. A common vocabulary of expected errors allows end-users to write code agnostic about the specific encoder/decoder implementation they are working with, whether first-party or third-party:

extension CocoaError.Code {
    public static var coderInvalidValue: CocoaError.Code
    public static var coderTypeMismatch: CocoaError.Code
    public static var coderReadCorrupt: CocoaError.Code
    public static var coderValueNotFound: CocoaError.Code
}

I find all the reasons compelling, but I don’t with the overall choice. A common vocabulary, side table info, and localization can be achieved with a dedicated error type for coders. I understand ABI (sorta) concerns are involved here, that older Foundations will continue to return the same NSError codes. However, it's never felt appropriate to create my own error instances with NSCocoaErrorDomain, and this feels very similar to that. What are our options here?

Zach Waldowski
zach@waldowski.me


(Itai Ferber) #11

Hi everyone,
This is a companion proposal to the Foundation Swift Archival & Serialization API. This introduces new encoders and decoders to be used as part of this system.
The proposal is available online and inlined below.

Executive summary: I like where you're going with this, but I'm worried about flexibility.

I'm not going to quote every bit of the JSON section because Apple Mail seems to destroy the formatting when I reply, but: I think you've identified several of the most important customization points (Date, Data, and illegal Floats). However, I think:

* People may want to map illegal Floats to legal floating-point values (say, `greatestFiniteMagnitude`, `-greatestFiniteMagnitude`, and `0`) or map them to `null`s. They may also want different behavior for different things: imagine `(positiveInfinity: Double.greatestFiniteMagnitude, negativeInfinity: -Double.greatestFiniteMagnitude, nan: .throw)`.

I agree, this may be something that users could want.

* Large integers are another big concern that you don't address. Because JSON only supports doubles, APIs that use 64-bit IDs often need them to be passed as strings, frequently with a different key ("id_str" instead of "id").

This is not true — JSON has no limitations on what numbers it can represent. 340282366920938463463374607431768211455 (2^128-1) is a perfectly legitimate number in JSON, though you may have a hard reading it in on some platforms. _Javascript_ numbers are IEEE 754 doubles, but that’s a Javascript problem, not a JSON problem.

If what you mean here is that some large numbers should be encoded as strings instead of integers for the benefit of the other side reading it in a valid way, then perhaps.

* For that matter, style and capitalization are a problem. JSON style varies, but it *tends* to be snake_case, where Cocoa favors camelCase. You can address this at the CodingKey level by manually specifying string equivalents of all the coding keys, but that's kind of a pain, and it affects all of your code and all of your serializations.

I'm sorely tempted to suggest that we give the JSON encoder and decoder a delegate:

  public protocol JSONCodingDelegate {
    /// Returns the string name to be used when encoding or decoding the given CodingKey as JSON.
    ///
    /// - Returns: The string to use, or `nil` for the default.
    func jsonName(for key: CodingKey, at keyPath: [CodingKey], in encoderOrDecoder: AnyObject) throws -> String?

    // These are used when encoding/decoding any of the integer types.
    func jsonValue(from integer: Int64, at keyPath: [CodingKey], in encoder: JSONEncoder) throws -> JSONValue?
    func integer(from jsonValue: JSONValue, at keyPath: [CodingKey], in decoder: JSONDecoder) throws -> Int64?
    
    // These are used when encoding/decoding any of the floating-point types.
    func jsonValue(from number: Double, at keyPath: [CodingKey], in encoder: JSONEncoder) throws -> JSONValue?
    func number(from jsonValue: JSONValue, at keyPath: [CodingKey], in decoder: JSONDecoder) throws -> Double?
    
    // These are used when encoding/decoding Date.
    func jsonValue(from date: Date, at keyPath: [CodingKey], in encoder: JSONEncoder) throws -> JSONValue?
    func date(from jsonValue: JSONValue, at keyPath: [CodingKey], in decoder: JSONDecoder) throws -> Date?
    
    // These are used when encoding/decoding Data.
    func jsonValue(from data: Data, at keyPath: [CodingKey], in encoder: JSONEncoder) throws -> JSONValue?
    func data(from jsonValue: JSONValue, at keyPath: [CodingKey], in decoder: JSONDecoder) throws -> Data?
    
    func jsonValue(from double: Double, at keyPath: [CodingKey], in encoder: JSONEncoder) throws -> JSONValue?
    func integer(from jsonValue: JSONValue, at keyPath: [CodingKey], in decoder: JSONDecoder) throws -> Double?
  }
  public enum JSONValue {
    case string(String)
    case number(Double)
    case bool(Bool)
    case object([String: JSONValue])
    case array([JSONValue])
    case null
  }

I disagree with generalizing this to the point of being on a delegate. This is all work that you could be doing in `encode(to:)` and `decode(from:)`. In `encode(to:)`, it’s always possible to clamp an invalid floating-point number to `Double.greatestFiniteMagnitude`, and always possible to `encode("\(id)", forKey: .id)` instead of `encode(id, forKey: .id)`.

The options that we have on `JSONEncoder` and `JSONDecoder` straddle a fine line between being pedantically correct (and refusing to break encapsulation for encoded types), and being pragmatically useful. In theory, it certainly feels "wrong" that we would allow someone to change the way in which a `Date` is encoded, or how `Double`s are represented; in a pragmatic sense, though, JSON has no native representation of such `Double` values, or a standardized representation of dates, and it’s useful to provide options to for controlling that.

However, allowing a delegate to intercept all such calls feels like it leans too much in the wrong direction. We’d like to offer as limited a set of knobs as possible while still being useful.

Or, perhaps, that a more general form of this delegate be available on all encoders and decoders. But that may be overkill, and even if it *is* a good idea, it's one we can add later.

Property List

We also intend to support the property list format, with PropertyListEncoder and PropertyListDecoder:

No complaints here.

Foundation-Provided Errors

Along with providing the above encoders and decoders, we would like to promote the use of a common set of error codes and messages across all new encoders and decoders. A common vocabulary of expected errors allows end-users to write code agnostic about the specific encoder/decoder implementation they are working with, whether first-party or third-party:

extension CocoaError.Code {
    /// Thrown when a value incompatible with the output format is encoded.
    public static var coderInvalidValue: CocoaError.Code

    /// Thrown when a value of a given type is requested but the encountered value is of an incompatible type.
    public static var coderTypeMismatch: CocoaError.Code

    /// Thrown when read data is corrupted or otherwise invalid for the format. This value already exists today.
    public static var coderReadCorrupt: CocoaError.Code

    /// Thrown when a requested key or value is unexpectedly null or missing. This value already exists today.
    public static var coderValueNotFound: CocoaError.Code
}

[snip]

All of these errors will include the coding key path that led to the failure in the error's userInfo dictionary under NSCodingKeyContextErrorKey, along with a non-localized, developer-facing failure reason under NSDebugDescriptionErrorKey.

Now comes the part where I whine like a four-year-old:

"Do we haaaaaaaave to use the `userInfo` dictionary, papa?"

An enum with an associated value would be a much more natural way to express these errors and the data that comes with them. Failing that, at least give us some convenience properties. The untyped bag of stuff in the `userInfo` dictionary fills developers who spend all their time in Swift with fear and loathing.

Actually, if you wanted to help us out with the "untyped bag of stuff" problem in general, I for one wouldn't say "no":

  public struct TypedKey<Key: Hashable, Value> {
    public var key: Key
    public init(key: Key, valueType: Value.Type) {
      self.key = key
    }
  }
  extension Dictionary where Value == Any {
    public subscript<CastedValue>(typedKey: TypedKey<Key, CastedValue>) -> CastedValue? {
      get {
        return self[typedKey.key] as? CastedValue
      }
      set {
        self[typedKey.key] = newValue
      }
    }
  }

As explained in a different email, `NSError` is Foundation’s common currency for errors, and we are not looking to change that as part of this proposal. If we were to add a `userInfo` dictionary to `Encoder` and `Decoder` mentioned in my other email to you, it would likely take on more of the form that you suggest here.

NSKeyedArchiver & NSKeyedUnarchiver Changes

Although our primary objectives for this new API revolve around Swift, we would like to make it easy for current consumers to make the transition to Codable where appropriate. As part of this, we would like to bridge compatibility between new Codabletypes (or newly-Codable-adopting types) and existing NSCoding types.

To do this, we want to introduce changes to NSKeyedArchiver and NSKeyedUnarchiver in Swift that allow archival of Codable types intermixed with NSCoding types:

// These are provided in the Swift overlay, and included in swift-corelibs-foundation.
extension NSKeyedArchiver {
    public func encodeCodable(_ codable: Codable?, forKey key: String) { ... }
}

extension NSKeyedUnarchiver {
    public func decodeCodable<T : Codable>(_ type: T.Type, forKey key: String) -> T? { ... }
}

NOTE: Since these changes are being made in extensions in the Swift overlay, it is not yet possible for these methods to be overridden. These can therefore not be added to NSCoder, since NSKeyedArchiver and NSKeyedUnarchiver would not be able to provide concrete implementations. In order to call these methods, it is necessary to downcast from an NSCoder to NSKeyedArchiver/NSKeyedUnarchiver directly. Since subclasses of NSKeyedArchiver and NSKeyedUnarchiver in Swift will inherit these implementations without being able to override them (which is wrong), we will NSRequiresConcreteImplementation() dynamically in subclasses.

The addition of these methods allows the introduction of Codable types into existing NSCoding structures, allowing for a transition to Codable types where appropriate.

I wonder about this.

Could `NSCoding` be imported in Swift as refining `Codable`? Then we could all just forget `NSCoding` exists, other than that certain types are less likely to properly handle being put into a JSONEncoder/Decoder. (Which, to tell the truth, is probably inevitable here; the Encoder and Decoder types look like they're probably too loosely defined to truly guarantee that you can mix-and-match coders and types without occasional problems.)

Likely not — `NSCoding` and `Codable` don’t support the same features:

* `NSCoding` implementations currently write type information into produced archives. Off the top of my head, I don’t think this is a strict necessity of the API, but a _lot_ of `NSCoding` implementations rely on this
* `Codable` requires type information on decode; `NSCoding` does not (because of the aforementioned type information in archives). We cannot translate decodes properly
* `NSCoding` has different primitive types than `Codable`

They don’t translate 1-to-1, and would have to stay completely distinct.

Semantics of Codable Types in Archives

There are a few things to note about including Codable values in NSKeyedArchiverarchives:

  • Bridgeable Foundation types will always bridge before encoding. This is to facilitate writing Foundation types in a compatible format from both Objective-C and Swift
    • On decode, these types will decode either as their Objective-C or Swift version, depending on user need (decodeObject(forKey:) will decode as an Objective-C object; decodeCodable(_:forKey:) as a Swift value)

This sounds sensible.

  • User types, which are not bridgeable, do not write out a $class and can only be decoded in Swift. In the future, we may add API to allow Swift types to provide an Objective-C class to decode as, effectively allowing for user bridging across archival

Even pure Swift class types? I guess that's probably necessary since even our private ability to look up classes at runtime doesn't cover things like generics, but...ugh.

I’m not sure what you mean by this comment. If you have an old codebase which you’re converting from Objective-C to Swift but want to support writing archives from newer versions of the codebase still readable by old versions, it isn’t unreasonable to provide an Objective-C class to encode as or from…

Along with these, the Array, Dictionary, and Set types will gain Codableconformance (as part of the Conditional Conformance feature), and encode through NSKeyedArchiver as NSArray, NSDictionary, and NSSet respectively.

You might need to be careful here—we'll need to make sure that data structures of Swift types bridge properly. I suppose that means `_SwiftValue` will need to support `NSCoding` after all...

It shouldn’t have to. NSKeyedArchiver can make callbacks into Swift for the encoding of `_SwiftValue`s it finds which did not bridge.

···

On 16 Mar 2017, at 1:00, Brent Royal-Gordon wrote:

On Mar 15, 2017, at 3:43 PM, Itai Ferber via swift-evolution >> <swift-evolution@swift.org> wrote:

--
Brent Royal-Gordon
Architechies


(Itai Ferber) #12

By stable ordering, you mean dictionaries with keys being output in the order that they were encoded?
Potentially, but this would require additional work as neither Swift dictionaries nor `NSMutableDictionary` support this (reasonable seeing as dictionaries are by definition not sorted), and the current implementation uses `JSONSerialization` as the concrete serializer. Certainly not impossible, though.

···

On 16 Mar 2017, at 8:49, Ben Rimmington wrote:

<https://github.com/apple/swift-evolution/pull/640>

Could the `JSONEncoder.OutputFormatting.prettyPrinted` case include the **stable ordering** of keys?

-- Ben


(Itai Ferber) #13

Thank you again for bringing these great proposals forward!

Thanks for reviewing it, and for your comments!

I only have a couple of questions about this proposal.

I noticed that the types in this proposal don’t conform to Encoder and Decoder. Is the plan to have them to provide private conforming types to Codable types they are asked to encode or decode?

Yes. This is because the top-level interface for encoding and decoding in JSON and plist is different from the intermediate interface that `Encoder` and `Decoder` offer. As such, the top-level types don’t conform to `Encoder` and `Decoder`, but vend out internal types which do.

Why are the strategy and format properties readwrite instead of configured at initialization time? Is the intent that the encoder / decoder can be re-used with a different configuration in a subsequent call to encode or decode?

Yes. It’s also a mouthful to have them all as params in the constructor, especially if we add more options in the future.

Finally, I agree with Brent’s comments regarding errors. I would prefer to see Foundation move away from NSError in favor of domain-specific error types. That said, the comment that this is a broader discussion for Foundation and not something to change in this proposal is reasonable. I hope Foundation will consider changing this in the future.

Thanks for your understanding — we will keep these concerns in mind.

···

On 16 Mar 2017, at 14:48, Matthew Johnson wrote:

Matthew


(Matthew Johnson) #14

Thank you again for bringing these great proposals forward!

Thanks for reviewing it, and for your comments!

I only have a couple of questions about this proposal.

I noticed that the types in this proposal don’t conform to Encoder and Decoder. Is the plan to have them to provide private conforming types to Codable types they are asked to encode or decode?

Yes. This is because the top-level interface for encoding and decoding in JSON and plist is different from the intermediate interface that Encoder and Decoder offer. As such, the top-level types don’t conform to Encoder and Decoder, but vend out internal types which do.

This makes sense. I was initially concerned about the meaning of mutating these values during encoding or decoding but it looks like that isn’t possible without some really nefarious code that passes a reference to the top-level encoder / decoder to an object that is getting encoded / decoded. What will you do if somebody actually does this?

Why are the strategy and format properties readwrite instead of configured at initialization time? Is the intent that the encoder / decoder can be re-used with a different configuration in a subsequent call to encode or decode?

Yes. It’s also a mouthful to have them all as params in the constructor, especially if we add more options in the future.

Taking them in an initializer would not need to be wordy - they could all specify default arguments.

···

On Mar 16, 2017, at 5:37 PM, Itai Ferber <iferber@apple.com> wrote:
On 16 Mar 2017, at 14:48, Matthew Johnson wrote:
Finally, I agree with Brent’s comments regarding errors. I would prefer to see Foundation move away from NSError in favor of domain-specific error types. That said, the comment that this is a broader discussion for Foundation and not something to change in this proposal is reasonable. I hope Foundation will consider changing this in the future.

Thanks for your understanding — we will keep these concerns in mind.

Matthew


(Ben Rimmington) #15

Sorry, I actually meant that keys would be ordered **alphabetically** (as they are with PropertyListSerialization).

I'm storing output from JSONSerialization in a git repository, and when I add/remove a single key-value the entire file changes, due to keys being reordered (seemingly at random).

-- Ben

···

Itai Ferber wrote:

By stable ordering, you mean dictionaries with keys being output in the order that they were encoded?
Potentially, but this would require additional work as neither Swift dictionaries nor NSMutableDictionary support this (reasonable seeing as dictionaries are by definition not sorted), and the current implementation uses JSONSerialization as the concrete serializer. Certainly not impossible, though.

On 16 Mar 2017, at 8:49, Ben Rimmington wrote:

<https://github.com/apple/swift-evolution/pull/640>

Could the `JSONEncoder.OutputFormatting.prettyPrinted` case include the **stable ordering** of keys?

-- Ben


(Brent Royal-Gordon) #16

This is a good revision to a good proposal.

I'm glad `CodingKey`s now require `stringValue`s; I think the intended semantics are now a lot clearer, and key behavior will be much more reliable.

I like the separation between keyed and unkeyed containers (and I think "unkeyed" is a good name, though not perfect), but I'm not quite happy with the unkeyed container API. Encoding a value into an unkeyed container appends it to the container's end; decoding a value from an unkeyed container removes it from the container's front. These are very important semantics that the method names in question do not imply at all. Certain aspects of `UnkeyedDecodingContainer` also feel like they do the same things as `Sequence` and `IteratorProtocol`, but in different and incompatible ways. And I certainly think that the `encode(contentsOf:)` methods on `UnkeyedEncodingContainer` could use equivalents on the `UnkeyedDecodingContainer`. Still, the design in this area is much improved compared to the previous iteration.

(Tiny nitpick: I keep finding myself saying "encode into", not "encode to" as the API name suggests. Would that be a better parameter label?)

I like the functionality of the `userInfo` dictionary, but I'm still not totally satisfied casting out of `Any` all the time. I might just have to get over that, though.

I wonder if `CodingKey` implementations might ever need access to the `userInfo`. I suppose you can just switch to a different set of `CodingKeys` if you do.

Should there be a way for an `init(from:)` implementation to determine the type of container in the encoder it's just been handed? Or perhaps the better question is, do we want to promise users that all decoders can tell the difference?

* * *

I went ahead and implemented a basic version of `Encoder` and `Encodable` in a Swift 3 playground, just to get a feel for this system in action and experiment with a few things. A few observations:

* I think it may make sense to class-constrain some of these protocols. `Encodable` and its containers seem to inherently have reference semantics—otherwise data could never be communicated from all those `encode` calls out to the ultimate caller of the API. Class-constraining would clearly communicate this to both the implementer and the compiler. `Decoder` and its containers don't *inherently* have reference semantics, but I'm not sure it's a good idea to potentially copy around a lot of state in a value type.

* I really think that including overloads for every primitive type in all three container types is serious overkill. In my implementation, the primitive types' `Encodable` conformances simply request a `SingleValueEncodingContainer` and write themselves into it. I can't imagine any coder doing anything in their overloads that wouldn't be compatible with that, especially since they can never be sure when someone will end up using the `Encodable` conformance directly instead of the primitive. So what are all these overloads buying us? Are they just avoiding a generic dispatch and the creation of a new `Encoder` and perhaps a `SingleValueEncodingContainer`? I don't think that's worth the increased API surface, the larger overload sets, or the danger that an encoder might accidentally implement one of the duplicative primitive encoding calls inconsistently with the others.

To be clear: In my previous comments, I suggested that we should radically reduce the number of primitive types. That is not what I'm saying here. I'm saying that we should always use a single value container to encode and decode primitives, and the other container types should always use `Encodable` or `Decodable`. This doesn't reduce the capabilities of the system at all; it just means you only have to write the code to handle a given primitive type one time instead of three.

* And then there's the big idea: Changing the type of the parameter to `encode(to:)` and `init(from:)`.

···

On Apr 3, 2017, at 1:31 PM, Itai Ferber via swift-evolution <swift-evolution@swift.org> wrote:
Hi everyone,

With feedback from swift-evolution and additional internal review, we've pushed updates to this proposal, and to the Swift Archival & Serialization proposal.
Changes to here mostly mirror the ones made to Swift Archival & Serialization, but you can see a specific diff of what's changed here. Full content below.

We'll be looking to start the official review process very soon, so we're interested in any additional feedback.

Thanks!

— Itai

***

While working with the prototype, I realized that the vast majority of conformances will immediately make a container and then never use the `encoder` or `decoder` again. I also noticed that it's illegal to create more than one container from the same coder, and there are unenforceable preconditions to that effect. So I'm wondering if it would make sense to not pass the coder at all, but instead have the conforming type declare what kind of container it wants:

  extension Pet: Codable {
    init(from container: KeyedDecodingContainer<CodingKeys>) throws {
      name = try container.decode(String.self, forKey: .name)
      age = try container.decode(Int.self, forKey: .age)
    }
    
    func encode(to container: KeyedEncodingContainer<CodingKeys>) throws {
      try container.encode(name, forKey: .name)
      try container.encode(age, forKey: .age)
    }
  }

  extension Array: Encodable where Element: Encodable {
    init(from container: UnkeyedDecodingContainer) throws {
      self.init()
      while !container.isAtEnd {
        append(try container.decode(Element.self))
      }
    }
    
    func encode(to container: UnkeyedEncodingContainer) throws {
      container.encode(contentsOf: self)
    }
  }

I think this could be implemented by doing the following:

  1. Adding an associated type to `Encodable` and `Decodable` for the type passed to `encode(to:)`/`init(from:)`.

  2. Creating protocols for the types that are permitted there. Call them `EncodingSink` and `DecodingSource` for now.

  3. Creating *simple* type-erased wrappers for the `Unkeyed*Container` and `SingleValue*Container` protocols and conforming them to `EncodingSink` and `DecodingSource`. These wouldn't need the full generic-subclass dance usually used for type-erased wrappers; they just exist so you can strap initializers to them. In a future version of Swift which allowed initializers on existentials, we could probably get rid of them.

(Incidentally, if our APIs always return a type-erased wrapper around the `Keyed*ContainerProtocol` types, there's no actual need for the underlying protocols to have a `Key` associated type; they can use `CodingKey` existentials and depend on the wrapper to enforce the strong key typing. That would allow us to use a simple type-erased wrapper for `Keyed*Container`, too.)

  4. For advanced use cases where you really *do* need to access the encoder in order to decide which container type to use, we would also need to create a simple type-erased wrapper around `Encoder` and `Decoder` themselves, conforming them to the `Sink`/`Source` protocols.

  5. The Source/Sink parameter would need to be `inout`, unless we *do* end up class-constraining things. (My prototype didn't.)

There are lots of little details that change too, but these are the broad strokes.

Although this technically introduces more types, I think it actually simplifies the design for people who are just using the `Codable` protocol. All they have to know about is the `Codable` protocol, the magic `CodingKeys` type, the three container types (realistically, probably just the `KeyedEncoding/DecodingContainer`), and the top-level encoders they want to use. Most users should never need to know about the members of the `Encoder` protocol; few even need to know about the other two container types. They don't need to do the "create a container" dance. The thing would just work with a minimum of fuss.

Meanwhile, folks who write encoders *do* deal with a bit more complexity, but only because they have to be aware of more type-erased wrappers. In other respects, it's simpler for them, too. Keyed containers don't need to be generic, and they have a layer of Foundation-provided wrappers above them that can help enforce good behavior and (probably) hide the implementation a little bit more. I think that overall, it's probably better for them, too.

Thoughts?

--
Brent Royal-Gordon
Architechies


#17

Hello,

I have two concerns with the proposal. An objective one, and a subjective one.

This message is about the objective one: flat vs. hierarchical encoders/decoders.

I don't know if you intend to support this difference in serialization formats, or not. If you do, then I have a concern. It looks like you encore values and objects in the same way. In the example below, Location can be flat-encoded, when Farm can not. Unfortunately, the Farm name (value) and location and animals (objects) are encoded with the same `encode(_:forKey:)` method. It is not quite clear that there is a problem:

public struct Location : Codable {
    private enum CodingKeys : CodingKey {
        case latitude
        case longitude
    }

    public func encode(to encoder: Encoder) throws {
        // Generic keyed encoder gives type-safe key access: cannot encode with keys of the wrong type.
        let container = encoder.container(keyedBy: CodingKeys.self)

        // The encoder is generic on the key -- free key autocompletion here.
        try container.encode(latitude, forKey: .latitude)
        try container.encode(longitude, forKey: .longitude)
    }
}
public struct Farm : Codable {
    private enum CodingKeys : CodingKey {
        case name
        case location
        case animals
    }

    public func encode(to encoder: Encoder) throws {
        let container = encoder.container(keyedBy: CodingKeys.self)
        try container.encode(name, forKey: .name)
        try container.encode(location, forKey: .location)
        try container.encode(animals, forKey: .animals)
    }
}

Gwendal Roué

···

Le 3 avr. 2017 à 22:31, Itai Ferber via swift-evolution <swift-evolution@swift.org> a écrit :

Hi everyone,

With feedback from swift-evolution and additional internal review, we've pushed updates to this proposal, and to the Swift Archival & Serialization <https://github.com/apple/swift-evolution/pull/639> proposal.
Changes to here mostly mirror the ones made to Swift Archival & Serialization, but you can see a specific diff of what's changed here <https://github.com/apple/swift-evolution/pull/640/commits/add1faad9e8005de91909bdc29f66984f4869898>. Full content below.

We'll be looking to start the official review process very soon, so we're interested in any additional feedback.

Thanks!

— Itai

Swift Encoders
Proposal: SE-NNNN
Author(s): Itai Ferber <https://github.com/itaiferber>, Michael LeHew <https://github.com/mlehew>, Tony Parker <https://github.com/parkera>
Review Manager: TBD
Status: Awaiting review
Associated PRs:
#8124 <https://github.com/apple/swift/pull/8124>
Introduction
As part of the proposal for a Swift archival and serialization API (SE-NNNN <https://github.com/itaiferber/swift-evolution/blob/swift-archival-serialization/proposals/XXXX-swift-archival-serialization.md>), we are also proposing new API for specific new encoders and decoders, as well as introducing support for new Codable types in NSKeyedArchiver and NSKeyedUnarchiver.

This proposal composes the latter two stages laid out in SE-NNNN <https://github.com/itaiferber/swift-evolution/blob/swift-archival-serialization/proposals/XXXX-swift-archival-serialization.md>.

Motivation
With the base API discussed in SE-NNNN <https://github.com/itaiferber/swift-evolution/blob/swift-archival-serialization/proposals/XXXX-swift-archival-serialization.md>, we want to provide new encoders for consumers of this API, as well as provide a consistent story for bridging this new API with our existing NSCoding implementations. We would like to offer a base level of support that users can depend on, and set a pattern that third parties can follow in implementing and extending their own encoders.

Proposed solution
We will:

Add two new encoders and decoders to support encoding Swift value trees in JSON and property list formats
Add support for passing Codable Swift values to NSKeyedArchiver and NSKeyedUnarchiver, and add Codableconformance to our Swift value types
Detailed design
New Encoders and Decoders

JSON

One of the key motivations for the introduction of this API was to allow safer interaction between Swift values and their JSON representations. For values which are Codable, users can encode to and decode from JSON with JSONEncoder and JSONDecoder:

open class JSONEncoder {
    // MARK: Top-Level Encoding

    /// Encodes the given top-level value and returns its JSON representation.
    ///
    /// - parameter value: The value to encode.
    /// - returns: A new `Data` value containing the encoded JSON data.
    /// - throws: `CocoaError.coderInvalidValue` if a non-comforming floating-point value is encountered during archiving, and the encoding strategy is `.throw`.
    /// - throws: An error if any value throws an error during encoding.
    open func encode<T : Encodable>(_ value: T) throws -> Data

    // MARK: Customization

    /// The formatting of the output JSON data.
    public enum OutputFormatting {
        /// Produce JSON compacted by removing whitespace. This is the default formatting.
        case compact

        /// Produce human-readable JSON with indented output.
        case prettyPrinted
    }

    /// The strategy to use for encoding `Date` values.
    public enum DateEncodingStrategy {
        /// Defer to `Date` for choosing an encoding. This is the default strategy.
        case deferredToDate

        /// Encode the `Date` as a UNIX timestamp (as a JSON number).
        case secondsSince1970

        /// Encode the `Date` as UNIX millisecond timestamp (as a JSON number).
        case millisecondsSince1970

        /// Encode the `Date` as an ISO-8601-formatted string (in RFC 3339 format).
        @available(OSX 10.12, iOS 10.0, watchOS 3.0, tvOS 10.0, *)
        case iso8601

        /// Encode the `Date` as a string formatted by the given formatter.
        case formatted(DateFormatter)

        /// Encode the `Date` as a custom value encoded by the given closure.
        ///
        /// If the closure fails to encode a value into the given encoder, the encoder will encode an empty `.default` container in its place.
        case custom((_ value: Date, _ encoder: Encoder) throws -> Void)
    }

    /// The strategy to use for encoding `Data` values.
    public enum DataEncodingStrategy {
        /// Encoded the `Data` as a Base64-encoded string. This is the default strategy.
        case base64

        /// Encode the `Data` as a custom value encoded by the given closure.
        ///
        /// If the closure fails to encode a value into the given encoder, the encoder will encode an empty `.default` container in its place.
        case custom((_ value: Data, _ encoder: Encoder) throws -> Void)
    }

    /// The strategy to use for non-JSON-conforming floating-point values (IEEE 754 infinity and NaN).
    public enum NonConformingFloatEncodingStrategy {
        /// Throw upon encountering non-conforming values. This is the default strategy.
        case `throw`

        /// Encode the values using the given representation strings.
        case convertToString(positiveInfinity: String, negativeInfinity: String, nan: String)
    }

    /// The output format to produce. Defaults to `.compact`.
    open var outputFormatting: OutputFormatting

    /// The strategy to use in encoding dates. Defaults to `.deferredToDate`.
    open var dateEncodingStrategy: DateEncodingStrategy

    /// The strategy to use in encoding binary data. Defaults to `.base64`.
    open var dataEncodingStrategy: DataEncodingStrategy

    /// The strategy to use in encoding non-conforming numbers. Defaults to `.throw`.
    open var nonConformingFloatEncodingStrategy: NonConformingFloatEncodingStrategy

    /// Contextual information to expose during encoding.
    open var userInfo: [CodingUserInfoKey : Any]
}

open class JSONDecoder {
    // MARK: Top-Level Decoding

    /// Decodes a top-level value of the given type from the given JSON representation.
    ///
    /// - parameter type: The type of the value to decode.
    /// - parameter data: The data to decode from.
    /// - returns: A value of the requested type.
    /// - throws: `CocoaError.coderReadCorrupt` if values requested from the payload are corrupted, or if the given data is not valid JSON.
    /// - throws: An error if any value throws an error during decoding.
    open func decode<T : Decodable>(_ type: T.Type, from data: Data) throws -> Value

    // MARK: Customization

    /// The strategy to use for decoding `Date` values.
    public enum DateDecodingStrategy {
        /// Defer to `Date` for decoding. This is the default strategy.
        case deferredToDate

        /// Decode the `Date` as a UNIX timestamp from a JSON number.
        case secondsSince1970

        /// Decode the `Date` as UNIX millisecond timestamp from a JSON number.
        case millisecondsSince1970

        /// Decode the `Date` as an ISO-8601-formatted string (in RFC 3339 format).
        @available(OSX 10.12, iOS 10.0, watchOS 3.0, tvOS 10.0, *)
        case iso8601

        /// Decode the `Date` as a string parsed by the given formatter.
        case formatted(DateFormatter)

        /// Decode the `Date` as a custom value decoded by the given closure.
        case custom((_ decoder: Decoder) throws -> Date)
    }

    /// The strategy to use for decoding `Data` values.
    public enum DataDecodingStrategy {
        /// Decode the `Data` from a Base64-encoded string. This is the default strategy.
        case base64

        /// Decode the `Data` as a custom value decoded by the given closure.
        case custom((_ decoder: Decoder) throws -> Data)
    }

    /// The strategy to use for non-JSON-conforming floating-point values (IEEE 754 infinity and NaN).
    public enum NonConformingFloatDecodingStrategy {
        /// Throw upon encountering non-conforming values. This is the default strategy.
        case `throw`

        /// Decode the values from the given representation strings.
        case convertFromString(positiveInfinity: String, negativeInfinity: String, nan: String)
    }

    /// The strategy to use in decoding dates. Defaults to `.deferredToDate`.
    open var dateDecodingStrategy: DateDecodingStrategy

    /// The strategy to use in decoding binary data. Defaults to `.base64`.
    open var dataDecodingStrategy: DataDecodingStrategy

    /// The strategy to use in decoding non-conforming numbers. Defaults to `.throw`.
    open var nonConformingFloatDecodingStrategy: NonConformingFloatDecodingStrategy

    /// Contextual information to expose during decoding.
    open var userInfo: [CodingUserInfoKey : Any]
}
Usage:

var encoder = JSONEncoder()
encoder.dateEncodingStrategy = .iso8601
encoder.dataEncodingStrategy = .custom(myBase85Encoder)

// Since JSON does not natively allow for infinite or NaN values, we can customize strategies for encoding these non-conforming values.
encoder.nonConformingFloatEncodingStrategy = .convertToString(positiveInfinity: "INF", negativeInfinity: "-INF", nan: "NaN")

// MyValue conforms to Codable
let topLevel = MyValue(...)

let payload: Data
do {
    payload = try encoder.encode(topLevel)
} catch {
    // Some value threw while encoding.
}

// ...

var decoder = JSONDecoder()
decoder.dateDecodingStrategy = .iso8601
decoder.dataDecodingStrategy = .custom(myBase85Decoder)

// Look for and match these values when decoding `Double`s or `Float`s.
decoder.nonConformingFloatDecodingStrategy = .convertFromString(positiveInfinity: "INF", negativeInfinity: "-INF", nan: "NaN")

let topLevel: MyValue
do {
    topLevel = try decoder.decode(MyValue.self, from: payload)
} catch {
    // Data was corrupted, or some value threw while decoding.
}
It should be noted here that JSONEncoder and JSONDecoder do not themselves conform to Encoder and Decoder; instead, they contain private nested types which do conform to Encoder and Decoder, which are passed to values' encode(to:) and init(from:). This is because JSONEncoder and JSONDecoder must present a different top-level API than they would at intermediate levels.

Property List

We also intend to support the property list format, with PropertyListEncoder and PropertyListDecoder:

open class PropertyListEncoder {
    // MARK: Top-Level Encoding

    /// Encodes the given top-level value and returns its property list representation.
    ///
    /// - parameter value: The value to encode.
    /// - returns: A new `Data` value containing the encoded property list data.
    /// - throws: An error if any value throws an error during encoding.
    open func encode<T : Encodable>(_ value: T) throws -> Data

    // MARK: Customization

    /// The output format to write the property list data in. Defaults to `.binary`.
    open var outputFormat: PropertyListSerialization.PropertyListFormat

    /// Contextual information to expose during encoding.
    open var userInfo: [CodingUserInfoKey : Any]
}

open class PropertyListDecoder {
    // MARK: Top-Level Decoding

    /// Decodes a top-level value of the given type from the given property list representation.
    ///
    /// - parameter type: The type of the value to decode.
    /// - parameter data: The data to decode from.
    /// - returns: A value of the requested type.
    /// - throws: `CocoaError.coderReadCorrupt` if values requested from the payload are corrupted, or if the given data is not a valid property list.
    /// - throws: An error if any value throws an error during decoding.
    open func decode<T : Decodable>(_ type: T.Type, from data: Data) throws -> Value

    /// Decodes a top-level value of the given type from the given property list representation.
    ///
    /// - parameter type: The type of the value to decode.
    /// - parameter data: The data to decode from.
    /// - parameter format: The parsed property list format.
    /// - returns: A value of the requested type along with the detected format of the property list.
    /// - throws: `CocoaError.coderReadCorrupt` if values requested from the payload are corrupted, or if the given data is not a valid property list.
    /// - throws: An error if any value throws an error during decoding.
    open func decode<T : Decodable>(_ type: Value.Type, from data: Data, format: inout PropertyListSerialization.PropertyListFormat) throws -> Value

    // MARK: Customization

    /// Contextual information to expose during decoding.
    open var userInfo: [CodingUserInfoKey : Any]
}
Usage:

let encoder = PropertyListEncoder()
let topLevel = MyValue(...)
let payload: Data
do {
    payload = try encoder.encode(topLevel)
} catch {
    // Some value threw while encoding.
}

// ...

let decoder = PropertyListDecoder()
let topLevel: MyValue
do {
    topLevel = try decoder.decode(MyValue.self, from: payload)
} catch {
    // Data was corrupted, or some value threw while decoding.
}
Like with JSON, PropertyListEncoder and PropertyListDecoder also provide private nested types which conform to Encoder and Decoder for performing the archival.

Foundation-Provided Errors

Along with providing the above encoders and decoders, we would like to promote the use of a common set of error codes and messages across all new encoders and decoders. A common vocabulary of expected errors allows end-users to write code agnostic about the specific encoder/decoder implementation they are working with, whether first-party or third-party:

extension CocoaError.Code {
    /// Thrown when a value incompatible with the output format is encoded.
    public static var coderInvalidValue: CocoaError.Code

    /// Thrown when a value of a given type is requested but the encountered value is of an incompatible type.
    public static var coderTypeMismatch: CocoaError.Code

    /// Thrown when read data is corrupted or otherwise invalid for the format. This value already exists today.
    public static var coderReadCorrupt: CocoaError.Code

    /// Thrown when a requested key or value is unexpectedly null or missing. This value already exists today.
    public static var coderValueNotFound: CocoaError.Code
}

// These reexpose the values above.
extension CocoaError {
    public static var coderInvalidValue: CocoaError.Code

    public static var coderTypeMismatch: CocoaError.Code
}
The localized description strings associated with the two new error codes are:

.coderInvalidValue: "The data is not valid for encoding in this format."
.coderTypeMismatch: "The data couldn't be read because it isn't in the correct format." (Precedent from NSCoderReadCorruptError.)
All of these errors will include the coding key path that led to the failure in the error's userInfo dictionary under NSCodingPathErrorKey, along with a non-localized, developer-facing failure reason under NSDebugDescriptionErrorKey.

NSKeyedArchiver & NSKeyedUnarchiver Changes

Although our primary objectives for this new API revolve around Swift, we would like to make it easy for current consumers to make the transition to Codable where appropriate. As part of this, we would like to bridge compatibility between new Codable types (or newly-Codable-adopting types) and existing NSCoding types.

To do this, we want to introduce changes to NSKeyedArchiver and NSKeyedUnarchiver in Swift that allow archival of Codable types intermixed with NSCoding types:

// These are provided in the Swift overlay, and included in swift-corelibs-foundation.
extension NSKeyedArchiver {
    public func encodeCodable(_ codable: Encodable?, forKey key: String) { ... }
}

extension NSKeyedUnarchiver {
    public func decodeCodable<T : Decodable>(_ type: T.Type, forKey key: String) -> T? { ... }
}
NOTE: Since these changes are being made in extensions in the Swift overlay, it is not yet possible for these methods to be overridden. These can therefore not be added to NSCoder, since NSKeyedArchiver and NSKeyedUnarchiver would not be able to provide concrete implementations. In order to call these methods, it is necessary to downcast from an NSCoder to NSKeyedArchiver/NSKeyedUnarchiver directly. Since subclasses of NSKeyedArchiver and NSKeyedUnarchiverin Swift will inherit these implementations without being able to override them (which is wrong), we will NSRequiresConcreteImplementation() dynamically in subclasses.
The addition of these methods allows the introduction of Codable types into existing NSCoding structures, allowing for a transition to Codable types where appropriate.

Refining encode(_:forKey:)

Along with these extensions, we would like to refine the import of -[NSCoder encodeObject:forKey:], which is currently imported into Swift as encode(_: Any?, forKey: String). This method currently accepts Objective-C and Swift objects conforming to NSCoding (non-conforming objects produce a runtime error), as well as bridgeable Swift types (Data, String, Array, etc.); we would like to extend it to support new Swift Codable types, which would otherwise produce a runtime error upon call.

-[NSCoder encodeObject:forKey:] will be given a new Swift name of encodeObject(_:forKey:), and we will provide a replacement encode(_: Any?, forKey: String) in the overlay which will funnel out to either encodeCodable(_:forKey:)or encodeObject(_:forKey:) as appropriate. This should maintain source compatibility for end users already calling encode(_:forKey:), as well as behavior compatibility for subclassers of NSCoder and NSKeyedArchiver who may be providing their own encode(_:forKey:).

Semantics of Codable Types in Archives

There are a few things to note about including Codable values in NSKeyedArchiver archives:

Bridgeable Foundation types will always bridge before encoding. This is to facilitate writing Foundation types in a compatible format from both Objective-C and Swift
On decode, these types will decode either as their Objective-C or Swift version, depending on user need (decodeObject(forKey:) will decode as an Objective-C object; decodeCodable(_:forKey:) as a Swift value)
User types, which are not bridgeable, do not write out a $class and can only be decoded in Swift. In the future, we may add API to allow Swift types to provide an Objective-C class to decode as, effectively allowing for user bridging across archival
Foundation Types Adopting Codable

The following Foundation Swift types will be adopting Codable, and will encode as their bridged types when encoded through NSKeyedArchiver, as mentioned above:

AffineTransform
Calendar
CharacterSet
Date
DateComponents
DateInterval
Decimal
IndexPath
IndexSet
Locale
Measurement
Notification
PersonNameComponents
TimeZone
URL
URLComponents
URLRequest
UUID
Along with these, the Array, Dictionary, and Set types will gain Codable conformance (as part of the Conditional Conformance feature), and encode through NSKeyedArchiver as NSArray, NSDictionary, and NSSet respectively.

Source compatibility
The majority of this proposal is additive. The changes to NSKeyedArchiver are intended to be non-source-breaking changes, and non-behavior-breaking changes for subclasses in Objective-C and Swift.

Effect on ABI stability
The addition of this API will not be an ABI-breaking change. However, this will add limitations for changes in future versions of Swift, as parts of the API will have to remain unchanged between versions of Swift (barring some additions, discussed below).

Effect on API resilience
Much like new API added to the standard library, once added, some changes to this API will be ABI- and source-breaking changes. Changes to the new encoder and decoder classes provided above will be restricted as described in the library evolution document <https://github.com/apple/swift/blob/master/docs/LibraryEvolution.rst> in the Swift repository; in particular, the removal of methods or nested types or changes to argument types will break client behavior. Additionally, additions to provided options enums will be a source-breaking change for users performing an exhaustive switch over their cases; removal of cases will be ABI-breaking.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


#18

Hello,

I have a deep subjective concern with this proposal.

I, as an application and library developer, feel something deeply unfair if a proposal with such a short scope would be blessed with automatic code generation right into the compiler.

This proposal should be able to live as an external library. If it needs automatic generation of code from the compiler, then something is deeply wrong in the Swift design process.

Gwendal Roué

···

Le 3 avr. 2017 à 22:31, Itai Ferber via swift-evolution <swift-evolution@swift.org> a écrit :

Hi everyone,

With feedback from swift-evolution and additional internal review, we've pushed updates to this proposal, and to the Swift Archival & Serialization <https://github.com/apple/swift-evolution/pull/639> proposal.
Changes to here mostly mirror the ones made to Swift Archival & Serialization, but you can see a specific diff of what's changed here <https://github.com/apple/swift-evolution/pull/640/commits/add1faad9e8005de91909bdc29f66984f4869898>. Full content below.

We'll be looking to start the official review process very soon, so we're interested in any additional feedback.

Thanks!

— Itai

Swift Encoders
Proposal: SE-NNNN
Author(s): Itai Ferber <https://github.com/itaiferber>, Michael LeHew <https://github.com/mlehew>, Tony Parker <https://github.com/parkera>
Review Manager: TBD
Status: Awaiting review
Associated PRs:
#8124 <https://github.com/apple/swift/pull/8124>
Introduction
As part of the proposal for a Swift archival and serialization API (SE-NNNN <https://github.com/itaiferber/swift-evolution/blob/swift-archival-serialization/proposals/XXXX-swift-archival-serialization.md>), we are also proposing new API for specific new encoders and decoders, as well as introducing support for new Codable types in NSKeyedArchiver and NSKeyedUnarchiver.

This proposal composes the latter two stages laid out in SE-NNNN <https://github.com/itaiferber/swift-evolution/blob/swift-archival-serialization/proposals/XXXX-swift-archival-serialization.md>.

Motivation
With the base API discussed in SE-NNNN <https://github.com/itaiferber/swift-evolution/blob/swift-archival-serialization/proposals/XXXX-swift-archival-serialization.md>, we want to provide new encoders for consumers of this API, as well as provide a consistent story for bridging this new API with our existing NSCoding implementations. We would like to offer a base level of support that users can depend on, and set a pattern that third parties can follow in implementing and extending their own encoders.

Proposed solution
We will:

Add two new encoders and decoders to support encoding Swift value trees in JSON and property list formats
Add support for passing Codable Swift values to NSKeyedArchiver and NSKeyedUnarchiver, and add Codableconformance to our Swift value types
Detailed design
New Encoders and Decoders

JSON

One of the key motivations for the introduction of this API was to allow safer interaction between Swift values and their JSON representations. For values which are Codable, users can encode to and decode from JSON with JSONEncoder and JSONDecoder:

open class JSONEncoder {
    // MARK: Top-Level Encoding

    /// Encodes the given top-level value and returns its JSON representation.
    ///
    /// - parameter value: The value to encode.
    /// - returns: A new `Data` value containing the encoded JSON data.
    /// - throws: `CocoaError.coderInvalidValue` if a non-comforming floating-point value is encountered during archiving, and the encoding strategy is `.throw`.
    /// - throws: An error if any value throws an error during encoding.
    open func encode<T : Encodable>(_ value: T) throws -> Data

    // MARK: Customization

    /// The formatting of the output JSON data.
    public enum OutputFormatting {
        /// Produce JSON compacted by removing whitespace. This is the default formatting.
        case compact

        /// Produce human-readable JSON with indented output.
        case prettyPrinted
    }

    /// The strategy to use for encoding `Date` values.
    public enum DateEncodingStrategy {
        /// Defer to `Date` for choosing an encoding. This is the default strategy.
        case deferredToDate

        /// Encode the `Date` as a UNIX timestamp (as a JSON number).
        case secondsSince1970

        /// Encode the `Date` as UNIX millisecond timestamp (as a JSON number).
        case millisecondsSince1970

        /// Encode the `Date` as an ISO-8601-formatted string (in RFC 3339 format).
        @available(OSX 10.12, iOS 10.0, watchOS 3.0, tvOS 10.0, *)
        case iso8601

        /// Encode the `Date` as a string formatted by the given formatter.
        case formatted(DateFormatter)

        /// Encode the `Date` as a custom value encoded by the given closure.
        ///
        /// If the closure fails to encode a value into the given encoder, the encoder will encode an empty `.default` container in its place.
        case custom((_ value: Date, _ encoder: Encoder) throws -> Void)
    }

    /// The strategy to use for encoding `Data` values.
    public enum DataEncodingStrategy {
        /// Encoded the `Data` as a Base64-encoded string. This is the default strategy.
        case base64

        /// Encode the `Data` as a custom value encoded by the given closure.
        ///
        /// If the closure fails to encode a value into the given encoder, the encoder will encode an empty `.default` container in its place.
        case custom((_ value: Data, _ encoder: Encoder) throws -> Void)
    }

    /// The strategy to use for non-JSON-conforming floating-point values (IEEE 754 infinity and NaN).
    public enum NonConformingFloatEncodingStrategy {
        /// Throw upon encountering non-conforming values. This is the default strategy.
        case `throw`

        /// Encode the values using the given representation strings.
        case convertToString(positiveInfinity: String, negativeInfinity: String, nan: String)
    }

    /// The output format to produce. Defaults to `.compact`.
    open var outputFormatting: OutputFormatting

    /// The strategy to use in encoding dates. Defaults to `.deferredToDate`.
    open var dateEncodingStrategy: DateEncodingStrategy

    /// The strategy to use in encoding binary data. Defaults to `.base64`.
    open var dataEncodingStrategy: DataEncodingStrategy

    /// The strategy to use in encoding non-conforming numbers. Defaults to `.throw`.
    open var nonConformingFloatEncodingStrategy: NonConformingFloatEncodingStrategy

    /// Contextual information to expose during encoding.
    open var userInfo: [CodingUserInfoKey : Any]
}

open class JSONDecoder {
    // MARK: Top-Level Decoding

    /// Decodes a top-level value of the given type from the given JSON representation.
    ///
    /// - parameter type: The type of the value to decode.
    /// - parameter data: The data to decode from.
    /// - returns: A value of the requested type.
    /// - throws: `CocoaError.coderReadCorrupt` if values requested from the payload are corrupted, or if the given data is not valid JSON.
    /// - throws: An error if any value throws an error during decoding.
    open func decode<T : Decodable>(_ type: T.Type, from data: Data) throws -> Value

    // MARK: Customization

    /// The strategy to use for decoding `Date` values.
    public enum DateDecodingStrategy {
        /// Defer to `Date` for decoding. This is the default strategy.
        case deferredToDate

        /// Decode the `Date` as a UNIX timestamp from a JSON number.
        case secondsSince1970

        /// Decode the `Date` as UNIX millisecond timestamp from a JSON number.
        case millisecondsSince1970

        /// Decode the `Date` as an ISO-8601-formatted string (in RFC 3339 format).
        @available(OSX 10.12, iOS 10.0, watchOS 3.0, tvOS 10.0, *)
        case iso8601

        /// Decode the `Date` as a string parsed by the given formatter.
        case formatted(DateFormatter)

        /// Decode the `Date` as a custom value decoded by the given closure.
        case custom((_ decoder: Decoder) throws -> Date)
    }

    /// The strategy to use for decoding `Data` values.
    public enum DataDecodingStrategy {
        /// Decode the `Data` from a Base64-encoded string. This is the default strategy.
        case base64

        /// Decode the `Data` as a custom value decoded by the given closure.
        case custom((_ decoder: Decoder) throws -> Data)
    }

    /// The strategy to use for non-JSON-conforming floating-point values (IEEE 754 infinity and NaN).
    public enum NonConformingFloatDecodingStrategy {
        /// Throw upon encountering non-conforming values. This is the default strategy.
        case `throw`

        /// Decode the values from the given representation strings.
        case convertFromString(positiveInfinity: String, negativeInfinity: String, nan: String)
    }

    /// The strategy to use in decoding dates. Defaults to `.deferredToDate`.
    open var dateDecodingStrategy: DateDecodingStrategy

    /// The strategy to use in decoding binary data. Defaults to `.base64`.
    open var dataDecodingStrategy: DataDecodingStrategy

    /// The strategy to use in decoding non-conforming numbers. Defaults to `.throw`.
    open var nonConformingFloatDecodingStrategy: NonConformingFloatDecodingStrategy

    /// Contextual information to expose during decoding.
    open var userInfo: [CodingUserInfoKey : Any]
}
Usage:

var encoder = JSONEncoder()
encoder.dateEncodingStrategy = .iso8601
encoder.dataEncodingStrategy = .custom(myBase85Encoder)

// Since JSON does not natively allow for infinite or NaN values, we can customize strategies for encoding these non-conforming values.
encoder.nonConformingFloatEncodingStrategy = .convertToString(positiveInfinity: "INF", negativeInfinity: "-INF", nan: "NaN")

// MyValue conforms to Codable
let topLevel = MyValue(...)

let payload: Data
do {
    payload = try encoder.encode(topLevel)
} catch {
    // Some value threw while encoding.
}

// ...

var decoder = JSONDecoder()
decoder.dateDecodingStrategy = .iso8601
decoder.dataDecodingStrategy = .custom(myBase85Decoder)

// Look for and match these values when decoding `Double`s or `Float`s.
decoder.nonConformingFloatDecodingStrategy = .convertFromString(positiveInfinity: "INF", negativeInfinity: "-INF", nan: "NaN")

let topLevel: MyValue
do {
    topLevel = try decoder.decode(MyValue.self, from: payload)
} catch {
    // Data was corrupted, or some value threw while decoding.
}
It should be noted here that JSONEncoder and JSONDecoder do not themselves conform to Encoder and Decoder; instead, they contain private nested types which do conform to Encoder and Decoder, which are passed to values' encode(to:) and init(from:). This is because JSONEncoder and JSONDecoder must present a different top-level API than they would at intermediate levels.

Property List

We also intend to support the property list format, with PropertyListEncoder and PropertyListDecoder:

open class PropertyListEncoder {
    // MARK: Top-Level Encoding

    /// Encodes the given top-level value and returns its property list representation.
    ///
    /// - parameter value: The value to encode.
    /// - returns: A new `Data` value containing the encoded property list data.
    /// - throws: An error if any value throws an error during encoding.
    open func encode<T : Encodable>(_ value: T) throws -> Data

    // MARK: Customization

    /// The output format to write the property list data in. Defaults to `.binary`.
    open var outputFormat: PropertyListSerialization.PropertyListFormat

    /// Contextual information to expose during encoding.
    open var userInfo: [CodingUserInfoKey : Any]
}

open class PropertyListDecoder {
    // MARK: Top-Level Decoding

    /// Decodes a top-level value of the given type from the given property list representation.
    ///
    /// - parameter type: The type of the value to decode.
    /// - parameter data: The data to decode from.
    /// - returns: A value of the requested type.
    /// - throws: `CocoaError.coderReadCorrupt` if values requested from the payload are corrupted, or if the given data is not a valid property list.
    /// - throws: An error if any value throws an error during decoding.
    open func decode<T : Decodable>(_ type: T.Type, from data: Data) throws -> Value

    /// Decodes a top-level value of the given type from the given property list representation.
    ///
    /// - parameter type: The type of the value to decode.
    /// - parameter data: The data to decode from.
    /// - parameter format: The parsed property list format.
    /// - returns: A value of the requested type along with the detected format of the property list.
    /// - throws: `CocoaError.coderReadCorrupt` if values requested from the payload are corrupted, or if the given data is not a valid property list.
    /// - throws: An error if any value throws an error during decoding.
    open func decode<T : Decodable>(_ type: Value.Type, from data: Data, format: inout PropertyListSerialization.PropertyListFormat) throws -> Value

    // MARK: Customization

    /// Contextual information to expose during decoding.
    open var userInfo: [CodingUserInfoKey : Any]
}
Usage:

let encoder = PropertyListEncoder()
let topLevel = MyValue(...)
let payload: Data
do {
    payload = try encoder.encode(topLevel)
} catch {
    // Some value threw while encoding.
}

// ...

let decoder = PropertyListDecoder()
let topLevel: MyValue
do {
    topLevel = try decoder.decode(MyValue.self, from: payload)
} catch {
    // Data was corrupted, or some value threw while decoding.
}
Like with JSON, PropertyListEncoder and PropertyListDecoder also provide private nested types which conform to Encoder and Decoder for performing the archival.

Foundation-Provided Errors

Along with providing the above encoders and decoders, we would like to promote the use of a common set of error codes and messages across all new encoders and decoders. A common vocabulary of expected errors allows end-users to write code agnostic about the specific encoder/decoder implementation they are working with, whether first-party or third-party:

extension CocoaError.Code {
    /// Thrown when a value incompatible with the output format is encoded.
    public static var coderInvalidValue: CocoaError.Code

    /// Thrown when a value of a given type is requested but the encountered value is of an incompatible type.
    public static var coderTypeMismatch: CocoaError.Code

    /// Thrown when read data is corrupted or otherwise invalid for the format. This value already exists today.
    public static var coderReadCorrupt: CocoaError.Code

    /// Thrown when a requested key or value is unexpectedly null or missing. This value already exists today.
    public static var coderValueNotFound: CocoaError.Code
}

// These reexpose the values above.
extension CocoaError {
    public static var coderInvalidValue: CocoaError.Code

    public static var coderTypeMismatch: CocoaError.Code
}
The localized description strings associated with the two new error codes are:

.coderInvalidValue: "The data is not valid for encoding in this format."
.coderTypeMismatch: "The data couldn't be read because it isn't in the correct format." (Precedent from NSCoderReadCorruptError.)
All of these errors will include the coding key path that led to the failure in the error's userInfo dictionary under NSCodingPathErrorKey, along with a non-localized, developer-facing failure reason under NSDebugDescriptionErrorKey.

NSKeyedArchiver & NSKeyedUnarchiver Changes

Although our primary objectives for this new API revolve around Swift, we would like to make it easy for current consumers to make the transition to Codable where appropriate. As part of this, we would like to bridge compatibility between new Codable types (or newly-Codable-adopting types) and existing NSCoding types.

To do this, we want to introduce changes to NSKeyedArchiver and NSKeyedUnarchiver in Swift that allow archival of Codable types intermixed with NSCoding types:

// These are provided in the Swift overlay, and included in swift-corelibs-foundation.
extension NSKeyedArchiver {
    public func encodeCodable(_ codable: Encodable?, forKey key: String) { ... }
}

extension NSKeyedUnarchiver {
    public func decodeCodable<T : Decodable>(_ type: T.Type, forKey key: String) -> T? { ... }
}
NOTE: Since these changes are being made in extensions in the Swift overlay, it is not yet possible for these methods to be overridden. These can therefore not be added to NSCoder, since NSKeyedArchiver and NSKeyedUnarchiver would not be able to provide concrete implementations. In order to call these methods, it is necessary to downcast from an NSCoder to NSKeyedArchiver/NSKeyedUnarchiver directly. Since subclasses of NSKeyedArchiver and NSKeyedUnarchiverin Swift will inherit these implementations without being able to override them (which is wrong), we will NSRequiresConcreteImplementation() dynamically in subclasses.
The addition of these methods allows the introduction of Codable types into existing NSCoding structures, allowing for a transition to Codable types where appropriate.

Refining encode(_:forKey:)

Along with these extensions, we would like to refine the import of -[NSCoder encodeObject:forKey:], which is currently imported into Swift as encode(_: Any?, forKey: String). This method currently accepts Objective-C and Swift objects conforming to NSCoding (non-conforming objects produce a runtime error), as well as bridgeable Swift types (Data, String, Array, etc.); we would like to extend it to support new Swift Codable types, which would otherwise produce a runtime error upon call.

-[NSCoder encodeObject:forKey:] will be given a new Swift name of encodeObject(_:forKey:), and we will provide a replacement encode(_: Any?, forKey: String) in the overlay which will funnel out to either encodeCodable(_:forKey:)or encodeObject(_:forKey:) as appropriate. This should maintain source compatibility for end users already calling encode(_:forKey:), as well as behavior compatibility for subclassers of NSCoder and NSKeyedArchiver who may be providing their own encode(_:forKey:).

Semantics of Codable Types in Archives

There are a few things to note about including Codable values in NSKeyedArchiver archives:

Bridgeable Foundation types will always bridge before encoding. This is to facilitate writing Foundation types in a compatible format from both Objective-C and Swift
On decode, these types will decode either as their Objective-C or Swift version, depending on user need (decodeObject(forKey:) will decode as an Objective-C object; decodeCodable(_:forKey:) as a Swift value)
User types, which are not bridgeable, do not write out a $class and can only be decoded in Swift. In the future, we may add API to allow Swift types to provide an Objective-C class to decode as, effectively allowing for user bridging across archival
Foundation Types Adopting Codable

The following Foundation Swift types will be adopting Codable, and will encode as their bridged types when encoded through NSKeyedArchiver, as mentioned above:

AffineTransform
Calendar
CharacterSet
Date
DateComponents
DateInterval
Decimal
IndexPath
IndexSet
Locale
Measurement
Notification
PersonNameComponents
TimeZone
URL
URLComponents
URLRequest
UUID
Along with these, the Array, Dictionary, and Set types will gain Codable conformance (as part of the Conditional Conformance feature), and encode through NSKeyedArchiver as NSArray, NSDictionary, and NSSet respectively.

Source compatibility
The majority of this proposal is additive. The changes to NSKeyedArchiver are intended to be non-source-breaking changes, and non-behavior-breaking changes for subclasses in Objective-C and Swift.

Effect on ABI stability
The addition of this API will not be an ABI-breaking change. However, this will add limitations for changes in future versions of Swift, as parts of the API will have to remain unchanged between versions of Swift (barring some additions, discussed below).

Effect on API resilience
Much like new API added to the standard library, once added, some changes to this API will be ABI- and source-breaking changes. Changes to the new encoder and decoder classes provided above will be restricted as described in the library evolution document <https://github.com/apple/swift/blob/master/docs/LibraryEvolution.rst> in the Swift repository; in particular, the removal of methods or nested types or changes to argument types will break client behavior. Additionally, additions to provided options enums will be a source-breaking change for users performing an exhaustive switch over their cases; removal of cases will be ABI-breaking.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Itai Ferber) #19

We’ll keep this in mind. :slight_smile:

···

On 15 Mar 2017, at 19:58, Will Stanton wrote:

Hello Itai,

Thanks for your response and its explanations!

Agreed that comprehension of multiple formats is important since there are a couple common ways of encoding JSON dates!

Still, ISO 8601 appears pretty often (though I don’t have data on that, Stack Overflow says RFC 7493 I-JSON prefers ISO 8601; https://tools.ietf.org/html/rfc7493#section-4.3), and as other servers might make/handle a lot of JSON produced to/from the API, I think it would be disadvantageous to default to `deferredToDate` (if `deferredToDate` doesn't use the ISO 8601 format).

As you mention, writers/readers have to agree on their format - my 2¢ is that ISO 8601 would be more common, and so a better default, than a Unix or reference date timestamp.

Regards,
Will Stanton

-—Gracias for the prediction :slight_smile:

On Mar 15, 2017, at 9:53 PM, Itai Ferber <iferber@apple.com> wrote:

Hi Will,

Thanks for your comments!
deferredToDate simply uses the default implementation that Date provides — since it is not a primitive type like Int or String and conforms to Codable itself, it will have an implementation of init(from:) and encode(to:). It will have an implementation that makes sense for Date in general, but since a big use-case for JSON lies in talking to external servers which you don't control, allowing for custom date formatting is important.

To that end, although ISO 8601 may make sense for some applications as the default, it is less efficient to format, encode, decode, and parse than, say, writing out a UNIX timestamp as a Double (or similar). Thus, the default is to allow Date to pick a representation that best suits itself, and if you need customization, you have the option to use it.

Since Date makes a concrete decision about how to encode, both sides will need to use deferredToDate for compatibility, in the same way that they would have to agree about ISO 8601, or any of the other options.

HTH!

— Itai

P.S. About Xcode autocompletion slowdown, I don't know to be honest, but I can't imagine it would be significant. Only certain types have enc... or dec... and even then, the list of methods isn't that long.


(Itai Ferber) #20

The sorted order of keys is an implementation detail of `PropertyListSerialization` which `JSONSerialization` does not share.
This would require a change in `JSONSerialization`, which is possible, but out of scope for this work specifically. It would have to go through review.

[FWIW, the key ordering isn’t changing at random; `JSONSerialization` writes keys in the order that it receives them (sorted in the order that `NSDictionary` stores them, based on hash). If you want this to not interfere with your Git repo (on a more immediate timescale), you can probably do something like adding a pre-commit Git hook to lint the file as needed. But I digress, this is totally off-topic.]

···

On 16 Mar 2017, at 14:57, Ben Rimmington wrote:

Sorry, I actually meant that keys would be ordered **alphabetically** (as they are with PropertyListSerialization).

I'm storing output from JSONSerialization in a git repository, and when I add/remove a single key-value the entire file changes, due to keys being reordered (seemingly at random).

-- Ben

Itai Ferber wrote:

By stable ordering, you mean dictionaries with keys being output in the order that they were encoded?
Potentially, but this would require additional work as neither Swift dictionaries nor NSMutableDictionary support this (reasonable seeing as dictionaries are by definition not sorted), and the current implementation uses JSONSerialization as the concrete serializer. Certainly not impossible, though.

On 16 Mar 2017, at 8:49, Ben Rimmington wrote:

<https://github.com/apple/swift-evolution/pull/640>

Could the `JSONEncoder.OutputFormatting.prettyPrinted` case include the **stable ordering** of keys?

-- Ben