Question regarding SE-0167 Swift Encoders

Hello,

I have already asked stupid questions about SE-0167 and SE-0166, but this time I hope this is a real one.

According so SE-0166, codable types themselves instantiate a single value decoder, or a keyed container:

  public struct Farm : Codable {
    public init(from decoder: Decoder) throws {
      let container = try decoder.container(keyedBy: CodingKeys.self
      ...
    }
  }

  public enum Animal : Int, Codable {
    public init(from decoder: Decoder) throws
            let intValue = try decoder.singleValueContainer().decode(Int.self)
      ...
    }
  }
  
According to SE-0167, decoder decode non-trivial types in their decode(_:forKey:) and decodeIfPresent(_:forKey:) methods:

  func decode<T>(_ type: T.Type, forKey key: Key) throws -> T where T : Decodable
  func decodeIfPresent<T>(_ type: T.Type, forKey key: Key) throws -> T? where T : Decodable

My trouble is that the decoder does not know whether the Decodable type will ask for a keyed container, or for a single value container.

Why is it a problem?

In the context of decoding of SQL rows, keys may refer to different things, depending on whether we are decoding a *value*, or a *complex object*:

- for values, keys are column names, as everybody can expect
- for complex objects, keys are names of "row scopes". Row scopes are a concept introduced by GRDB.swift and allows a type that knows how to consume `SELECT * FROM table1` to consume as well the results of `SELECT table1.*, table2.* FROM table1 JOIN table2` through a "scope" that presents the row in the shape expected by the consumer (here, only columns from table1).

This is supposed to allow support for types that contain both nested types and values (one of the goals of SE-0166 and SE-0167):

  struct Compound : Codable {
    let someStruct: SomeStruct // object that feeds on the "someStruct" scope
    let name: String // value that feeds on the "name" column
  }

The two decoding methods decode(_:forKey:) and decodeIfPresent(_:forKey:) can't be implemented nicely, because they don't know whether the decodable type will ask for a keyed container or a single value container, and thus they don't know whether they should look for the presence of a row scope, or of a column:

A workaround is to perform runtime checks on the GRDB protocols adopted by T, as below. But it's still impossible to support other codable types:

  if let valueType = T.self as? DatabaseValueConvertible.Type {
    // if column is missing, trigger the "missing key" error or return nil.
  } else if let complexType = T.self as? RowConvertible.Type {
    // if row scope is missing, trigger the "missing key" error or return nil.
  } else {
    // don't know what to do
    fatalError("unsupported")
  }

Do you have any advice?

Gwendal Roué

This is the general topic of "row adapters", something that I don't know if any other SQL libraries implements. You may thus not be familiar with it, and may even doubt if that concept has any value.

Row adapters are indeed quite unusual, but they're very useful, and solve hard problems.

Please check GitHub - groue/GRDB.swift: A toolkit for SQLite databases, with a focus on application development for more information.

Gwendal

···

Le 29 mai 2017 à 13:51, Gwendal Roué <gwendal.roue@gmail.com> a écrit :

In the context of decoding of SQL rows, keys may refer to different things, depending on whether we are decoding a *value*, or a *complex object*:

- for values, keys are column names, as everybody can expect
- for complex objects, keys are names of "row scopes". Row scopes are a concept introduced by GRDB.swift and allows a type that knows how to consume `SELECT * FROM table1` to consume as well the results of `SELECT table1.*, table2.* FROM table1 JOIN table2` through a "scope" that presents the row in the shape expected by the consumer (here, only columns from table1).

Hi Gwendal,

There are no stupid questions — everything helps hammer out this API, so I appreciate you taking the time to look at this so deeply.
I have to confess that I’m not familiar with this concept, but let’s take a look:

if let valueType = T.self as? DatabaseValueConvertible.Type {
     // if column is missing, trigger the "missing key" error or return nil.
} else if let complexType = T.self as? RowConvertible.Type {
     // if row scope is missing, trigger the "missing key" error or return nil.
} else {
     // don't know what to do
     fatalError("unsupported")
}

Is it appropriate for a type which is neither `DatabaseValueConvertible` nor `RowConvertible` to be decoded with your decoder? If not, then this warrants a `preconditionFailure` or an error of some sort, right? In this case, that would be valid.

You also mention that "it’s still impossible to support other Codable types" — what do you mean by this? Perhaps there’s a way to accomplish what you’re looking to do.
In any case, one option (which is not recommended unless if there are other avenues to solve this by) is to perform a "dry run" decoding. Attempt to decode the type with a dummy decoder to see what container it will need, then prepare your approach and do it again for real. Obviously, this isn’t a clean way to do it if we can find alternatives, but it’s an option.

— Itai

···

On 29 May 2017, at 4:51, Gwendal Roué via swift-evolution wrote:

Hello,

I have already asked stupid questions about SE-0167 and SE-0166, but this time I hope this is a real one.

According so SE-0166, codable types themselves instantiate a single value decoder, or a keyed container:

  public struct Farm : Codable {
    public init(from decoder: Decoder) throws {
      let container = try decoder.container(keyedBy: CodingKeys.self
      ...
    }
  }

  public enum Animal : Int, Codable {
    public init(from decoder: Decoder) throws
            let intValue = try decoder.singleValueContainer().decode(Int.self)
      ...
    }
  }
  
According to SE-0167, decoder decode non-trivial types in their decode(_:forKey:) and decodeIfPresent(_:forKey:) methods:

  func decode<T>(_ type: T.Type, forKey key: Key) throws -> T where T : Decodable
  func decodeIfPresent<T>(_ type: T.Type, forKey key: Key) throws -> T? where T : Decodable

My trouble is that the decoder does not know whether the Decodable type will ask for a keyed container, or for a single value container.

Why is it a problem?

In the context of decoding of SQL rows, keys may refer to different things, depending on whether we are decoding a *value*, or a *complex object*:

- for values, keys are column names, as everybody can expect
- for complex objects, keys are names of "row scopes". Row scopes are a concept introduced by GRDB.swift and allows a type that knows how to consume `SELECT * FROM table1` to consume as well the results of `SELECT table1.*, table2.* FROM table1 JOIN table2` through a "scope" that presents the row in the shape expected by the consumer (here, only columns from table1).

This is supposed to allow support for types that contain both nested types and values (one of the goals of SE-0166 and SE-0167):

  struct Compound : Codable {
    let someStruct: SomeStruct // object that feeds on the "someStruct" scope
    let name: String // value that feeds on the "name" column
  }

The two decoding methods decode(_:forKey:) and decodeIfPresent(_:forKey:) can't be implemented nicely, because they don't know whether the decodable type will ask for a keyed container or a single value container, and thus they don't know whether they should look for the presence of a row scope, or of a column:

A workaround is to perform runtime checks on the GRDB protocols adopted by T, as below. But it's still impossible to support other codable types:

  if let valueType = T.self as? DatabaseValueConvertible.Type {
    // if column is missing, trigger the "missing key" error or return nil.
  } else if let complexType = T.self as? RowConvertible.Type {
    // if row scope is missing, trigger the "missing key" error or return nil.
  } else {
    // don't know what to do
    fatalError("unsupported")
  }

Do you have any advice?

Gwendal Roué

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Hello Itai,

Thanks for helping sorting things out.

I have since my initial question a better understanding of Codable, and I hope I can better express the trouble. I bump against the fact that SE-0166 and SE-0167 assume that there is a single kind of coding keys.

This is the case for JSON and Plist:
  
  {
    "a": "foo"
    "b": { ... }
  }

But imagine a different serialization format where we don't use the same kind of keys for values and objects. Values are stored on red keys, and objects on blue keys:
  
  {
    "a" (red): "foo"
    "b" (blue): { ... }
  }

This serialization format accepts keys with the same name, as long as they don't have the same color:
  
  {
    "a" (red): "foo"
    "a" (blue): { ... }
  }

This format is used by SQL rows in GRDB. A SQL row is both a set of columns with associated values (the "red" keys), plus a set of scopes with associated "view" on the row (the blue keys):

    let row = ...
    row["id"] // 1
    let scopedRpw = row.scoped(on: "foo")! // <Row "id":2, "foo": "bar">
    scopedRow["id"] // 2

If you wonder: "but why???": columns and scopes are what can make rows a suitable base for hierarchical decoding, just like JSON and PList. When a flat SQL row fetched from a joined query is seen as a hierarchical structure, several simple `init(row:)` initializers can get the rows they expect, and we load a complex graph of objects. I have high hopes (Joins in the Query Interface · Issue #176 · groue/GRDB.swift · GitHub).

I care about Codable because of the code generation is has been blessed with. I expect GRDB users to rush on Codable since they won't have any longer to write the decoding boilerplate.

I have to confess that I’m not familiar with this concept, but let’s take a look:
if let valueType = T.self as? DatabaseValueConvertible.Type {
    // if column is missing, trigger the "missing key" error or return nil.
} else if let complexType = T.self as? RowConvertible.Type {
    // if row scope is missing, trigger the "missing key" error or return nil.
} else {
    // don't know what to do
    fatalError("unsupported")
}
Is it appropriate for a type which is neither DatabaseValueConvertible nor RowConvertible to be decoded with your decoder? If not, then this warrants a preconditionFailure or an error of some sort, right? In this case, that would be valid.

Yes it is, there's no point preventing this.

We can forget the GRDB DatabaseValueConvertible and RowConvertible protocals in this discussion - they have their purpose, but are irrelevant here, and I was wrong letting them in the discussion. Will you look at some updated code?

In practice, let's consider the `KeyedDecodingContainerProtocol.decode(_:forKey:)` method. The decoding container is asked for a type T. It does not know yet if T is a single-value or a keyed type. No problem: it delays the decision until the Decoder is asked for a container:
    
    struct RowKeyedDecodingContainer<Key: CodingKey>: KeyedDecodingContainerProtocol {
        func decode<T>(_ type: T.Type, forKey key: Key) throws -> T where T : Decodable {
            // Push the key, and wait until the decoder is asked for a container
            // so that we know if T is keyed, or single-value:
            return try T(from: RowDecoder(row: row, codingPath: codingPath + [key]))
        }
    }

    struct RowDecoder: Decoder {
        func container<Key>(keyedBy type: Key.Type) throws -> KeyedDecodingContainer<Key> {
            if let key = codingPath.last {
                // Asked for a keyed type: look for a row scope
                if let scopedRow = row.scoped(on: key!.stringValue) {
                    let container = RowKeyedDecodingContainer<Key>(row: scopedRow, codingPath: codingPath)
                    return KeyedDecodingContainer(container)
                } else {
                    throw DecodingError.keyNotFound...
                }
            } else {
                // Asked for a keyed type at the top level
                let container = RowKeyedDecodingContainer<Key>(row: row, codingPath: codingPath)
                return KeyedDecodingContainer(container)
            }
        }
        
        func singleValueContainer() throws -> SingleValueDecodingContainer {
            // Asked for a single-value type: look for a column
            return RowColumnDecodingContainer(row: row, column: codingPath.last!!.stringValue)
        }
    }

(Sorry for the bangs, I still have to understand how I should deal with nil coding keys)

This works pretty well so far.

But now let's consider the `KeyedDecodingContainerProtocol.decodeIfPresent(_:forKey:)` method. Now we have a problem. This method must return nil if the key is missing. But which key? We don't know if the decoded type is keyed, or single-value. We can't postpone the decision, as above. So we have to double guess:

My current implementation is the following:

    func decodeIfPresent<T>(_ type: T.Type, forKey key: Key) throws -> T? where T : Decodable {
        if let dbValue: DatabaseValue = row[key.stringValue] {
            // We don't know if T(from: Decoder) will request a single value
            // container, or a keyed container.
            //
            // Since the column is present, let's assume that T will ask for a
            // single value container (a column). This is our only opportunity
            // to turn NULL into nil. If T eventually asks for a keyed container
            // (a row scope), then the user will face a weird error.
            if dbValue.isNull {
                return nil
            } else {
                return try T(from: RowDecoder(row: row, codingPath: codingPath + [key]))
            }
        } else if row.scoped(on: key.stringValue) != nil {
            // We don't know if T(from: Decoder) will request a single value
            // container, or a keyed container.
            //
            // Since the row scope is present, let's assume that T will ask for
            // a keyed container (a row scope). If T eventually asks for a
            // single value container (a column), then the user will face a
            // weird error.
            return try T(from: RowDecoder(row: row, codingPath: codingPath + [key]))
        } else {
            // Both column and row scope are missing: we are sure that the value
            // is missing.
            return nil
        }
    }

But it's less than ideal, as expressed by the inline comments.

We could discuss potential solutions, but I first hope that I was able to clearly express the topic.

Gwendal

Itai,

(This email is not technical)

I'm not claiming that SE-0166 should be able to address all archival formats. I've been talking about GRDB to show at least one format that SE-0166 doesn't cover well. And should SE-0166 be fixed to support SQL (in the GRDB fashion), this does not mean that other developers won't eventually fight with SE-0166 until they understand it does not fit their bill.

But there's something very special with SE-0166:

It's in the standard library, with all the backward-compatibility constraints that come with such a position.

IT'S BLESSED WITH CODE GENERATION.

I don't know if you, Michael LeHew, Tony Parker, and the core team, realize the importance of this insanely great privilege granted to this proposal.

The lack of introspection and macros in Swift makes SE-0166 immensely attractive for a whole category of libraries.

When SE-0166 is lacking, should those libs ignore it, and lose CODE GENERATION, which means looking like it's still Swift 3?

Should those libs claim SE-0166 conformance, and raise runtime errors for invalid inputs (where "invalid" does not mean "invalid data", or "invalid code", but "impossible to fit in SE-0166" <=> "invalid library")?

I'd like to hear a little better than that :-) GRDB is a library of unusual quality (sorry for the auto-congratulation). Until now, fatal errors thrown by GRDB were always a sign of programmer mistake. Not of defects in the foundations. I wish this would remain true.

Less caveats and runtime/fatal errors mean less user frustration.

Less caveats also mean less documentation to write. Ideally, this should be enough: https://github.com/groue/GRDB.swift/tree/Swift4#codable-records

Gwendal
Just a guy that write Swift apps and libraries

···

Le 30 mai 2017 à 20:49, Itai Ferber <iferber@apple.com> a écrit :

Hi Gwendal,

There are no stupid questions — everything helps hammer out this API, so I appreciate you taking the time to look at this so deeply.
I have to confess that I’m not familiar with this concept, but let’s take a look:

if let valueType = T.self as? DatabaseValueConvertible.Type {
    // if column is missing, trigger the "missing key" error or return nil.
} else if let complexType = T.self as? RowConvertible.Type {
    // if row scope is missing, trigger the "missing key" error or return nil.
} else {
    // don't know what to do
    fatalError("unsupported")
}
Is it appropriate for a type which is neither DatabaseValueConvertible nor RowConvertible to be decoded with your decoder? If not, then this warrants a preconditionFailure or an error of some sort, right? In this case, that would be valid.

You also mention that "it’s still impossible to support other Codable types" — what do you mean by this? Perhaps there’s a way to accomplish what you’re looking to do.
In any case, one option (which is not recommended unless if there are other avenues to solve this by) is to perform a "dry run" decoding. Attempt to decode the type with a dummy decoder to see what container it will need, then prepare your approach and do it again for real. Obviously, this isn’t a clean way to do it if we can find alternatives, but it’s an option.

— Itai

On 29 May 2017, at 4:51, Gwendal Roué via swift-evolution wrote:

Hello,

I have already asked stupid questions about SE-0167 and SE-0166, but this time I hope this is a real one.

According so SE-0166, codable types themselves instantiate a single value decoder, or a keyed container:

public struct Farm : Codable {
public init(from decoder: Decoder) throws {
let container = try decoder.container(keyedBy: CodingKeys.self
...
}
}

public enum Animal : Int, Codable {
public init(from decoder: Decoder) throws
let intValue = try decoder.singleValueContainer().decode(Int.self)
...
}
}

According to SE-0167, decoder decode non-trivial types in their decode(_:forKey:) and decodeIfPresent(_:forKey:) methods:

func decode<T>(_ type: T.Type, forKey key: Key) throws -> T where T : Decodable
func decodeIfPresent<T>(_ type: T.Type, forKey key: Key) throws -> T? where T : Decodable

My trouble is that the decoder does not know whether the Decodable type will ask for a keyed container, or for a single value container.

Why is it a problem?

In the context of decoding of SQL rows, keys may refer to different things, depending on whether we are decoding a *value*, or a *complex object*:

- for values, keys are column names, as everybody can expect
- for complex objects, keys are names of "row scopes". Row scopes are a concept introduced by GRDB.swift and allows a type that knows how to consume `SELECT * FROM table1` to consume as well the results of `SELECT table1.*, table2.* FROM table1 JOIN table2` through a "scope" that presents the row in the shape expected by the consumer (here, only columns from table1).

This is supposed to allow support for types that contain both nested types and values (one of the goals of SE-0166 and SE-0167):

struct Compound : Codable {
let someStruct: SomeStruct // object that feeds on the "someStruct" scope
let name: String // value that feeds on the "name" column
}

The two decoding methods decode(_:forKey:) and decodeIfPresent(_:forKey:) can't be implemented nicely, because they don't know whether the decodable type will ask for a keyed container or a single value container, and thus they don't know whether they should look for the presence of a row scope, or of a column:

A workaround is to perform runtime checks on the GRDB protocols adopted by T, as below. But it's still impossible to support other codable types:

if let valueType = T.self as? DatabaseValueConvertible.Type {
// if column is missing, trigger the "missing key" error or return nil.
} else if let complexType = T.self as? RowConvertible.Type {
// if row scope is missing, trigger the "missing key" error or return nil.
} else {
// don't know what to do
fatalError("unsupported")
}

Do you have any advice?

Gwendal Roué

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Hi Gwendal,

I hear your frustration. Some comments inline.

Itai,

(This email is not technical)

I'm not claiming that SE-0166 should be able to address all archival formats. I've been talking about GRDB to show at least one format that SE-0166 doesn't cover well. And should SE-0166 be fixed to support SQL (in the GRDB fashion), this does not mean that other developers won't eventually fight with SE-0166 until they understand it does not fit their bill.

I’ll respond to the technical portion of this thread in the other email, but let me at least provide some background here. When working on this feature, we thought for a very long time about what we were looking to support with this feature, and how (feel free to take a look at the Alternatives Considered section of the proposal, though of course, there were more attempts and approaches before that).
The majority of this thought was put into figuring out what the proper abstractions were for applying this new API — how can we abstract over different archival and serialization formats in a way that makes this useful?

In truth, if you try to abstract over all archival and serialization formats, the abstraction that you get is... the empty set. :) There are simply so many different things at odds with one another across different formats (JSON supports null values, plist does not; numbers are arbitrary precision in JSON, but not in plist or MessagePack or others; plist and MessagePack and others support binary data blobs, but JSON does not; etc.) that if you try to abstract over them all, you end up with nothing useful — an empty protocol that covers nothing.

So the key here is to try to strike a pragmatic balance between supporting some of the most common archival and serialization formats in a way that makes them useful, even if we have to handle special cases in some of them (e.g. null values in plist, binary data in JSON, etc.). It’s true that we cannot support them all, but in fact, we’re not looking to, because it would weaken the API.

I will respond to the comments specific to GRDB in the other thread, but this is a bit of background. Yes, there will always developers who will not be able to fit a serialization format into this API because it is fundamentally different in a way that cannot fit with the rest of the formats we’re looking to support. There’s nothing to be done about that. But, you mention this yourself.

But there's something very special with SE-0166:

It's in the standard library, with all the backward-compatibility constraints that come with such a position.

IT'S BLESSED WITH CODE GENERATION.

I don't know if you, Michael LeHew, Tony Parker, and the core team, realize the importance of this insanely great privilege granted to this proposal.

Believe me, I do, because we considered a lot of different approaches before settling on this. We wanted to avoid code generation for this reason — it has a privileged place within the compiler, it generates code which the user may not be able to introspect, etc.
At the end of the day, though, we decided on this option because it provided the best user experience as part of the language in the vast majority of cases. There’s a lot to be said for that, and you mention this yourself, too.

The lack of introspection and macros in Swift makes SE-0166 immensely attractive for a whole category of libraries.

When SE-0166 is lacking, should those libs ignore it, and lose CODE GENERATION, which means looking like it's still Swift 3?

Should those libs claim SE-0166 conformance, and raise runtime errors for invalid inputs (where "invalid" does not mean "invalid data", or "invalid code", but "impossible to fit in SE-0166" <=> "invalid library")?

That being said, let’s separate the capabilities of the Codable API itself from the code generated by the compiler for it. While the code generation is a huge convenience for the majority of simple cases, it does just that — generate code for the simple cases. We cannot arbitrarily generate code to match arbitrary applications. Much more is possible with custom encode/decode implementations and custom CodingKeys than you might imagine, rather than just sticking to the default, compiler-generated implementation. (Data migration, format-specific encoded representations, multiple sets of CodingKeys, etc.)

If a library finds use for the Codable APIs only for the code generation, then I think that’s likely misapplication of the API. Attempting to use the Codable API to fit a square peg into a round hole will be frustrating because, well, it was designed for a singular purpose.
The code generation that comes with Codable is meant for archival and serialization, not for arbitrary introspection. You’re right in that there is an overlap here (and I think the key pain point is that we need better tools for doing introspection, macros, and compile-time metaprogramming), but this is not a problem that this API is meant to solve. If a library cannot use the code generated by the Codable API, it’s not an "invalid library" — it’s just a poor fit. A full-featured compile-time macro/introspection system will require further thought and discussion, but the current Codable feature will eventually fit into that.

I'd like to hear a little better than that :-) GRDB is a library of unusual quality (sorry for the auto-congratulation). Until now, fatal errors thrown by GRDB were always a sign of programmer mistake. Not of defects in the foundations. I wish this would remain true.

Less caveats and runtime/fatal errors mean less user frustration.

Less caveats also mean less documentation to write. Ideally, this should be enough: https://github.com/groue/GRDB.swift/tree/Swift4#codable-recordsThere are always going to be cases where certain input cannot be limited in the type system. Consider JSON, which does not support NaNs or Infinity. There is no way to statically prevent that input via the type system, so yes, there has to be a runtime error thrown for that. That doesn’t make JSON invalid for this API; we just add a strategy to the encoder to allow users to customize what happens there. These errors don’t have to be fatal errors — we have EncodingError.invalidValue to cover just this issue; it is just not possible to cover all possible properties and all possible values of inputs statically.

It’s up to Encoders and Decoders to make the decisions that allow them to fit nicely within this. If they cannot, then it might be that they are not a good fit for this API.

···

On May 31, 2017, at 5:36 AM, Gwendal Roué <gwendal.roue@gmail.com> wrote:

Gwendal
Just a guy that write Swift apps and libraries

Le 30 mai 2017 à 20:49, Itai Ferber <iferber@apple.com <mailto:iferber@apple.com>> a écrit :

Hi Gwendal,

There are no stupid questions — everything helps hammer out this API, so I appreciate you taking the time to look at this so deeply.
I have to confess that I’m not familiar with this concept, but let’s take a look:

if let valueType = T.self as? DatabaseValueConvertible.Type {
    // if column is missing, trigger the "missing key" error or return nil.
} else if let complexType = T.self as? RowConvertible.Type {
    // if row scope is missing, trigger the "missing key" error or return nil.
} else {
    // don't know what to do
    fatalError("unsupported")
}
Is it appropriate for a type which is neither DatabaseValueConvertible nor RowConvertible to be decoded with your decoder? If not, then this warrants a preconditionFailure or an error of some sort, right? In this case, that would be valid.

You also mention that "it’s still impossible to support other Codable types" — what do you mean by this? Perhaps there’s a way to accomplish what you’re looking to do.
In any case, one option (which is not recommended unless if there are other avenues to solve this by) is to perform a "dry run" decoding. Attempt to decode the type with a dummy decoder to see what container it will need, then prepare your approach and do it again for real. Obviously, this isn’t a clean way to do it if we can find alternatives, but it’s an option.

— Itai

On 29 May 2017, at 4:51, Gwendal Roué via swift-evolution wrote:

Hello,

I have already asked stupid questions about SE-0167 and SE-0166, but this time I hope this is a real one.

According so SE-0166, codable types themselves instantiate a single value decoder, or a keyed container:

public struct Farm : Codable {
public init(from decoder: Decoder) throws {
let container = try decoder.container(keyedBy: CodingKeys.self
...
}
}

public enum Animal : Int, Codable {
public init(from decoder: Decoder) throws
let intValue = try decoder.singleValueContainer().decode(Int.self)
...
}
}

According to SE-0167, decoder decode non-trivial types in their decode(_:forKey:) and decodeIfPresent(_:forKey:) methods:

func decode<T>(_ type: T.Type, forKey key: Key) throws -> T where T : Decodable
func decodeIfPresent<T>(_ type: T.Type, forKey key: Key) throws -> T? where T : Decodable

My trouble is that the decoder does not know whether the Decodable type will ask for a keyed container, or for a single value container.

Why is it a problem?

In the context of decoding of SQL rows, keys may refer to different things, depending on whether we are decoding a *value*, or a *complex object*:

- for values, keys are column names, as everybody can expect
- for complex objects, keys are names of "row scopes". Row scopes are a concept introduced by GRDB.swift and allows a type that knows how to consume `SELECT * FROM table1` to consume as well the results of `SELECT table1.*, table2.* FROM table1 JOIN table2` through a "scope" that presents the row in the shape expected by the consumer (here, only columns from table1).

This is supposed to allow support for types that contain both nested types and values (one of the goals of SE-0166 and SE-0167):

struct Compound : Codable {
let someStruct: SomeStruct // object that feeds on the "someStruct" scope
let name: String // value that feeds on the "name" column
}

The two decoding methods decode(_:forKey:) and decodeIfPresent(_:forKey:) can't be implemented nicely, because they don't know whether the decodable type will ask for a keyed container or a single value container, and thus they don't know whether they should look for the presence of a row scope, or of a column:

A workaround is to perform runtime checks on the GRDB protocols adopted by T, as below. But it's still impossible to support other codable types:

if let valueType = T.self as? DatabaseValueConvertible.Type {
// if column is missing, trigger the "missing key" error or return nil.
} else if let complexType = T.self as? RowConvertible.Type {
// if row scope is missing, trigger the "missing key" error or return nil.
} else {
// don't know what to do
fatalError("unsupported")
}

Do you have any advice?

Gwendal Roué

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

Itai,

(This email is not technical)

I'm not claiming that SE-0166 should be able to address all archival formats. I've been talking about GRDB to show at least one format that SE-0166 doesn't cover well. And should SE-0166 be fixed to support SQL (in the GRDB fashion), this does not mean that other developers won't eventually fight with SE-0166 until they understand it does not fit their bill.

But there's something very special with SE-0166:

It's in the standard library, with all the backward-compatibility constraints that come with such a position.

IT'S BLESSED WITH CODE GENERATION.

One important future goal I hope we can address in future versions of Swift is powerful macro/meta-programming features. I think like Sourcery (https://github.com/krzysztofzablocki/Sourcery\) but at compile-time, in the language. Once we have that, I hope we can re-implement SE-0166’s code generation using those meta-programming features into the Standard Library and tear it out of the compiler. If we can achieve that, we will have a strong toolbox for any third-library which have similar needs.

PS: With meta-programming, we could also re-implement the automatic Equatable/Hashable conformances. More magic in the compiler is usually a bad thing I think.

···

On 31 May 2017, at 14:36, Gwendal Roué via swift-evolution <swift-evolution@swift.org> wrote:

I don't know if you, Michael LeHew, Tony Parker, and the core team, realize the importance of this insanely great privilege granted to this proposal.

The lack of introspection and macros in Swift makes SE-0166 immensely attractive for a whole category of libraries.

When SE-0166 is lacking, should those libs ignore it, and lose CODE GENERATION, which means looking like it's still Swift 3?

Should those libs claim SE-0166 conformance, and raise runtime errors for invalid inputs (where "invalid" does not mean "invalid data", or "invalid code", but "impossible to fit in SE-0166" <=> "invalid library")?

I'd like to hear a little better than that :-) GRDB is a library of unusual quality (sorry for the auto-congratulation). Until now, fatal errors thrown by GRDB were always a sign of programmer mistake. Not of defects in the foundations. I wish this would remain true.

Less caveats and runtime/fatal errors mean less user frustration.

Less caveats also mean less documentation to write. Ideally, this should be enough: https://github.com/groue/GRDB.swift/tree/Swift4#codable-records

Gwendal
Just a guy that write Swift apps and libraries

Le 30 mai 2017 à 20:49, Itai Ferber <iferber@apple.com <mailto:iferber@apple.com>> a écrit :

Hi Gwendal,

There are no stupid questions — everything helps hammer out this API, so I appreciate you taking the time to look at this so deeply.
I have to confess that I’m not familiar with this concept, but let’s take a look:

if let valueType = T.self as? DatabaseValueConvertible.Type {
    // if column is missing, trigger the "missing key" error or return nil.
} else if let complexType = T.self as? RowConvertible.Type {
    // if row scope is missing, trigger the "missing key" error or return nil.
} else {
    // don't know what to do
    fatalError("unsupported")
}
Is it appropriate for a type which is neither DatabaseValueConvertible nor RowConvertible to be decoded with your decoder? If not, then this warrants a preconditionFailure or an error of some sort, right? In this case, that would be valid.

You also mention that "it’s still impossible to support other Codable types" — what do you mean by this? Perhaps there’s a way to accomplish what you’re looking to do.
In any case, one option (which is not recommended unless if there are other avenues to solve this by) is to perform a "dry run" decoding. Attempt to decode the type with a dummy decoder to see what container it will need, then prepare your approach and do it again for real. Obviously, this isn’t a clean way to do it if we can find alternatives, but it’s an option.

— Itai

On 29 May 2017, at 4:51, Gwendal Roué via swift-evolution wrote:

Hello,

I have already asked stupid questions about SE-0167 and SE-0166, but this time I hope this is a real one.

According so SE-0166, codable types themselves instantiate a single value decoder, or a keyed container:

public struct Farm : Codable {
public init(from decoder: Decoder) throws {
let container = try decoder.container(keyedBy: CodingKeys.self
...
}
}

public enum Animal : Int, Codable {
public init(from decoder: Decoder) throws
let intValue = try decoder.singleValueContainer().decode(Int.self)
...
}
}

According to SE-0167, decoder decode non-trivial types in their decode(_:forKey:) and decodeIfPresent(_:forKey:) methods:

func decode<T>(_ type: T.Type, forKey key: Key) throws -> T where T : Decodable
func decodeIfPresent<T>(_ type: T.Type, forKey key: Key) throws -> T? where T : Decodable

My trouble is that the decoder does not know whether the Decodable type will ask for a keyed container, or for a single value container.

Why is it a problem?

In the context of decoding of SQL rows, keys may refer to different things, depending on whether we are decoding a *value*, or a *complex object*:

- for values, keys are column names, as everybody can expect
- for complex objects, keys are names of "row scopes". Row scopes are a concept introduced by GRDB.swift and allows a type that knows how to consume `SELECT * FROM table1` to consume as well the results of `SELECT table1.*, table2.* FROM table1 JOIN table2` through a "scope" that presents the row in the shape expected by the consumer (here, only columns from table1).

This is supposed to allow support for types that contain both nested types and values (one of the goals of SE-0166 and SE-0167):

struct Compound : Codable {
let someStruct: SomeStruct // object that feeds on the "someStruct" scope
let name: String // value that feeds on the "name" column
}

The two decoding methods decode(_:forKey:) and decodeIfPresent(_:forKey:) can't be implemented nicely, because they don't know whether the decodable type will ask for a keyed container or a single value container, and thus they don't know whether they should look for the presence of a row scope, or of a column:

A workaround is to perform runtime checks on the GRDB protocols adopted by T, as below. But it's still impossible to support other codable types:

if let valueType = T.self as? DatabaseValueConvertible.Type {
// if column is missing, trigger the "missing key" error or return nil.
} else if let complexType = T.self as? RowConvertible.Type {
// if row scope is missing, trigger the "missing key" error or return nil.
} else {
// don't know what to do
fatalError("unsupported")
}

Do you have any advice?

Gwendal Roué

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Hi Gwendal,

I hear your frustration. Some comments inline.

And I apologize for having written a rant :-)

Itai,

(This email is not technical)

I'm not claiming that SE-0166 should be able to address all archival formats. I've been talking about GRDB to show at least one format that SE-0166 doesn't cover well. And should SE-0166 be fixed to support SQL (in the GRDB fashion), this does not mean that other developers won't eventually fight with SE-0166 until they understand it does not fit their bill.

I’ll respond to the technical portion of this thread in the other email, but let me at least provide some background here. When working on this feature, we thought for a very long time about what we were looking to support with this feature, and how (feel free to take a look at the Alternatives Considered section of the proposal, though of course, there were more attempts and approaches before that).
The majority of this thought was put into figuring out what the proper abstractions were for applying this new API — how can we abstract over different archival and serialization formats in a way that makes this useful?

In truth, if you try to abstract over all archival and serialization formats, the abstraction that you get is... the empty set. :) There are simply so many different things at odds with one another across different formats (JSON supports null values, plist does not; numbers are arbitrary precision in JSON, but not in plist or MessagePack or others; plist and MessagePack and others support binary data blobs, but JSON does not; etc.) that if you try to abstract over them all, you end up with nothing useful — an empty protocol that covers nothing.

So the key here is to try to strike a pragmatic balance between supporting some of the most common archival and serialization formats in a way that makes them useful, even if we have to handle special cases in some of them (e.g. null values in plist, binary data in JSON, etc.). It’s true that we cannot support them all, but in fact, we’re not looking to, because it would weaken the API.

I will respond to the comments specific to GRDB in the other thread, but this is a bit of background. Yes, there will always developers who will not be able to fit a serialization format into this API because it is fundamentally different in a way that cannot fit with the rest of the formats we’re looking to support. There’s nothing to be done about that. But, you mention this yourself.

I totally support this point of view. And JSON/Plist/NSCoding were the expected fundamentals.

For me the consequence of these intrinsic limitations / admitted caveats / necessary humbleness, should have blocked the inclusion of the this archival mechanism in the Standard Lib. Or at least its *fast* inclusion in the stdlib.

The reason for this is that unlike many others parts of the standard lib that are under constant scrutiny from the community, chances are low that many Coder/Decoder are/will be written.

To be short: don't you think it's too early to declare the archival library *stable*?

But there's something very special with SE-0166:

It's in the standard library, with all the backward-compatibility constraints that come with such a position.

IT'S BLESSED WITH CODE GENERATION.

I don't know if you, Michael LeHew, Tony Parker, and the core team, realize the importance of this insanely great privilege granted to this proposal.

Believe me, I do, because we considered a lot of different approaches before settling on this. We wanted to avoid code generation for this reason — it has a privileged place within the compiler, it generates code which the user may not be able to introspect, etc.
At the end of the day, though, we decided on this option because it provided the best user experience as part of the language in the vast majority of cases. There’s a lot to be said for that, and you mention this yourself, too.

So you want to improve the user experience... But below you attempt to lower the impact of code generation:

The lack of introspection and macros in Swift makes SE-0166 immensely attractive for a whole category of libraries.

When SE-0166 is lacking, should those libs ignore it, and lose CODE GENERATION, which means looking like it's still Swift 3?

Should those libs claim SE-0166 conformance, and raise runtime errors for invalid inputs (where "invalid" does not mean "invalid data", or "invalid code", but "impossible to fit in SE-0166" <=> "invalid library")?

That being said, let’s separate the capabilities of the Codable API itself from the code generated by the compiler for it. While the code generation is a huge convenience for the majority of simple cases, it does just that — generate code for the simple cases. We cannot arbitrarily generate code to match arbitrary applications. Much more is possible with custom encode/decode implementations and custom CodingKeys than you might imagine, rather than just sticking to the default, compiler-generated implementation. (Data migration, format-specific encoded representations, multiple sets of CodingKeys, etc.)

If a library finds use for the Codable APIs only for the code generation, then I think that’s likely misapplication of the API.

That's what you hope. I predict that it will be abused in all kinds of awful ways, much worse than the GRDB attempt.

One year ago, all the rage was about dynamic runtime (http://inessential.com/2016/05/15/the_case_for_dynamic-swift_optimism\), because memories were full of Objective-C. Next came countless blog posts about Swift Mirror hackery. Developers are *craving* for code generation.

There is a huge existing ecosystem for archival mechanisms that already avoid boilerplate code. Swift is a modern language, and does not err on the side of Java and C++, unless I'm mistaken. Scripting languages (Python, Ruby) have been generating code for years. Rust has Diesel (http://diesel.rs <http://diesel.rs/&gt;\). Some GRDB users use Sourcery (https://github.com/krzysztofzablocki/Sourcery\).

So I wouldn't talk about misapplication of the API, but more about a mismatch between the API and the developer needs.

Some libs will fit into SE-0166, and will be blessed with code generation. Some other libs won't fit, and won't be blessed with code generation. This will have a huge impact on their perceived quality.

Attempting to use the Codable API to fit a square peg into a round hole will be frustrating because, well, it was designed for a singular purpose.

Yes, I understand. And your point of view is better than mine, because SE-0166 provides an actual *solution*, *today*, for *very frequent* use case. I can't help thinking that it's a little too soon for SE-0166, but I know very well that nobody will want to lose it, now that it is there.

The only reasonable way to reconcile our views, assuming I can be of any influence, is that SE-0166 should exist, but *not in the standard lib*. It's a matter of avoiding crippling stdlib with ad-hoc code that will not age very well and are nearly impossible to deprecate. It may be time to think about extending the number of core libraries.

Gwendal

···

Le 31 mai 2017 à 19:15, Itai Ferber <iferber@apple.com> a écrit :

On May 31, 2017, at 5:36 AM, Gwendal Roué <gwendal.roue@gmail.com <mailto:gwendal.roue@gmail.com>> wrote:

Hi David,

You're right, it's possible that SE-0166 eventually uses future macro-like features.

When this happens, the code generation blessing will be available to all Swift users, and SE-0166 will no longer be the necessary underlying foundation for libraries that want to use its code generation features (regardless of whether they are right or wrong: library developers don't think like compiler developers).

Gwendal

···

Le 31 mai 2017 à 15:39, David Hart <david@hartbit.com> a écrit :

On 31 May 2017, at 14:36, Gwendal Roué via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Itai,

(This email is not technical)

I'm not claiming that SE-0166 should be able to address all archival formats. I've been talking about GRDB to show at least one format that SE-0166 doesn't cover well. And should SE-0166 be fixed to support SQL (in the GRDB fashion), this does not mean that other developers won't eventually fight with SE-0166 until they understand it does not fit their bill.

But there's something very special with SE-0166:

It's in the standard library, with all the backward-compatibility constraints that come with such a position.

IT'S BLESSED WITH CODE GENERATION.

One important future goal I hope we can address in future versions of Swift is powerful macro/meta-programming features. I think like Sourcery (https://github.com/krzysztofzablocki/Sourcery\) but at compile-time, in the language. Once we have that, I hope we can re-implement SE-0166’s code generation using those meta-programming features into the Standard Library and tear it out of the compiler. If we can achieve that, we will have a strong toolbox for any third-library which have similar needs.

PS: With meta-programming, we could also re-implement the automatic Equatable/Hashable conformances. More magic in the compiler is usually a bad thing I think.