dealing with heterogenous lists/dictionary with Codable

Oh! Yeah, my bad... You are correct; I'd started thinking like I was on -evolution instead of -users.

···

On Oct 19, 2017, at 1:29 PM, Itai Ferber <iferber@apple.com> wrote:

Mm, the point I’m trying to get at here is that JSON is inherently untyped while Swift is strongly typed, and the two don’t line up.
It doesn’t really make sense to ask to decode an existential because there’s not enough type information to influence what you get back.

On 19 Oct 2017, at 13:20, David Sweeris wrote:

I think if you can figure that out, you’re halfway to letting protocols conform to themselves.

(Syntactically, I would probably say that something like “Codable.Self” would read well, but I think that already means something. Maybe the answer will become clearer when we rework the reflection APIs?)

Sent from my iPhone

On Oct 19, 2017, at 13:13, Itai Ferber <iferber@apple.com <mailto:iferber@apple.com>> wrote:

Even then, that wouldn’t necessarily help in the general case. If you decode {"key" : 1} as [String : Codable], what concrete type would 1 have? Int? Double? Int8? (Arguments can be made for any one of these, but the key here is that it is inherently ambiguous and there isn’t necessarily a good answer.)

On 19 Oct 2017, at 12:57, David Sweeris wrote:

On Oct 19, 2017, at 12:50 PM, David Baraff via swift-users <swift-users@swift.org <mailto:swift-users@swift.org>> wrote:

Yes; this is a case where anywhere in the code base I want to just say
struct MyNewType : Codable {
// add codable datatypes
}

and don’t want/can’t always go to the centralized place to add it in.
Is there some extension-like trick I can pull off that lets me spread the implementation out over different files/libraries?

Ah, ok.

No, I don't think you'll be able to do that until/unless Swift gets more macro/metaprogramming features. Maybe if protocols ever get to conform to themselves? That's a common request, but implementing it is apparently beyond tricky. I'm pretty sure somebody's working on it, but "bigger fish" and all that...

- Dave Sweeris

So, what I see as the problem here is contradictory compiler statements:

MyPlayground.playground:5:9: note: cannot automatically synthesize 'Encodable' because '[String : Any]' does not conform to 'Encodable'
    let someDict: [String : Any]
        ^

warning: MyPlayground.playground:8:18: warning: 'is' test is always true
[String : Any]() is Codable
                 ^

So is it codable or isn't it? You can use a [String: Any] in encode() and decode() directly, so why doesn't it get compiler support? At the least, the message needs to be updated to something that's actually true.

···

On Oct 19, 2017, at 1:38 PM, David Sweeris via swift-users <swift-users@swift.org> wrote:

Oh! Yeah, my bad... You are correct; I'd started thinking like I was on -evolution instead of -users.

On Oct 19, 2017, at 1:29 PM, Itai Ferber <iferber@apple.com <mailto:iferber@apple.com>> wrote:

Mm, the point I’m trying to get at here is that JSON is inherently untyped while Swift is strongly typed, and the two don’t line up.
It doesn’t really make sense to ask to decode an existential because there’s not enough type information to influence what you get back.

On 19 Oct 2017, at 13:20, David Sweeris wrote:

I think if you can figure that out, you’re halfway to letting protocols conform to themselves.

(Syntactically, I would probably say that something like “Codable.Self” would read well, but I think that already means something. Maybe the answer will become clearer when we rework the reflection APIs?)

Sent from my iPhone

On Oct 19, 2017, at 13:13, Itai Ferber <iferber@apple.com <mailto:iferber@apple.com>> wrote:

Even then, that wouldn’t necessarily help in the general case. If you decode {"key" : 1} as [String : Codable], what concrete type would 1 have? Int? Double? Int8? (Arguments can be made for any one of these, but the key here is that it is inherently ambiguous and there isn’t necessarily a good answer.)

On 19 Oct 2017, at 12:57, David Sweeris wrote:

On Oct 19, 2017, at 12:50 PM, David Baraff via swift-users <swift-users@swift.org <mailto:swift-users@swift.org>> wrote:

Yes; this is a case where anywhere in the code base I want to just say
struct MyNewType : Codable {
// add codable datatypes
}

and don’t want/can’t always go to the centralized place to add it in.
Is there some extension-like trick I can pull off that lets me spread the implementation out over different files/libraries?

Ah, ok.

No, I don't think you'll be able to do that until/unless Swift gets more macro/metaprogramming features. Maybe if protocols ever get to conform to themselves? That's a common request, but implementing it is apparently beyond tricky. I'm pretty sure somebody's working on it, but "bigger fish" and all that...

- Dave Sweeris

_______________________________________________
swift-users mailing list
swift-users@swift.org
https://lists.swift.org/mailman/listinfo/swift-users

Right now, collection types in the standard library (`Array`, `Dictionary`, and `Set`) employ a workaround for the lack of conditional conformance, which would allow us to expose things like `extension Array : Codable where Element : Codable` (i.e. `Array<T>` is `Codable` if and only if `T` is `Codable`).
These collections unconditionally conform to `Codable` and trap at runtime if the type is not actually `Codable`. The synthesis complaint is an explicit check in the compiler so we don’t go ahead and happily synthesize code on your behalf that would just trap at runtime.

When the conditional conformance feature lands, `[String : Any]() is Codable` should return `false`.

···

On 19 Oct 2017, at 14:15, Kevin Nattinger wrote:

So, what I see as the problem here is contradictory compiler statements:

MyPlayground.playground:5:9: note: cannot automatically synthesize 'Encodable' because '[String : Any]' does not conform to 'Encodable'
    let someDict: [String : Any]
        ^

warning: MyPlayground.playground:8:18: warning: 'is' test is always true
[String : Any]() is Codable
                 ^

So is it codable or isn't it? You can use a [String: Any] in encode() and decode() directly, so why doesn't it get compiler support? At the least, the message needs to be updated to something that's actually true.

On Oct 19, 2017, at 1:38 PM, David Sweeris via swift-users >> <swift-users@swift.org> wrote:

Oh! Yeah, my bad... You are correct; I'd started thinking like I was on -evolution instead of -users.

On Oct 19, 2017, at 1:29 PM, Itai Ferber <iferber@apple.com >>> <mailto:iferber@apple.com>> wrote:

Mm, the point I’m trying to get at here is that JSON is inherently untyped while Swift is strongly typed, and the two don’t line up.
It doesn’t really make sense to ask to decode an existential because there’s not enough type information to influence what you get back.

On 19 Oct 2017, at 13:20, David Sweeris wrote:

I think if you can figure that out, you’re halfway to letting protocols conform to themselves.

(Syntactically, I would probably say that something like “Codable.Self” would read well, but I think that already means something. Maybe the answer will become clearer when we rework the reflection APIs?)

Sent from my iPhone

On Oct 19, 2017, at 13:13, Itai Ferber <iferber@apple.com >>> <mailto:iferber@apple.com>> wrote:

Even then, that wouldn’t necessarily help in the general case. If you decode {"key" : 1} as [String : Codable], what concrete type would 1 have? Int? Double? Int8? (Arguments can be made for any one of these, but the key here is that it is inherently ambiguous and there isn’t necessarily a good answer.)

On 19 Oct 2017, at 12:57, David Sweeris wrote:

On Oct 19, 2017, at 12:50 PM, David Baraff via swift-users >>>> <swift-users@swift.org <mailto:swift-users@swift.org>> wrote:

Yes; this is a case where anywhere in the code base I want to just say
struct MyNewType : Codable {
// add codable datatypes
}

and don’t want/can’t always go to the centralized place to add it in.
Is there some extension-like trick I can pull off that lets me spread the implementation out over different files/libraries?

Ah, ok.

No, I don't think you'll be able to do that until/unless Swift gets more macro/metaprogramming features. Maybe if protocols ever get to conform to themselves? That's a common request, but implementing it is apparently beyond tricky. I'm pretty sure somebody's working on it, but "bigger fish" and all that...

- Dave Sweeris

_______________________________________________
swift-users mailing list
swift-users@swift.org
https://lists.swift.org/mailman/listinfo/swift-users

String, [Int], etc. but also allow structures where

struct NewThingy : Codable {
let data1: T1
let data2: T2
}

where T1 and T2 are themselves Codable.

This is already possible, just not with dictionaries of unknown types
(because they’re not known to be Codable)

So basically, back to wanting to let the compiler do the work, when I make
new structures, while still allowing for heterogenous containers.

It’s also possible to give the compiler hints as to what decodes into what.
Have you looked at the docs on the Apple foundation page?

Geordie

···

David Baraff <davidbaraff@gmail.com> schrieb am Do. 19. Okt. 2017 um 21:14: > My apologies. I misstated the problem: I don’t want to just limit to Int,

Begin forwarded message:

From: Itai Ferber <iferber@apple.com>

Subject: Re: [swift-users] dealing with heterogenous lists/dictionary with
Codable

Date: October 19, 2017 at 10:40:28 AM PDT

To: David Baraff <davidbaraff@gmail.com>

Cc: Geordie Jay <geojay@gmail.com>, swift-users <swift-users@swift.org>

Why are you stuck? I think the following matches your needs, no?

import Foundation
enum MyType : Codable, Equatable {
    case int(Int)
    case string(String)
    case list([MyType])
    case dictionary([String : MyType])

    public init(from decoder: Decoder) throws {
        // Can be made prettier, but as a simple example:
        let container = try decoder.singleValueContainer()
        do {
            self = .int(try container.decode(Int.self))
        } catch DecodingError.typeMismatch {
            do {
                self = .string(try container.decode(String.self))
            } catch DecodingError.typeMismatch {
                do {
                    self = .list(try container.decode([MyType].self))
                } catch DecodingError.typeMismatch {
                    self = .dictionary(try container.decode([String : MyType].self))
                }
            }
        }
    }

    public func encode(to encoder: Encoder) throws {
        var container = encoder.singleValueContainer()
        switch self {
        case .int(let int): try container.encode(int)
        case .string(let string): try container.encode(string)
        case .list(let list): try container.encode(list)
        case .dictionary(let dictionary): try container.encode(dictionary)
        }
    }

    static func ==(_ lhs: MyType, _ rhs: MyType) -> Bool {
        switch (lhs, rhs) {
        case (.int(let int1), .int(let int2)): return int1 == int2
        case (.string(let string1), .string(let string2)): return string1 == string2
        case (.list(let list1), .list(let list2)): return list1 == list2
        case (.dictionary(let dict1), .dictionary(let dict2)): return dict1 == dict2
        default: return false
        }
    }
}
let values: MyType = .list([.int(42), .string("hello!"), .list([.int(9), .string("hi")]), .dictionary(["zero": .int(0), "one": .int(1)])])print(values)
let encoder = JSONEncoder()let data = try encoder.encode(values)print(String(data: data, encoding: .utf8)!) // => [42,"hello!",[9,"hi"],{"zero":0,"one":1}]
let decoder = JSONDecoder()let decoded = try decoder.decode(MyType.self, from: data)print(decoded)
print(values == decoded) // => true

On 19 Oct 2017, at 20:15, David Baraff wrote:

Begin forwarded message:

From: Itai Ferber <iferber@apple.com>

Subject: Re: [swift-users] dealing with heterogenous lists/dictionary with
Codable

Date: October 19, 2017 at 9:39:25 AM PDT

To: David Baraff <davidbaraff@gmail.com>

Cc: Geordie Jay <geojay@gmail.com>, swift-users <swift-users@swift.org>

Hi David and Geordie,

That approach won’t work — encoders and decoders only work directly with
concrete Codable types (e.g. String, Int, MyFoo [where MyFoo is Codable],
etc.).
This is by design: since there is no type information stored in the JSON
payload, there isn’t necessarily a way to tell how to decode the type
you’re looking at, so asking for a generalCodable` isn’t helpful.

Since it’s unlikely that what you truly need is a [String : Any] but
really a [String : <one of String, Int, MyFoo, etc.>], one easy way to
decode this type is to create a wrapper enum or similar which overrides
init(from:) to be able to decode from one of those types. You can then
ask to decode a [String : MyWrapperType] and use that instead.

What types are you expecting in the dictionary?

The problem is that I want to be able to encode types T where
(a) T is String, Int
(b) lists of T
(c ) dictionaries of type <String, T>

The problem is the recursive nature: yes, my types are simple (say only
base types String and Int) but the “nesting” level may be quite deep (a
list of list of dictionaries of <etc.).

Let’s turn this around: in addition to the JSONEncoder, one can also use
the PropertyListEncoder.

Are we saying that something one could pull from a property list file
(which is pretty much what i want: arbitrary deep nesting of basic types)
is also not Codable? So a PropertyListEncoder could not encode actual
property lists?

I really do want a heterogenous container. I think I am stuck.

— Itai

On 19 Oct 2017, at 18:11, David Baraff via swift-users wrote:

I’ll try. Is that cast smart enough to apply recursively? We shall see.

Sent from my iPad

On Oct 19, 2017, at 7:34 AM, Geordie Jay <geojay@gmail.com> wrote:

I mean can you do something along the lines of

let codableDict = stringAnyDict as? [String : Codable]

?

I’m not at a computer to test it myself

David Baraff <davidbaraff@gmail.com> schrieb am Do. 19. Okt. 2017 um > 15:45:

That’s exactly what I want. The ironic part is that I got my dictionary
by decoding a Json file. If that’s where my dictionary came from, is there
a simple way of coercing the Json serialization routines to give me back
codables, rather than Anys?

Sent from my iPad

On Oct 19, 2017, at 3:38 AM, Geordie Jay <geojay@gmail.com> wrote:

David Baraff via swift-users <swift-users@swift.org> schrieb am Do. 19. >> Okt. 2017 um 03:47:

So I have simple structs like this:

        struct Library: Codable {
                let domain: String
                let unit: String
        }

and it’s super-simple to serialize. Yay.

But:

        struct LibraryGroup : Codable { // I wish...
           let libraries: [Library]
           let someDict: [String : Any]
        }

I haven’t tried this, but is it possible to have a dictionary of [String
: Codable] ? Because that’s exactly the type requirements you’re
describing, no?

Geordie

So what I’m looking for is something where if the values in someDict are
themselves Codable, I can serialize things, and if they’re not, I can’t.
In my previous scheme, I was using NSKeyedArchiver to serialize everything,
manualy, including someDict; in trying to switch to Codable I ran smack
into the fact that Codable wants to know what all the types are, in advance.

Am I just stuck? How do I get the best of both worlds, where the
compiler can make use of the fact that it can see the data types of my
structures, while still being able to serialize heterogenous data like is
found in LibraryGroup?

Is my only alternative to write a custom coder for LibraryGroup? Is
there any hope I could teach Codable what to do with
        [String: Any]

?

_______________________________________________
swift-users mailing list
swift-users@swift.org
https://lists.swift.org/mailman/listinfo/swift-users

_______________________________________________

swift-users mailing list
swift-users@swift.org
https://lists.swift.org/mailman/listinfo/swift-users

From: Geordie Jay <geojay@gmail.com>

Subject: Re: [swift-users] dealing with heterogenous lists/dictionary with
Codable

Date: October 19, 2017 at 12:24:44 PM PDT

To: David Baraff <davidbaraff@gmail.com>, Itai Ferber <iferber@apple.com>

Cc: swift-users <swift-users@swift.org>

My apologies. I misstated the problem: I don’t want to just limit to
Int, String, [Int], etc. but also allow structures where

struct NewThingy : Codable {
let data1: T1
let data2: T2
}

where T1 and T2 are themselves Codable.

This is already possible, just not with dictionaries of unknown types
(because they’re not known to be Codable)

Sure, but I don’t want to give a dictionary of unknown types: i’m very
happy to say that my dictionary is
[String : Codable]

but
struct Foo : Codable {
let d: [String : Codable]
}

doesn’t work; the d inside F is not itself Codable.

That’s strange. We’re actually doing exactly this and it works for us
(although we are using a concrete Codable type rather than the Codable
metatype itself).

Maybe it’s worth filing a bug on Jira

Good luck.

···

David Baraff <davidbaraff@gmail.com> schrieb am Do. 19. Okt. 2017 um 21:35: > > > Begin forwarded message:

David Baraff <davidbaraff@gmail.com> schrieb am Do. 19. Okt. 2017 um > 21:14:

So basically, back to wanting to let the compiler do the work, when I
make new structures, while still allowing for heterogenous containers.

It’s also possible to give the compiler hints as to what decodes into
what. Have you looked at the docs on the Apple foundation page?

Encoding and Decoding Custom Types | Apple Developer Documentation

Geordie

Begin forwarded message:

From: Itai Ferber <iferber@apple.com>

Subject: Re: [swift-users] dealing with heterogenous lists/dictionary
with Codable

Date: October 19, 2017 at 10:40:28 AM PDT

To: David Baraff <davidbaraff@gmail.com>

Cc: Geordie Jay <geojay@gmail.com>, swift-users <swift-users@swift.org>

Why are you stuck? I think the following matches your needs, no?

import Foundation
enum MyType : Codable, Equatable {
    case int(Int)
    case string(String)
    case list([MyType])
    case dictionary([String : MyType])

    public init(from decoder: Decoder) throws {
        // Can be made prettier, but as a simple example:
        let container = try decoder.singleValueContainer()
        do {
            self = .int(try container.decode(Int.self))
        } catch DecodingError.typeMismatch {
            do {
                self = .string(try container.decode(String.self))
            } catch DecodingError.typeMismatch {
                do {
                    self = .list(try container.decode([MyType].self))
                } catch DecodingError.typeMismatch {
                    self = .dictionary(try container.decode([String : MyType].self))
                }
            }
        }
    }

    public func encode(to encoder: Encoder) throws {
        var container = encoder.singleValueContainer()
        switch self {
        case .int(let int): try container.encode(int)
        case .string(let string): try container.encode(string)
        case .list(let list): try container.encode(list)
        case .dictionary(let dictionary): try container.encode(dictionary)
        }
    }

    static func ==(_ lhs: MyType, _ rhs: MyType) -> Bool {
        switch (lhs, rhs) {
        case (.int(let int1), .int(let int2)): return int1 == int2
        case (.string(let string1), .string(let string2)): return string1 == string2
        case (.list(let list1), .list(let list2)): return list1 == list2
        case (.dictionary(let dict1), .dictionary(let dict2)): return dict1 == dict2
        default: return false
        }
    }
}
let values: MyType = .list([.int(42), .string("hello!"), .list([.int(9), .string("hi")]), .dictionary(["zero": .int(0), "one": .int(1)])])print(values)
let encoder = JSONEncoder()let data = try encoder.encode(values)print(String(data: data, encoding: .utf8)!) // => [42,"hello!",[9,"hi"],{"zero":0,"one":1}]
let decoder = JSONDecoder()let decoded = try decoder.decode(MyType.self, from: data)print(decoded)
print(values == decoded) // => true

On 19 Oct 2017, at 20:15, David Baraff wrote:

Begin forwarded message:

From: Itai Ferber <iferber@apple.com>

Subject: Re: [swift-users] dealing with heterogenous lists/dictionary
with Codable

Date: October 19, 2017 at 9:39:25 AM PDT

To: David Baraff <davidbaraff@gmail.com>

Cc: Geordie Jay <geojay@gmail.com>, swift-users <swift-users@swift.org>

Hi David and Geordie,

That approach won’t work — encoders and decoders only work directly with
concrete Codable types (e.g. String, Int, MyFoo [where MyFoo is Codable],
etc.).
This is by design: since there is no type information stored in the JSON
payload, there isn’t necessarily a way to tell how to decode the type
you’re looking at, so asking for a generalCodable` isn’t helpful.

Since it’s unlikely that what you truly need is a [String : Any] but
really a [String : <one of String, Int, MyFoo, etc.>], one easy way to
decode this type is to create a wrapper enum or similar which overrides
init(from:) to be able to decode from one of those types. You can then
ask to decode a [String : MyWrapperType] and use that instead.

What types are you expecting in the dictionary?

The problem is that I want to be able to encode types T where
(a) T is String, Int
(b) lists of T
(c ) dictionaries of type <String, T>

The problem is the recursive nature: yes, my types are simple (say only
base types String and Int) but the “nesting” level may be quite deep (a
list of list of dictionaries of <etc.).

Let’s turn this around: in addition to the JSONEncoder, one can also use
the PropertyListEncoder.

Are we saying that something one could pull from a property list file
(which is pretty much what i want: arbitrary deep nesting of basic types)
is also not Codable? So a PropertyListEncoder could not encode actual
property lists?

I really do want a heterogenous container. I think I am stuck.

— Itai

On 19 Oct 2017, at 18:11, David Baraff via swift-users wrote:

I’ll try. Is that cast smart enough to apply recursively? We shall see.

Sent from my iPad

On Oct 19, 2017, at 7:34 AM, Geordie Jay <geojay@gmail.com> wrote:

I mean can you do something along the lines of

let codableDict = stringAnyDict as? [String : Codable]

?

I’m not at a computer to test it myself

David Baraff <davidbaraff@gmail.com> schrieb am Do. 19. Okt. 2017 um >> 15:45:

That’s exactly what I want. The ironic part is that I got my dictionary
by decoding a Json file. If that’s where my dictionary came from, is there
a simple way of coercing the Json serialization routines to give me back
codables, rather than Anys?

Sent from my iPad

On Oct 19, 2017, at 3:38 AM, Geordie Jay <geojay@gmail.com> wrote:

David Baraff via swift-users <swift-users@swift.org> schrieb am Do. 19. >>> Okt. 2017 um 03:47:

So I have simple structs like this:

        struct Library: Codable {
                let domain: String
                let unit: String
        }

and it’s super-simple to serialize. Yay.

But:

        struct LibraryGroup : Codable { // I wish...
           let libraries: [Library]
           let someDict: [String : Any]
        }

I haven’t tried this, but is it possible to have a dictionary of [String
: Codable] ? Because that’s exactly the type requirements you’re
describing, no?

Geordie

So what I’m looking for is something where if the values in someDict
are themselves Codable, I can serialize things, and if they’re not, I
can’t. In my previous scheme, I was using NSKeyedArchiver to serialize
everything, manualy, including someDict; in trying to switch to Codable I
ran smack into the fact that Codable wants to know what all the types are,
in advance.

Am I just stuck? How do I get the best of both worlds, where the
compiler can make use of the fact that it can see the data types of my
structures, while still being able to serialize heterogenous data like is
found in LibraryGroup?

Is my only alternative to write a custom coder for LibraryGroup? Is
there any hope I could teach Codable what to do with
        [String: Any]

?

_______________________________________________
swift-users mailing list
swift-users@swift.org
https://lists.swift.org/mailman/listinfo/swift-users

_______________________________________________

swift-users mailing list
swift-users@swift.org
https://lists.swift.org/mailman/listinfo/swift-users