SE-0195 — Introduce User-defined "Dynamic Member Lookup" Types

Evolution Proposal Reviews
21

What is your evaluation of the proposal?

+1. It's great!

Is the problem being addressed significant enough to warrant a change to Swift?

Absolutely. Swift will benefit greatly from being able to piggy back off other languages' strengths

Does this proposal fit well with the feel and direction of Swift?

It appears contrary to the dominate-the-world goal, but I'm not opposed. I'd rather have easy interop now, while we slowly assimilate the rest of the world >:)

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

N/A

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

Read the whole proposal, and some forum threads from its development phase

Additional Comments

We should introduce a protocol that defines conversions to/from Swift's currency types (Bool, (U)Int(|8|16|32|64), Float, Double, String, Array, Dictionary, Set), so that there is a standardized interface to bridging back and forth between native Swift types, and the dynamic language's types. ExpressibleByXLiteral is a good first step, but doesn't help for non literal values, and doesn't help with conversion back to Swift types

22

Yes, the combination of these two proposals allows that, and it just naturally falls out of the model. You look up the function, cache it into a variable, and invoke it inside the loop. It is as simple as changing this:

for ...
   a.method()

Into something like this:

let tmp = a.method
for ...
  tmp()

-Chris

1 Like
23

What is your evaluation of the proposal?

In my opinion the proposal doesn't solve the problem at hand in a reasonable manner. I'm mostly concerned about the type-safety of the code. Given that a given entity can only implement such protocol once most of the time the subscript method would return either a protocol or a strict (but only one) type. In reality, we would be looking at something like PythonValue which the consumer would need to know how unpack. I don't believe this problem can be solved with current Swift capabilities, but an ideal solution could make use of a generic protocol concept.

I'm also not a huge fan of stringly typed APIs and I believe that the bridging code should be auto-generated somehow to contain possible keys that can be accessed.

Given all of the above in an ideal world, we could have something like this:

public protocol DynamicMemberLookupProtocol<LookupValue> {
    associatedtype KeywordType: CodingKey

    subscript<KeywordType: ExpressibleByStringLiteral, LookupValue>
        (dynamicMember key: KeywordType) -> LookupValue { get }
}


class Bridged {}

extension Bridged: DynamicMemberLookupProtocol<Bool> {
    typealias KeywordType = Key

    enum Key {
        case someBool
        case otherBool
    }

    subscript<KeywordType: ExpressibleByStringLiteral, LookupValue>
        (dynamicMember name: KeywordType) -> LookupValue { 

        return underlying[key].boolValue
    }
} 

extension Bridged: DynamicMemberLookupProtocol<String> {
    typealias KeywordType = Key

    enum Key {
        case someString
        case otherString
    }

    subscript<KeywordType: ExpressibleByStringLiteral, LookupValue>
        (dynamicMember name: KeywordType) -> LookupValue { 

        return underlying[key].stringValue
    }
}

It could still allow for some form of dynamic lookup by String type when needed, but I believe that would be rarely used. This would also allow SourceKit to generate some form of code completion for these dynamic members.

Is the problem being addressed significant enough to warrant a change to Swift?

It's definitely and interesting feature, but as I stated above, I don't believe it can be implemented currently in Swift in a proper manner.

Does this proposal fit well with the feel and direction of Swift?

I don't believe so, for me Swift is all about type-safety and this proposal, in current form, doesn't solve the problem at hand with type safety in mind.

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

I've seen similar approach being used in languages like PHP or JS and from the point of the consumer of such API I can't say I'm a huge fan. Most of the time, it feels like passing in strings and retrieving some value, but what that value actually is is left to the consumer to specify.

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

I've read the current version of the proposal and glanced through the existing replies to the thread.

1 Like
24

This is essentially equivalent to manually writing wrappers around all types/functions/members. The main issue with this is that such a wrapper targets the unstable Python API, rather than the relatively stable Python C API. Such wrappers are cumbersome to write in the first place (placing a HUGE barrier to importing a new python library), APIs change making wrappers get out of sync, and the whole thing is just a big mess.

Plus, who will own/maintain such wrappers? E.g. if we want to wrap Pandas, will the wrapper be part of the Pandas repo (no chance), Swift repo (slim chance), or some repo for common Python wrappers? How is the versioning between Swift/Python versions handled? Yikes.

1 Like
25

Such wrappers could be autogenerated using tools like Sourcery. I'm not an expert on Python so I have no idea how Python modules are built, but I believe such task shouldn't be impossible to achieve.

Anyway, we're not discussing Python (or any other specific programming language here) bridging but a dynamic member lookup for Swift types. My proposal (which is strictly theoretical at this point) would still allow you to simply implement the protocol with KeywordType of String and LookupValue of Any (or PythonValue).

26

How would the wrapper generation work? From the proposal:

Use "automatically generated wrappers" to interface with Python

This approach has numerous problems. Beyond that, wrappers also fundamentally require that we take (something like) this proposal to work in the first place. The primary issue is that Python (and other dynamic languages) require no property declarations. If you have no property declaration, there is nothing for the wrapper generator to [w]rap or generate.

And my focus on Python to facilitate more concrete terms of discussion, but any other dyanmic language (the target of this proposal) would have the exact same issues.

For which dynamic language would you ever have a KeywordType that's not String, and a LookupValue that isn't PyVal or equivalent?

I like strong styping, and dislike stringly typing too, but we're talking about dynamic languages here. That's the name of the game.

27

One example that comes to my mind right now would be protobuf entities that are actually indexed by Int and have a known value type most of the time. Using DynamicMemberLookupProtocol and DynamicCallable one could implement protobuf services which are now missing from swift-protobuf package.

28

Footnote: This is true for languages like Python and JS where method invocation is modeled as property lookup that returns a function.

It would not work for Ruby and other languages using the “Smalltalk family” method invocation API in the DynamicCallable proposal — but then Ruby itself does not support what you describe!

1 Like
29

You and I both want to be able to use fixed key sets, and you and I both think arbitrary string literals should be supported too. So please consider taking my position on this proposal—that it should be accepted in its current form and that we should later extend the feature to support fixed key sets in addition to ExpressibleByStringLiteral types. I really think SE-0195 could be an important step towards our goal.

3 Likes
30

Hi Maku, the idea of generating wrappers automatically was extensively discussed in the pitch phase and the problems with it are explained at the bottom of the proposal. Among other problems explained in the section and the linked post, Python doesn't have property declarations and other things for a tool to generate Swift declarations from, so wrappers would have to be done by hand, and you need something to build the wrappers out of :-)

1 Like
31

What is your evaluation of the proposal?

A hesitant +1.

Is the problem being addressed significant enough to warrant a change to Swift?

Possibly. Subscripts don't seem to be any more cumbersome to me than a property. ie object.someValue vs object["someValue"]. The only benefit I can think of is that it frees plain subscripts to be converted into another languages subscripts.

On the matter of interrupting with other languages, I think this is something that can provide benefit (although not necessarily a required feature).

In one of my iOS apps, I've actually been using the moment.js library alongside native code. Here's what that looks like. We were having issues with our relative timestamps being out of sync with our web interface (one would say "yesterday" while the other would say "12 hours ago" etc.). But after switching to moment and tuning the performance a bit, I found that it was actually faster than our ObjC implementation, loaded just as quickly (and lazily), and had a smaller binary size. That probably isn't the norm, but goes to show that not all operations are automatically faster in pure Swift/ObjC. Not to mention that for a lot of things it just doesn't make sense to reimplement them from scratch.

Does this proposal fit well with the feel and direction of Swift?

It think the biggest challenge with this proposal is that you are asking a bunch of people that love Swift, a very strongly typed language, to accept interop with languages that are inherently dynamic and unenforced.

2 Likes
32

What is your evaluation of the proposal?

+1. I'm really happy to see this proposal move forward. It has been really well thought, and it has gone through many reviews to endup being in this final form, which I really like. Adding this powers to the language and having access to other ecosystems will be great for Swift.

Furthermore I really like that it gives to user level code some language level powers, and with minimal impact in the compiler. As a Swift user I really care about the burden that the compiler has to take with each new feature as it affects my day to day directly (we all want a good performant error-free compiler), so adding this powerful feature with minimal invasion seems like a good deal.

Is the problem being addressed significant enough to warrant a change to Swift?

I think so. This proposal won't only allow current Swift developers like us to use a new bast number of libraries but, and for me much more important, will facilitate people from other communities to start peeking into the Swift ecosystem. Would make it easier for them to start writing Swift if they know they can still use their favourite libraries.

Does this proposal fit well with the feel and direction of Swift?

This makes the language much more powerful, which will help with world domination.
And what I really like is that it still does it in a way that allows dynamic layers to be written in a type safe manner by returning Optional for example. It doesn't throw out the door the nice and safer things about Swift.

That said, and is not rare coming from me, I still would love to see in the future really powerful metaprogramming facilities that would allow user level code to implement something like type providers to give progressive typing to APIs implemented with this Dynamic protocols.

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

I haven't used anything similar in a strongly static typed language. But this is similar to Objective-C forwarding selector mechanism, and it's obviously how Ruby works all the time.
The point with that feature of Objective-C is that 99% of the time in all my experience we just use the typed APIs, but is nice to have a tool that allows you to do runtime magic when necessary.

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

I've been interested in this proposal since Chris suggested it with a first post in the mailing list. I participated in the conversation during all this time, I read all the versions of the proposal and tried the different versions of the Python playground.

I want to thank Chris for taking the time to evolve the proposal properly, accepting and acting on the feedback and not rushing it. Even when the initial feedback was pretty negative. Thanks!

Question:

That said, I have a couple of questions that won't affect my review but I'm still curious to know the answer for:

  1. The proposal mentions proxies. One think I would really like is to have a class that proxies method calls to an injected value. But thinking how to do it with this proposal the only way I can think of is to have a big if/else checking for the function name and then forward them manually with "static" (like written in the source) calls to the underlaying object. Is there any piece I'm missing that would allow this to work without the big if/else?
  2. Given that the protocol doesn't have formal requirements, what would happen if the subscript has multiple overloads? Would the actual Swift resolution machinery just kick in?

Thanks.

2 Likes
33

Thank you for writing this up so thoroughly, Chris. You've done a very good job making it clear what is and isn't being proposed, and motivating the change.

I remain pretty unhappy with this, but I'll separate my specific unhappiness from my general unhappiness so that the proposal can be improved without having to deal with my overall -1 at the same time.

Specific Concerns

  • Swift member names have labels, like foo(bar:). While none of the dynamic languages you've mentioned here have similar multi-part names, it's worth describing how they will be handled, if at all. (I assume the flip side of the problem, how to handle keyword arguments in languages that support them, will be described in the forthcoming DynamicCallable proposal.)

  • I am concerned about the use of DynamicMemberLookupProtocol on existential or generic values. Because the feature necessarily looks statically for a subscript(dynamicMember:), it's possible that it will pick a different implementation than what would happen. This isn't a new problem for Swift; it's what happens with anything that's not defined as a protocol requirement. But it feels more surprising in this case, because it feels like the subscript is part of the protocol, even though formally it isn't.

  • I am convinced as to your points about heavier syntax not really buying us anything, for two reasons:

    • Stroustrup's observation that goes something like "people want new features to be explicitly called out and old features to be syntactically lightweight, but then new features become old features". (I can't find a citation for this but I remember the original context being C++'s template.)

    • We used a very similar argument to avoid putting new or alloc in front of instantiation in Swift 1. (That is, new NSView(frame: …), to make it clear that an allocation was happening.) The specific point is that any method can do allocations, or do large amounts of work, and you should be able to understand that from context anyway, so the same should be true of regular instantiation.

    But that said I don't think AnyObject is a great example to cite. AnyObject is terrible. It's the compromise we calculated to make Swift 1 support nearly everything Objective-C did. If we were doing it today, there's be a much higher chance it would use explicit optionals, or be restricted only to subscripts. On that note...

  • The Objective-C interoperability approach to handling the "inherently dynamic" parts of Objective-C is a feature called "AnyObject dispatch".

    This is not correct. The "inherently dynamic" parts of Objective-C feature sending new messages to objects with known type, but AnyObject dispatch lets you send existing messages to objects with unknown type. Even in Objective-C, the preferred way to send messages to id is to declare a protocol declaring those methods, even if nothing formally adopts that protocol.

    In practice, we think it is very rare that AnyObject dispatch is used these days, since Swift now imports id as Any rather than AnyObject. It can't even be used to dig into nested dictionaries the way it used to be.

    I don't think this invalidates the larger points you are making. The Clang importer is huge and is indeed a source of bugs. The existence of AnyObject lookup, as little-used as it is these days, is another source. But it's poor justification for Swift/ObjC interop being memory-unsafe or unsound.

  • Such integration would require that we do something like this proposal anyway, because Python (like Objective-C) is a fundamentally dynamic language, and we need a way to express that fundamental dynamism: either this proposal or something like "AnyObject for Python".

    This is nonsense. Such an importer could easily synthesize direct calls to the Python runtime using computed properties, exactly what you have in your sample subscript(dynamicMember:). We already do something like this with the init(rawValue:) initializers synthesized for imported C enums.

    I'm not saying (here) that the dynamic version is better or worse, only that it is not necessary in order to implement an importer- or type-provider-based approach.

General Concerns

  • Your "explicit member lookup" code is really more verbose than necessary. Where you have this:

    // dog2 = Dog("Kaylee").add_trick("snore")
let dog2 = Dog("Kaylee").get(member: "add_trick")("snore")

    I would expect to see subscripts, for exactly the same reasons you outline of why this protocol uses subscripts:

    // dog2 = Dog("Kaylee").add_trick("snore")
let dog2 = Dog("Kaylee")["add_trick"]("snore")

    This syntax is certainly not as nice as the one with DynamicMemberLookupProtocol, but it works today. (Well, the lookup part works today. The calling part still needs your DynamicCallable.)

    I can see this not being sufficient to lure Python programmers to Swift, but I don't know if that's a goal of the Swift project. As you say, the idea of Swift being a better Python than Python itself (or Ruby, or JavaScript) is not something worth striving for, and therefore I'm inclined to say this doesn't meet the bar since it isn't significantly better than just using subscripts directly. That's already a very good level of interop.

  • A thought: This feature is clearly designed for Swift clients of Python libraries. If you were writing a Swift library that depended on Python but was intended to be used by other Swift clients, you have a choice: expose bare Python values, or wrap them in Swift structs. If you choose the latter, it seems way less important to provide sugar for the direct access to Python, especially if we ever get a language feature like property behaviors. That said, there are always an order of magnitude more direct clients than libraries, so this may not be a major consideration.

  • On a minor note, I object to the emphasis on "type safe" throughout this proposal. The proposal is "type safe" in that it does not lead to accessing values of one type as if they had another type, but that's just the default for Swift. It would be fine to mention this once and be done with it. For example:

    Now clients are able to write more natural code like: json[0]?.name?.first?.stringValue which is close to the expressivity of Javascript... while being fully type safe!

    This is no more type-safe than it is in JavaScript. Either way it's a check at run time.

TLDR: This is a well-built proposal, but the whole feature is ultimately syntactic sugar for a subscript, and I'm inclined to say it's not sufficiently better than just using a subscript to be worth adding to the language. But were we to accept it, I have a few specific concerns around compound member names and the use of dynamic lookup on non-concrete types (detailed above).

P.S. DynamicCallable is more interesting to me. I haven't read the proposal thoroughly, but that one's closer to pulling its own weight in my mind.

6 Likes
34

There is no design for proxies included, and admittedly the thoughts on that are half baked. It isn't clear that this proposal is the best way to improve proxy systems, so I'd suggest not focusing on that. I should take it out of the proposal if it is distracting.

Yes. As implemented, the compiler checks the requirements of the protocol (even though it is not spelled in the protocol itself). You're allowed to have multiple implementations of the subscript, and the compiler uses overload resolution to pick the right one. You can check out the testcases attached to the patch for an example of that.

-Chris

35

Thank you for the detailed response Jordan!

Python does have keyword arguments and are supported in the prototype. In the pitch phases, support for Ruby, Squeak and other Smalltalk derived languages (e.g. Swift itself :-) came up and I'm committed to supporting them. The reason you don't see that in the proposal is that keyword arguments and basename+keyword issues are the subject of the DynamicCallable proposal, not DynamicMemberLookup.

DynamicMemberLookup as a generic or existential constraint itself is not enough to enable the new behavior (as you say, it does not have witness table entries for the compiler to talk to). It is useful as a basis for further derived protocols which themselves satisfy DynamicMemberLookup's constraints, providing the necessary witnesses. This all composes very nicely and is demonstrated in the testcases included with the implementation.

Perhaps I misunderstand what you're saying here, but wrapper generation (either through an offline process, through a clang-importer, or type-provider-like approach) isn't a feasible solution. This is covered at the end of the proposal and it links to an email with more details why (one issue is that Python doesn't have property declarations to synthesize these from!)

Thank you again for your thoughtful response.

-Chris

36

Two things occurred to me:

#1:

It also extends the language such that member lookup syntax (x.y) - when it otherwise fails
(because there is no member y defined on the type of x)

What happens if there is a name collision between the bridging object and the thing you're bridging? This may go unnoticed when you're writing it, or the compiler might complain if the types are not the ones expected. I guess the solution would be to directly use the subscript syntax if that's what you mean.

#2: What is the effect on using KeyPaths with these pseudo properties? Would they always succeed, or always fail?

37

The proposal covers this here: https://github.com/apple/swift-evolution/blob/master/proposals/0195-dynamic-member-lookup.md#effect-on-api-resilience

Basically, it isn't a good idea to use this on protocols that have lots of statically defined members or an evolving interface. When a conflict happens, the compiler picks the statically defined member. If you'd like to override that to get the dynamic one, you can use the subscript explicitly (just as you say).

This proposal doesn't provide keypath support for these members, but it is possible that we could extend keypaths to support this in the future if there is a compelling use-case.

-Chris

1 Like
38

What is your evaluation of the proposal?

I'm hugely in its favor. This is a functionality which Swift needs; it will add powerful new capabilities to the language.

Is the problem being addressed significant enough to warrant a change to Swift?

Yes, absolutely. As-strongly-as-possible-typed dynamic lookup is definitely worth it.

Does this proposal fit well with the feel and direction of Swift?

Maybe. I feel that Swift has, as a rule, always striven to be as static and strongly-typed as possible, and this feels in may ways the opposite of those goals. That being said, the functionality being added improves Swift's interoperability and flexibility enough to more than offset what is at worst a very minor step backwards.

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

As with others, Objective-C is where I have the most experience with dynamism; this proposal is considerably more usable and reliable than that support.

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

I gave it a quick reading; I don't feel qualified to comment on a more in-depth study at the moment anyway :)

1 Like
39

I didn't realize this. Personally, I would consider lack of keypath support to be a significant defect in the proposal. We should not have properties which can't be addressed through keypaths.

3 Likes
40

Sorry for being unclear! I'll try to clarify these points.

Python's keyword arguments (and Ruby's) really aren't the same as Swift's and Objective-C's compound names, though. Python allows you to use keyword arguments to name existing ordered arguments, allowing reordering. Ruby puts keyword arguments in a Hash bucket (and transparently explodes them back out in Ruby 2.0+). In both cases, it's the base name that selects the function, and the argument labels are part of the call. In Swift, the argument labels participate in overload resolution, something that becomes clear when you don't call the function:

let fn1 = obj.foo(bar:)
let fn2 = obj.foo(baz:)

I'm fine with saying this isn't supported in DynamicMemberLookupProtocol, but I think it should be explicitly called out if that's the case. On the other hand, if anyone tries to make a Squeak interface with this, they might run into trouble.

The case I am concerned about is when the derived protocol has a subscript(dynamicMember:) defined in an extension, and then some concrete model object also defines subscript(dynamicMember:). In that case the generic/existential code will behave differently from code that acts on the concrete type. Again, this isn't a new problem, and it's probably unlikely to come up in practice, but it is a caveat of the static approach.

Yes, again, I'm not saying that one of those approaches is better. I'm saying that "Such integration would require that we do something like this proposal anyway", which you say in your objections to both alternate approaches, is incorrect. Your other objections are still valid, but this one should be removed.