SE-0195 — Introduce User-defined "Dynamic Member Lookup" Types

The review of "SE-0195 — Introduce User-defined "Dynamic Member Lookup" Types" begins now and runs through January 29, 2018.

The proposal is available here:

https://github.com/apple/swift-evolution/blob/master/proposals/0195-dynamic-member-lookup.md

Reviews are an important part of the Swift evolution process. All review feedback should be either on this forum thread or, if you would like to keep your feedback private, directly to the review manager. When emailing the review manager directly, please keep the proposal link at the top of the message.

What goes into a review of a proposal?

The goal of the review process is to improve the proposal under review through constructive criticism and, eventually, determine the direction of Swift.

When reviewing a proposal, here are some questions to consider:

  • What is your evaluation of the proposal?

  • Is the problem being addressed significant enough to warrant a change to Swift?

  • Does this proposal fit well with the feel and direction of Swift?

  • If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

  • How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

Thanks,

Ted Kremenek

Review Manager

24 Likes

Thanks Ted! One additional request for the discussion: this proposal has benefited from many rounds are extensive discussion in the pitch phase, and much of that feedback and alternatives considered are incorporated into the proposal. I'd really appreciate it if reviewers would take a look at the new proposal - even if you've looked at it before - since it has evolved quite a bit over time.

In particular, in the pitch phase several people were concern about potential abuse of the feature. The proposal specifically provides several ways to "dial it back" in the alternatives section, so if that is your concern, please +1 one of those alternatives in your feedback.

Thank you!

-Chris

5 Likes

What is your evaluation of the proposal?

Huge +1 on this.

Is the problem being addressed significant enough to warrant a change to Swift?

Yes, I believe that in order to truly make it pleasant to interface with dynamic languages we need to be able to reference members that may have been attached dynamically in a way that is natural in Swift. Otherwise the interface (like the example in the proposal) is less than ideal, and the chances of winning over people from these libraries is much less.

Does this proposal fit well with the feel and direction of Swift?

Yes. Much of the discussion around this (and the related protocol) was around possible abuse of these features. And while there is always the possibility of abuse, this feature does not bring in a dynamic typing model that many people seemed to be worried about. This proposal is fully in line with Swift's type system and thus is no worse than typing everything with Any

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

I've used Scala's implementation, and this proposal is very much in-line. Both allow for expressivity of dynamic language models, as well as the possibility of other interesting uses.

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

I've followed this proposal from the start.

Kudos to @Chris_Lattner3 for driving this forward

What is your evaluation of the proposal

+1 I would love to use Python scientific libraries through Swift. Moreover, I like the JSON use case.

Does this proposal fit well with the feel and direction of Swift?

Yes, the proposed design seems rather 'Swifty' to me. Also, I would like to see Swift 'regaining' some of the dynamism of Obj-C.

How much effort did you put into your review?

I read the proposal.

-g.

What is your evaluation of the proposal?

+1. This will be very useful for interop with lots of languages. As someone who has to deal with a ton of old code written in scripting languages I'd love to be able to start writing newer code in Swift while still being able to leverage existing libraries.

I'm sure people will also find a bunch of clever ways to use it in Swift-only projects, as well. Yeah, some people will complain about those things being an abuse of the feature, but just about every new feature can be abused so I don't consider that a particularly worthwhile argument. The prohibition on retroactive conformance should stop anyone from doing anything too disgusting.

Is the problem being addressed significant enough to warrant a change to Swift?

Yes. Being able to leverage existing Python libraries for areas where native Swift libraries are sparse (particularly for scientific computing or computational biology) would make Swift a viable language for many areas where it's not yet useful.

Does this proposal fit well with the feel and direction of Swift?

It feels like a very Swifty to me. I'd rather not see much work done on native dynamic features right now, but this feels like a good way to put only a minimal shim into the compiler to enable user code to do most of the heavy lifting of bridging dynamic languages.

This feels much more appropriate than either putting lots of special-cased Python-specific code into the compiler or depending on outside tools to mechanically generate a ton of wrapper code.

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

I don't think I've encountered a similar feature in a strongly typed language before.

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

I've been following this very closely ever since Chris introduced the idea. I'd been very much hoping that the Discourse forum would be open before this came up for review so that I could more easily jump into the discussion. :smiley:

What is your evaluation of the proposal?

+1

Is the problem being addressed significant enough to warrant a change to Swift?

Yes.
It is a syntactic addition, everything could also be achieved by explicitly using a subscript.
However, the proposed feature greatly improves the ease of use when bridging to dynamic languages.

Does this proposal fit well with the feel and direction of Swift?

Yes.
It allows the natural use of dynamic objects, with the same look-and-feel as a native Swift object.
There were a lot of concerns about this proposal during the discussion, but all of them have been addressed and I am convinced that the fear of misuse will not become a real problem.

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

It is very similar to __getitem__() in Python. The ability to statically define the type of these members and to specify error behavior makes the Swift feature much safer.

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

Read the various versions of the proposal and followed the discussion.

What is your evaluation of the proposal?

+1 , I like number 3 for the limit of abuse. Specially since they will be other protocols that are related to the same idea, it would be good to tide them all together under a common umbrella like @dynamic in the call site.

Is the problem being addressed significant enough to warrant a change to Swift?

yes

Does this proposal fit well with the feel and direction of Swift?

yes, world domination

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

TypeScript impoting typeless js. not the same though.

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

participated during review

What is your evaluation of the proposal?
I'm in general in favor of this proposal. It will make Swift a more powerful programming language by allowing users to interoperate with existing code written in languages like Python, which is a huge gain.
I'm also a fan of the implementation being a protocol that a type can implement, it feels Swifty and fits within the rest of the language design.

Is the problem being addressed significant enough to warrant a change to Swift?
The problem being addressed is significant. However, I wonder if the nature of the problem warrants tackling this at this present moment as opposed to further down the line.
One thing that I miss from the proposal is a mention about the impact on the complexity of the compiler implementation that adding this to the language would have.
As someone who writes Swift full-time, and has done so almost exclusively for the past 3-or-so years, it's frustrating to see that so much of the tooling (Xcode, lldb), but also the compiler still has huge performance and stability issues that I encounter and affect my productivity on a daily basis.
Part of it I'm sure is a direct consequence of the fast-pace of evolution of Swift from 1.0 to 4.0. Precisely because of this, I would like to see the Swift team take a step back, and given that the language is stabilizing in some ways (at least when it comes to syntax and features?), be mindful of what "nice-to-haves" get added at this point, and turn the focus towards maturity.

That been said, I am not familiar with compiler implementation, and it's possible that I'm making wrong assumptions about the impact this feature would have on the swiftc codebase, which is why perhaps this proposal would benefit from calling that out.

Does this proposal fit well with the feel and direction of Swift?
As mentioned before, the approach taken fits within the rest of the language.

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
N/A

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?
I read (not always in the deepest detail) all of the versions of this proposal up to and including this one, and followed some of the discussion on the mailing list.

3 Likes

One thing that I miss from the proposal is a mention about the impact on the complexity of the compiler implementation that adding this to the language would have.

Hi Javi,

Sorry for not being clear about that. An implementation of the feature is available in the PR at the top of the proposal: It is a very simple extension that is non-invasive in the compiler. It is ~200 lines of code in the compiler, and ~300 lines of test cases added IIRC.

-Chris

8 Likes

What is your evaluation of the proposal?

Huge +1. I will also start (ab)using this feature for some DSL stuff.

Is the problem being addressed significant enough to warrant a change to Swift?

Yes.

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

Rather in-depth study and was following the discussions since the beginning.

Thanks @Chris_Lattner3 for pushing this forward. By the way, Is callable type proposal also coming up after this?

Thanks @Chris_Lattner3 for pushing this forward. By the way, Is callable type proposal also coming up after this?

Yep, I'll spin up DynamicCallable after this proposal settles. I have a draft available on my public gists, but will need to dust it off and update it to align best with the final version of this proposal.

-Chris

1 Like

I think this proposal is going to be one of the biggest hits for Swift 5. When paired with its companion proposal and an implementation of Python bridging, it'll be utterly transformative by allowing those who want to do numerical computing, scientific computing, machine learning work, etc., to use both very mature Python libraries and Swift.

Yes, very much so.

Yes, I think the proposal justifies very well how this is the most "Swifty" way to implement the desired feature. I will not recite the extensive language in the proposal to that effect.

I think no other "dial it back" alternatives are required to prevent abuse, as restricting conformance so that third parties cannot retroactively change how a type is dispatched will remove the most feared consequences. Moreover, while there is good precedent (among synthesized conformances) for the no-retroactive-conformance limit, some of the other alternatives would be rather idiosyncratic.

I have used Python for scientific computing; I do believe that this proposal, paired with its companion proposal, will allow Swift to be suitable for use with Python libraries without obvious major drawbacks.

A quick reading of this version of the proposal; an in-depth study of previous versions.

3 Likes

+1 It's a big step towards gathering a much larger user base.

It really fits well with DSL's and integrating with other languages, but not really anywhere else. I don't agree with the JSON example. I would hope people would get more time to discover/adopt something like Codable. However, the gains from this proposal, for me, outway the things people might do which are bad practice.

The problem of language integration is certainly being addressed well with this proposal.

Yes, using protocol conformance to get sugar is a well-developed pattern in Swift.

Not type-safe versions, but in comparison to those, much better :)

Read multiple proposal versions and participated in the discussion.

Just a query. Are you also thinking about making a proposal to get 'PythonGlue' into Swift?

Yes, after DynamicMemberLookup and DynamicCallable are settled, I plan to discuss refinement and integration of the PythonGlue prototype - I'd love to see it eventually get standardized in the default Python module overlay.

That said, it has many open design discussions, so I'm not sure exactly how long that iteration will take, and I'm not sure if swift-evolution is the right process for the Python module. I'm sure we'll figure out a solution to the process issues though - the core team has reassured me several times that integration with dynamic languages like Python is something they want to see happen.

-Chris

1 Like

I'd also like to see a Swift overlay for JavaScriptCore.framework, but this depends on the team at Apple:
https://github.com/apple/swift/commit/babc84f4263f3149b22a12417992d3cbe57ba7fb

An overlay for JavaScriptCore would be really really interesting. While it is beyond my experience to drive, it could be really profound!

3 Likes

What is your evaluation of the proposal?

This proposal has the potential to open up vast new design spaces for Swift types and libraries. I am very much in favor of it, and I would be very disappointed to see it closed off to users through something like abuse-mitigation alternative #2.

I would prefer to see this implemented with an attribute instead of a protocol. The proposal already captures and attempts to rebut many of my arguments on this point, so I won't repeat them. However, I would like to respond to this point:

Protocols describe semantics of a conforming type, and this proposal provides key behavior to the type that conforms to it.

While it's true that this protocol marks types which have a particular semantic, the protocol doesn't actually capture any of that semantic's requirements. That means, for instance, that a DynamicMemberLookupProtocol existential or generic constraint does not give access to the dynamic functionality. I don't like this at all. Pure marker protocols are occasionally a good idea, but not often.

Is the problem being addressed significant enough to warrant a change to Swift?

Yes. We need features like this for dynamic language bridging, but I'm actually more excited about extending this feature for boilerplate reduction.

Suppose that today, you're generating a bunch of code like:

extension Account {
    var friends: Resource<[Account]> {
        return Resource("account", self, "friends")
    }
    var blocks: Resource<[Account]> {
        return Resource("account", self, "blocks")
    }
    var posts: Resource<[Post]> {
        return Resource("account", self, "posts")
    }
}

There might be a whole bunch of different extensions like this on different parts of your model.

I envision a world where you could build a type which functions as a sort of schema of all the properties you want to have:

struct ResourceSchema<Parent, Child> {
    var name: String
}
extension ResourceSchema where Parent == Account, Child == [Account] {
    static var friends: ResourceSchema { return ResourceSchema(name: "friends") }
    static var blocks: ResourceSchema { return ResourceSchema(name: "blocks") }
}
extension ResourceSchema where Parent == Account, Child == [Post] {
    static var posts: ResourceSchema { return ResourceSchema(name: "posts") }
}

(In an ideal world, those lines could look like static let friends = ResourceSchema(name: "friends") instead. Remind me to propose allowing stored static variables on generic types in extensions which fully constrain all the parameters.)

And then you could write a single getter implementation which was used for all of these properties:

struct Account: DynamicMemberLookupProtocol {
    ...
    
   subscript<Child>(dynamicMember key: ResourceSchema<Account, Child>) -> Resource<Child> {
      return Resource("account", self, key.name)
   }
}

This could replace a lot of things which currently require code generation without having to go all the way to adding hygienic macros. This is not something that I think the proposal can do today, but I see this as a straightforward extension of the proposed functionality.

Does this proposal fit well with the feel and direction of Swift?

Yes. The use of a subscript is an elegant way to represent member lookup, and the overall feature fits Swift's goal of ensuring code is uncluttered and easy to read. The mechanism is currently limited, but it can be expanded in the future.

I remain unconvinced that a protocol is the right way to represent this feature, but this is ultimately a relatively unimportant detail.

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

Most languages I've used with a similar feature work something like Objective-C's -forwardInvocation:, where all types—whether they have dynamic behavior or not—call some sort of fallback method when a member is not found. Compared to those languages, the proposed mechanism is much better because the compiler can reliably emit errors when a type does not use the fallback lookup feature. For example, Objective-C warns for methods it doesn't know about, but which may eventually be found at runtime; Ruby does not warn about methods it doesn't think exists, so only a method lookup failure at runtime produces an error. The proposed solution is much better than either of these.

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

I've participated in discussion of this feature at every public stage of its design. I gave the final proposal a quick reading.

1 Like

Proposed solution:

In the discussion cycle, there was significant concern about abuse of this feature, particularly if someone retroactively conforms a type to DynamicMemberLookupProtocol. Further, it is easy to argue that dynamic behavior is a core part of the contract of a type's behavior, not something that should be changeable retroactively. For this reason, the compiler only permits conformance of this protocol on the original type definition, not extensions.

This requirement would prevent a JavaScriptCore overlay from adding DynamicMemberLookupProtocol conformance to JSContext and JSValue. The overlay (or a third-party library) could use wrapper classes instead.

Naming:

Suggestions for a better name for the protocol and the subscript (along with rationale to support them) are more than welcome.

This might depend on the names chosen for the DynamicCallable proposal. For example, if callable types are interchangeable with closure types, the Dynamic part of the name could be dropped. And the ability to accept arbitrarily labelled arguments could be a separate feature (perhaps an alternative to variadic parameters). Both features might use attributes instead of a protocol.

Does it make sense to the have DynamicCallable and DynamicMemberLookupProtocol be separate proposals. One would be quite fruitless without the other!

Off topic, but does the DynamicCallable proposal allow for the lookup, but not evaluation of functions? E.g. allow a function to be dynamically looked up once, and called repeatedly in a hot loop, without duplicating dynamic look ups