SE-0195 — Introduce User-defined "Dynamic Member Lookup" Types

[rex-remind (Rex)]

I'm a fan of @dynamic(memberLookup, callable), it's easy to digest and points to the interplay of the two.

[michelf (Michel Fortin)]

@tim1724

It should be used rarely enough that a bit of ugliness at the declaration site is no big deal.

That could have been the logic behind the super verbose template syntax in C++. No one knows how many uses people will find for this feature in the future. It could become quite common. Ugliness is not much of a barrier.

[xwu (Xiaodi Wu)]

@Chris_Lattner3

The attribute still works with existentials. You can define something like:

@dynamicMemberLookup
protocol MyThing {
  func randomRequirement()

  subscript(dynamicMember: String) -> Self? { get set }
}

… and all types that conform to MyThing will get the dynamic behavior, and have their requirement spelled according to the protocol requirement - all without needing an explicit associatedtypes (which make existentials complicated). This sort of example was mentioned by Joe Groff, and is what really made me realize that this is better as an attribute that protocols can use, rather than be a protocol itself.

It is quite possible I’m misunderstanding you. If so, I apologize, please restate and I’ll try again :-)

Well, kind of what I'm getting at, but not entirely. As a jumping off point, the protocol in your example would work equally well refining a bona fide DynamicMemberLookupProtocol. In fact, it's kind of weird that the protocol's subscript(dynamicMember:) is "inherited" from and refining an attribute.

To build on this notion that @dynamicMemberLookup is a protocol, with its own requirements, masquerading as an attribute: you answered "yes" to the question of whether protocols can use this attribute, which would imply that we could recapitulate the previously proposed design simply by writing:

@dynamicMemberLookup
public protocol DynamicMemberLookupProtocol { }

As to existentials: Since Self requirements and associated types both complicate existentials, the @dynamicMemberLookup spelling doesn't make the protocol in your example any more flexible than it would be when the proposed feature is spelled as DynamicMemberLookupProtocol. Nor, as I think the proposal stated, does there seem to be much utility in enabling such protocol existentials in the first place.

[kiel (Kiel Gillard)]

Personally not a fan of 'magic' because it suggest the compiler is doing something mysterious for or incomprehensible to you, the developer. Perhaps, for example:

@synthesized(dynamicMemberLookup, callable)

...since it better expresses the compiler is "filling in the blanks" for you.

[QuinceyMorris]

I don't know if anyone cares, but there's really no "dynamic member lookup" in this proposal. In a way, the point is that there isn't any member lookup in what the compiler does here. (The dynamic member lookup happens in DynamicCallable, I guess.)

I suppose there is "dynamicMember lookup" going on here, but that's not really the thrust.

I'm inclined to think the dynamic behavior is similar to the "dynamic" modifier behavior, though obviously via a different implementation mechanism. So, my suggestion would be:

@dynamic (implicit, callable)

[hexdreamer (Kenny Leung)]

I guess if we were to call it what it really is, it would be

 @sugar(memberToSubscript)

[hashemi (Ahmad Alhashemi)]

@runtime or @hijack also come to mind

[saagarjha]

I don't think @hijack really means quite the same thing, since it implied that something's being overridden or taken over. This is just a dynamic lookup.

[Chris_Lattner3 (Chris Lattner)]

@xwu

To build on this notion that @dynamicMemberLookup is a protocol, with its own requirements, masquerading as an attribute: you answered “yes” to the question of whether protocols can use this attribute, which would imply that we could recapitulate the previously proposed design simply by writing:

@dynamicMemberLookup
public protocol DynamicMemberLookupProtocol { }

I'm sorry I missed this earlier. The above will be rejected by the compiler, because the protocol does not define the subscript required by the attribute. You can use the attribute on protocols, but they have to specify the requirement to use, which is what allows them to be used as existentials etc.

[xwu (Xiaodi Wu)]

@Chris_Lattner3

I’m sorry I missed this earlier. The above will be rejected by the compiler, because the protocol does not define the subscript required by the attribute. You can use the attribute on protocols, but they have to specify the requirement to use, which is what allows them to be used as existentials etc.

Even if so, I'd still be able to do the same thing with minor modifications:

@dynamicMemberLookup
protocol DynamicMemberLookupProtocol {
  associatedtype Result
  subscript(dynamicMember: String) -> Result { get set }
}

[hashemi (Ahmad Alhashemi)]

@saagarjha

I don't think @hijack really means quite the same thing, since it implied that something's being overridden or taken over. This is just a dynamic lookup.

I see what you mean. I wouldn’t have called it hijack if this was a dynamic language. Python's __getattr__ or Ruby's method_missing feel more natural in those languages.

In Swift though, "dynamic" makes me think of dynamic dispatch in generics and protocols, which is still handled by the compiler with some static type compatibility guarantees. The changes in this proposal go a step beyond that. Thinking of it as "just a dynamic lookup" is perhaps one of the reasons people are concerned about the potential for abuse.

[Chris_Lattner3 (Chris Lattner)]

@xwu

Even if so, I’d still be able to do the same thing with minor modifications:

@dynamicMemberLookup
protocol DynamicMemberLookupProtocol {
associatedtype Result
subscript(dynamicMember: String) -> Result { get set }
}

Yep, that should work fine. That implementation is deciding a number of things that are not ok for the proposal in general, but could be useful for specific applications (please don't call it DMLP though! :-). The things it is deciding is:

1. Mutability settings for the getter and setter.
2. That there is an associated type named Result.
3. That it cannot be used as an existential - at least until we have generalized existentials.

It can still be a super useful abstraction point though.

-Chris

[xwu (Xiaodi Wu)]

@Chris_Lattner3

Yep, that should work fine. That implementation is deciding a number of things that are not ok for the proposal in general, but could be useful for specific applications (please don’t call it DMLP though! :-). The things it is deciding is:

Mutability settings for the getter and setter.
That there is an associated type named Result.
That it cannot be used as an existential - at least until we have generalized existentials.

It can still be a super useful abstraction point though.

But that's my point, though. Actually, two points:

First: besides the complications with (1) that you've addressed in both versions of the proposal, why aren't (2) and (3) "OK for the proposal in general"? That is, what does being usable as an existential get you?

Second: even if we accept that (2) and (3) aren't OK in general, isn't the fact that I can trivially re-express @dynamicMemberLookup as a protocol without additional magic suggestive that the design is, in fact, very much appropriately spelled as a protocol in the first place?

[masters3d]

@xwu

Second: even if we accept that (2) and (3) aren’t OK in general, isn’t the fact that I can trivially re-express @dynamicMemberLookup as a protocol without additional magic suggestive that the design is, in fact, very much appropriately spelled as a protocol in the first place?

I disagree with this. Having to use @dynamicMemberLookup in the protocol definition is the whole point of having this feature behind the attribute. The additional "magic" is the use of the attribute. It should be very clear to the reader that this protocol is special because the attribute says so.

[xwu (Xiaodi Wu)]

@masters3d

I disagree with this. Having to use @dynamicMemberLookup in the protocol definition is the whole point of having this feature behind the attribute. The additional “magic” is the use of the attribute. It should be very clear to the reader that this protocol is special because the attribute says so.

If you believe that an attribute is needed to clarify this, then you shouldn't be satisfied with this revised proposal, because (as I demonstrated) I can remove the need for an attribute in a large swath of cases by writing five lines of code, recapitulating the previous design.

Which is why I'm asking why the revised proposal goes with the attribute to begin with, when it's intended to be "inherited" as though it's a protocol that can be refined.

[masters3d]

@xwu

If you believe that an attribute is needed to clarify this, then you shouldn’t be satisfied with this revised proposal, because (as I demonstrated) I can remove the need for an attribute in a large swath of cases by writing five lines of code, recapitulating the previous design.

But I can alway navigate to this root declaration where I will find the attribute. That is not the case when the marker is just the protocol itself.

I'm inclined to agree with you on how easily it is to hide this but it is just as easily to hide the @objc in a swift class even though @objc on a class requires conforming to the NSObject. Otherwise you get this error:
Only classes that inherit from NSObject can be declared @objc.

We could go the same route with Dynamic Member Lookup and friends by requiring an attribute and a protocol but I don't see the advantage.

It would seem to me that all attributes are always effectively inherited. This seems to be the case with @objc . Let me know if I am wrong.

[Chris_Lattner3 (Chris Lattner)]

@xwu

But that’s my point, though. Actually, two points:

First: besides the complications with (1) that you’ve addressed in both versions of the proposal, why aren’t (2) and (3) “OK for the proposal in general”? That is, what does being usable as an existential get you?

There are some advantages in some domains for expressing things as an existential. In particular, you get implicit conversions from values that conform to the existential's protocol to the existential type. In an earlier version of the Python interop design, PyVal was an existential. I ultimately moved away from that, but I wouldn't be surprised that it would be useful for some other application.

To be clear though, I'm just saying that there is a thing that exists and it could be useful for some cases. I'm not saying that this is strong motivation.

@xwu

Second: even if we accept that (2) and (3) aren’t OK in general, isn’t the fact that I can trivially re-express @dynamicMemberLookup as a protocol without additional magic suggestive that the design is, in fact, very much appropriately spelled as a protocol in the first place?

We debated this at length in the core team meeting, and my interpretation of the discussion was "yes, this could be modeled as a protocol" but "this would mislead people". The problem is that being a protocol would naturally lead people to think that they could use it (e.g.) as a generic constraint, but the underlying model does not support that. If we provided it as a protocol, people would see it as a deficiency that the implementation didn't support that and that they would keep asking for it. Given the implementation approach, these approaches would be "wrong", but they wouldn't be wrong for asking, given how protocols fit into Swift.

Given that, and given that it could also be very reasonably modeled as an attribute, it seemed better to model it as an attribute: doing so doesn't lead people into a path where it feels like an uncanny valley where it is somewhat like a protocol but not really. It really is compiler magic, and an attribute is a good way to model that. The thing that eventually really convinced me is the fact that you'll be able to define a protocol with this attribute on it (assuming the protocol provides the subscript) and that THAT protocol will work properly with generics and existentials of desired for a specific application.

While there are reasonable arguments on all sides, this feels "just right" on balance. I'm sure that others will feel different of course, but given that these spelling decisions don't fundamentally affect the expressivity of the proposal, it seems like a good place to land.

-Chris

[tkremenek (Ted Kremenek)]

Proposal Accepted

On February 8, 2018 the Core Team decided to accept this proposal, given the form in its second version where an attribute is used to designate types that participate in the sugared dot-syntax described in this proposal.

After reviewing the feedback from the community, the Core Team felt the spelling of the attribute should remain as proposed — @dynamicMemberLookup. The alternative of @dynamic(...) was also considered, but the Core Team felt that spelling to be too close to the dynamic keyword and a potential source of confusion. This concern was brought up in the review thread as well.

The rationale for the marker being an attribute instead of a protocol (as in the original proposal) was well-articulated by @Chris_Lattner3 in the review thread in that the marker protocol would not provide the expected affordances of a normal protocol:

@Chris_Lattner3

We debated this at length in the core team meeting, and my interpretation of the discussion was “yes, this could be modeled as a protocol” but “this would mislead people”. The problem is that being a protocol would naturally lead people to think that they could use it (e.g.) as a generic constraint, but the underlying model does not support that. If we provided it as a protocol, people would see it as a deficiency that the implementation didn’t support that and that they would keep asking for it. Given the implementation approach, these approaches would be “wrong”, but they wouldn’t be wrong for asking, given how protocols fit into Swift.

Thank you to everyone who participated in discussing this language change, both during the pitch phase and the formal review!

[nuclearace (Erik Little)]

@Chris_Lattner3 Do we have any timeline for the DynamicCallable proposal? I know you said that you still needed to touch it up to reflect changes that have been discussed, but it'll be nice to have that on the table quickly so that we have time to discuss it in time for March 1st (if that is a goal).

[Chris_Lattner3 (Chris Lattner)]

I'm hoping to find time this weekend to update the DynamicMememberLookup patch and submit it. I'm not sure when I'll have time to start implementing DynamicCallable, but I will do so as soon as I can. I don't have a projection for how long that will take.

All that said, yes, I'd very much like to get DynamicCallable into Swift 5 and will do what I can to make it.