SE-0195 — Introduce User-defined "Dynamic Member Lookup" Types

My impression is that there are a bunch of languages with member lookup syntax that's mostly the same as Swift's. Individual members of that bunch of languages will have variants on the syntax where you'd need to fall back to the subscript+string form.

// No problem to represent in Swift. 95% of member lookups look like this.
myObj.foo = 42

// Namespaced members might be dicey. 4% look like this.
myObj.intrinsic::identity = 'myObj'
// Swift: myObj[dynamicMember: "intrinsic::identity"] = "myObj"

// Dynamic namespaces definitely fall back to subscript+string. 1% look like this.
var ns = new Namespace('myNamespace')
print(myObj.ns::bar)

// Oh no!
default xml namespace = 'http://www.w3.org/2000/svg'
myXML.@attr = 'preserveAspectRatio'

Implementers would have the responsibility and discretion to choose the extent to which people feel at home, but it's going to be more difficult when the syntax diverges.

Jordan, I think you're missing my point. I'm saying that neither the Python nor the Squeak style of arguments are handled by this proposal, because both are the subject of the DynamicCallable proposal. In that proposal, both Python and Squeak style dispatch will be handled as I mentioned.

It is worth pointing out that Smalltalk derived languages have properties as well (for which DynamicMemberLookup is still useful). It requires the composition of both proposals to cleanly handle method invocations and naming.

To reiterate, this came up and was discussed at length in the pitch phases, please refer back to those for more details, or wait until DynamicCallable is dusted off.

-Chris

1 Like

What is your evaluation of the proposal?

Huge +1

Is the problem being addressed significant enough to warrant a change to Swift?

Absolutely, being able to call into Python libraries would make Swift open to the vast readily-available data engineering/machine learning ecosystem. That's in addition being able to call code in other dynamic languages and handling other use cases mentioned previously.

Does this proposal fit well with the feel and direction of Swift?

Yes, it fits well, proposed syntax doesn't get in the way of getting things done, especially if DynamicCallable is accepted in addition to this one.

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

I've tracked this proposal from the start.

1 Like

I agree with Jordan, but I'll add some points:

  • I'm not really happy with how the proposal keeps throwing the words "type safe" around. It overplays the benefits. You don't need this proposal for type-safety; you could also get that with a subscript.

  • I feel that this proposal is much more than just an "interface" to dynamic languages; it integrates them wholesale in to the language, and makes too many affordances to accommodate for languages where there is an independence mismatch with the Swift type-system. We have protocols with no requirements, using special subscript-matching rules in an attempt to shoe-horn another model of calling in.

  • Dynamic member lookup is almost-always a failable operation, but this model makes it hidden at the point of use. You mention that it's an implementation decision, and somebody could decide to return optionals, but you also point out that it looks awful. So, in practice, the fallibility will be hidden at the point of use.

I think the alternative presented under the "increasing visibility" section is more verbose than it needs to be. I think we could do better. Let me take your example and change it from using ^ to making dynamic-lookups throwable. So instead of marking up every lookup, we are able to just mark up every line:

public protocol DynamicMemberLookupProtocol {
    subscript<KeywordType: ExpressibleByStringLiteral, LookupValue>
      (dynamicMember name: KeywordType) throws -> LookupValue { get }
}
try Python { py -> Void in
  let np = try py.import("numpy")
  let b = try np.array([6, 7, 8])
  let y = try np.arange(24).reshape(2, 3, 4)

  let a = try np.ones(3, dtype: np.int32)
  let b = try np.linspace(0, pi, 3)
  let c = try a+b
  let d = try np.exp(c)
}

So that's better than the example, but still not great, I accept. However, we have now we've reduced the problem to making dynamic-lookup failures implicit within a particular scope, and explicit outside of it. That goes against the Swift Error model (which calls for explicit trys everywhere), but we can adjust that if we limit it really specifically:

public protocol DynamicLookupScope {
    associatedtype Value: DynamicMemberLookupProtocol where Scope == Self

    /// Inside `scope`, lookups via Value.subscript(dynamicMember:) are implicitly called with a `try`, as a syntax convenience.
    ///
    static func call<T>(_ scope: (Self) throws -> T) rethrows -> T
}
public protocol DynamicMemberLookupProtocol {
    associatedtype Scope: DynamicLookupScope where Value == Self

    subscript<KeywordType: ExpressibleByStringLiteral, LookupValue>
      (dynamicMember name: KeywordType) throws -> LookupValue { get }
}

class Python {
    typealias Value = PyVal
    static func call<T>(_ scope: (Python)->T) throws -> T { ... }
}
class PyVal: DynamicMemberLookupProtocol {
    typealias Scope = Python
    ...
}

Now we can try and re-write your example again:

do {
    try Python { py -> Void in
      let np = py.import("numpy")
      let b = np.array([6, 7, 8])
      let y = np.arange(24).reshape(2, 3, 4)

      let a = np.ones(3, dtype: np.int32)
      let b = np.linspace(0, pi, 3)
      let c = a+b
      let d = np.exp(c)
    }
} catch Python.memberNotFound {
    preconditionFailure("Uh-oh")
}

So that's another way to remove the boilerplate, and one which does not hide the possibility of lookup failure.

1 Like

In Swift, throwing is for handling recoverable errors. This, on the other hand, is six lines of boilerplate to print "Uh oh"--there's no meaningful recovery possible. How could there be? Every possible Python call could throw, and the only choice is to trap, which Chris's proposal already does.

5 Likes

Assuming the python code returned a value, your error handler might return a default, or disable this part of the application which depends on this python module. It doesn’t matter; you’re taking it too literally. I’m just putting this out as an illustration of how we can make failability explicit while giving an option to skip the boilerplate if you are in a special context which already makes it obvious.

If you don’t want it to use the normal Error system - fine, we can create a parallel system. Maybe you have to write “dynamic” instead of “try”, and we don’t provide an Error type. It would me like an optional, but instead of optional-chaining or special operators, you can mark the entire expression with a single keyword and handle failures if appropriate.

Swift is a great language because it does not hide the underlying complexity of the system; it just provides high-level APIs which guide you to use them. I feel like this foreign code should live in its own, clearly-marked scope, (and be failable outside of it for adhoc calls), but I don’t care about forcing boilerplate on people. If there is a way to make it clear without being verbose, I’d be happy.

1 Like

This area has been covered at length in the previous discussions of earlier drafts. I encourage you to read through those very interesting conversations.

One very good argument is simple, so I'll restate it here: if it's not already clear by importing a Python module that you're working with dynamic features, then it's hard to see how additional syntax would clear that up.

2 Likes

I think it should be "try-or-dy" :-)

One very good argument is simple, so I’ll restate it here: if it’s not already clear by importing a Python module that you’re working with dynamic features, then it’s hard to see how additional syntax would clear that up.

This argument is a straw man. The proposal as written includes no requirement that anything remotely approaching import Python is present in a source file for which dynamic features are available. All that is required is that the source work with a type which conforms to DynamicMemberLookupProtocol.

If this proposal is accepted as-is it will be impossible to look at an arbitrary Swift file and know whether dynamic member lookup is used or not. Determining whether or not that is the case will require nontrivial effort. It is likely that most programmers will simply assume it is not used unless they have reason to believe otherwise. This lack of clarity and potential for faulty assumptions is what motivated the inclusion of the "Increasing Visibility of Dynamic Member Lookups" section of the proposal.

1 Like

This has been covered ad nauseam, and the proposal text itself answers this claim very well. But I'll repeat a small part of it here:

First, let's re-emphasize again that it is already impossible to look at an arbitrary Swift file and know whether dynamic member lookup (which already exists in Swift) is used or not.

Second, if the proposal were accepted, it's absolutely the case that a statement such as import Python would have to be written before the proposed feature could be used: no type in the standard library will conform to this protocol, so an import must be present in every file that uses these dynamic features (or else the user would have to implement the dynamic bridging themselves in the project--which would be the ultimate way of screaming "dynamic features here").

3 Likes

Second, if the proposal were accepted, it’s absolutely the case that a statement such as import Python would have to be written before the proposed feature could be used: no type in the standard library will conform to this protocol, so an import must be present in every file that uses these dynamic features (or else the user would have to implement the dynamic bridging themselves in the project–which would be the ultimate way of screaming “dynamic features here”).

Sticking with examples discussed in the proposal, import Python much more readily implies dynamic lookup than import JSON. There are already hundreds of JSON libraries in Swift and none of them use dynamic lookup unless they are abusing AnyObject dispatch. Further, I definitely do not want to have to assume dynamic lookup may be happening anywhere I see import JSON. Additionally, there will be other libraries using the feature which are even less obviously related to dynamic lookup.

Of course there is absolutely nothing in the proposal that requires an import statement at all. If the proposal is accepted as-is there may be types within any module that provide dynamic lookup. The fact that the standard library does not provide any conforming types is irrelevant.

You may not walk into large Swift code bases and need to get up to speed quickly but there are many people who do. Introducing this feature without any kind of usage-site annotation will make that more difficult, at least for code bases in which it becomes widely used. You may think the benefits of the proposal outweigh the costs, but there are costs.

1 Like

Why is the use or non-use of non-AnyObject dynamic lookup the one characteristic of a library's implementation that you simply must know on inspection, and not anything else about the library? If you're working on the project, don't you have to know so much more about the library even to understand what you're reading?

It's very relevant. If there are no conforming types in the standard library, then an import statement must be required or the conforming type must be part of the same module. An import statement at the top of a file provides as much information as any proposed syntax to surround the entire file with some dynamicFeaturesHere { /* whole file */ } catch { print("Oops") } annotation.

3 Likes

Why is the use or non-use of non-AnyObject dynamic lookup the one characteristic of a library’s implementation that you simply must know on inspection, and not anything else about the library? If you’re working on the project, don’t you have to know so much more about the library even to understand what you’re reading?

What I am interested in knowing upon inspection is whether a member name is actually a string literal or whether the compiler has verified that a member actually exists for the name in use.

An import statement at the top of a file provides as much information as any proposed syntax to surround the entire file with some dynamicFeaturesHere { /* whole file */ } catch { print("Oops") } annotation.

No it does not. An import statement does not tell me whether a library implements dynamic lookup. If the library does implement dynamic lookup for on of its types, the import statement does not tell me whether code in the present file actually uses that type. Even further, if the code in a file works with a type that implements dynamic lookup the import statement does not tell me whether the code in question intends to use dynamic lookup or in fact is intended to only use members that are statically visible and could be verified by the compiler.

1 Like

What is your evaluation of the proposal?

The proposal is well motivated and addresses an important problem but I remain unconvinced that the details of the design are the best solution for Swift. I would like to see further revision before the proposal is accepted. More on that below.

Is the problem being addressed significant enough to warrant a change to Swift?

The problem addressed is indeed significant. The primary use case presented in the proposal is well motivated and should be addressed eventually.

However, I also believe that getting the design right is far more important than introducing a solution in the Swift 5 timeframe. This is a feature that has a significant impact on the surface of the language and we now have a relatively strict source compatibiliy requirement for future changes. We need to get this feature right the first time.

Does this proposal fit well with the feel and direction of Swift?

Not in its current form.

The primary aspect of the design that I find unsatisfactory is that I strongly believe that in a language such as Swift member lookup is an operation that should not ever fail at runtime without some form of annotation at the call site.

By "disguising" a string literal as an ordinary member name (that is usually statically checked) it hides potential mistakes that would otherwise be visible at the usage site. The proposal provides sufficient motivation for introducing the sugar but unfortunately does not provide any alternative mechanism for making this potential for mistake visible to readers of code that utilizes the sugar. It asks us to sacrifice a "feature" many Swift programmers (and users of strongly typed langauges in general) enjoy: if a piece of code compiles there is no chance that you mistyped or renamed a member such that lookup will fail at runtime.

Swift does include dynamic lookup in the limited context of AnyObject dispatch. Despite having used the language full time for more than 3 years (and since the initial beta release more casually) I have not found a case where I thought using this feature would make a piece of code better. In fact, I believe the teams I have worked with would not approve a pull request that used this feature without an extremely strong and well-motivated argument for making an exception.

This is simply not a feature that the Swift programmers I know and work with think about very often and it is even mroe rarely used. It is a relatively obscure feature whose use is generally discouraged. IMO this is a reasonable argument for making AnyObject dispatch also be more visible at usage sites. The primary argument (again, IMO) against doing so is source compatibility. With this in mind, I do not believe the lack of a usage-site annotation for AnyObject dispatch is a good precedent to rely on when designing a much more general dynamic lookup feature.

One of the examples included in the proposal actually illustrates my concern very well. It includes a JSON type that conforms to DynamicMemberLookupProtocol with a code sample of json[0]?.name?.first?.stringValue using using the dynamic loookup capability of the type. This example uses string literals (disguised as members) to extract data from the JSON.

In my experience, using literals rather than constants in a context like this is widely viewed as an anti-pattern, particularly when the same literal is repeated multiple times. The potential for mistake is not theoretical: people do get bitten by this problem in shipping code. Further, it provides a good example of the kind of library people will write and use in pure Swift if the proposal is accepted.

Setting aside all other concerns regarding this kind of API for working with JSON, if we rewrite the example assuming constants were defined for the dictionary keys the example becomes json[0]?[name]?[first]?.stringValue. When we do that the sugar in the current proposal trades two braces for one dot while disguising a string literal as a member name.

Does the Swift evolution community really believe that this it is actually a good tradeoff to make? Do we want to allow libraries like this to be written and used without at least providing some hint to the reader that member names within a scope may actually be disguised string literals?

I believe a usage-site annotation appropriately provides a subtle indication that there are usually better ways to work with JSON and also informs a reader that the "member names" are actually string literals that are used to dynamically look up a value. This kind of subtle guidance and clarity is one of the things I love most about Swift.

While usage-site annotation provides important clarity in code that uses dynamic lookup the most significant benefit is in Swift code that does not use the feature at all. Readers of this code will be able to continue to rely on the fact that the compiler has verified the member names present in the code. This will not be possible (with the same degree of certainty that it is today) if this proposal is accepted as-is.

The proposal does contain a section called "Increasing Visibility of Dynamic Member Lookups". This section presents a good argument against usage-site annotation modelled after optional chaining. However, it does not mention other alternatives which have been discussed.

One alternative is an expression-level modifier mmodelled after try. Another alternative is to introduce a scope where the dynamic lookup sugar is enabled. There may be others. The important point that should be carefully considered is that it is possible to have usage-site annotation that does not impose a significant burden on users.

In general, I don't believe the design space for usage-site annotation has been sufficiently explored. It is clear that there are options available that are significantly less burdensome than the alternative included in the proposal while still communicating clearly to a reader that a member name might be incorrect within a specific context. I strongly believe that this design space should be carefully considered before a dynamic lookup feature is introduced into Swift.

If this proposal is revised to include a usage-site annotation feature I would enthusiastically support a proposal to adopt the same annotation for AnyObject dispatch if the potential impact on existing code was deemed acceptable.

Moving on, one other aspect of the design about which I am uncertain is addressed in the section "Make this be a attribute on a type, instead of a protocol conformance". I don't have a strong opinion here but do beleive we should very carefully evaluate the alternative before introducing this feature. We need to get it right.

In particular, I find several points in the rebuttal to the attribute alternative to be unconvincing.

Protocols describe semantics of a conforming type, and this proposal provides key behavior to the type that conforms to it.

This does not imply that attributes never describe semantics. @discardableResult, @escaping, @frozen (if accepted), and many other attributes describe semantics. Protocols don't have an exclusive claim for describing semantics.

When a type uses this proposal, it provides a fundamental change to the type's behavior. While it isn't perfectly followed, attributes generally do not have this sort of effect on a type.

IMO this is actually an argument in favor of an attribute. This kind of change seems much better aligned with the kind of semantics described by @frozen than it does with most protocols.

Attributes are syntactically very light-weight, which makes this easier to overlook - given the significant effect on a type, we prefer it to be more visible.

I find the concern with syntactic weigth at the type-declaration site and lack of concern with the absence of any indication of the presence of this feature at usage sites to be very puzzling. This is exactly backwards in my opinion. Whether an attribute or protocol is used in the type declaration is immaterial for the purposes of clarity.

If someone is looking for this information they will find it. If they are glancing at the declaration I think they are about equally as likely to notice either form of declaration. However, at the usage site where this feature will lead to mistakes (as is inevitable with dynamic lookup), people are extremely likely to miss the presence of this feature if no annotation at all is present.

Finally, I would like to briefly discuss the following:

The result type may also be any type the implementation desires, including an Optional, ImplicitlyUnwrappedOptional or some other type, which allows the implementation to reflect dynamic failures in a way the user can be expected to process (e.g., see the JSON example below).

This section of the proposal does not say anything about allowing an implementation to choose to use Swift's error system to report dynamic failures. This seems like an important omission. If the proposal does not support that it should explain why. If it does that should be documented. I appologize if this topic was addressed in earlier discussions - I don't remember whether it was or not and don't have time to read the archives.

In any case, I believe the proposal should be updated to includ this information. FWIW, my personal opinion is that a dynamic member lookup should be well integrated with Swift's error model, at least as an option for the author of a type supporting dynamic lookup.

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

I have used dynamic languages such as Ruby and Javascript a lot on the past. I am very familiar with both the capabilities people enjoy when using dynamism including a wide range of design techniques enabled by its use (Rails-like DSLs, etc). I am also very familiar with the disadvantages of dynamism, including member lookup bugs introduced by typos and refactoring / renaming.

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

In-depth consideration and participation in discussions on the list.

10 Likes

Hi Jordan.

I am convinced as to your points about heavier syntax not really buying us anything, for two reasons:

Stroustrup’s observation that goes something like “people want new features to be explicitly called out and old features to be syntactically lightweight, but then new features become old features”. (I can’t find a citation for this but I remember the original context being C++'s template.)

I can of course only speak for myself, but my concern is very specifically not that this is a new feature but that it is a feature I have used heavily in the past in other languages. It is from a place of intimate familiarity with the feature that I am urging caution. I want to be able to trust member lookup in Swift code without having to first figure out whether the code in question is using dynamic member lookup.

We used a very similar argument to avoid putting new or alloc in front of instantiation in Swift 1. (That is, new NSView(frame: …), to make it clear that an allocation was happening.) The specific point is that any method can do allocations, or do large amounts of work, and you should be able to understand that from context anyway, so the same should be true of regular instantiation.

I didn’t know this was debated and am glad you decided to leave it out. I don’t see new adding any useful information here. As you observe, it is far from sufficient to make allocations visible. That would require a much more heavyweight feature involving an effect in the type system.

On the other hand, I think it is very useful to know that the compiler is able to check member lookup. If the language includes dynamic lookup and that is not visible in any way at the usage site we will no longer know immediately whether or not the compiler is able to verify that the members used in a piece of code are valid or not unless we know the declaration of all types involved. This is a significant loss IMO, especially for people who are new to a large code base.

In practice most people will probably continue to assume members are statically verified unless they have reason to believe dynamic member lookup is in use. This will probably even work out ok most of the time. But IMO that doesn’t mean it is the best design we can come up with.

4 Likes

How would this work for Python?

struct PyKey: LosslessStringConvertible {
    var description: String { get }
    init(_ description: String)
}

struct PyVal {
    subscript(key: PyKey) -> PyVal { get set }
    subscript(val: PyVal) -> PyVal { get set }
}

You'd need a tool to generate PyKey extensions for each imported library:

extension PyKey {
    static let `arange`   = PyKey("arange")
    static let `array`    = PyKey("array")
    static let `exp`      = PyKey("exp")
    static let `int32`    = PyKey("int32")
    static let `linspace` = PyKey("linspace")
    static let `ones`     = PyKey("ones")
    static let `reshape`  = PyKey("reshape")
}

Then the example from Increasing Visibility of Dynamic Member Lookups could be written as:

let np = Python.import("numpy")
let x = np[.array]([6, 7, 8])
let y = np[.arange](24)[.reshape](2, 3, 4)

let a = np[.ones](3, dtype: np[.int32])
let b = np[.linspace](0, pi, 3)
let c = a+b
let d = np[.exp](c)
2 Likes

Thanks for demonstrating how this might work with the motivating Python use case @benrimmington. I should have thought to do that myself!

I would like to consider the example Ben gives above relative to the current proposal. We can see that this code pays a syntactic penalty of the subscript braces surrounding every member name. In exchange for a heavier syntax you get help from the compiler in verifying that you didn’t accidentally make a typo. Even better, if the library changes and the member name constants are updated the compiler will produce an error at broken call sites that must be addressed. As a bonus, you get code completion.

It is still possible to use a member on an object of the wrong type, but the scope of potential errors is significantly narrower. It is also clear to all readers that member lookup is happening dynamically and may therefore fail.

Of course there is also a tradeoff in that the member name constants must be declared. The good news is that they only need be declared once. There is a lot of potential to automate this and the declarations necessary for a Python library can be published as a Swift module and shared by the Swift community. They can also be extended by anyone at any point in their code where an extension declaration is valid.

These are tradeoffs that I can say with confidence the Swift programmers I work with would be happy to make. Even with dynamic member lookup available I believe many programmers would prefer the alternative where the compiler provides more assistance. It is interesting that this approach which offers nontrivial advantages is already viable in Swift today with no change to the language necessary.

I think the tradeoff of dynamic member lookup is reasonable in some settings. That is why I do not oppose the proposal altogether. But I do think this tradeoff should be a choice that individual bits of Swift code opt-in to with a usage-site annotation.

We certainly should not reduce the static guarantees provided by the language for the sake of supporting dynamic lookup with no syntactic footprint at all at the usage site. Symbols are symbols. String literals are string literals.

We should not have to think about whether something that looks like a symbol actually is a symbol when reading Swift code. At least not unless there is an indication in the surrounding code that some symbols are actually sugared string literals.

We already have the dynamic keyword. It would be acceptable to me to just reuse that and allow it to introduce a scope in which this kind of syntactic sugar is available. Is it really that bad to have to place dynamic {} around code that uses dynamic member lookup (or other similar features introduced in the future)? This would be an indication that the code contained within that scope leaves behind some of the guarantees that are present for most Swift code.

Such a distinction allows us to have confidence in relying on those guarantees when reading code not enclosed in a dynamic {} context and also calls our attention to the need to watch out for the issues that can arise in dynamic code when we are reading code that is in such a context. IMO this is an extremely useful distinction to make if a single language is going to support both the static and dynamic approaches.

2 Likes

Swift is great, because it forces people to think about errors in the "appropriate" conditions according to an opinionated set of guidelines. After all, it would have been possible for every integer addition to produce an optional result, but the outcome of that would not have led to more reliable code - that is why integer arithmetic defaults to trapping on overflow.

If you take a look at the Python interoperability prototype I have posted on the mailing list several times, you can see that there is a well developed and documented way of handling Python errors in an approach that dovetails well with Swift error handling. That approach composes on top of this
proposal perfectly. Likewise, as the strawman JSON example shows, it is perfectly reasonable for a specific API to force an optional result when it is the right thing to do.

This proposal is an underlying mechanism necessary to enable all of these policy decisions -- the proposal is not about a specific Python interoperability design. The Python interop approach in my prototype is carefully considered and I believe it to be the right way to properly integrate with Swift (a language I know well) but if it is not, that will be sorted out in the pitch, proposal, and review phases of the actual Python interop proposal. None of your concerns about Python interop appear to reflect on design of this proposal itself.

-Chris

3 Likes

This proposal does not include or imply any changes to a specific JSON library, and I believe it makes this clear (let me know if you were confused and thought the proposal was about extending some specific JSON library, and I'll fix the proposal).

The JSON example is just meant to show how something could use the proposal with an optional result. Perhaps I'm mistaken, but I think that you' are reading too much into it.

-Chris

2 Likes

Hi Matthew,

FWIW, the proposal completely supports the concept of a throwing member. The problem is that Swift has a current limitation (aka, bug in my opinion) wherein lvalues like properties and subscripts are not allowed to throw ([SR-238] Support throwing subscripts · Issue #42860 · apple/swift · GitHub). When this gets fixed, DynamicMemberLookups will be allowed to throw and will directly as a result of the conforming subscript throwing. This limitation has nothing to do with the proposal, it has to do with preexisting limitations of Swift - yes, I know this is frustrating.

-Chris

6 Likes