SE-0195 — Introduce User-defined "Dynamic Member Lookup" Types

Evolution Proposal Reviews
102

+1

What is your evaluation of the proposal?

I am very much in favor of this proposal.

Is the problem being addressed significant enough to warrant a change to Swift?

Yes.

My experience is that this type of feature would allow Swift programmers to develop solutions which in more traditional static languages (C/C++) would be accomplished via code generators. However, code generators are not a very approachable solution for many programmers and users, their implementations often are significantly immature relative to the languages they target (for example, compare protobuf to any of the languages it targets, or LLVM's tblgen to C++). While code generators do have other advantages (e.g. performance of the generated code), enabling this category of problem to be solved completely in the language has an incredibly high value.

In my opinion, this type of capability (in conjunction with the DynamicCallable) will greatly enhance the ability for Swift library authors to compose rich frameworks for dynamic problems, including ones which go above and beyond the original aspirations of the proposal.

For example, a Swift web framework could use this mechanism to develop a convention "syntax" for defining routes on a website. Similarly, a database mapping framework might use this capability to provide a convenient, type-safe way of working with SQL statements. I believe there are many such examples waiting for innovative developers to unlock, and that the dynamic nature does not compromise the safety of the language, but makes it easier for framework developers to create a very usable system which ultimately will improve the adoption of Swift.

Does this proposal fit well with the feel and direction of Swift?

Yes, the protocol-based extension makes clear sense to me as a user of Swift.

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

I have seen similar features used extensively in Python (which is already dynamic, but has similar dynamic capabilities on top of the general class lookup and dispatch mechanisms) to build clever frameworks. These have proved very popular. Some examples of Python libraries that leverage features like this are Pandas (a data processing library), Flask (a small web app framework), and SQLAlchemy (an object relational mapping layer).

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

Moderate (between a quick reading and an in-depth study). My feedback is primarily based on the need for this general feature in Swift, and less on the specific concrete proposal.

2 Likes
103

What is your evaluation of the proposal?

Enthusiastic +1. I like it as is.

I am opposed to all the strategies listed under “Reducing Potential Abuse” in the proposal.

In the future, I would very much like to see lookup keys other than String, as Brent suggested. However, the proposal is worthwhile even without that feature.

I would also like lookups to be able to throw, as Matthew wanted. However, because of the technical limitations Chris laid out, I would not want this proposal to be delayed for that.

It is a little troubling that the feature will not support keypaths. My gut says that will need attention soon. However, as Chris says, we should have a use case in hand.

Is the problem being addressed significant enough to warrant a change to Swift?

Yes, there are two independent problems each of which could justify this feature:

  1. Dynamic language interop opens many exciting doors for Swift — but only if it is syntactically ergonomic. It currently is not. No, not even string subscripts. (Try actually writing out some Python API calls with it; it's a hot information design mess. And yes, code syntax and formatting is a form of information design.) Statically generated API bindings might be feasible (and preferable) in some cases, but many dynamic APIs are not even knowable at compile time. Wrapper generation is at best a partial solution.
  2. Unstructured data access and config DSLs, while often best served by clever use of statically typed constructs (enums enums enums), can make good use of dynamism even in an otherwise statically typed language. In particular, I can imagine this being very nice for things like routing config which (1) is parsed and checked in its entirety at startup and (2) benefits from concise references by name to the things which it has configured. The cumbersome SwiftPM manifest format might even profit from this proposal.

Yes, I’m sure that some people (likely including me) will go too far experimenting with #2. And when those experiments turn out to be bad, the community will abandon them — if the original author doesn’t ditch them first. (Remember in Swift’s early days the torrent of goofy novelty operator libraries? That ran its course, the few that were actually useful survived, and the language landscape has only improved since then.)

Does this proposal fit well with the feel and direction of Swift?

This is the contentious one, isn’t it?

I’ve heard a number of concerns about this — some reflexive sneering at dynamic languages, some entirely valid. Some have made me think hard. None have swayed me. My thoughts on the greatest hits:

“It’s hard to spot dynamic calls in code!”

Two words: syntax highlighting.

Swift is already a language that relies heavily on either static analysis from tools or static investigation by the developer to be understood. Its unusually extensive type inference alone already makes it incredibly difficult to know without the help of a tool what the type of $1 is or what type .foo belongs too.

“I don’t want to use it, even by accident!”

Fair enough. Set a linter rule. Wish granted!

“People will write terrible code with it!”

This is entirely true. They will also write terrible code without it.

Most of the concerns I’ve heard expressed could be addressed by sensible developers and good tools. The latter are feasible; the former no language can enforce.

“This proposal undermines the compile-time safety we’ve come to expect from Swift!”

This is the one objection that carries real weight for me. I was concerned about it at first too.

The thing that swayed me was Chris giving examples of other things that are not checked at compile time, yet have no syntactic “danger” marker at the usage site:

  • array subscripting,
  • integer overflow, and
  • invoking a method that can cause a fatalError or precondition failure, which can of course be any method.

AnyObject dispatch has come up a lot in this discussion, but it’s a bit of a red herring: obscure, rarely used, a necessary hack. The bullet points above are core language features that we use all the time, and none of them involve any of the syntactic markers or other usage obstacles people have proposed in this thread. Not one.

On objective analysis, Swift is full of runtime dangers. Yet I’ve always felt like it is an especially safe language, in a way that is closer to ML than C. Why? I’ll dredge up what I posted back in early December; it still stands now:

Sentiments like this subjectively fit the experience of using Swift. When using the language in practice, it’s fairly clear to a mindful developer which operations carry some inherent level of danger, what kind of danger that is, how to reason about it, and how to mitigate it. (Usually, anyway.)

Having all had this pleasant experience, I think we may be too quick to look for the explanation in the design of the language itself. We fail to notice that it comes not from the language alone, but from the whole ecosystem that surrounds it: standard library design, naming conventions, tools, libraries, culture, Erica’s books, etc.

Optionals present themselves as a specially privileged part of the language; withUnsafeBytes presents itself as a library call using existing language features. But both share a similar definitively Swift-ish aesthetic in how they guide our attention, and how they circumscribe programmer error while leaving the language ergonomic and situationally adaptable.

[Chris’s] reminder is a good one: Swift already does not have, and never has had, a sense of safety that depends on Haskell-like strictness in the language itself. The design question at hand in this thread thus becomes not “how can we prevent dynamic dispatch from ruining everything” but rather “how can we introduce this new element in a way that does not disrupt the ecosystem.” In considering that, we’d do well to remember the long-established language features that have so far failed to ruin everything despite their “impurity.”

Underlying many objections is a fear that creeping dynamism will ruin the whole neighborhood and destroy all we hold dear. I don’t think that will happen. I see Chris’s examples of other unmarked dangerous behavior, and the line of thought above, as an optimistic answer to that fear: Swift’s Swiftiness has always depended on good judgement. It has always depended on an ecosystem that extends beyond the language features, and in some particular cases even pushes toward mindfulness and safety in spite of language features. A language feature alone will not erase the good sense of the language’s stewards, or poison its community.

I think there is a place for dynamic dispatch in Swift. There are a few problems where it is the right answer. And I’m sure that the ecosystem can integrate it in a way that does not fundamentally alter what we like about Swift. This is a language that’s managed to integrate raw memory access, for heaven’s sake, and still feel “closer to ML than C!” If it can do that, it can certainly withstand the far smaller disturbance in the force of what Chris is proposing.

How much effort did you put into your review?

More than was wise.

4 Likes
104 
What is your evaluation of the proposal?

A strong +1. As a Ruby user, this will make the bridging much better.

Is the problem being addressed significant enough to warrant a change to Swift?

Yes. This not only make using Ruby in Swift easier, but also make using Swift in Ruby easier.
There's a C extension system in Ruby, just like Python's, which allow us to write critical part in C for speed boost. But the extension APIs are in C style, which scare many people without experience away. With this change, we can make the extension APIs more Swifty.

Does this proposal fit well with the feel and direction of Swift?

Yes. It'll made Swift a more generalized language.

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

Ruby has method_missing which allow dynamic behavior on any object.
DynamicMemberLookupProtocol restrict the behavior to the conformed type, which will reduce the chance of footgun.
Although the subscript requirement is not explicit, but It still got compile time error when omitted, which is great.

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

Start following this since the initial pitch stage.

1 Like
105

I am for the proposal but find that it doesn’t fit well with Swift as it stands, in particular an expression that can throw must be marked with try but a dynamic expression isn’t marked. I would find it more consistent to mark expressions that contain a dynmamic element be likewise marked with dynamic. Perhaps also dynamic { ... } for a block of dynamic expressions.

My reasoning is that a throw is already contained in a do block and therefore the try isn’t necessary, but deemed worthwhile. I find a mismatch between try worthwhile but dynamic not.

106

Per Chris’s comment above, when subscripts can throw (which is a current limitation of Swift not related to this proposal), then dynamic member accesses will also be able to throw.

For a dynamic API where “no such method” errors cause a potentially recoverable error, then the existing try { … } syntax would be the correct way to contain and handle those errors. No new language construct needed.

For APIs where dynamic calls always succeed — and such APIs will exist — requiring a wrapper block is inappropriate. The language should be able to differentiate this situation. Declaring the dynamic accessor throws is the Swifty way to make that differentiation.

For dynamic APIs where “no such method” errors do not present any kind of reasonable recovery action for callers, e.g. malformed routing config that is a fatalError, then an additional dynamic { } wrapper servers no functional purpose. It is just annoying “Simon says” language design — and Swift does not require any such Simon says construct for method calls that may result in a fatalError, or potentially out of bounds array access, or integer operations that might overflow.

If the concern is not allowing error recovery but merely drawing attention to the fact that dynamic access is present, then please see my comments above about syntax highlighting.

I don’t see that a new dynamic { } construct adds any value in any of these cases.

107

@Paul_Cantrell distinguishing dynamic member lookup from statically verified member names seems like a pretty ambitious task for a linter or a syntax highlighter to me. Is it common for linters and syntax highlighters to have this level of semantic knowledge about a language?

I have also been thinking more about the implementation of types that conform to DynamicMemberLookupProtocol. I find it hard to believe that authors of these types would want dynamic lookup to be in effect within the implementation of the type. Imagine the following:

class MyDynamicType: DynamicMemberLookupProtocol {
    subscript (dynamicMember name: String) -> MyDynamicType {
        // oops!  typo!  infinite loop!
        let member = lookupMmeber() 
        return member.invoke()
    }
    
    private lookupMember() -> Member { ... }
}

Of course the above example is a bit of a straw man but it serves to demonstrate the concern. In practice mistakes would usually be more subtle and perhaps less likely to be detected quickly.

It is possible to avoid this issue by moving the implementation to a private type that does not support dynamic lookup and forwarding to that. But do we really want implementations to have to do that?

If a body of code is required to opt-in to dynamic member lookup this is a non-issue. The implementation of the type would simply not opt-in.

1 Like
108

SourceKit can do this; it builds an AST with static type info. I expect Xcode and AppCode could thus do the syntax highlighting.

SwiftLint uses SourceKitten, a wrapper around SourceKit. I'm not sure whether SourceKitten currently exposes enough information in its output for a linter to implement this feature. However, it does expose type-dependent autocomplete suggestions (the same ones as Xcode), so it clearly has the necessary info internally — and I’m sure JP would be game for adding it if there were demand.

In short, yes, I think this is entirely within the realm of possibility.

2 Likes
109

Hi Howard,

This is discussed in the proposal. If you have specific other suggestions, I'm happy to incorporate and discuss them.

-Chris

110

As an observer, it's getting quite frustrating to see @Chris_Lattner3 tell people to read the proposal. I'd guess he's tired of asking people to read it, too.

If you've got a question or concern, please first try to find it addressed in the proposal before posting.

1 Like
111

I'd have to disagree with you there; not everyone can benefit from SourceKitService to perform analysis of their code. The examples you've provided are things I'd consider to be IDEs, but there are many people who don't have these: many people use text editors, which are generally limited to syntax highlighting based on keywords or regular expressions. You're totally ignoring these people here.

112

I think it’s Swift as a whole that is ignoring them — or more accurately not prioritizing them. Please read the comments in my review about the numerous other features already existing in the language that assume that developers either have ready static analysis at their fingertips, or are going to do a lot of manual digging. This is a defining part of Swift’s aesthetic.

113

Your examples above don’t suggest that Swift favorites IDE’s with its syntax. For the .someCase example, you’d still have to look up the possible enum cases in a language like c. And $0 is essentially a function argument which you would also need to lookup if you didn’t know what it expected.

IMHO I don’t think something involving an IDE or linter should be a valid solution for a Swift Language issue.

That said my opinion for this proposal is that it doesn’t necessarily need syntax highlighting, but would actually be very much appreciated ;-)

114

C enum values are globally scoped. If, say, we had enum KeyCode and enum Compass, giving both of them a value named up would create a name conflict. C developers therefore typically give enum values names like keycode_up and compass_up.

In Java and similar languages, enum values are type-scoped. You can therefore reference KeyCode.UP and Compass.UP — and that type qualification is required for references from outside the type.

In Swift however, you can just type .up and the language will figure out whether you mean KeyCode or Compass from the compile-time type of the context. You therefore need to know the types in the AST to know what .up means — and those types can be inferred through many levels of indirection. In fact, it’s sometimes not even possible to know a particular type without doing the simultaneous resolution of a bunch of, say, chained closures.

The language doesn’t help much with nearby explicit declarations; it rather favors make good, clear code less verbose, and leaves you to seek help from an IDE when the context doesn’t make inferred types obvious.

…and the type of that function argument again depends on the results of type inference, which is right at hand in an IDE and darned hard to find in a bare text editor.

This is a language that already favors those with static analysis at hand. And that is the right decision in the year 2018.

115

I still have to know something starts with keycode or compass, or know a property accepts KeyCode or Compass types. Swift is just better at helping you get what you need with less digging. This is though pure convenience and doesn’t harm the language if you don’t have it. However with regards to this proposal someone suggested dynamic lookup could be harmful and your recommendation was let an IDE handle it. Now whether or not I think it is I don’t think we should be using solutions that don’t involve the actual language design.

I would actually really like to know how much IDE integration a feature can rely on, if anyone could shed some light?

116

In the following code:

let items = (1...100)
    .map(foo)
    .map(bar)
    .filter { $0.baz == .up }

…what type does .up come from? To know, I need to know the type of baz, which means knowing the type of bar, which means knowing the type of foo. But I can’t just start at the top and follow the chain down: if foo and bar are overloaded, then I need to find which unique simultaneous combination of the two returns some type that has a member baz that has a value .up.

There is no upper limit to this complexity, except for the compiler timing itself out trying to solve the types.

I can’t speak to whether the language’s designers specifically intended to rely on IDEs, but I can say the behavior of Swift’s type resolver is not something I expect to be able to reason about without the help of tools.

3 Likes
117

But it's still possible to do the lookup by yourself. If you're chaining then you should have some idea of what you're expecting to happen (i.e. know the intermediate types you want). You're not just typing in closures and hoping for the best.

118

The (important IMO) difference is that in the above example when the code compiles you know these symbols have been statically checked. You didn’t make a typo and the code wasn’t broken by a renaming or other refactoring. That will not be true of dynamic lookup.

Type inference is an incredibly important technology. I don’t mind relying on tools to learn more about the types involved. But importantly, I can also choose to add explicit type information and recompile. I do not have to rely on an additional tool. Further, the presence of type inference is obvious in the source by the omission of explicit type information.

None of this is true of dynamic lookup as it has been proposed. The feature as it is proposed hides the presence of dynamic lookup in source and offers no assistance in determining even a broad scope where it is used. My personal opinion is that this goes too far for a feature which reduces the static guarantees provided by the language.

We should not have to rely on external tools to discover code that may be silently broken due to typo or refactoring. I also think we should be able to use dynamic language interop without this syntactic sugar and without having to write our own wrapper that passes along a member name to types like PyVal and wraps the result if we want to choose static guarantees over syntactic sugar. This sugar should be opt-in.

2 Likes
119

This feature replaces the use of string subscripts, which have this same problem.

The problem with automatic wrapper generation, which has been stated before, is that some dynamic languages don't declare all their properties upfront. How would you work around this issue?

That said, I think there is an argument to be made for generating a static interface from what information is available (my uninformed impression is that type providers allow this), and providing a subscript fallback.

In any case, this proposal seems to be designed to make writing wrapper classes easier. If you don't use wrapper classes, you'll get no static guarantees with or without this proposal.

120

Likewise it can be quite frustrating to be pointed at the proposal as soon as a concern or an idea for mitigation of such concerns comes up when the proposal does not address them.

In this example the idea of making dynamic call sites more visible is only discussed in general in the proposal. The specific suggestion made by @hlovatt and others, to mark expressions with dynamic or dyn, which is relatively lightweight, is not discussed at all in the proposal.

Instead the proposal chooses an unnecessary heavyweight method of marking the use sites as an example which is misleading IMHO.
Let me explain why I think that the method chosen as an example in the proposal is unnecessarily heavyweight:

  • the chosen ^ sigil for the example is visually intrusive, especially when combined with . (because ^ is a sigil in the upper region of a glyph whereas . is in the lower region, so they don't combine well visually)
  • the sigil is not only applied to the name lookup but in addition to the call itself which is unnecessary for the concern raised most often: that the dynamic name lookup might fail due to a typo or broken refactoring. It is not a concern that the call itself might fail (I expect here that the dynamic name lookup for a method respects the parameter list for the method, i.e. looking up foo(1, 2) would fail if only a method foo(PyVal x) would exist, but not a method foo(PyVal x, PyVal y)).

Choosing another sigil for dynamic name lookup, i.e. -> and not applying it to the call itself would then result in the following example which is much more readable. For operators let's choose a prefix ' (all sigil suggestions are only straw men, of course):

let np = Python.import("numpy")
let x = np->array([6, 7, 8])
let y = np->arange(24)->reshape(2, 3, 4)

let a = np->ones(3, dtype: np->int32)
let b = np->linspace(0, pi, 3)
let c = a '+ b
let d = np->exp(c)
print(d)

The suggestion of using something similar to try would look like follows (and is missing from the discussion in the proposal although it has been mentioned several times):

let np = Python.import("numpy")
let x = dyn np.array([6, 7, 8])
let y = dyn np.arange(24).reshape(2, 3, 4)

let a = dyn np.ones(3, dtype: np.int32)
let b = dyn np.linspace(0, pi, 3)
let c = dyn a+b
let d = dyn np.exp(c)
print(d)

The advantage of this approach is that there is no separate sigil necessary for operators. The proposal states

Requiring additional syntax for a.b but not a + b (which can be just as dynamic) would be inconsistent.

This clearly ignores the fact that marking expressions with dyn (similar to try) would solve this inconsistency.

Furthermore the whole discussion of whether or not to make call sites more visible is not representing the concerns raised on Swift Evolution well, so let me restate the concern:

  • Static type checking ensures that member access or method calls cannot fail due to the member or method not being present on the receiver, i.e. static type checking protects from typos and errors made when renaming things. For dynamic member lookup this is not true anymore and it is not visible where dynamic lookup is used.

IOUs are somewhat of an exception to this rule although even there the member accesses or method calls are checked statically and only fail when the IOU is nil (and in that case all would fail).

The discussion in the proposal rightly notes that

Swift's type system already includes features (optionals, IUOs, runtime failure) for handling failability.

and

Adding punctuation to the lookup itself reduces the likelihood that an API author would make the lookups return strong optionals, because of increased syntactic noise.

This might be true but

  1. something like dyn would not visually conflict with the ? introduced by strong optionals.
  2. I do not consider it good design to conflate the failure of member lookup (remember: due to a typo, i.e. a programming error) with a return value being optional because of application logic, i.e. the failures have a different nature.

A similar point raised against marking the use sites of dynamic member lookup is that Swift already contains features that might fail at runtime like array subscripts, integer overflow or IOUs.

IMHO this, too, ignores the fact that these runtime failures have a different nature than failures when looking up a member: the existing features do never fail because of a misspelled name. Of course their failures are programming errors, too, and they might even be the result of typos (e.g. using + instead of -) but inherently they are errors in the program logic (which are generally difficult to detect automatically) whereas lookup failures due to misspelled names can be found quite easily automatically by static type checking.

This failure due to typos is why I do not agree with the claim of type safety made by the proposal. IMHO it is only dynamically type safe but not statically type safe.

3 Likes
121

This feature replaces the use of string subscripts, which have this same problem.

The potential issue is quite clear when a string literal is used as a subscript.

If you don’t use wrapper classes, you’ll get no static guarantees with or without this proposal.

Of course you will never get the same kind of guarantees that you do with static dispatch. For example, the underlying Python type could change a member name.

However, if you declare constants for the members in use you can get a guarantee that all uses of that member are using the same string value. In other words, you avoid duplicating string literals throughout your code. There is value in doing this whether you are writing a wrapper or not. For example, if the underlying Python type did change a member name you only need to update a single constant to bring your Swift code back in sync with Python.

1 Like