SE-0216: User-defined dynamically callable types

There’s a point where I talk about having a subscript(dynamicMember:argumentLabels:) which returns a closure and a subscript(dynamicMember:) which returns an instance. That’s how it supports Ruby. If you try out manually-desugared versions in a playground, the type checker has no trouble selecting the appropriate overload.

Brent, can you explain why your counter-proposal uses a subscript? It seems to achieve nothing except creating questions about what it means to try to set it.

It uses a subscript because most languages (including Swift!) put properties and methods in the same namespace, and so the implementations of property and method lookup are often identical. Python is one example of such a language.

(The languages I can think of which don't do this still only expose the method namespace to the outside world for most uses. Ruby falls into this category, but bridging it idiomatically requires that we pretend otherwise.)

Rather than thinking of the subscript as a property lookup or method lookup, think of it as a member lookup. There is one subscript handling both properties and methods because that subscript handles members in general. That subscript receives argument labels because in Swift, argument labels are part of a function's name.

1 Like

Given these two desugarings from your gist:

myLangObj.someProp
  // => myLangObj[dynamicMember: "someProp"]
myLangObj.nullaryMethod()
  // => myLangObj[dynamicMember: "nullaryMethod"].__call()

…how does myLangObj[dynamicMember: "foo"] know whether it's being invoked as myLangObj.foo or myLangObj.foo()? For Ruby support to work, both the former and the latter need to return the same value. In other words, the [dynamicMember:] subscript implementation needs to know whether its return value is about to be called.

Many languages semantically distinguish property and method lookup:

  • A JavaScript method call is not semantically equivalent to a property lookup followed by a call: the latter will use null as the this argument to the method.
  • A Ruby method call is not semantically equivalent to a property lookup followed by a call: a property lookup will implicitly call a method if it finds one, so the latter will attempt to call the return value of that.
  • Java allows properties and methods to be overloaded, and it distinguishes whether you meant the property or the method based on use.
  • Some languages allow functions and methods to be overloaded by type. The types of arguments are an aspect of the lookup; the lookup may be ambiguous on its own.
  • Some languages that don't allow type-based overloading do allow overloading by arity. The argument count is an aspect of the lookup; the lookup may be ambiguous on its own.
  • Many other OO languages do not allow a method to be looked up without calling it. There is no valid return value that represents a looked-up method.

In addition, combining property and method lookup into a single operation, or attempting to distinguish them via overloads, creates many new corner cases:

  • There is no natural language construct that corresponds to setting a method name with argument labels, but it becomes a case of the setter.
  • The call arguments can be constructed separately from the argument label list and may differ in length.
  • A lookup for a method call with no formal arguments must be distinguished from a property lookup for a variety of reasons.

Overall, the approach seems like it would greatly increase the actual complexity of working with these language features in Swift.

3 Likes

If you look at the Ruby section instead of the Python section, you'll notice that the subscript there has two overloads:

@dynamicMemberLookup
struct RubyValue {
  fileprivate let value: Ruby.VALUE
  fileprivate func sendMessage(_ method: String, with args: [RubyValue]) -> RubyValue { ... }
  
  // Ruby only exposes methods on its objects; properties are accessed
  // through accessor methods. Here's the subscript for method lookups:
  subscript(dynamicMember name: String, argumentLabels labels: String...) -> (RubyValue...) -> RubyValue {
    get {
      return { (args: RubyValue...) in
        sendMessage(name, with: extractOptionHash(labels, args))
      }
    }
  }
  
  // When we directly access a name with no call parentheses or context 
  // suggesting a function type, we want to immeidately call the accessor,
  // emulating a Swift property.
  subscript(dynamicMember name: String) -> RubyValue {
    get {
      return sendMessage(name, with: [])
    }
    set {
      // The concatenation here forms a Ruby setter name.
      sendMessage(name + "=", with: [newValue])
    }
  }
}

When Swift sees myLangObj.nullaryMethod() and tries to type-check it, it first tries to make the subscript(dynamicMember:) overload work, but that fails because RubyValue can't be called. (It doesn't have a __call method; you can't call a closure with just parens in Ruby, either, which is part of the duct tape which makes its syntax work.) It then tries to make the subscript(dynamicMember:argumentLabels:) overload work, which succeeds because it returns a function type which can be called with zero parameters.

When Swift sees myLangObj.someProp, on the other hand, there is nothing to stop it from using subscript(dynamicMember:), so it selects that overload.


These three can all be handled by having a value-returning subscript(dynamicMember:) and a function-returning subscript(dynamicMember:argumentLabels:). Just as it would for a native Swift type which had a method whose base name was also a property, it will disambiguate with context but default to assuming it's a property.

These two would have to be handled by returning some sort of value representing the set of possible callees—possibly a closure which encapsulates the lookup. Not ideal, admittedly, but very workable, and it corresponds to Swift's partially-completed call syntax.

This one actually isn't a problem—the count of the argument labels corresponds to the function's arity.

In the past, we've discussed permitting variables (particularly parameters) with names like foo(bar:) with an arity matching their function type; this would correspond to that. It's not something the language supports today, though, and we don't know if it's something we ever will, so this is a fair criticism.

That's true. In most cases, Sema would synthesize the lookup while the call was visible to it, but not always. (Most dynamic languages are tolerant of arity mistakes, though—they'll usually throw an exception or use default values—so this is a correctness problem but not a safety problem.)

I'll take your word for it, but I'm not really sure how much of a difference this makes for these dynamic features. Either way, Swift will have to guess whether it's a call or property access from extremely limited contextual type information.


I'm glad we've explored this design. I still prefer it, but you (collectively) are right that it's not as strong as I thought.

One last comment: if the core team does adopt SE-0216, I hope it at least considers removing dynamicCall(withArguments:) (and probably giving its name to the withKeywordArguments: form). The justifications given for including it don't seem very strong, and about 10% of the patch seems to be devoted to supporting it.

2 Likes

I agree Brent's proposal is workable in Ruby. Moving the type signature out of the compiler and decoupling it from the labels is interesting because it gets very close to transparently permitting the "and also pass a trailing closure [ruby block]" syntax I wrote about during the pitch thread.

So this compiles OK:

struct S {
    subscript(dynamicMember name: String, argumentLabels labels: String...) -> (Int..., (Int) -> String) -> String {
        get {
            return { (args: Int..., clo: (Int) -> String) in
                "a"
            }
        }
    }
}
let s = S()
s[dynamicMember: "m", argumentLabels: "O", "T"](1, 2) { anInt in
    "b"
}
// ruby: s.m(1, 2) { 'b' }

But providing a further overload to take just the closure [pleasantly surprised this worked at all]:

extension S {
    subscript(dynamicMember name: String, argumentLabels labels: String...) -> ((Int) -> String) -> String {
        get {
            return { (clo: (Int) -> String) in
                "c"
            }
        }
     }
}

...requires parens at the point of use to avoid treating the closure as a subscript arg:

s[dynamicMember: "m2"] ({ anInt in
    "d"
})
// ruby: s.m2 { 'd' }

A potential evolution of @dynamicMemberCallable wouldn't have this tiny problem but would need a chunk of proposal and compiler work.

(fwiw I have a Ruby client)

For the record: the motivation for separating the withArguments: and withKeywordArguments: methods is to disallow keyword-argument-calls for languages that don't support keywords. The example used in the proposal is JavaScript.
For languages without keyword arguments, defining a withKeywordArguments: method would be unnatural and a "no argument labels" precondition would be required at run time.

1 Like

I know, but you also note that JavaScript has a convention for mimicking keyword arguments with an object. I can’t think of a dynamic language which doesn’t have keyword arguments at least informally.

Agreed that it is better to have one common feature instead of multiple different ones if possible!

Thank you for writing up the gist, I really appreciate it, I now understand what you are thinking. I don't think that this achieves the goal you claim though, because with your proposal it seems like we end up with three things:

  1. @dynamicMemberLookup as it currently exists.
  2. Your new extension to DML that introduces a subscript for method calls.
  3. Your new __call thing for direct calls.

This is similar to the @dynamicMemberLookup + @dynamicCallable implemention along with dynamicCallable's future direction.

From my view, the fundamental difference seems to be that you think the __call work can be extended to an C++ operator() static dispatch feature, which the @dynamicMemberLookup + @dynamicCallable proposal doesn't attempt to cover. I consider this to be a separate feature (which may or may not ever happen) because it is inherently statically resolved, and therefore (arguably) very different in spirit from the @dynamicFoo proposals.

This is a pretty big problem, given it doesn't solve the Python use-case. Python callables allow keyword arguments. This is a really important point to our interop layer, not something that is "nice to have thing".

I personally consider this to be a non-starter.

-Chris

2 Likes

What is your evaluation of the proposal?

My evaluation of the proposal is very positive. I feel that implementing the @dynamicCallable attribute is the natuaral next step after implementing the @dynamicMemberLookup attribute in 0195 to provide support for dynamic languages.

I also like the proposed ambiguity solution for types that contain withArguments: and withKeywordArgument methods. This was a question that came up in my mind while reading through the proposed solution.

Is the problem being addressed significant enough to warrant a change to Swift?

I believe it is. Since the @dynamicMemberLookup attribute was added to Swift, accessing a property of a Python instance is relatively straight forward. However, calling a method on that instance is a lot more cumbersome. For example:

Python:

class compute:

    def __init__(self, text):
        self.data = text

    def get_data(self):
        return self.data

Swift:

let computeModule = Python.import("Compute.compute")

// The cumbersome call method to create the computedInstance
let computedInstance: PyVal = computeModule.get(member: "compute").call(args: "Swift Data")

// Accessing a property on that Python instance
print(computedInstance.data)

Does this proposal fit well with the feel and direction of Swift?

In my opinion, yes. I feel Swift is starting to gain ground on the server and also in popular machine learning libraries so providing support for Swift to interop with dynamic languages to me only makes sense.

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

N/A

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

I did an in-depth study of the proposal on Github as well as putting together live code samples to at least test out @dynamicMemberLookup. I am also a Python fan, so putting together the code samples was fun!

1 Like

I briefly thought about also using subscripts for the callable features. The show stopper for me was not being able to throw from a subscript.

2 Likes

This seems to be an excellent proposal that provides parity with SE-195 and improves cross-language compatibility, which should be (in my opinion) a longterm goal of Swift. +1

1 Like

Hi Ted. I can't seem to get the Xcode toolchain for this proposal working. I've selected the toolchain in Xcode (10 beta 2) but it's still building with Xcode's default toolchain.

I just observed the same behavior. Can you try switching back to the legacy build system to see if that causes the toolchain to get picked up?

File > Project Settings (and then select Legacy build system)

When I tried that it worked for me. Can you see if that workaround works for you?

@hartbit confirmed on Twitter that the workaround works. I have filed rdar://problem/41404260

I'll be taking last-minute feedback on this proposal up until the end of the day.

One concern about @dynamicCallable (and @dynamicMemberLookup) is that the attribute requirements are solely name-based.

To satisfy @dynamicCallable, a nominal type must have a method named dynamicallyCall(withArguments:) or dynamicallyCall(withKeywordArguments:), which is kind of inflexible.

Alternatively, the requirement-satisfying methods could be specifically marked:

@dynamicCallable
struct PythonObject {
  // Explicit marker. I can name the "dynamic call" method whatever I want now.
  @dynamicCallableMethod
  func call(withKeywordArguments: [String : PythonObject]) -> PythonObject { ... }

  // Minor win: I can now name a function `dynamicallyCall` that's irrelevant
  // to `@dynamicCallable`. Overload resolution of the dynamic call method isn't
  // name-based now.
  func dynamicallyCall(withArguments: [Int]) -> Int { ... }
}

That's a lot of attributes though. A crazy alternative: we could do away with @dynamicCallable and add a new method keyword like dynamicCall to be used instead of func (kudos to @rxwei for this idea, sorry if I misrepresented it):

struct PythonObject {
  // `dynamicCall` is a special method keyword like `init` or `subscript` that
  // denotes a "dynamic call method". 
  dynamicCall(args: [String : PythonObject]) -> PythonObject { ... }

  // Similarly, for dynamic member lookup:
  dynamicMember(name: String) -> PythonObject { ... }
}

Just some last-minute thoughts.
EDIT: I originally misrepresented the syntax dynamicCall/dynamicMember syntax.

1 Like

Is there some particular motivation for allowing arbitrary names? You could basically say

about any actual protocol requirement, so why is this one special? A future way for satisfying protocol requirements using a member with a different name on the concrete type (e.g. for conforming to two protocols with a naming clash) would handle this use case. This mechanism is already “privately” implemented as @_implements(Protocol, DeclName), so it seems likely that it will become user-facing in future.

2 Likes

Right. I think having a consistent and utterable name for the method — e.g. in case you want to call it directly instead of through the syntax, which isn't hard to imagine for these features — is actually very beneficial here.

3 Likes