SE-0216: User-defined dynamically callable types

Yes, the initial reason for leaving out "dynamic member callable" was honestly because of implementation difference/difficulty.

(@dynamicCallable is implemented by adding a new rule for simplifying the "applicable function" constraint: it doesn't touch member lookup or overload choice logic. I imagine @dynamicMemberCallable would be implemented more similarly to @dynamicMemberLookup, since it does involve member lookup.)

I do believe separating the two features into different proposals is a good idea for modularity.
It's easier to review just @dynamicCallable as a standalone idea and the design for @dynamicMemberCallable may change/be informed by the feedback on @dynamicCallable.

Edit: if someone is interested in tackling the implementation of "dynamic member callable", I would be happy to discuss and share specific implementation detail ideas.

2 Likes

Thanks for bringing this up! Some people discussed JavaScript interopability (including the binding problem) in the pitch threads for @dynamicCallable (here and here).

A useful starting point for JavaScript interop is working through how to add callable behavior for a specific implementation (e.g. extending JSValue from JavaScriptCore with callable behavior).

JSValue in particular defines the following method for performing JavaScript method calls:

extension JSValue {
  func invokeMethod(_ method: String!, withArguments arguments: [Any]!) -> JSValue!
}

At a glance, this seems exactly like something @dynamicMemberCallable can sugar.


I wonder if there are other implementations of JavaScript interop besides JSValue; thinking about how to extend those would be useful too.

2 Likes

Thanks Dan for the thoughtful reply. Looking at JSValue, then what I am looking at is

extension JSValue {
  func call(withArguments arguments: [Any]!) -> JSValue!
}

https://developer.apple.com/documentation/javascriptcore/jsvalue/1451648-call

Oddly there is no this context argument here, so you would have to use the C API with its thisObject argument:

func JSObjectCallAsFunction(_ ctx: JSContextRef!, 
                          _ object: JSObjectRef!, 
                          _ thisObject: JSObjectRef!, 
                          _ argumentCount: Int, 
                          _ arguments: UnsafePointer<JSValueRef?>!, 
                          _ exception: UnsafeMutablePointer<JSValueRef?>!) -> JSValueRef!

https://developer.apple.com/documentation/javascriptcore/1451407-jsobjectcallasfunction

edit

Those links were handy. I think I’ve confused myself. I reread the proposal, and see that the member version is marked for the future. So I don’t have concerns. Specifically for JavaScript, it has .call() and .apply() on functions, which could be used to set the this context (clumsily). The other issues can be discussed when the member proposal comes.

My impression is that:

  • call(withArguments:) is for calling top-level JavaScript functions.
    • This can be modeled by @dynamicCallable.
  • invokeMethod(_:withArguments:) is for invoking methods where this needs to be bound.
    • This can be modeled by @dynamicMemberCallable when implemented.

From the "Discussion" section regarding invokeMethod:

Calling this Objective-C method first uses the forProperty(_:) method to look up the named field of the JavaScript value. Then, JavaScriptCore treats that field’s contents as a JavaScript function and sets the JavaScript this keyword to refer to this JSValue instance.

1 Like

@Chris_Lattner3 and @dan-zheng, in the proposed solution:

We propose introducing a new @dynamicCallable attribute to the Swift language which may be applied to structs, classes, enums, and protocols. This follows the precedent of SE-0195.

But the @dynamicMemberLookup attribute cannot be applied to protocols, AFAIK.

import Foundation
import JavaScriptCore

@dynamicMemberLookup
public protocol JSDynamicMemberLookup {

  subscript(dynamicMember name: String) -> JSValue { get set }
}

// error: @dynamicMemberLookup attribute requires 'JSDynamicMemberLookup'
// to have a 'subscript(dynamicMember:)' member with a string index

(I was trying to add this to existing Objective-C classes, such as JSContext and JSValue).

The errors occurs with the current logic because the JSDynamicMemberLookup protocol does not define a subscript(dynamicMember:) method, it only lists it as a requirement.

The following does compile, however:

@dynamicMemberLookup
public protocol JSDynamicMemberLookup {}
extension JSDynamicMemberLookup {
  subscript(dynamicMember name: String) -> JSValue {
    get { fatalError() }
    set { fatalError() }
  }
}

This behavior is easier to implement: @dynamicMemberLookup (and @dynamicCallable) is simply a sugar for a concrete method. Supporting the protocol requirement case is non-trivial.

I'm curious what your use case for a protocol like JSDynamicMemberLookup is?

What is your evaluation of the proposal?

Strong +1.

Is the problem being addressed significant enough to warrant a change to Swift?

Yes.

Does this proposal fit well with the feel and direction of Swift?

Yes, it complements @dynamicMemberLookup and hugely improves Swift's dynamic languages interoperability story.

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

Reading the proposal, following the pitches and using the implementation available with Swift for TensorFlow.

Fair enough; it did dawn on me as I read your patch that 'these two things are not the same'...

May take you up on the tips offer later in the summer.

1 Like

My use case is @dynamicMemberLookup for the JSContext and JSValue classes (imported from Objective-C) without subclassing.

import Foundation
import JavaScriptCore

@dynamicMemberLookup
public protocol JSDynamicMemberLookup: AnyObject {}

extension JSDynamicMemberLookup where Self: JSValue {

  public subscript(dynamicMember name: String) -> JSValue {
    get {
      return objectForKeyedSubscript(name as NSString)
    }
    set {
      setObject(newValue, forKeyedSubscript: name as NSString)
    }
  }
}

extension JSValue: JSDynamicMemberLookup {}

extension JSDynamicMemberLookup where Self: JSContext {

  public subscript(dynamicMember name: String) -> JSValue {
    get {
      return globalObject![dynamicMember: name]
    }
    set {
      globalObject![dynamicMember: name] = newValue
    }
  }
}

extension JSContext: JSDynamicMemberLookup {}

let context = JSContext()!
context.Math.PI.toDouble() == .pi // true
1 Like

I’m worried about the fact that the proposal has relegated support for Smalltalk-like languages to the future directions section. I think it’s important for this proposal be able to support writing bindings to Ruby and I’m concerned that if it’s not done now, it may never be done.

5 Likes

I see, good point!
Indeed, with the current design there is no way to (usefully) extend an existing type with @dynamicMemberLookup or @dynamicCallable behavior.

This is a notable deficiency in cases where one can't change the declaration of the existing type.
Here, I might argue that the proper solution is for JavaScriptCore to implement @dynamicMemberLookup and @dynamicCallable behavior. :slight_smile:

Your concern is valid. It seems to me the only major language interop fully supported by @dynamicCallable at the moment is Python. However, many people are interested in Ruby/Objective-C/JavaScript interop, which all require @dynamicMemberCallable behavior.

Personally, I don't see @dynamicMemberCallable as something that's in danger of never being done. I have a clear idea of how to implement it and I'm happy to share ideas if anyone's interested in tackling it. Existing code for dynamic member lookup/callable behavior should definitely be reused/generalized.

I might tackle it myself over a weekend, when I get a chance.

6 Likes

+1 to the proposal. It's an important feature with many intriguing applications, and seems like a good design.

The use of ExpressibleByDictionaryLiteral is particularly elegant, allowing implementations which can assume unique keywords to use dictionaries, while not ruling out those that can't.


I'll echo David’s concerns about @dynamicMemberCallable being left behind. It does seem best to review it separately, but that proposal should not lag too far behind the heels of this one.


This confused me:

This proposal does not introduce the ability to provide dynamically callable static/class members.

Does this refer to something like MyType.dynamicThing("foo")? If so, wouldn’t that be supported via static var dynamicThing: MyCallableType? If not, what does it mean?

The general class of functionality this provides is important, but I have some misgivings about this design.

Swift is moving towards a model where argument labels are part of the name of a function, not part of the function itself. We've discussed this direction many times in the past—for instance, when we removed argument labels from function types and made associated value labels part of a case's name.

dynamicCall(withKeywordArguments:) ignores this direction in the language's evolution. The semantics of these calls are fundamentally quite different from a normal Swift function. I worry that, as we try to evolve the AST to improve the extremely messy way we currently handle Swift calls, we may hit limits imposed by this proposal. I don't want to end up in a situation where we're handling Swift code worse in order to handle Python code better.

My other concern about this design is that it is laser-focused on Python. The proposal gestures at how one feature might be used in Javascript, but it's not ideal for that. It doesn't handle Smalltalk and Ruby at all (and IMHO, misses the main challenge of Ruby—it doesn't need to access argument labels early, it needs to handle ambiguity between calls and property accesses). It doesn't help us with any problems outside language bridging.

On a bikeshedding front, I worry that the distinction between "dynamic callable" and "dynamic function" (as in the dynamic keyword) may be too subtle, and that if the @dynamicMemberCallable feature suggested in "future directions" is later adopted, the distinction between @dynamicCallable, @dynamicMemberCallable, and @dynamicMemberLookup will also be too subtle.


I think this functionality should be implemented as two orthogonal features:

  1. Add a variadic list of argument labels to the lookup done by @dynamicMemberLookup. This would let you look up a member whose name includes argument labels.

  2. Add a way to use plain function call syntax on a non-function type, along the lines of C++'s operator (). Python would declare one of these with a variadic list of arguments, but the feature would be able to serve many other use cases, too.

This design would be worse for Python, but better for everything else. It would slot more readily into Swift's semantics and it would cover many more use cases. The downsides are that it makes the Python bridge's job harder (it would need to save the labels seen in the @dynamicMemberLookup call inside the instance it returned), and that a user would not be able to assign a PyObject representing a function to a variable and then call it with keyword arguments using the ordinary function syntax (which they wouldn't be able to do with a Swift closure, either). I think a little bridge awkwardness is worth having a more flexible, semantically simpler feature.


I think good dynamic bridging is a great thing to add to the language, but I don't think this particular design is right for Swift. It is too limiting and too limited. I urge the core team to reject SE-0216 and ask for a different solution to the same problem.

7 Likes

As I understand them based on this proposal and past discussion, @dynamicMemberCallable plus @dynamicMemberLookup will handle Ruby just fine. A Ruby bridge would not use @dynamicCallable at all.

2 Likes

@dynamicMemberCallable is not part of this proposal, so if we need @dynamicMemberCallable to handle Ruby, this proposal doesn't handle Ruby. That's my point—the proposal is very narrowly targeted at handling the needs of a Python bridge, and is too inflexible to handle even minor variations in requirements.

Could you expand on possible "limits imposed by this proposal", or problematic AST changes that you envision?

I think, from an implementation perspective, @dynamicMemberLookup and @dynamicCallable are exceptionally simple because they are syntactic sugar. This would make them flexible to changes in the AST.

Can you share a use-case where it's necessary in a language to look up a "member whose name includes argument labels"?

I explored "regular callable" behavior and I agree that it is more general.
However, for keyword-argument calls, it doesn't desugar argument labels like @dynamicCallable does. This is unnatural for language interop:

// With @dynamicCallable.
np.random.randint(-10, 10, dtype: np.float)
// With regular callable sugar. The empty strings are killer.
np.random.randint(["": -10, "": 10, "dtype": np.float])

I think "regular callable sugar" and @dynamicCallable solve different problems and should be evaluated separately.

Perhaps the solution is as simple as not creating a new attribute @dynamicMemberCallable, but rather extending @dynamicCallable to handle the two new dynamicallyCallMethod methods using ambiguity resolution. Would this address your concern?

To be fair, the proposal does say as much. I see this as a family of three proposed features which when taken together do appear to handle all the dynamic language bridging needs, at least all the ones I'm aware of — and do so within an overall division of labor that makes sense to me.

Now if the third (@dynamicMemberCallable) proposal never materializes, then I do share your concern! Assuming it does, however, I don’t see a problem with this overall design vis-a-vis Ruby.

That’s not to comment on your other concerns; I just don’t see that lack of Ruby support in particular should count against this proposal. That’s more of a comment on Ruby itself, which has does not have any notion of callable things separate from method dispatch.

@beccadax's argument seems to me to be that due to SE-0111 and SE-0155, Swift itself is becoming a language where it's necessary to look up a "member whose name includes argument labels".

2 Likes