SE-0216: User-defined dynamically callable types

The review of SE-0216: User-defined dynamically callable types begins now and runs through Tuesday, June 26th, 2018.

Reviews are an important part of the Swift evolution process. All review feedback should be either on this forum thread or, if you would like to keep your feedback private, directly to me as the review manager via email or direct message on the forums. If you send me email, please put "SE-0216" somewhere in the subject line.

What goes into a review of a proposal?

The goal of the review process is to improve the proposal under review through constructive criticism and, eventually, determine the direction of Swift.

When reviewing a proposal, here are some questions to consider:

  • What is your evaluation of the proposal?

  • Is the problem being addressed significant enough to warrant a change to Swift?

  • Does this proposal fit well with the feel and direction of Swift?

  • If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

  • How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

As always, thank you for contributing to Swift.

John McCall
Review Manager

11 Likes

Just realized that the proposal now includes dynamicMemberCallable as a future direction.
Imho this is an important change which definitely lessens my concerns regarding the complexity of the feature (I didn't see a discussion about this split).

Two choices for method signatures is better than four, and although a single signature would be preferable, the two remaining options offer some abilities which func dynamicallyCall(withArguments: [(String?, T1)]) -> T2 can't deliver.

As I don't need Python interoperability, there are topics I consider much more important than the whole "dynamic" story - but assuming that integration of this change doesn't have significant impact on other proposals, that is no reason to speak against inclusion of a finished implementation (but imho we should hurry adding "regular" callable types...).

Bottom line:
I was about to write a downvote for the original proposal, but the new version changed my mind, so I'm at least neutral now.

Without commenting on the proposal review, can you explain to me what you mean by "regular"? (Statically callable types?)

Huge +1. This provides parity with the @dynamicMemberLookup proposal, without them both, either feature would feel like it's missing its other half.

And to speak the actual uses of this, I can see a few beyond the motivator for this. For example, I could see this being used as part of some Mock type to model function/method calls, which I think would be hugely useful.

Looking forward, is the goal to move the TensorFlow Python interop into the upstream Swift? If so, I think this proposal is probably a prerequisite for that one. Without that, I don't think the interop experience would be good enough for inclusion into the main toolchain.

I think "statically callable" and "dynamically callable" are misleading names because (to me) they suggest that the call is statically/dynamically dispatched, which isn't the case.

I'm not sure about the best names for the two concepts but I'll refer to them as "regular callable" and "dynamically callable" in this post.

"Dynamically callable" is the idea described in the proposal: @dynamicCallable types are required to define at least one of two dynamicallyCall methods.

func dynamicallyCall(withArguments: A) -> X
// - `A` can be any type where `A : ExpressibleByArrayLiteral`.
// - `X` can be any type.
func dynamicallyCall(withKeywordArguments: D) -> X
// - `D` can be any type where `D : ExpressibleByDictionaryLiteral` and
//   `D.Key : ExpressibleByStringLiteral`.
// - `X` can be any type.

Calls to an instance of a @dynamicCallable type are desugared to an application of one of the two methods.

@dynamicCallable is limited for the following reasons:

  1. By definition, the dynamicallyCall methods take a variable number of arguments, so it's not possible to enforce a dynamic call to take a fixed number of arguments within the type system. To implement something like std::plus and std::minus, I can only do the following:
@dynamicCallable
enum BinaryOperation<T : Numeric> {
  case add, subtract, multiply

  func dynamicallyCall(withArguments arguments: [T]) -> T {
    // I can't enforce the two argument precondition using the type system.
    // What I really want is: `call(withArguments: (T, T))`.
    precondition(arguments.count == 2, "Must have 2 arguments")
    let x = arguments[0]
    let y = arguments[1]
    switch self {
    case .add:
      return x + y
    case .subtract:
      return x - y
    case .multiply:
      return x * y
    }
  }
}
let add: BinaryOperation = .add
add(1, 2) // works
add(1, 2, 3) // fails at run time, not compile time
  1. The dynamicallyCall methods require all arguments to have the same type. This limits @dynamicCallable to a few specific use cases like dynamic language interoperability.

The "regular callable" concept (I'll refer to it as @callable) address both of these limitations.
@callable types can mark any method as callable: there's no constraint on the type of the method whatsoever.

Calls to an instance of a @callable type are simply forwarded to the callable method.
For the BinaryOperation example above, @callable would be more ideal to enforce a two argument precondition:

@callable
enum BinaryOperation<T : Numeric> {
  case add, subtract, multiply

  @callableMethod // Mark this method as sugar-able.
  func call(withArguments arguments: (T, T)) -> T { ... }
}
let add: BinaryOperation = .add
add(1, 2) // works
add(1, 2, 3) // fails to type-check at compile time

@callable is analogous to operator() in C++ and is more general than @dynamicCallable.
However, it doesn't provide a great answer for dynamic language interoperability because argument labels aren't desugared in the same way.

Here's a Python interop demonstration:

// Testing the Python function `str.format(*args, **kwargs)`.
let greeting: PythonObject = "Hello {name}!"

// With @dynamicCallable: desugared to
// `dynamicallyCall(withKeywordArguments: ["name": "John"])`.
//
// Looks natural and similar to Python: `greeting.format(name="John")`.
greeting.format(name: "John")

// With @callable: there is no argument label sugar.
// Need to construct dictionary manually.
greeting.format(["name": "John"])

To me, it seems the main question is: is dynamic language interoperability important enough to Swift to justify @dynamicCallable?

Note that @dynamicMemberLookup doesn't have the same problem of generality as @dynamicCallable, it is just as useful for language interop as other use cases.

2 Likes

I think @dynamicCallable could be useful even outside of the realm of language interops. I could see it being used in conjuction with the @dynamicMethodLookup to implement a Mock type that would be useful for mock testing.

But my input on the question is yes. I think a great interop experience for these languages would be a requirement to have any hope of using frameworks and libraries written in them from Swift.

2 Likes

what @dan-zheng wrote (without any detailed plans for syntax)

2 Likes

What is your evaluation of the proposal?

Strong +1.

Is the problem being addressed significant enough to warrant a change to Swift?

Yes.

Does this proposal fit well with the feel and direction of Swift?

Yes. This proposal enables seamless syntactic sugar with dynamic languages and aligns naturally with the future direction after @dynamicMemberLookup.

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

n/a

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

In-depth study and helping @dan-zheng with the implementation.

1 Like

Yes definitely! Details are TBD though since it isn't clear how it should participate in swift-evolution process, but you can play with the implementation right now on the tensorflow branch. It is a single source file.

-Chris

3 Likes

+1

This is a big win for interop with JS, Python, etc, and it seems relatively non-invasive to the compiler as a whole.

1 Like

+1

This is very well thought out, and the two method version is much better than the previous four method iteration.

+1

This is a fairly simple, straightforward addition but allows for powerful interoperability with a variety of languages, allowing Swift to leverage many useful and popular libraries.

I'd like to see the @dynamicMemberCallable mentioned in the future directions, but that can probably come later, as the current proposal should cover the more pressing use cases.

1 Like

Downloadable Toolchains Available for this Proposal

We have downloadable toolchains/binaries of Swift that contains an implementation of this proposal available for download:

Follow the instructions on Swift.org for using built Swift binaries/toolchains.

12 Likes

The proposal itself looks fine.

Sad to see that the dynamicallyCallMethod part has dropped out since the previous threads -- was looking forward to adopting it for Ruby.

What/where was the thinking behind leaving it out? Implementations too different?

Let's add this back now while we're concentrating on this aspect of the language: not sure how much enthusiasm there'll be for revisiting this 'future direction' without a project like TF to drive it.

1 Like

For JavaScript specifically, there’s a difference between calling a method and ‘detaching’ a method into a function and calling it. It loses the reference to the receiver (this in JavaScript, self in Swift).

Swift ‘binds’ the receiver when detaching an instance method:

let abc = "abc"
let reversedMethod = abc.reversed
reversedMethod()
// => "cba"

let abcArray = ["a", "b", "c"]
abcArray.joined()
// => "a | b | c"
let joinMethod = abcArray.joined
joinMethod(" | ")
// => "a | b | c"

Where as in JavaScript, the this (called context in JavaScript) will have to be passed in using .call():

const abcArray = ["a", "b", "c"]
abcArray.join(" | ")
// => "a | b | c"
const joinMethod = abcArray.join
joinMethod(" | ")
// => Exception
joinMethod.call(abcArray, " | ")
// => "a | b | c"

This means you can even pass in a different context:

joinMethod.call(["x", "y", "z"], " | ")
// => "x | y | z"

The receiver is only known when you call it using receiver.method() but not remembered when const method = receiver.method; method().

Objects in JavaScript are really just key-value pairs, and methods are just where the value is a function!

Not sure if this a huge concern and if it applies to other languages, but I just thought I’d bring it up. Would we want to ability to detach methods and call them later? What would happen with a @dynamicCallable instance if we ‘detached’ a method and then called it?

I just want to finish saying I think this functionality is low priority compared to the overall benefit of this addition. But if adding something like this later would actually mean a much different approach, then perhaps it’s worth considering thinking about now.

1 Like

Yes, the initial reason for leaving out "dynamic member callable" was honestly because of implementation difference/difficulty.

(@dynamicCallable is implemented by adding a new rule for simplifying the "applicable function" constraint: it doesn't touch member lookup or overload choice logic. I imagine @dynamicMemberCallable would be implemented more similarly to @dynamicMemberLookup, since it does involve member lookup.)

I do believe separating the two features into different proposals is a good idea for modularity.
It's easier to review just @dynamicCallable as a standalone idea and the design for @dynamicMemberCallable may change/be informed by the feedback on @dynamicCallable.

Edit: if someone is interested in tackling the implementation of "dynamic member callable", I would be happy to discuss and share specific implementation detail ideas.

2 Likes

Thanks for bringing this up! Some people discussed JavaScript interopability (including the binding problem) in the pitch threads for @dynamicCallable (here and here).

A useful starting point for JavaScript interop is working through how to add callable behavior for a specific implementation (e.g. extending JSValue from JavaScriptCore with callable behavior).

JSValue in particular defines the following method for performing JavaScript method calls:

extension JSValue {
  func invokeMethod(_ method: String!, withArguments arguments: [Any]!) -> JSValue!
}

At a glance, this seems exactly like something @dynamicMemberCallable can sugar.


I wonder if there are other implementations of JavaScript interop besides JSValue; thinking about how to extend those would be useful too.

2 Likes

Thanks Dan for the thoughtful reply. Looking at JSValue, then what I am looking at is

extension JSValue {
  func call(withArguments arguments: [Any]!) -> JSValue!
}

https://developer.apple.com/documentation/javascriptcore/jsvalue/1451648-call

Oddly there is no this context argument here, so you would have to use the C API with its thisObject argument:

func JSObjectCallAsFunction(_ ctx: JSContextRef!, 
                          _ object: JSObjectRef!, 
                          _ thisObject: JSObjectRef!, 
                          _ argumentCount: Int, 
                          _ arguments: UnsafePointer<JSValueRef?>!, 
                          _ exception: UnsafeMutablePointer<JSValueRef?>!) -> JSValueRef!

https://developer.apple.com/documentation/javascriptcore/1451407-jsobjectcallasfunction

edit

Those links were handy. I think I’ve confused myself. I reread the proposal, and see that the member version is marked for the future. So I don’t have concerns. Specifically for JavaScript, it has .call() and .apply() on functions, which could be used to set the this context (clumsily). The other issues can be discussed when the member proposal comes.

My impression is that:

  • call(withArguments:) is for calling top-level JavaScript functions.
    • This can be modeled by @dynamicCallable.
  • invokeMethod(_:withArguments:) is for invoking methods where this needs to be bound.
    • This can be modeled by @dynamicMemberCallable when implemented.

From the "Discussion" section regarding invokeMethod:

Calling this Objective-C method first uses the forProperty(_:) method to look up the named field of the JavaScript value. Then, JavaScriptCore treats that field’s contents as a JavaScript function and sets the JavaScript this keyword to refer to this JSValue instance.

1 Like

@Chris_Lattner3 and @dan-zheng, in the proposed solution:

We propose introducing a new @dynamicCallable attribute to the Swift language which may be applied to structs, classes, enums, and protocols. This follows the precedent of SE-0195.

But the @dynamicMemberLookup attribute cannot be applied to protocols, AFAIK.

import Foundation
import JavaScriptCore

@dynamicMemberLookup
public protocol JSDynamicMemberLookup {

  subscript(dynamicMember name: String) -> JSValue { get set }
}

// error: @dynamicMemberLookup attribute requires 'JSDynamicMemberLookup'
// to have a 'subscript(dynamicMember:)' member with a string index

(I was trying to add this to existing Objective-C classes, such as JSContext and JSValue).