[Pitch #3] Introduce user-defined dynamically "callable" types

Evolution Pitches
1

Hi All,

The field of machine learning is highly dependent on some popular data science libraries written in Python, and interoperability with these libraries is a high priority for the Swift for TensorFlow project. You can read more background about this in our Python interoperability whitepaper.

This work originally motivated SE-0195 - Introduce User-defined "Dynamic Member
Lookup" Types which was accepted, implemented, and shipped in Swift 4.2. OTOH, its sister proposal has been sitting idle since then. It introduces a @dynamicCallable attribute to further improve the bindings.

I still haven't had time to implement it, but I just posted an updated version of the proposal and I'd be curious to get any feedback on the new design. Recent changes make it align with the SE-0195 design, incorporated feedback from the last pitch cycle (notably, support for Smalltalk derived languages), and shortened up some of the discussion.

If you are curious, you can see our current Python interop code, which should work with Swift 4.2. After @dynamicCallable is settled, we'll open discussion on formalizing and standardizing the Python binding itself - our ideal end game is for it (or something like it) to replace the existing Python module.

In any case, I'd really love feedback on the new proposal!

-Chris

14 Likes
2

In my opinion, we should make a decision that applies to all types, irrespective of whether they are dynamic or static. That is, either Swift should say “Yes, it should be possible to make instances callable”, or “No it should not be possible”. Carving this out specifically for dynamic types seems unprincipled.

Currently, if one makes a type such as:

final class DifferentiableFunction<T: FloatingPoint> {
  var f: (T) -> T
  var df: DifferentiableFunction<T>?
}

then to evaluate the function an instance foo represents, one can either write foo.f(x), or define a subscript to enable syntax like foo[x]. It would be nice to write foo(x) instead, and the question is whether to permit that.

As I see it, if the square-bracket subscript spelling is good enough for native Swift types, then it is good enough for imported dynamic types. And if the square-bracket syntax is *not* good enough for dynamic types, then it is not good enough for native types either.

If we do decide to introduce the ability to make instances of a type callable, it could be implemented as an attribute like @callable on the method which is substituted at the call-site. Multiple methods could have that attribute, and it would work as long as the call-site is unambiguous.

8 Likes
3

I think this is a great idea! I was more sad that it didn't land along-side the dynamic member lookup stuff. But now that we have that, I think it's actually harmful to not have this feature. It does a nice job of rounding out those kinds of behaviors.

I agree that retroactive conformance to this should be banned, like the the member lookup proposal.

4

I think this is a different issue entirety. Lets create a new thread discussing these concerns about being able to call an instance with () in a similar fashion as subscripts. This proposal is about calling methods/funcs using the . notation.

I do not think it would be valid swift code to say foo.() in the same way it is not valid to say foo.[]

5

From the proposal:

The proposal is about allowing the use of parentheses immediately after an instance of a type, with no “.” involved, and parsing it as a function call—but only in the specific case of dynamic types. I am saying that we should decide whether to allow that syntax, and apply the decision uniformly to all types.

1 Like
6

I am on board with dynamicMethodCall but I think this should be a subscript. This is great extension to the member look up proposal.

  1. Other than initializes, when would dynamicCall be used?

I think dynamicCall should be folded into the dynamicMethodCall for the use for constructors/initializer using special names like foo.new(), foo.init().

Since in Swift we do not have new Foo() then it is not possible to provide this functionality to a dynamicCall when this call is a constructors/initializer. Sometimes in my swift code I have used Type.init so I do not think it would be so much to ask people interacting with python/ruby/etc code to use the same heuristics. I recognize that some languages like python use different names for their explicit constructor.

Perhaps we could just instead support:

  init (arguments: [T1]) {}

  init(keywordArguments: [(String, T3)]){}

instead of dynamicCall.

7

I see. I was focused on dynamicMethodCall. I do not think dynamicCall should be allowed outside constructors/initializers but I may be missing other use cases for other dynamic languages.

8

IIRC, dynamicMethodCall behavior is already possible, just have a dynamicMember of function type.

9

Does the ‘dynamic’ in ‘@dynamicCallable’ carry semantic weight? That is, does the compiler use that to understand it should not statically try to find the matching method?

I am otherwise unsure why not just go with ‘@callable’. Being able to apply this to native Swift types too seems intriguing.

I think this should go forward. But I do wonder about unintended effects. Were we to get this capability, what gaps in Swift might it be used to work around? Would it be abused to provide some dynamic or meta programming desires, and if so would that have any long-term negative effects on Swift’s path? (My uneducated guess right now is, that would be mostly fine. This is a small and reasoned feature with a high return in power provided. If it solves other problems too, even better. )

3 Likes
10

I'm a bit concerned about the mix of requirements - there are 4 implementation strategies, based on how much information you need, starting from everything (basename+keywords) down to just the argument values. So you pick your lowest-level, and there are default implementations which can lower the call information down to what you need.

Why do we even need to model how the foreign language accepts its function calls? I would think we should be modelling the Swift call-site, and allowing implementors like PyValue or RubyValue decide which information they want to glean from that.

That means reducing these 4 requirements down to the one with the most information. Something like:

protocol DynamicCallable {
  associatedtype ArgumentType, ResultType
  
  func dynamicCall<S1, S2>(basename: S1, keywordArguments: [(S2, ArgumentType)]) -> ResultType
   where S1: StringProtocol, S2: StringProtocol
}

The example in the proposal about optimising Python's basename and non-basename forms could be done by making the above function @inlineable. I suppose most of the time it will be checking whether or not a static string is empty.

I don't have any issue with the idea behind the proposal, just the implementation is very awkward.

1 Like
11

How about supporting trailing closure syntax on the dynamic call? Ruby has a similar concept that could take advantage of this and, beyond dynamic language interop, it would let us build more APIs that look Swift-like.

To get the compile-time checking consistent with the rest of the proposal I think this would add an additional four possible spellings. On one hand this feels a bit ugly; on the other hand any given implementation would likely implement only a couple. I realize there's an ambiguous parse lurking given particular argument-type choices -- feels like not an issue in practice, we would just need to rule on the precedence.

(Is the explicit String in the second dynamicCall variant a typo?)

12

Well if we reduce it to one requirement, if that's even feasible, you'll be putting more logic into that single method to handle the correct call types. Which will increase the likelihood of someone incorrectly implementing the method.

1 Like
13

I suppose the proposal is not exactly clear on this:

 func dynamicCall(arguments: [T1]) -> T2
 func dynamicCall(keywordArguments: [(String, T3)]) -> T4
 func dynamicMethodCall(baseName: S1, arguments: [T5]) -> T6
 func dynamicMethodCall(baseName: S2, keywordArguments: [(S3, T7)]) -> T8

Can it be that T1 != T3, or T2 != T4? Or are these basically "Argument" and "Result" associated types (limited to 1 per conformer)? Otherwise you could create some weird stuff:

@dynamicCallable
struct RubyAndPython {

 func dynamicCall(arguments: [PyVal]) -> PyVal { ... }

 func dynamicMethodCall(baseName: String, arguments: [RubyVal]) -> RubyVal { ... }
}

There is a mistake in the proposal where it says:

Python does support keyword arguments. While it could just implement that hook, where we implement the keyword and non-keyword forms to avoid allocating a temporary dictionary in the non-keyword case:

There are no dictionaries in this proposal. Keyword arguments are passed as an array of tuples.

Also, how are you going to call this function with an Int or String without explicitly boxing them as PyVals? You would need T1 to be a protocol (PythonConvertible or something).

As it happens, there was some talk about allowing protocols to satisfy associated-types, exactly for these kind of architectures. [Pitch] Generalized supertype constraints

EDIT: Indeed, this is what the Playground does: https://github.com/google/swift/blob/a988b64966b1830cd379419e401d11efdca092d1/stdlib/public/Python/Python.swift#L272

14

While true, this would mean you couldn't have any non-function dynamicMember, thus rendering it impractical for dynamic language interop.

This is not an error: To pass the kwargs along to e.g. Python, the @dynamicCallable implementation would have to reduce them into a Python dictionary type (or at least iterate over the keywords to make sure none were given). A variant that only takes arguments would be able to skip this step and probably pass the arguments more or less directly into the Python API.

I'm not sure how important of a goal performance for these interop-layers is, but this seems to be the reasoning.

Also @Chris_Lattner3 what is the reasoning behind passing an empty string where no keyword was given instead of changing the signature to take S? where S: ExpressibleByStringLiteral and passing nil instead?

15

This seems like some very Python-specific logic to be encoding in the requirements. As I said before, I think that rather than offering dispatch points for every kind of invocation strategy we can think of, we should offer a model of Swift method calling and let the implementors handle their details, with performance coming from things like aggressive inlining.

One approach they might take would be to keep dynamicCall really small, and forward to a bunch of specialised functions. Most of the time the compiler should be able to statically figure out if there is a base-name or argument labels anyway.

So if you consider all of that, why not do something like the following?

struct Python {

  @inlineable // or maybe even @inline(__always)
  func dynamicCall<S1, S2>(basename: S1, arguments: [(S2, PyVal)]) -> PyVal {
    if basename.isEmpty {
      // maybe even split basename/noBasename handling as separate functions to promote inlining.
      if arguments.contains(where: { !$0.0.isEmpty }) {
        return _call_noBasename_withLabels(arguments)
      } else {
        return _call_noBasename_noLabels(arguments.map { $0.1 })
      }
    } else {
      // ..etc
    }
  }

  @usableFromInline
  internal func _call_noBasename_withLabels([S2, PyVal]) -> PyVal { ... }

  @usableFromInline
  internal func _call_noBasename_noLabels([PyVal]) -> PyVal { ... }
}
16

A part of Swift methods, as in many other languages supporting function types, is that they can be assigned to variables and passed around. This piece of code works in Python. Translating it to swift would work the same.

l = [0, 1, 2, 4, 8]
a = l.append
a(16) # l is now [0, 1, 2, 4, 8, 16]

Using your proposed approach, this would not be possible to do using @dynamicCallable. This is why a variant with the semantic of calling the function represented by the receiver is needed.

I'm not sure that is the case right now. Surely not when using something other than Array as container for the arguments, which the proposal aims to allow, I think for good reason:

We write the arguments and keywordArguments parameter as an array type, but these will actually be allowed to be any type that conforms to the ExpressibleByArrayLiteral protocol, where the element type has the specified constraints.

Now, we could just solve these problems by providing one methods with both the base-name and the argument labels as optionals, but that would (a) muddle the semantics of the method (am I being called or should I call a member of mine?) and (b) prevent us from making things that are impossible in a language compile-time errors.

So I think it is justified to have a number of different options when it comes to what semantics types implementing @dynamicCallable can support.

One might argue there should be @dynamicCallable and @dynamicMemberCallable or something like that, but I don't think that really provides much additional information over what one can tell from the proposed method and parameter names, and trying to do something a type doesn't support would be a compile-time error either way.

1 Like
17

Hi Nevin,

I'm not sure what you mean. The proposal isn't making any such distinction. What do you mean by "dynamic" and "static" types?

I think you're confusing this proposal with something like operator() in C++, which this proposal has very little to do with. There is a short cross reference in the "Future directions" section of the proposal though.

Please take a look. If I'm completely misunderstanding you, then please try again :slight_smile:. This happens to me pretty frequently.

-Chris

18

Please elaborate why you think this should be the case.

In the case of @dynamicMemberLookup, it uses subscript because the members are allowed to be lvalues that are identified by their name. Subscripts are the model Swift has for "properties that have indices" which is why it uses them.

In contrast, calls never produce an lvalue. I don't see a motivation to make them a subscript.

-Chris

1 Like
19

This proposal applies to native swift types. PyValue and others are standard Swift structs written with pure Swift code. The decision of @dynamicCallable vs @callable is discussed in the "Future directions" section. I don't have a strong opinion either way.

It seems that there is some confusion in the writing, please let me know what caused that confusion and I'll fix it ASAP, thank you!

-Chris

1 Like
20

The answer is straightforward: we want to produce a compiler time error (not a runtime error) when someone uses (e.g.) keyword arguments with a language binding that does not support them (e.g. Javascript).

-Chris