[Pitch] Introduce user-defined dynamically "callable" types

Hello all,

I have a couple of proposals cooking in a quest to make Swift interoperate with dynamically typed languages like Python better. Instead of baking in hard coded support for one language or the other, I’m preferring to add a few small but general purpose capabilities to Swift. This is the first, which allows a Swift type to become “callable”.

The proposal is here:

I’ve also attached a snapshot below, but it will drift out of date as the proposal is refined. Feedback and thoughts are appreciated, thanks!

-Chris

Introduce user-defined dynamically "callable" types

Proposal: SE-NNNN <https://gist.github.com/lattner/NNNN-DynamicCallable.md>
Author: Chris Lattner <https://github.com/lattner>
Review Manager: TBD
Status: Awaiting implementation
<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#introduction>Introduction

This proposal introduces a new DynamicCallable protocol to the standard library. Types that conform to it are "callable" with the function call syntax. It is simple syntactic sugar which allows the user to write:

    a = someValue(keyword1: 42, "foo", keyword2: 19)
and have it be interpreted by the compiler as:

  a = someValue.dynamicCall(arguments: [
    ("keyword1", 42), ("", "foo"), ("keyword2", 19)
  ])
Other languages have analogous features (e.g. Python "callables"), but the primary motivation of this proposal is to allow elegant and natural interoperation with dynamic languages in Swift.

Swift-evolution thread: Discussion thread topic for that proposal <https://lists.swift.org/pipermail/swift-evolution/>
<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#motivation>Motivation

Swift is well known for being exceptional at interworking with existing C and Objective-C APIs, but its support for calling APIs written in scripting langauges like Python, Perl, and Ruby is quite lacking. These languages provide an extremely dynamic programming model where almost everything is discovered at runtime.

Through the introduction of this proposal, and the related DynamicMemberLookupProtocol proposal, we seek to fix this problem. We believe we can make many common APIs feel very natural to use directly from Swift without all the complexity of implementing something like the Clang importer. For example, consider this Python code:

class Dog:
    def __init__(self, name):
        self.name = name
        self.tricks = [] # creates a new empty list for each dog
        
    def add_trick(self, trick):
        self.tricks.append(trick)
we would like to be able to use this from Swift like this (the comments show the corresponding syntax you would use in Python):

  // import DogModule
  // import DogModule.Dog as Dog // an alternate
  let Dog = Python.import(“DogModule.Dog")

  // dog = Dog("Brianna")
  let dog = Dog("Brianna")

  // dog.add_trick("Roll over")
  dog.add_trick("Roll over")

  // dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog("Kaylee").add_trick("snore")
Of course, this would also apply to standard Python APIs as well. Here is an example working with the Python pickleAPI and the builtin Python function open:

  // import pickle
  let pickle = Python.import("pickle")

  // file = open(filename)
  let file = Python.open(filename)

  // blob = file.read()
  let blob = file.read()

  // result = pickle.loads(blob)
  let result = pickle.loads(blob)
This can all be expressed today as library functionality written in Swift, but without this proposal, the code required is unnecessarily verbose and gross. Without it (but with the related dynamic member lookup proposal) the code would have a method name (like call) all over the code:

  // import pickle
  let pickle = Python.import("pickle") // normal method in Swift, no change.

  // file = open(filename)
  let file = Python.open.call(filename)

  // blob = file.read()
  let blob = file.read.call()

  // result = pickle.loads(blob)
  let result = pickle.loads.call(blob)

  // dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog.call("Kaylee").add_trick.call("snore")
While this is a syntactic sugar proposal, we believe that this expands Swift to be usable in important new domains. This sort of capability is also highly precedented in other languages, and is a generally useful language feature that could be used for other purposes as well.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#proposed-solution>Proposed solution

We propose introducing this protocol to the standard library:

protocol DynamicCallable {
  associatedtype DynamicCallableArgument
  associatedtype DynamicCallableResult

  func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws -> DynamicCallableResult
}
It also extends the language such that function call syntax - when applied to a value of DynamicCallable type - is accepted and transformed into a call to the dynamicCall member. The dynamicCall method takes a list of tuples: the first element is the keyword label (or an empty string if absent) and the second value is the formal parameter specified at the call site.

Before this proposal, the Swift language has two types that participate in call syntax: functions and metatypes (for initialization). Neither of those may conform to protocols at the moment, so this introduces no possible ambiguity into the language.

It is worth noting that this does not introduce the ability to provide dynamicly callable static/class members. We don't believe that this is important given the goal of supporting dynamic languages like Python, but if there is a usecase discovered in the future, it could be explored as future work. Such future work should keep in mind that call syntax on metatypes is already meaningful, and that ambiguity would have to be resolved somehow.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#discussion>Discussion

While the signature for dynamicCall is highly general we expect the most common use will be clients who are programming against concrete types that implement this proposal. One very nice aspect of this is that, as a result of Swift's existing subtyping mechanics, implementations of this type can choose whether they can actually throw an error or not. For example, consider this silly implementation:

struct ParameterSummer : DynamicCallable {
  func dynamicCall(arguments: [(String, Int)]) -> Int {
    return arguments.reduce(0) { $0+$1.1 }
  }
}

let x = ParameterSummer()
print(x(1, 7, 12)) // prints 20
Because ParameterSummer's implementation of dynamicCall does not throw, the call site is known not to throw either, so the print doesn't need to be marked with try.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#example-usage>Example Usage

A more realistic (and motivating) example comes from a prototype Python interop layer. While the concrete details of this use case are subject to change and not important for this proposal, it is perhaps useful to have a concrete example to see how this comes together.

That prototype currently has two types which model Python values, one of which handles Python exceptions and one of which does not. Their conformances would look like this, enabling the use cases described in the Motivation section above:

extension ThrowingPyRef: DynamicCallable {
  func dynamicCall(arguments: [(String, PythonConvertible)]) throws
      -> PythonConvertible {
    // Make sure state errors are not around.
    assert(PyErr_Occurred() == nil, "Python threw an error but wasn't handled")

    // Count how many keyword arguments are in the list.
    let numKeywords = arguments.reduce(0) {
      $0 + ($1.0.isEmpty ? 0 : 1)
    }

    let kwdict = numKeywords != 0 ? PyDict_New() : nil

    // Non-keyword arguments are passed as a tuple of values.
    let argTuple = PyTuple_New(arguments.count-numKeywords)!
    var nonKeywordIndex = 0
    for (keyword, argValue) in arguments {
      if keyword.isEmpty {
        PyTuple_SetItem(argTuple, nonKeywordIndex, argValue.toPython())
        nonKeywordIndex += 1
      } else {
        PyDict_SetItem(kwdict!, keyword.toPython(), argValue.toPython())
      }
    }

    // Python calls always return a non-null value when successful. If the
    // Python function produces the equivalent of C "void", it returns the None
    // value. A null result of PyObjectCall happens when there is an error,
    // like 'self' not being a Python callable.
    guard let resultPtr = PyObject_Call(state, argTuple, kwdict) else {
      throw PythonError.invalidCall(self)
    }

    let result = PyRef(owned: resultPtr)

    // Translate a Python exception into a Swift error if one was thrown.
    if let exception = PyErr_Occurred() {
      PyErr_Clear()
      throw PythonError.exception(PyRef(borrowed: exception))
    }

    return result
  }
}

extension PyRef: DynamicCallable {
  func dynamicCall(arguments: [(String, PythonConvertible)])
      -> PythonConvertible {
    // Same as above, but internally aborts instead of throwing Swift
    // errors.
  }
}
<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#source-compatibility>Source compatibility

This is a strictly additive proposal with no source breaking changes.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#effect-on-abi-stability>Effect on ABI stability

This is a strictly additive proposal with no ABI breaking changes.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#effect-on-api-resilience>Effect on API resilience

This has no impact on API resilience which is not already captured by other language features.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#alternatives-considered>Alternatives considered

A few alternatives were considered:

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#add-ability-to-reject-parameter-labels>Add ability to reject parameter labels

The implementation above does not allow an implementation to staticly reject argument labels. If this was important to add, we could add another protocol to model this, along the lines of:

/// A type conforming just to this protocol would not accept parameter
/// labels in its calls.
protocol DynamicCallable {
  associatedtype DynamicCallableArgument
  associatedtype DynamicCallableResult

  func dynamicCall(arguments: [DynamicCallableArgument]) throws -> DynamicCallableResult
}

/// A type conforming to this protocol does allow optional parameter
/// labels.
protocol DynamicCallableWithKeywordsToo : DynamicCallable {
  func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws -> DynamicCallableResult
}
This would allow a type to implement one or the other based on their capabilities. This proposal is going with a very simple design, but if there is demand for this, the author is happy to switch.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#staticly-checking-for-exact-signatures>Staticly checking for exact signatures

This protocol does not allow a type to specify an exact signature for the callable - a specific number of parameters with specific types. If we went down that route, the best approach would be to introduce a new declaration kind (which would end up being very similar to get-only subscripts) since, in general, a type could want multiple concrete callable signatures, and those signatures should participate in overload resolution.

While such a feature could be interesting for some use cases, it is almost entirely orthogonal from this proposal: it addresses different use cases and does not solve the needs of this proposal. It does not address our needs because even a variadic callable declaration would not provide access to the keyword argument labels we need.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#direct-language-support-for-python>Direct language support for Python

We considered implementing something analogous to the Clang importer for Python, which would add a first class Python specific type(s) to Swift language or standard library. We rejected this option because it would be significantly more invasive in the compiler, would set the precedent for all other dynamic languages to get first class language support, and because that first class support doesn't substantially improve the experience of working with Python over existing Swift with a couple small "generally useful" extensions like this one.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#naming>Naming

The most fertile ground for bikeshedding is the naming of the protocol and the members. We welcome other ideas and suggestions for naming, but here are some thoughts on obvious options to consider:

We considered but rejected the name CustomCallable, because the existing Custom* protocols in the standard library (CustomStringConvertible, CustomReflectable, etc) provide a way to override and custom existing builtin abilities of Swift. In contrast, this feature grants a new capability to a type.

We considered but rejected a name like ExpressibleByCalling to fit with the ExpressibleBy* family of protocols (like ExpressibleByFloatLiteral, ExpressibleByStringLiteral, etc). This name family is specifically used by literal syntax, and calls are not literals. Additionally the type itself is not "expressible by calling" - instead, instances of the type may be called.

On member and associated type naming, we intentionally gave these long and verbose names so they stay out of the way of user code completion. The members of this protocol are really just compiler interoperability glue. If there was a Swift attribute to disable the members from showing up in code completion, we would use it (such an attribute would also be useful for the LiteralConvertible and other compiler magic protocols).

I don't like the idea of some calls having wildly different semantics from others; it's difficult enough to tell what exactly a call might be doing already. Since we also lack the more obvious static "Callable" protocol idea to give even well-typed call syntax to user-defined types, this also seems like it'd be easily abused for that purpose too.

I think a much better general solution to the problem of "make dynamic systems interact with type systems" is something like F#'s type providers which lets you write your own importers that look at dynamic information from a database, dynamic language VM, or some other system and generate type information usable by the compiler. Integration at the importer level could let you produce more well-typed Swift declarations by looking at the runtime information you get by importing a Python module.

-Joe

···

On Nov 10, 2017, at 9:37 AM, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

Hello all,

I have a couple of proposals cooking in a quest to make Swift interoperate with dynamically typed languages like Python better. Instead of baking in hard coded support for one language or the other, I’m preferring to add a few small but general purpose capabilities to Swift. This is the first, which allows a Swift type to become “callable”.

The proposal is here:
https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d

I’ve also attached a snapshot below, but it will drift out of date as the proposal is refined. Feedback and thoughts are appreciated, thanks!

-Chris

Introduce user-defined dynamically "callable" types

  • Proposal: SE-NNNN
  • Author: Chris Lattner
  • Review Manager: TBD
  • Status: Awaiting implementation
Introduction

This proposal introduces a new DynamicCallable protocol to the standard library. Types that conform to it are "callable" with the function call syntax. It is simple syntactic sugar which allows the user to write:

    a = someValue(keyword1: 42, "foo", keyword2: 19)
and have it be interpreted by the compiler as:

  a = someValue.dynamicCall(arguments
: [
    (
"keyword1", 42), ("", "foo"), ("keyword2", 19
)
  ])

Other languages have analogous features (e.g. Python "callables"), but the primary motivation of this proposal is to allow elegant and natural interoperation with dynamic languages in Swift.

Swift-evolution thread: Discussion thread topic for that proposal

Motivation

Swift is well known for being exceptional at interworking with existing C and Objective-C APIs, but its support for calling APIs written in scripting langauges like Python, Perl, and Ruby is quite lacking. These languages provide an extremely dynamic programming model where almost everything is discovered at runtime.

Through the introduction of this proposal, and the related DynamicMemberLookupProtocol proposal, we seek to fix this problem. We believe we can make many common APIs feel very natural to use directly from Swift without all the complexity of implementing something like the Clang importer. For example, consider this Python code:

class Dog
:
    
def __init__(self, name
):
        
self.name =
name
        
self.tricks = [] # creates a new empty list for each dog

def add_trick(self, trick
):
        
self.tricks.append(trick)
we would like to be able to use this from Swift like this (the comments show the corresponding syntax you would use in Python):

  // import DogModule
  // import DogModule.Dog as Dog // an alternate
  let Dog = Python.import(“DogModule.Dog")

  // dog = Dog("Brianna")
  let dog = Dog("Brianna")

  // dog.add_trick("Roll over")
  dog.add_trick("Roll over")

  // dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog("Kaylee").add_trick("snore")
Of course, this would also apply to standard Python APIs as well. Here is an example working with the Python pickleAPI and the builtin Python function open:

  // import pickle
  let pickle = Python.import("pickle"
)

// file = open(filename)
  let file = Python.open
(filename)

// blob = file.read()
  let blob = file.read
()

// result = pickle.loads(blob)
  let result = pickle.loads(blob)
This can all be expressed today as library functionality written in Swift, but without this proposal, the code required is unnecessarily verbose and gross. Without it (but with the related dynamic member lookup proposal) the code would have a method name (like call) all over the code:

  // import pickle
  let pickle = Python.import("pickle") // normal method in Swift, no change.

// file = open(filename)
  let file = Python.open.call
(filename)

// blob = file.read()
  let blob = file.read.call
()

// result = pickle.loads(blob)
  let result = pickle.loads.call
(blob)

// dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog.call("Kaylee").add_trick.call("snore")
While this is a syntactic sugar proposal, we believe that this expands Swift to be usable in important new domains. This sort of capability is also highly precedented in other languages, and is a generally useful language feature that could be used for other purposes as well.

Proposed solution

We propose introducing this protocol to the standard library:

protocol DynamicCallable
{
  
associatedtype DynamicCallableArgument

associatedtype DynamicCallableResult

func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws ->
DynamicCallableResult
}

It also extends the language such that function call syntax - when applied to a value of DynamicCallable type - is accepted and transformed into a call to the dynamicCall member. The dynamicCall method takes a list of tuples: the first element is the keyword label (or an empty string if absent) and the second value is the formal parameter specified at the call site.

Before this proposal, the Swift language has two types that participate in call syntax: functions and metatypes (for initialization). Neither of those may conform to protocols at the moment, so this introduces no possible ambiguity into the language.

It is worth noting that this does not introduce the ability to provide dynamicly callable static/class members. We don't believe that this is important given the goal of supporting dynamic languages like Python, but if there is a usecase discovered in the future, it could be explored as future work. Such future work should keep in mind that call syntax on metatypes is already meaningful, and that ambiguity would have to be resolved somehow.

Discussion

While the signature for dynamicCall is highly general we expect the most common use will be clients who are programming against concrete types that implement this proposal. One very nice aspect of this is that, as a result of Swift's existing subtyping mechanics, implementations of this type can choose whether they can actually throw an error or not. For example, consider this silly implementation:

struct ParameterSummer : DynamicCallable
{
  
func dynamicCall(arguments: [(String, Int)]) -> Int
{
    
return arguments.reduce(0) { $0+$1
.1 }
  }
}

let x = ParameterSummer
()

print(x(1, 7, 12)) // prints 20
Because ParameterSummer's implementation of dynamicCall does not throw, the call site is known not to throw either, so the print doesn't need to be marked with try.

Example Usage

A more realistic (and motivating) example comes from a prototype Python interop layer. While the concrete details of this use case are subject to change and not important for this proposal, it is perhaps useful to have a concrete example to see how this comes together.

That prototype currently has two types which model Python values, one of which handles Python exceptions and one of which does not. Their conformances would look like this, enabling the use cases described in the Motivation section above:

extension ThrowingPyRef: DynamicCallable
{
  
func dynamicCall(arguments: [(String, PythonConvertible)]) throws

->
PythonConvertible {
    
// Make sure state errors are not around.
    assert(PyErr_Occurred() == nil, "Python threw an error but wasn't handled"
)

// Count how many keyword arguments are in the list.
    let numKeywords = arguments.reduce(0
) {
      
$0 + ($1.0.isEmpty ? 0 : 1
)
    }

let kwdict = numKeywords != 0 ? PyDict_New() : nil

// Non-keyword arguments are passed as a tuple of values.
    let argTuple = PyTuple_New(arguments.count-numKeywords)!

var nonKeywordIndex = 0

for (keyword, argValue) in
arguments {
      
if keyword.isEmpty
{
        
PyTuple_SetItem(argTuple, nonKeywordIndex, argValue.toPython
())
        nonKeywordIndex
+= 1

      }
else
{
        
PyDict_SetItem(kwdict!, keyword.toPython(), argValue.toPython
())
      }
    }

// Python calls always return a non-null value when successful. If the
    // Python function produces the equivalent of C "void", it returns the None
    // value. A null result of PyObjectCall happens when there is an error,
    // like 'self' not being a Python callable.
    guard let resultPtr = PyObject_Call(state, argTuple, kwdict) else
{
      
throw PythonError.invalidCall(self
)
    }

let result = PyRef(owned
: resultPtr)

// Translate a Python exception into a Swift error if one was thrown.
    if let exception = PyErr_Occurred
() {
      
PyErr_Clear
()
      
throw PythonError.exception(PyRef(borrowed
: exception))
    }

return
result
  }
}

extension PyRef: DynamicCallable
{
  
func dynamicCall(arguments: [(String
, PythonConvertible)])
      
->
PythonConvertible {
    
// Same as above, but internally aborts instead of throwing Swift
    // errors.
  }
}

Source compatibility

This is a strictly additive proposal with no source breaking changes.

Effect on ABI stability

This is a strictly additive proposal with no ABI breaking changes.

Effect on API resilience

This has no impact on API resilience which is not already captured by other language features.

Alternatives considered

A few alternatives were considered:

Add ability to reject parameter labels

The implementation above does not allow an implementation to staticly reject argument labels. If this was important to add, we could add another protocol to model this, along the lines of:

/// A type conforming just to this protocol would not accept parameter
/// labels in its calls.
protocol DynamicCallable
{
  
associatedtype DynamicCallableArgument

associatedtype DynamicCallableResult

func dynamicCall(arguments: [DynamicCallableArgument]) throws ->
DynamicCallableResult
}

/// A type conforming to this protocol does allow optional parameter
/// labels.
protocol DynamicCallableWithKeywordsToo : DynamicCallable
{
  
func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws ->
DynamicCallableResult
}

This would allow a type to implement one or the other based on their capabilities. This proposal is going with a very simple design, but if there is demand for this, the author is happy to switch.

Staticly checking for exact signatures

This protocol does not allow a type to specify an exact signature for the callable - a specific number of parameters with specific types. If we went down that route, the best approach would be to introduce a new declaration kind (which would end up being very similar to get-only subscripts) since, in general, a type could want multiple concrete callable signatures, and those signatures should participate in overload resolution.

While such a feature could be interesting for some use cases, it is almost entirely orthogonal from this proposal: it addresses different use cases and does not solve the needs of this proposal. It does not address our needs because even a variadic callable declaration would not provide access to the keyword argument labels we need.

Direct language support for Python

We considered implementing something analogous to the Clang importer for Python, which would add a first class Python specific type(s) to Swift language or standard library. We rejected this option because it would be significantly more invasive in the compiler, would set the precedent for all other dynamic languages to get first class language support, and because that first class support doesn't substantially improve the experience of working with Python over existing Swift with a couple small "generally useful" extensions like this one.

Naming

The most fertile ground for bikeshedding is the naming of the protocol and the members. We welcome other ideas and suggestions for naming, but here are some thoughts on obvious options to consider:

We considered but rejected the name CustomCallable, because the existing Custom* protocols in the standard library (CustomStringConvertible, CustomReflectable, etc) provide a way to override and custom existing builtin abilities of Swift. In contrast, this feature grants a new capability to a type.

We considered but rejected a name like ExpressibleByCalling to fit with the ExpressibleBy* family of protocols (like ExpressibleByFloatLiteral, ExpressibleByStringLiteral, etc). This name family is specifically used by literal syntax, and calls are not literals. Additionally the type itself is not "expressible by calling" - instead, instances of the type may be called.

On member and associated type naming, we intentionally gave these long and verbose names so they stay out of the way of user code completion. The members of this protocol are really just compiler interoperability glue. If there was a Swift attribute to disable the members from showing up in code completion, we would use it (such an attribute would also be useful for the LiteralConvertible and other compiler magic protocols).

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Hi Chris,

Hello all,

I have a couple of proposals cooking in a quest to make Swift interoperate with dynamically typed languages like Python better. Instead of baking in hard coded support for one language or the other, I’m preferring to add a few small but general purpose capabilities to Swift. This is the first, which allows a Swift type to become “callable”.

I’m generally in favor of adding new features if they simplify the language model or subsume existing special cases, either moving them into the standard library or simplifying their implementation. However this proposal looks like a strictly additive feature, which introduces yet another kind of callable thing. We already have:

- Statically dispatched functions
- VTable-dispatched class methods
- Witness table dispatched protocol methods
- ObjC methods
- Dynamic method dispatch on AnyObject
- Enum case constructors
- Curried functions and various thunks, etc

I don’t see the new dynamic callable you are proposing replacing or generalizing any of the above, it will simply be a whole new code path.

This all comes at a great cost. If you look at the implementation of calls in lib/SILGen/SILGenApply.cpp you will see there is a great deal of complexity there to deal with all the different special cases. The type checker also has a lot of complexity related to method calls and member accesses.

I would be against adding yet another new type of call to the language at this point. Even if the implementation is relatively small it is yet another special case we have to maintain forever.

Swift is well known for being exceptional at interworking with existing C and Objective-C APIs, but its support for calling APIs written in scripting langauges like Python, Perl, and Ruby is quite lacking. These languages provide an extremely dynamic programming model where almost everything is discovered at runtime.

Most other statically compiled languages don’t attempt to solve this problem of interoperating with Python and Ruby either. I’m not sure this is a feature users expect or one that should be prioritized, given all the other work that remains in the implementation that will actually improve the day to day experience of developers.

We propose introducing this protocol to the standard library:

protocol DynamicCallable {
  associatedtype DynamicCallableArgument
  associatedtype DynamicCallableResult

  func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws -> DynamicCallableResult
}

This is not really very general at all, because it assumes all arguments have the same type, along with all results. Why would arguments and results have different types if they’re type erased anyway? And why are string keyword names privileged in any way? What about varargs, inout parameters and other Swift-specific modifiers on calls?

Before this proposal, the Swift language has two types that participate in call syntax: functions and metatypes (for initialization). Neither of those may conform to protocols at the moment, so this introduces no possible ambiguity into the language.

However, it rules out a possible type checker optimization — currently we can assume that if a type variable is the subject of both a conformance constraint and an apply constraint, the constraint system is invalid. With your change we could not make this assumption anymore.

It is worth noting that this does not introduce the ability to provide dynamicly callable static/class members. We don't believe that this is important given the goal of supporting dynamic languages like Python, but if there is a usecase discovered in the future, it could be explored as future work. Such future work should keep in mind that call syntax on metatypes is already meaningful, and that ambiguity would have to be resolved somehow.

This is the problem with this proposal — it solves a very narrow use case, but it introduces a piece of ABI that we have to maintain forever. When future generalizations are discovered, we will have to add yet another mechanism, complicating the language model further.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#discussion>
Effect on ABI stability

This is a strictly additive proposal with no ABI breaking changes.

However, it adds *new* ABI which we will have to maintain forever.

A strong -1 from me here.

Slava

···

On Nov 10, 2017, at 9:37 AM, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

1 Like

Leaving the "should we do it at all" question aside…

We propose introducing this protocol to the standard library:

protocol DynamicCallable {
  associatedtype DynamicCallableArgument
  associatedtype DynamicCallableResult

  func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws -> DynamicCallableResult
}

It also extends the language such that function call syntax - when applied to a value of DynamicCallable type - is accepted and transformed into a call to the dynamicCall member. The dynamicCall method takes a list of tuples: the first element is the keyword label (or an empty string if absent) and the second value is the formal parameter specified at the call site.

I don't really like passing the argument names to the parameter list. Instead, I would prefer to see them handled in the member-resolution method. You don't give a design for DynamicMemberLookupProtocol here, but assuming that you're imagining that a call like this:

  foreignObject.someValue(keyword1: 42, "foo", keyword2: 19)

Would translate to something like this:

  foreignObject.dynamicMethodLookup(name: "someValue").dynamicCall(arguments: [("keyword1", 42), ("", "foo"), ("keyword2", 19)])

I would instead like it to look like this:

  foreignObject.dynamicMethodLookup(name: "someValue", arguments: ["keyword1", "", "keyword2"]).dynamicCall(arguments: [42, "foo", 19])

This would better suit languages like Squeak (or Swift!) where you cannot fully look up a method without its keywords. (Languages like Python would simply store the keywords inside the object returned by `dynamicMethodLookup`.) It would also allow you to use our "get-a-closure" syntax:

  let uncalledMethod = foreignObject.someValue(keyword1:_:keyword2:)

···

On Nov 10, 2017, at 9:37 AM, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

--
Brent Royal-Gordon
Architechies

I generally dislike „protocol magic“, but I think in this case, there is also a precedence for an (imho) better way:
There’s no „Subscriptable“ protocol, and I would prefer callable types to be handled in the same way (afaics, subscripts are already used as „callable with different braces“ today in some cases).

You mention subscripts in „Alternatives considered“, but without an explanation why you don’t want a new declaration kind for dynamicCall — and I think it could co-exist with the orthogonal use case that you mention.

- Tino

Example Usage

Swift is quite flexible on what can act as a closure — would it be allowed to use a dynamic callable in that context?
I guess forwarding of

let closure: ([(String, Int)]) -> Int = DynamicCallableType()

to the dynamicCall method of DynamicCallableType isn’t that hard, but wouldn’t it be odd if the value of closure changes when you leave out its type?

HI, this proposal looks really interesting!

I have a few questions:

*Clarity on the proposal's intent*
*Nice cheap bridges, or lowering barriers to bridging?*

I can see this providing a nice quick interface to Python from Swift, but
I'm not sure if the exposed interface will be very Swifty (you probably
have a much better idea of what is Swifty :wink: than I do though). It seems
you want it to be possible for everything to be dynamically exposed, I've
used similar with Lua's meta methods, and I found it to be very powerful,
you could basically implement inheritance in the language, which wasn't
necessarily a good thing in retrospect.

Is it common for the the argument labels in other languages to be open
ended, or are labels typically finite? If the answer is finite, why not use
a Swift method as the wrapper?
Do you want duck typing, and would it be better to expose this via a
protocol?

It seems like in almost every case you could do something like this:

func myMethod<X: PythonConvertible & CanQuack, Y: PythonConvertible>(a: X?
= nil, b: Y) {

    pythonBridge.call("myMethod", arguments: ["a": X, "b": Y])

}

It might be good to add some *use-cases* (a popular Python library perhaps)
to the proposal where this type of bridge would be insufficient :).

It seems like this proposal pushes the responsibility of Swifty-ness and
type-safety to the caller. At some point you'll have to write a type-safe
bridging layer, or write your entire program in non-Swifty code ("The most
obvious way to write code should also behave in a safe manner"). Is the
main goal to *lower the barrier to Python* and other dynamic languages? or
is it to provide a cheap nice Swifty bridge? I have the above concerns
about the latter.

*Alternative sugar*

Ruby has Keyword Arguments for similar sugar:

*def* foo(regular, hash={})
    *puts* "the hash: #{hash}"

I'm sure you're aware of it, but I'll explain for completeness, any
trailing argument labels are stored and passed as a hash:

foo(regular, bar: "hello", bas: 123) *# outputs 'the hash: [**bar:
"hello", bas: 123]'*

Have you considered an alternative like this? For example:

func myMethod(regular: Int, store: @argcapture [String: PythonConvertible])
-> PythonConvertible

I'm sure you have good reasons, it might make the implementation bleed out
into other parts of the codebase. It would be good to include it in the
proposal *alternatives* section though. At the moment most of the
"alternatives" in the proposal just seem to be extensions to the same
solution :slight_smile:

*Clarity*
*Perhaps just that things are more clear to me now*

If my extrapolation is correct a user will implement a single type that
will allow a subset of a good portion of another language to be exposed
(object method and property bridging). I'm guessing that the dynamic member
proposal you're planning will not work with methods, it will require a
property, I think this helps explain some of the motivations. It might be
nice to have a more complete example that includes dynamic members. I
didn't find it clear from the proposal that it would only be necessary to
implement this protocol once per language.

Thanks for considering my questions,
Andrew Bennett

···

On Sat, Nov 11, 2017 at 4:37 AM, Chris Lattner via swift-evolution < swift-evolution@swift.org> wrote:

Hello all,

I have a couple of proposals cooking in a quest to make Swift interoperate
with dynamically typed languages like Python better. Instead of baking in
hard coded support for one language or the other, I’m preferring to add a
few small but general purpose capabilities to Swift. This is the first,
which allows a Swift type to become “callable”.

The proposal is here:
https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d

I’ve also attached a snapshot below, but it will drift out of date as the
proposal is refined. Feedback and thoughts are appreciated, thanks!

-Chris

Introduce user-defined dynamically "callable" types

   - Proposal: SE-NNNN
   <https://gist.github.com/lattner/NNNN-DynamicCallable.md>
   - Author: Chris Lattner <https://github.com/lattner>
   - Review Manager: TBD
   - Status: Awaiting implementation

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#introduction>
Introduction

This proposal introduces a new DynamicCallable protocol to the standard
library. Types that conform to it are "callable" with the function call
syntax. It is simple syntactic sugar which allows the user to write:

    a = someValue(keyword1: 42, "foo", keyword2: 19)

and have it be interpreted by the compiler as:

  a = someValue.dynamicCall(arguments: [
    ("keyword1", 42), ("", "foo"), ("keyword2", 19)
  ])

Other languages have analogous features (e.g. Python "callables"), but the
primary motivation of this proposal is to allow elegant and natural
interoperation with dynamic languages in Swift.

Swift-evolution thread: Discussion thread topic for that proposal
<https://lists.swift.org/pipermail/swift-evolution/>

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#motivation>
Motivation

Swift is well known for being exceptional at interworking with existing C
and Objective-C APIs, but its support for calling APIs written in scripting
langauges like Python, Perl, and Ruby is quite lacking. These languages
provide an extremely dynamic programming model where almost everything is
discovered at runtime.

Through the introduction of this proposal, and the related
DynamicMemberLookupProtocol proposal, we seek to fix this problem. We
believe we can make many common APIs feel very natural to use directly from
Swift without all the complexity of implementing something like the Clang
importer. For example, consider this Python code:

class Dog:
    def __init__(self, name):
        self.name = name
        self.tricks = [] # creates a new empty list for each dog

    def add_trick(self, trick):
        self.tricks.append(trick)

we would like to be able to use this from Swift like this (the comments
show the corresponding syntax you would use in Python):

  // import DogModule // import DogModule.Dog as Dog // an alternate let Dog = Python.import(“DogModule.Dog") // dog = Dog("Brianna") let dog = Dog("Brianna") // dog.add_trick("Roll over") dog.add_trick("Roll over") // dog2 = Dog("Kaylee").add_trick("snore") let dog2 = Dog("Kaylee").add_trick("snore")

Of course, this would also apply to standard Python APIs as well. Here is
an example working with the Python pickleAPI and the builtin Python
function open:

  // import pickle let pickle = Python.import("pickle")

  // file = open(filename) let file = Python.open(filename)

  // blob = file.read() let blob = file.read()

  // result = pickle.loads(blob) let result = pickle.loads(blob)

This can all be expressed today as library functionality written in Swift,
but without this proposal, the code required is unnecessarily verbose and
gross. Without it (but with the related dynamic member lookup proposal) the
code would have a method name (like call) all over the code:

  // import pickle let pickle = Python.import("pickle") // normal method in Swift, no change.
  // file = open(filename) let file = Python.open.call(filename)

  // blob = file.read() let blob = file.read.call()

  // result = pickle.loads(blob) let result = pickle.loads.call(blob)

  // dog2 = Dog("Kaylee").add_trick("snore") let dog2 = Dog.call("Kaylee").add_trick.call("snore")

While this is a syntactic sugar proposal, we believe that this expands
Swift to be usable in important new domains. This sort of capability is
also highly precedented in other languages, and is a generally useful
language feature that could be used for other purposes as well.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#proposed-solution>Proposed
solution

We propose introducing this protocol to the standard library:

protocol DynamicCallable {
  associatedtype DynamicCallableArgument
  associatedtype DynamicCallableResult

  func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws -> DynamicCallableResult
}

It also extends the language such that function call syntax - when applied
to a value of DynamicCallable type - is accepted and transformed into a
call to the dynamicCall member. The dynamicCall method takes a list of
tuples: the first element is the keyword label (or an empty string if
absent) and the second value is the formal parameter specified at the call
site.

Before this proposal, the Swift language has two types that participate in
call syntax: functions and metatypes (for initialization). Neither of those
may conform to protocols at the moment, so this introduces no possible
ambiguity into the language.

It is worth noting that this does not introduce the ability to provide
dynamicly callable static/class members. We don't believe that this is
important given the goal of supporting dynamic languages like Python, but
if there is a usecase discovered in the future, it could be explored as
future work. Such future work should keep in mind that call syntax on
metatypes is already meaningful, and that ambiguity would have to be
resolved somehow.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#discussion>
Discussion

While the signature for dynamicCall is highly general we expect the most
common use will be clients who are programming against concrete types that
implement this proposal. One very nice aspect of this is that, as a result
of Swift's existing subtyping mechanics, implementations of this type can
choose whether they can actually throw an error or not. For example,
consider this silly implementation:

struct ParameterSummer : DynamicCallable {
  func dynamicCall(arguments: [(String, Int)]) -> Int {
    return arguments.reduce(0) { $0+$1.1 }
  }
}
let x = ParameterSummer()print(x(1, 7, 12)) // prints 20

Because ParameterSummer's implementation of dynamicCall does not throw,
the call site is known not to throw either, so the print doesn't need to be
marked with try.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#example-usage>Example
Usage

A more realistic (and motivating) example comes from a prototype Python
interop layer. While the concrete details of this use case are subject to
change and not important for this proposal, it is perhaps useful to have a
concrete example to see how this comes together.

That prototype currently has two types which model Python values, one of
which handles Python exceptions and one of which does not. Their
conformances would look like this, enabling the use cases described in the
Motivation section above:

extension ThrowingPyRef: DynamicCallable {
  func dynamicCall(arguments: [(String, PythonConvertible)]) throws
      -> PythonConvertible {
    // Make sure state errors are not around. assert(PyErr_Occurred() == nil, "Python threw an error but wasn't handled")

    // Count how many keyword arguments are in the list. let numKeywords = arguments.reduce(0) {
      $0 + ($1.0.isEmpty ? 0 : 1)
    }

    let kwdict = numKeywords != 0 ? PyDict_New() : nil

    // Non-keyword arguments are passed as a tuple of values. let argTuple = PyTuple_New(arguments.count-numKeywords)!
    var nonKeywordIndex = 0
    for (keyword, argValue) in arguments {
      if keyword.isEmpty {
        PyTuple_SetItem(argTuple, nonKeywordIndex, argValue.toPython())
        nonKeywordIndex += 1
      } else {
        PyDict_SetItem(kwdict!, keyword.toPython(), argValue.toPython())
      }
    }

    // Python calls always return a non-null value when successful. If the // Python function produces the equivalent of C "void", it returns the None // value. A null result of PyObjectCall happens when there is an error, // like 'self' not being a Python callable. guard let resultPtr = PyObject_Call(state, argTuple, kwdict) else {
      throw PythonError.invalidCall(self)
    }

    let result = PyRef(owned: resultPtr)

    // Translate a Python exception into a Swift error if one was thrown. if let exception = PyErr_Occurred() {
      PyErr_Clear()
      throw PythonError.exception(PyRef(borrowed: exception))
    }

    return result
  }
}
extension PyRef: DynamicCallable {
  func dynamicCall(arguments: [(String, PythonConvertible)])
      -> PythonConvertible {
    // Same as above, but internally aborts instead of throwing Swift // errors. }
}

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#source-compatibility>Source
compatibility

This is a strictly additive proposal with no source breaking changes.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#effect-on-abi-stability>Effect
on ABI stability

This is a strictly additive proposal with no ABI breaking changes.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#effect-on-api-resilience>Effect
on API resilience

This has no impact on API resilience which is not already captured by
other language features.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#alternatives-considered>Alternatives
considered

A few alternatives were considered:

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#add-ability-to-reject-parameter-labels>Add
ability to reject parameter labels

The implementation above does not allow an implementation to staticly
reject argument labels. If this was important to add, we could add another
protocol to model this, along the lines of:

/// A type conforming just to this protocol would not accept parameter/// labels in its calls.protocol DynamicCallable {
  associatedtype DynamicCallableArgument
  associatedtype DynamicCallableResult

  func dynamicCall(arguments: [DynamicCallableArgument]) throws -> DynamicCallableResult
}
/// A type conforming to this protocol does allow optional parameter/// labels.protocol DynamicCallableWithKeywordsToo : DynamicCallable {
  func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws -> DynamicCallableResult
}

This would allow a type to implement one or the other based on their
capabilities. This proposal is going with a very simple design, but if
there is demand for this, the author is happy to switch.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#staticly-checking-for-exact-signatures>Staticly
checking for exact signatures

This protocol does not allow a type to specify an exact signature for the
callable - a specific number of parameters with specific types. If we went
down that route, the best approach would be to introduce a new declaration
kind (which would end up being very similar to get-only subscripts) since,
in general, a type could want multiple concrete callable signatures, and
those signatures should participate in overload resolution.

While such a feature could be interesting for some use cases, it is almost
entirely orthogonal from this proposal: it addresses different use cases
and does not solve the needs of this proposal. It does not address our
needs because even a variadic callable declaration would not provide access
to the keyword argument labels we need.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#direct-language-support-for-python>Direct
language support for Python

We considered implementing something analogous to the Clang importer for
Python, which would add a first class Python specific type(s) to Swift
language or standard library. We rejected this option because it would be
significantly more invasive in the compiler, would set the precedent for
all other dynamic languages to get first class language support, and
because that first class support doesn't substantially improve the
experience of working with Python over existing Swift with a couple small
"generally useful" extensions like this one.
<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#naming>
Naming

The most fertile ground for bikeshedding is the naming of the protocol and
the members. We welcome other ideas and suggestions for naming, but here
are some thoughts on obvious options to consider:

We considered but rejected the name CustomCallable, because the existing
Custom* protocols in the standard library (CustomStringConvertible, Cust
omReflectable, etc) provide a way to override and custom existing builtin
abilities of Swift. In contrast, this feature grants a new capability to a
type.

We considered but rejected a name like ExpressibleByCalling to fit with
the ExpressibleBy* family of protocols (like ExpressibleByFloatLiteral,
ExpressibleByStringLiteral, etc). This name family is specifically used
by literal syntax, and calls are not literals. Additionally the type itself
is not "expressible by calling" - instead, instances of the type may be
called.
On member and associated type naming, we intentionally gave these long and
verbose names so they stay out of the way of user code completion. The
members of this protocol are really just compiler interoperability glue. If
there was a Swift attribute to disable the members from showing up in code
completion, we would use it (such an attribute would also be useful for the
LiteralConvertible and other compiler magic protocols).

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Hello Chris, I have some questions about this passage:

Before this proposal, the Swift language has two types that participate in call syntax: functions and metatypes (for initialization). Neither of those may conform to protocols at the moment, so this introduces no possible ambiguity into the language.
Can you shortly describe why it would be ambiguous if metatypes would conform to protocols?

If metatypes were allowed to conform to protocols, how would this affect your proposal?

Last year I pitched a the idea to make metatypes conform to Hashable which seemed to be a welcome change.

···

Am 10. November 2017 um 18:37:26, Chris Lattner via swift-evolution (swift-evolution@swift.org) schrieb:

Hello all,

I have a couple of proposals cooking in a quest to make Swift interoperate with dynamically typed languages like Python better. Instead of baking in hard coded support for one language or the other, I’m preferring to add a few small but general purpose capabilities to Swift. This is the first, which allows a Swift type to become “callable”.

The proposal is here:
https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d

I’ve also attached a snapshot below, but it will drift out of date as the proposal is refined. Feedback and thoughts are appreciated, thanks!

-Chris

Introduce user-defined dynamically "callable" types
Proposal: SE-NNNN
Author: Chris Lattner
Review Manager: TBD
Status: Awaiting implementation
Introduction

This proposal introduces a new DynamicCallable protocol to the standard library. Types that conform to it are "callable" with the function call syntax. It is simple syntactic sugar which allows the user to write:

    a = someValue(keyword1: 42, "foo", keyword2: 19)
and have it be interpreted by the compiler as:

  a = someValue.dynamicCall(arguments: [
    ("keyword1", 42), ("", "foo"), ("keyword2", 19)
  ])
Other languages have analogous features (e.g. Python "callables"), but the primary motivation of this proposal is to allow elegant and natural interoperation with dynamic languages in Swift.

Swift-evolution thread: Discussion thread topic for that proposal

Motivation

Swift is well known for being exceptional at interworking with existing C and Objective-C APIs, but its support for calling APIs written in scripting langauges like Python, Perl, and Ruby is quite lacking. These languages provide an extremely dynamic programming model where almost everything is discovered at runtime.

Through the introduction of this proposal, and the related DynamicMemberLookupProtocol proposal, we seek to fix this problem. We believe we can make many common APIs feel very natural to use directly from Swift without all the complexity of implementing something like the Clang importer. For example, consider this Python code:

class Dog:
    def __init__(self, name):
        self.name = name
        self.tricks = [] # creates a new empty list for each dog
         
    def add_trick(self, trick):
        self.tricks.append(trick)
we would like to be able to use this from Swift like this (the comments show the corresponding syntax you would use in Python):

  // import DogModule
  // import DogModule.Dog as Dog // an alternate
  let Dog = Python.import(“DogModule.Dog")

  // dog = Dog("Brianna")
  let dog = Dog("Brianna")

  // dog.add_trick("Roll over")
  dog.add_trick("Roll over")

  // dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog("Kaylee").add_trick("snore")
Of course, this would also apply to standard Python APIs as well. Here is an example working with the Python pickleAPI and the builtin Python function open:

  // import pickle
  let pickle = Python.import("pickle")

  // file = open(filename)
  let file = Python.open(filename)

  // blob = file.read()
  let blob = file.read()

  // result = pickle.loads(blob)
  let result = pickle.loads(blob)
This can all be expressed today as library functionality written in Swift, but without this proposal, the code required is unnecessarily verbose and gross. Without it (but with the related dynamic member lookup proposal) the code would have a method name (like call) all over the code:

  // import pickle
  let pickle = Python.import("pickle") // normal method in Swift, no change.

  // file = open(filename)
  let file = Python.open.call(filename)

  // blob = file.read()
  let blob = file.read.call()

  // result = pickle.loads(blob)
  let result = pickle.loads.call(blob)

  // dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog.call("Kaylee").add_trick.call("snore")
While this is a syntactic sugar proposal, we believe that this expands Swift to be usable in important new domains. This sort of capability is also highly precedented in other languages, and is a generally useful language feature that could be used for other purposes as well.

Proposed solution

We propose introducing this protocol to the standard library:

protocol DynamicCallable {
  associatedtype DynamicCallableArgument
  associatedtype DynamicCallableResult

  func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws -> DynamicCallableResult
}
It also extends the language such that function call syntax - when applied to a value of DynamicCallable type - is accepted and transformed into a call to the dynamicCall member. The dynamicCall method takes a list of tuples: the first element is the keyword label (or an empty string if absent) and the second value is the formal parameter specified at the call site.

Before this proposal, the Swift language has two types that participate in call syntax: functions and metatypes (for initialization). Neither of those may conform to protocols at the moment, so this introduces no possible ambiguity into the language.

It is worth noting that this does not introduce the ability to provide dynamicly callable static/class members. We don't believe that this is important given the goal of supporting dynamic languages like Python, but if there is a usecase discovered in the future, it could be explored as future work. Such future work should keep in mind that call syntax on metatypes is already meaningful, and that ambiguity would have to be resolved somehow.

Discussion

While the signature for dynamicCall is highly general we expect the most common use will be clients who are programming against concrete types that implement this proposal. One very nice aspect of this is that, as a result of Swift's existing subtyping mechanics, implementations of this type can choose whether they can actually throw an error or not. For example, consider this silly implementation:

struct ParameterSummer : DynamicCallable {
  func dynamicCall(arguments: [(String, Int)]) -> Int {
    return arguments.reduce(0) { $0+$1.1 }
  }
}

let x = ParameterSummer()
print(x(1, 7, 12)) // prints 20
Because ParameterSummer's implementation of dynamicCall does not throw, the call site is known not to throw either, so the print doesn't need to be marked with try.

Example Usage

A more realistic (and motivating) example comes from a prototype Python interop layer. While the concrete details of this use case are subject to change and not important for this proposal, it is perhaps useful to have a concrete example to see how this comes together.

That prototype currently has two types which model Python values, one of which handles Python exceptions and one of which does not. Their conformances would look like this, enabling the use cases described in the Motivation section above:

extension ThrowingPyRef: DynamicCallable {
  func dynamicCall(arguments: [(String, PythonConvertible)]) throws
      -> PythonConvertible {
    // Make sure state errors are not around.
    assert(PyErr_Occurred() == nil, "Python threw an error but wasn't handled")

    // Count how many keyword arguments are in the list.
    let numKeywords = arguments.reduce(0) {
      $0 + ($1.0.isEmpty ? 0 : 1)
    }

    let kwdict = numKeywords != 0 ? PyDict_New() : nil

    // Non-keyword arguments are passed as a tuple of values.
    let argTuple = PyTuple_New(arguments.count-numKeywords)!
    var nonKeywordIndex = 0
    for (keyword, argValue) in arguments {
      if keyword.isEmpty {
        PyTuple_SetItem(argTuple, nonKeywordIndex, argValue.toPython())
        nonKeywordIndex += 1
      } else {
        PyDict_SetItem(kwdict!, keyword.toPython(), argValue.toPython())
      }
    }

    // Python calls always return a non-null value when successful. If the
    // Python function produces the equivalent of C "void", it returns the None
    // value. A null result of PyObjectCall happens when there is an error,
    // like 'self' not being a Python callable.
    guard let resultPtr = PyObject_Call(state, argTuple, kwdict) else {
      throw PythonError.invalidCall(self)
    }

    let result = PyRef(owned: resultPtr)

    // Translate a Python exception into a Swift error if one was thrown.
    if let exception = PyErr_Occurred() {
      PyErr_Clear()
      throw PythonError.exception(PyRef(borrowed: exception))
    }

    return result
  }
}

extension PyRef: DynamicCallable {
  func dynamicCall(arguments: [(String, PythonConvertible)])
      -> PythonConvertible {
    // Same as above, but internally aborts instead of throwing Swift
    // errors.
  }
}
Source compatibility

This is a strictly additive proposal with no source breaking changes.

Effect on ABI stability

This is a strictly additive proposal with no ABI breaking changes.

Effect on API resilience

This has no impact on API resilience which is not already captured by other language features.

Alternatives considered

A few alternatives were considered:

Add ability to reject parameter labels

The implementation above does not allow an implementation to staticly reject argument labels. If this was important to add, we could add another protocol to model this, along the lines of:

/// A type conforming just to this protocol would not accept parameter
/// labels in its calls.
protocol DynamicCallable {
  associatedtype DynamicCallableArgument
  associatedtype DynamicCallableResult

  func dynamicCall(arguments: [DynamicCallableArgument]) throws -> DynamicCallableResult
}

/// A type conforming to this protocol does allow optional parameter
/// labels.
protocol DynamicCallableWithKeywordsToo : DynamicCallable {
  func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws -> DynamicCallableResult
}
This would allow a type to implement one or the other based on their capabilities. This proposal is going with a very simple design, but if there is demand for this, the author is happy to switch.

Staticly checking for exact signatures

This protocol does not allow a type to specify an exact signature for the callable - a specific number of parameters with specific types. If we went down that route, the best approach would be to introduce a new declaration kind (which would end up being very similar to get-only subscripts) since, in general, a type could want multiple concrete callable signatures, and those signatures should participate in overload resolution.

While such a feature could be interesting for some use cases, it is almost entirely orthogonal from this proposal: it addresses different use cases and does not solve the needs of this proposal. It does not address our needs because even a variadic callable declaration would not provide access to the keyword argument labels we need.

Direct language support for Python

We considered implementing something analogous to the Clang importer for Python, which would add a first class Python specific type(s) to Swift language or standard library. We rejected this option because it would be significantly more invasive in the compiler, would set the precedent for all other dynamic languages to get first class language support, and because that first class support doesn't substantially improve the experience of working with Python over existing Swift with a couple small "generally useful" extensions like this one.

Naming

The most fertile ground for bikeshedding is the naming of the protocol and the members. We welcome other ideas and suggestions for naming, but here are some thoughts on obvious options to consider:

We considered but rejected the name CustomCallable, because the existing Custom* protocols in the standard library (CustomStringConvertible, CustomReflectable, etc) provide a way to override and custom existing builtin abilities of Swift. In contrast, this feature grants a new capability to a type.

We considered but rejected a name like ExpressibleByCalling to fit with the ExpressibleBy* family of protocols (like ExpressibleByFloatLiteral, ExpressibleByStringLiteral, etc). This name family is specifically used by literal syntax, and calls are not literals. Additionally the type itself is not "expressible by calling" - instead, instances of the type may be called.

On member and associated type naming, we intentionally gave these long and verbose names so they stay out of the way of user code completion. The members of this protocol are really just compiler interoperability glue. If there was a Swift attribute to disable the members from showing up in code completion, we would use it (such an attribute would also be useful for the LiteralConvertible and other compiler magic protocols).

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

To close the loop on this pitch, I’ve gotten a lot of useful feedback from the discussion. I’ll prepare a revised version and do pitch #2 when I have time, perhaps later this week. Thanks!

-Chris

···

On Nov 10, 2017, at 9:37 AM, Chris Lattner <sabre@nondot.org> wrote:

Hello all,

I have a couple of proposals cooking in a quest to make Swift interoperate with dynamically typed languages like Python better. Instead of baking in hard coded support for one language or the other, I’m preferring to add a few small but general purpose capabilities to Swift. This is the first, which allows a Swift type to become “callable”.

The proposal is here:
https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d

I’ve also attached a snapshot below, but it will drift out of date as the proposal is refined. Feedback and thoughts are appreciated, thanks!

-Chris

Introduce user-defined dynamically "callable" types

Proposal: SE-NNNN <https://gist.github.com/lattner/NNNN-DynamicCallable.md>
Author: Chris Lattner <https://github.com/lattner>
Review Manager: TBD
Status: Awaiting implementation
<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#introduction>Introduction

This proposal introduces a new DynamicCallable protocol to the standard library. Types that conform to it are "callable" with the function call syntax. It is simple syntactic sugar which allows the user to write:

    a = someValue(keyword1: 42, "foo", keyword2: 19)
and have it be interpreted by the compiler as:

  a = someValue.dynamicCall(arguments: [
    ("keyword1", 42), ("", "foo"), ("keyword2", 19)
  ])
Other languages have analogous features (e.g. Python "callables"), but the primary motivation of this proposal is to allow elegant and natural interoperation with dynamic languages in Swift.

Swift-evolution thread: Discussion thread topic for that proposal <https://lists.swift.org/pipermail/swift-evolution/>
<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#motivation>Motivation

Swift is well known for being exceptional at interworking with existing C and Objective-C APIs, but its support for calling APIs written in scripting langauges like Python, Perl, and Ruby is quite lacking. These languages provide an extremely dynamic programming model where almost everything is discovered at runtime.

Through the introduction of this proposal, and the related DynamicMemberLookupProtocol proposal, we seek to fix this problem. We believe we can make many common APIs feel very natural to use directly from Swift without all the complexity of implementing something like the Clang importer. For example, consider this Python code:

class Dog:
    def __init__(self, name):
        self.name = name
        self.tricks = [] # creates a new empty list for each dog
        
    def add_trick(self, trick):
        self.tricks.append(trick)
we would like to be able to use this from Swift like this (the comments show the corresponding syntax you would use in Python):

  // import DogModule
  // import DogModule.Dog as Dog // an alternate
  let Dog = Python.import(“DogModule.Dog")

  // dog = Dog("Brianna")
  let dog = Dog("Brianna")

  // dog.add_trick("Roll over")
  dog.add_trick("Roll over")

  // dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog("Kaylee").add_trick("snore")
Of course, this would also apply to standard Python APIs as well. Here is an example working with the Python pickleAPI and the builtin Python function open:

  // import pickle
  let pickle = Python.import("pickle")

  // file = open(filename)
  let file = Python.open(filename)

  // blob = file.read()
  let blob = file.read()

  // result = pickle.loads(blob)
  let result = pickle.loads(blob)
This can all be expressed today as library functionality written in Swift, but without this proposal, the code required is unnecessarily verbose and gross. Without it (but with the related dynamic member lookup proposal) the code would have a method name (like call) all over the code:

  // import pickle
  let pickle = Python.import("pickle") // normal method in Swift, no change.

  // file = open(filename)
  let file = Python.open.call(filename)

  // blob = file.read()
  let blob = file.read.call()

  // result = pickle.loads(blob)
  let result = pickle.loads.call(blob)

  // dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog.call("Kaylee").add_trick.call("snore")
While this is a syntactic sugar proposal, we believe that this expands Swift to be usable in important new domains. This sort of capability is also highly precedented in other languages, and is a generally useful language feature that could be used for other purposes as well.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#proposed-solution>Proposed solution

We propose introducing this protocol to the standard library:

protocol DynamicCallable {
  associatedtype DynamicCallableArgument
  associatedtype DynamicCallableResult

  func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws -> DynamicCallableResult
}
It also extends the language such that function call syntax - when applied to a value of DynamicCallable type - is accepted and transformed into a call to the dynamicCall member. The dynamicCall method takes a list of tuples: the first element is the keyword label (or an empty string if absent) and the second value is the formal parameter specified at the call site.

Before this proposal, the Swift language has two types that participate in call syntax: functions and metatypes (for initialization). Neither of those may conform to protocols at the moment, so this introduces no possible ambiguity into the language.

It is worth noting that this does not introduce the ability to provide dynamicly callable static/class members. We don't believe that this is important given the goal of supporting dynamic languages like Python, but if there is a usecase discovered in the future, it could be explored as future work. Such future work should keep in mind that call syntax on metatypes is already meaningful, and that ambiguity would have to be resolved somehow.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#discussion>Discussion

While the signature for dynamicCall is highly general we expect the most common use will be clients who are programming against concrete types that implement this proposal. One very nice aspect of this is that, as a result of Swift's existing subtyping mechanics, implementations of this type can choose whether they can actually throw an error or not. For example, consider this silly implementation:

struct ParameterSummer : DynamicCallable {
  func dynamicCall(arguments: [(String, Int)]) -> Int {
    return arguments.reduce(0) { $0+$1.1 }
  }
}

let x = ParameterSummer()
print(x(1, 7, 12)) // prints 20
Because ParameterSummer's implementation of dynamicCall does not throw, the call site is known not to throw either, so the print doesn't need to be marked with try.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#example-usage>Example Usage

A more realistic (and motivating) example comes from a prototype Python interop layer. While the concrete details of this use case are subject to change and not important for this proposal, it is perhaps useful to have a concrete example to see how this comes together.

That prototype currently has two types which model Python values, one of which handles Python exceptions and one of which does not. Their conformances would look like this, enabling the use cases described in the Motivation section above:

extension ThrowingPyRef: DynamicCallable {
  func dynamicCall(arguments: [(String, PythonConvertible)]) throws
      -> PythonConvertible {
    // Make sure state errors are not around.
    assert(PyErr_Occurred() == nil, "Python threw an error but wasn't handled")

    // Count how many keyword arguments are in the list.
    let numKeywords = arguments.reduce(0) {
      $0 + ($1.0.isEmpty ? 0 : 1)
    }

    let kwdict = numKeywords != 0 ? PyDict_New() : nil

    // Non-keyword arguments are passed as a tuple of values.
    let argTuple = PyTuple_New(arguments.count-numKeywords)!
    var nonKeywordIndex = 0
    for (keyword, argValue) in arguments {
      if keyword.isEmpty {
        PyTuple_SetItem(argTuple, nonKeywordIndex, argValue.toPython())
        nonKeywordIndex += 1
      } else {
        PyDict_SetItem(kwdict!, keyword.toPython(), argValue.toPython())
      }
    }

    // Python calls always return a non-null value when successful. If the
    // Python function produces the equivalent of C "void", it returns the None
    // value. A null result of PyObjectCall happens when there is an error,
    // like 'self' not being a Python callable.
    guard let resultPtr = PyObject_Call(state, argTuple, kwdict) else {
      throw PythonError.invalidCall(self)
    }

    let result = PyRef(owned: resultPtr)

    // Translate a Python exception into a Swift error if one was thrown.
    if let exception = PyErr_Occurred() {
      PyErr_Clear()
      throw PythonError.exception(PyRef(borrowed: exception))
    }

    return result
  }
}

extension PyRef: DynamicCallable {
  func dynamicCall(arguments: [(String, PythonConvertible)])
      -> PythonConvertible {
    // Same as above, but internally aborts instead of throwing Swift
    // errors.
  }
}
<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#source-compatibility>Source compatibility

This is a strictly additive proposal with no source breaking changes.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#effect-on-abi-stability>Effect on ABI stability

This is a strictly additive proposal with no ABI breaking changes.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#effect-on-api-resilience>Effect on API resilience

This has no impact on API resilience which is not already captured by other language features.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#alternatives-considered>Alternatives considered

A few alternatives were considered:

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#add-ability-to-reject-parameter-labels>Add ability to reject parameter labels

The implementation above does not allow an implementation to staticly reject argument labels. If this was important to add, we could add another protocol to model this, along the lines of:

/// A type conforming just to this protocol would not accept parameter
/// labels in its calls.
protocol DynamicCallable {
  associatedtype DynamicCallableArgument
  associatedtype DynamicCallableResult

  func dynamicCall(arguments: [DynamicCallableArgument]) throws -> DynamicCallableResult
}

/// A type conforming to this protocol does allow optional parameter
/// labels.
protocol DynamicCallableWithKeywordsToo : DynamicCallable {
  func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws -> DynamicCallableResult
}
This would allow a type to implement one or the other based on their capabilities. This proposal is going with a very simple design, but if there is demand for this, the author is happy to switch.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#staticly-checking-for-exact-signatures>Staticly checking for exact signatures

This protocol does not allow a type to specify an exact signature for the callable - a specific number of parameters with specific types. If we went down that route, the best approach would be to introduce a new declaration kind (which would end up being very similar to get-only subscripts) since, in general, a type could want multiple concrete callable signatures, and those signatures should participate in overload resolution.

While such a feature could be interesting for some use cases, it is almost entirely orthogonal from this proposal: it addresses different use cases and does not solve the needs of this proposal. It does not address our needs because even a variadic callable declaration would not provide access to the keyword argument labels we need.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#direct-language-support-for-python>Direct language support for Python

We considered implementing something analogous to the Clang importer for Python, which would add a first class Python specific type(s) to Swift language or standard library. We rejected this option because it would be significantly more invasive in the compiler, would set the precedent for all other dynamic languages to get first class language support, and because that first class support doesn't substantially improve the experience of working with Python over existing Swift with a couple small "generally useful" extensions like this one.

<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#naming>Naming

The most fertile ground for bikeshedding is the naming of the protocol and the members. We welcome other ideas and suggestions for naming, but here are some thoughts on obvious options to consider:

We considered but rejected the name CustomCallable, because the existing Custom* protocols in the standard library (CustomStringConvertible, CustomReflectable, etc) provide a way to override and custom existing builtin abilities of Swift. In contrast, this feature grants a new capability to a type.

We considered but rejected a name like ExpressibleByCalling to fit with the ExpressibleBy* family of protocols (like ExpressibleByFloatLiteral, ExpressibleByStringLiteral, etc). This name family is specifically used by literal syntax, and calls are not literals. Additionally the type itself is not "expressible by calling" - instead, instances of the type may be called.

On member and associated type naming, we intentionally gave these long and verbose names so they stay out of the way of user code completion. The members of this protocol are really just compiler interoperability glue. If there was a Swift attribute to disable the members from showing up in code completion, we would use it (such an attribute would also be useful for the LiteralConvertible and other compiler magic protocols).

I don't like the idea of some calls having wildly different semantics from others; it's difficult enough to tell what exactly a call might be doing already.

This isn’t particularly useful feedback. Can you elaborate more on what your concern is, and how calls are unlike anything else in Swift that would have this potential problem?

Since we also lack the more obvious static "Callable" protocol idea to give even well-typed call syntax to user-defined types, this also seems like it'd be easily abused for that purpose too.

Similarly, I’d love for you to elaborate on how the potential for abuse of this feature is different than anything else in Swift (e.g. operator overloading). Potential for abuse hasn’t been a particularly guiding force in the design on Swift, and for good reasons.

I also don’t understand what you mean by a static Callable protocol. I specifically address what I think you might mean in the “alternatives” part of the proposal, did you read that?

I think a much better general solution to the problem of "make dynamic systems interact with type systems" is something like F#'s type providers which lets you write your own importers that look at dynamic information from a database, dynamic language VM, or some other system and generate type information usable by the compiler.

Thanks! I wasn’t aware of type providers, I’ll investigate them.

Integration at the importer level could let you produce more well-typed Swift declarations by looking at the runtime information you get by importing a Python module.

This is also addressed directly in the proposal. I don’t want to add Python specific support to Swift. The motivations are explained in the proposal.

Do you have any actual feedback on the proposal?

-Chris

···

On Nov 10, 2017, at 10:03 AM, Joe Groff <jgroff@apple.com> wrote:

-Joe

On Nov 10, 2017, at 9:37 AM, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

Hello all,

I have a couple of proposals cooking in a quest to make Swift interoperate with dynamically typed languages like Python better. Instead of baking in hard coded support for one language or the other, I’m preferring to add a few small but general purpose capabilities to Swift. This is the first, which allows a Swift type to become “callable”.

The proposal is here:
https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d

I’ve also attached a snapshot below, but it will drift out of date as the proposal is refined. Feedback and thoughts are appreciated, thanks!

-Chris

Introduce user-defined dynamically "callable" types

  • Proposal: SE-NNNN
  • Author: Chris Lattner
  • Review Manager: TBD
  • Status: Awaiting implementation
Introduction

This proposal introduces a new DynamicCallable protocol to the standard library. Types that conform to it are "callable" with the function call syntax. It is simple syntactic sugar which allows the user to write:

   a = someValue(keyword1: 42, "foo", keyword2: 19)
and have it be interpreted by the compiler as:

a = someValue.dynamicCall(arguments
: [
   (
"keyword1", 42), ("", "foo"), ("keyword2", 19
)
])

Other languages have analogous features (e.g. Python "callables"), but the primary motivation of this proposal is to allow elegant and natural interoperation with dynamic languages in Swift.

Swift-evolution thread: Discussion thread topic for that proposal

Motivation

Swift is well known for being exceptional at interworking with existing C and Objective-C APIs, but its support for calling APIs written in scripting langauges like Python, Perl, and Ruby is quite lacking. These languages provide an extremely dynamic programming model where almost everything is discovered at runtime.

Through the introduction of this proposal, and the related DynamicMemberLookupProtocol proposal, we seek to fix this problem. We believe we can make many common APIs feel very natural to use directly from Swift without all the complexity of implementing something like the Clang importer. For example, consider this Python code:

class Dog
:

def __init__(self, name
):

self.name =
name

self.tricks = [] # creates a new empty list for each dog

def add_trick(self, trick
):

self.tricks.append(trick)
we would like to be able to use this from Swift like this (the comments show the corresponding syntax you would use in Python):

// import DogModule
// import DogModule.Dog as Dog // an alternate
let Dog = Python.import(“DogModule.Dog")

// dog = Dog("Brianna")
let dog = Dog("Brianna")

// dog.add_trick("Roll over")
dog.add_trick("Roll over")

// dog2 = Dog("Kaylee").add_trick("snore")
let dog2 = Dog("Kaylee").add_trick("snore")
Of course, this would also apply to standard Python APIs as well. Here is an example working with the Python pickleAPI and the builtin Python function open:

// import pickle
let pickle = Python.import("pickle"
)

// file = open(filename)
let file = Python.open
(filename)

// blob = file.read()
let blob = file.read
()

// result = pickle.loads(blob)
let result = pickle.loads(blob)
This can all be expressed today as library functionality written in Swift, but without this proposal, the code required is unnecessarily verbose and gross. Without it (but with the related dynamic member lookup proposal) the code would have a method name (like call) all over the code:

// import pickle
let pickle = Python.import("pickle") // normal method in Swift, no change.

// file = open(filename)
let file = Python.open.call
(filename)

// blob = file.read()
let blob = file.read.call
()

// result = pickle.loads(blob)
let result = pickle.loads.call
(blob)

// dog2 = Dog("Kaylee").add_trick("snore")
let dog2 = Dog.call("Kaylee").add_trick.call("snore")
While this is a syntactic sugar proposal, we believe that this expands Swift to be usable in important new domains. This sort of capability is also highly precedented in other languages, and is a generally useful language feature that could be used for other purposes as well.

Proposed solution

We propose introducing this protocol to the standard library:

protocol DynamicCallable
{

associatedtype DynamicCallableArgument

associatedtype DynamicCallableResult

func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws ->
DynamicCallableResult
}

It also extends the language such that function call syntax - when applied to a value of DynamicCallable type - is accepted and transformed into a call to the dynamicCall member. The dynamicCall method takes a list of tuples: the first element is the keyword label (or an empty string if absent) and the second value is the formal parameter specified at the call site.

Before this proposal, the Swift language has two types that participate in call syntax: functions and metatypes (for initialization). Neither of those may conform to protocols at the moment, so this introduces no possible ambiguity into the language.

It is worth noting that this does not introduce the ability to provide dynamicly callable static/class members. We don't believe that this is important given the goal of supporting dynamic languages like Python, but if there is a usecase discovered in the future, it could be explored as future work. Such future work should keep in mind that call syntax on metatypes is already meaningful, and that ambiguity would have to be resolved somehow.

Discussion

While the signature for dynamicCall is highly general we expect the most common use will be clients who are programming against concrete types that implement this proposal. One very nice aspect of this is that, as a result of Swift's existing subtyping mechanics, implementations of this type can choose whether they can actually throw an error or not. For example, consider this silly implementation:

struct ParameterSummer : DynamicCallable
{

func dynamicCall(arguments: [(String, Int)]) -> Int
{

return arguments.reduce(0) { $0+$1
.1 }
}
}

let x = ParameterSummer
()

print(x(1, 7, 12)) // prints 20
Because ParameterSummer's implementation of dynamicCall does not throw, the call site is known not to throw either, so the print doesn't need to be marked with try.

Example Usage

A more realistic (and motivating) example comes from a prototype Python interop layer. While the concrete details of this use case are subject to change and not important for this proposal, it is perhaps useful to have a concrete example to see how this comes together.

That prototype currently has two types which model Python values, one of which handles Python exceptions and one of which does not. Their conformances would look like this, enabling the use cases described in the Motivation section above:

extension ThrowingPyRef: DynamicCallable
{

func dynamicCall(arguments: [(String, PythonConvertible)]) throws

->
PythonConvertible {

// Make sure state errors are not around.
   assert(PyErr_Occurred() == nil, "Python threw an error but wasn't handled"
)

// Count how many keyword arguments are in the list.
   let numKeywords = arguments.reduce(0
) {

$0 + ($1.0.isEmpty ? 0 : 1
)
   }

let kwdict = numKeywords != 0 ? PyDict_New() : nil

// Non-keyword arguments are passed as a tuple of values.
   let argTuple = PyTuple_New(arguments.count-numKeywords)!

var nonKeywordIndex = 0

for (keyword, argValue) in
arguments {

if keyword.isEmpty
{

PyTuple_SetItem(argTuple, nonKeywordIndex, argValue.toPython
())
       nonKeywordIndex
+= 1

     }
else
{

PyDict_SetItem(kwdict!, keyword.toPython(), argValue.toPython
())
     }
   }

// Python calls always return a non-null value when successful. If the
   // Python function produces the equivalent of C "void", it returns the None
   // value. A null result of PyObjectCall happens when there is an error,
   // like 'self' not being a Python callable.
   guard let resultPtr = PyObject_Call(state, argTuple, kwdict) else
{

throw PythonError.invalidCall(self
)
   }

let result = PyRef(owned
: resultPtr)

// Translate a Python exception into a Swift error if one was thrown.
   if let exception = PyErr_Occurred
() {

PyErr_Clear
()

throw PythonError.exception(PyRef(borrowed
: exception))
   }

return
result
}
}

extension PyRef: DynamicCallable
{

func dynamicCall(arguments: [(String
, PythonConvertible)])

->
PythonConvertible {

// Same as above, but internally aborts instead of throwing Swift
   // errors.
}
}

Source compatibility

This is a strictly additive proposal with no source breaking changes.

Effect on ABI stability

This is a strictly additive proposal with no ABI breaking changes.

Effect on API resilience

This has no impact on API resilience which is not already captured by other language features.

Alternatives considered

A few alternatives were considered:

Add ability to reject parameter labels

The implementation above does not allow an implementation to staticly reject argument labels. If this was important to add, we could add another protocol to model this, along the lines of:

/// A type conforming just to this protocol would not accept parameter
/// labels in its calls.
protocol DynamicCallable
{

associatedtype DynamicCallableArgument

associatedtype DynamicCallableResult

func dynamicCall(arguments: [DynamicCallableArgument]) throws ->
DynamicCallableResult
}

/// A type conforming to this protocol does allow optional parameter
/// labels.
protocol DynamicCallableWithKeywordsToo : DynamicCallable
{

func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws ->
DynamicCallableResult
}

This would allow a type to implement one or the other based on their capabilities. This proposal is going with a very simple design, but if there is demand for this, the author is happy to switch.

Staticly checking for exact signatures

This protocol does not allow a type to specify an exact signature for the callable - a specific number of parameters with specific types. If we went down that route, the best approach would be to introduce a new declaration kind (which would end up being very similar to get-only subscripts) since, in general, a type could want multiple concrete callable signatures, and those signatures should participate in overload resolution.

While such a feature could be interesting for some use cases, it is almost entirely orthogonal from this proposal: it addresses different use cases and does not solve the needs of this proposal. It does not address our needs because even a variadic callable declaration would not provide access to the keyword argument labels we need.

Direct language support for Python

We considered implementing something analogous to the Clang importer for Python, which would add a first class Python specific type(s) to Swift language or standard library. We rejected this option because it would be significantly more invasive in the compiler, would set the precedent for all other dynamic languages to get first class language support, and because that first class support doesn't substantially improve the experience of working with Python over existing Swift with a couple small "generally useful" extensions like this one.

Naming

The most fertile ground for bikeshedding is the naming of the protocol and the members. We welcome other ideas and suggestions for naming, but here are some thoughts on obvious options to consider:

We considered but rejected the name CustomCallable, because the existing Custom* protocols in the standard library (CustomStringConvertible, CustomReflectable, etc) provide a way to override and custom existing builtin abilities of Swift. In contrast, this feature grants a new capability to a type.

We considered but rejected a name like ExpressibleByCalling to fit with the ExpressibleBy* family of protocols (like ExpressibleByFloatLiteral, ExpressibleByStringLiteral, etc). This name family is specifically used by literal syntax, and calls are not literals. Additionally the type itself is not "expressible by calling" - instead, instances of the type may be called.

On member and associated type naming, we intentionally gave these long and verbose names so they stay out of the way of user code completion. The members of this protocol are really just compiler interoperability glue. If there was a Swift attribute to disable the members from showing up in code completion, we would use it (such an attribute would also be useful for the LiteralConvertible and other compiler magic protocols).

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Object that are functions too, Amazing! I wanted that in Javascript for a while!

···

On Nov 10, 2017, 1:04 PM -0500, Joe Groff via swift-evolution <swift-evolution@swift.org>, wrote:

I don't like the idea of some calls having wildly different semantics from others; it's difficult enough to tell what exactly a call might be doing already. Since we also lack the more obvious static "Callable" protocol idea to give even well-typed call syntax to user-defined types, this also seems like it'd be easily abused for that purpose too.

I think a much better general solution to the problem of "make dynamic systems interact with type systems" is something like F#'s type providers which lets you write your own importers that look at dynamic information from a database, dynamic language VM, or some other system and generate type information usable by the compiler. Integration at the importer level could let you produce more well-typed Swift declarations by looking at the runtime information you get by importing a Python module.

-Joe

> On Nov 10, 2017, at 9:37 AM, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:
>
> Hello all,
>
> I have a couple of proposals cooking in a quest to make Swift interoperate with dynamically typed languages like Python better. Instead of baking in hard coded support for one language or the other, I’m preferring to add a few small but general purpose capabilities to Swift. This is the first, which allows a Swift type to become “callable”.
>
> The proposal is here:
> https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d
>
> I’ve also attached a snapshot below, but it will drift out of date as the proposal is refined. Feedback and thoughts are appreciated, thanks!
>
> -Chris
>
>
>
>
>
> Introduce user-defined dynamically "callable" types
>
> • Proposal: SE-NNNN
> • Author: Chris Lattner
> • Review Manager: TBD
> • Status: Awaiting implementation
> Introduction
>
> This proposal introduces a new DynamicCallable protocol to the standard library. Types that conform to it are "callable" with the function call syntax. It is simple syntactic sugar which allows the user to write:
>
> a = someValue(keyword1: 42, "foo", keyword2: 19)
> and have it be interpreted by the compiler as:
>
> a = someValue.dynamicCall(arguments
> : [
> (
> "keyword1", 42), ("", "foo"), ("keyword2", 19
> )
> ])
>
> Other languages have analogous features (e.g. Python "callables"), but the primary motivation of this proposal is to allow elegant and natural interoperation with dynamic languages in Swift.
>
> Swift-evolution thread: Discussion thread topic for that proposal
>
> Motivation
>
> Swift is well known for being exceptional at interworking with existing C and Objective-C APIs, but its support for calling APIs written in scripting langauges like Python, Perl, and Ruby is quite lacking. These languages provide an extremely dynamic programming model where almost everything is discovered at runtime.
>
> Through the introduction of this proposal, and the related DynamicMemberLookupProtocol proposal, we seek to fix this problem. We believe we can make many common APIs feel very natural to use directly from Swift without all the complexity of implementing something like the Clang importer. For example, consider this Python code:
>
> class Dog
> :
>
> def __init__(self, name
> ):
>
> self.name =
> name
>
> self.tricks = [] # creates a new empty list for each dog
>
>
>
> def add_trick(self, trick
> ):
>
> self.tricks.append(trick)
> we would like to be able to use this from Swift like this (the comments show the corresponding syntax you would use in Python):
>
> // import DogModule
> // import DogModule.Dog as Dog // an alternate
> let Dog = Python.import(“DogModule.Dog")
>
> // dog = Dog("Brianna")
> let dog = Dog("Brianna")
>
> // dog.add_trick("Roll over")
> dog.add_trick("Roll over")
>
> // dog2 = Dog("Kaylee").add_trick("snore")
> let dog2 = Dog("Kaylee").add_trick("snore")
> Of course, this would also apply to standard Python APIs as well. Here is an example working with the Python pickleAPI and the builtin Python function open:
>
> // import pickle
> let pickle = Python.import("pickle"
> )
>
>
> // file = open(filename)
> let file = Python.open
> (filename)
>
>
> // blob = file.read()
> let blob = file.read
> ()
>
>
> // result = pickle.loads(blob)
> let result = pickle.loads(blob)
> This can all be expressed today as library functionality written in Swift, but without this proposal, the code required is unnecessarily verbose and gross. Without it (but with the related dynamic member lookup proposal) the code would have a method name (like call) all over the code:
>
> // import pickle
> let pickle = Python.import("pickle") // normal method in Swift, no change.
>
>
> // file = open(filename)
> let file = Python.open.call
> (filename)
>
>
> // blob = file.read()
> let blob = file.read.call
> ()
>
>
> // result = pickle.loads(blob)
> let result = pickle.loads.call
> (blob)
>
>
> // dog2 = Dog("Kaylee").add_trick("snore")
> let dog2 = Dog.call("Kaylee").add_trick.call("snore")
> While this is a syntactic sugar proposal, we believe that this expands Swift to be usable in important new domains. This sort of capability is also highly precedented in other languages, and is a generally useful language feature that could be used for other purposes as well.
>
> Proposed solution
>
> We propose introducing this protocol to the standard library:
>
> protocol DynamicCallable
> {
>
> associatedtype DynamicCallableArgument
>
>
> associatedtype DynamicCallableResult
>
>
>
> func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws -
> DynamicCallableResult
> }
>
> It also extends the language such that function call syntax - when applied to a value of DynamicCallable type - is accepted and transformed into a call to the dynamicCall member. The dynamicCall method takes a list of tuples: the first element is the keyword label (or an empty string if absent) and the second value is the formal parameter specified at the call site.
>
> Before this proposal, the Swift language has two types that participate in call syntax: functions and metatypes (for initialization). Neither of those may conform to protocols at the moment, so this introduces no possible ambiguity into the language.
>
> It is worth noting that this does not introduce the ability to provide dynamicly callable static/class members. We don't believe that this is important given the goal of supporting dynamic languages like Python, but if there is a usecase discovered in the future, it could be explored as future work. Such future work should keep in mind that call syntax on metatypes is already meaningful, and that ambiguity would have to be resolved somehow.
>
> Discussion
>
> While the signature for dynamicCall is highly general we expect the most common use will be clients who are programming against concrete types that implement this proposal. One very nice aspect of this is that, as a result of Swift's existing subtyping mechanics, implementations of this type can choose whether they can actually throw an error or not. For example, consider this silly implementation:
>
> struct ParameterSummer : DynamicCallable
> {
>
> func dynamicCall(arguments: [(String, Int)]) -> Int
> {
>
> return arguments.reduce(0) { $0+$1
> .1 }
> }
> }
>
>
> let x = ParameterSummer
> ()
>
> print(x(1, 7, 12)) // prints 20
> Because ParameterSummer's implementation of dynamicCall does not throw, the call site is known not to throw either, so the print doesn't need to be marked with try.
>
> Example Usage
>
> A more realistic (and motivating) example comes from a prototype Python interop layer. While the concrete details of this use case are subject to change and not important for this proposal, it is perhaps useful to have a concrete example to see how this comes together.
>
> That prototype currently has two types which model Python values, one of which handles Python exceptions and one of which does not. Their conformances would look like this, enabling the use cases described in the Motivation section above:
>
> extension ThrowingPyRef: DynamicCallable
> {
>
> func dynamicCall(arguments: [(String, PythonConvertible)]) throws
>
>
> -
> PythonConvertible {
>
> // Make sure state errors are not around.
> assert(PyErr_Occurred() == nil, "Python threw an error but wasn't handled"
> )
>
>
> // Count how many keyword arguments are in the list.
> let numKeywords = arguments.reduce(0
> ) {
>
> $0 + ($1.0.isEmpty ? 0 : 1
> )
> }
>
>
> let kwdict = numKeywords != 0 ? PyDict_New() : nil
>
>
>
> // Non-keyword arguments are passed as a tuple of values.
> let argTuple = PyTuple_New(arguments.count-numKeywords)!
>
>
> var nonKeywordIndex = 0
>
>
> for (keyword, argValue) in
> arguments {
>
> if keyword.isEmpty
> {
>
> PyTuple_SetItem(argTuple, nonKeywordIndex, argValue.toPython
> ())
> nonKeywordIndex
> += 1
>
> }
> else
> {
>
> PyDict_SetItem(kwdict!, keyword.toPython(), argValue.toPython
> ())
> }
> }
>
>
> // Python calls always return a non-null value when successful. If the
> // Python function produces the equivalent of C "void", it returns the None
> // value. A null result of PyObjectCall happens when there is an error,
> // like 'self' not being a Python callable.
> guard let resultPtr = PyObject_Call(state, argTuple, kwdict) else
> {
>
> throw PythonError.invalidCall(self
> )
> }
>
>
> let result = PyRef(owned
> : resultPtr)
>
>
> // Translate a Python exception into a Swift error if one was thrown.
> if let exception = PyErr_Occurred
> () {
>
> PyErr_Clear
> ()
>
> throw PythonError.exception(PyRef(borrowed
> : exception))
> }
>
>
> return
> result
> }
> }
>
>
> extension PyRef: DynamicCallable
> {
>
> func dynamicCall(arguments: [(String
> , PythonConvertible)])
>
> -
> PythonConvertible {
>
> // Same as above, but internally aborts instead of throwing Swift
> // errors.
> }
> }
>
> Source compatibility
>
> This is a strictly additive proposal with no source breaking changes.
>
> Effect on ABI stability
>
> This is a strictly additive proposal with no ABI breaking changes.
>
> Effect on API resilience
>
> This has no impact on API resilience which is not already captured by other language features.
>
> Alternatives considered
>
> A few alternatives were considered:
>
> Add ability to reject parameter labels
>
> The implementation above does not allow an implementation to staticly reject argument labels. If this was important to add, we could add another protocol to model this, along the lines of:
>
> /// A type conforming just to this protocol would not accept parameter
> /// labels in its calls.
> protocol DynamicCallable
> {
>
> associatedtype DynamicCallableArgument
>
>
> associatedtype DynamicCallableResult
>
>
>
> func dynamicCall(arguments: [DynamicCallableArgument]) throws -
> DynamicCallableResult
> }
>
>
> /// A type conforming to this protocol does allow optional parameter
> /// labels.
> protocol DynamicCallableWithKeywordsToo : DynamicCallable
> {
>
> func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws -
> DynamicCallableResult
> }
>
> This would allow a type to implement one or the other based on their capabilities. This proposal is going with a very simple design, but if there is demand for this, the author is happy to switch.
>
> Staticly checking for exact signatures
>
> This protocol does not allow a type to specify an exact signature for the callable - a specific number of parameters with specific types. If we went down that route, the best approach would be to introduce a new declaration kind (which would end up being very similar to get-only subscripts) since, in general, a type could want multiple concrete callable signatures, and those signatures should participate in overload resolution.
>
> While such a feature could be interesting for some use cases, it is almost entirely orthogonal from this proposal: it addresses different use cases and does not solve the needs of this proposal. It does not address our needs because even a variadic callable declaration would not provide access to the keyword argument labels we need.
>
> Direct language support for Python
>
> We considered implementing something analogous to the Clang importer for Python, which would add a first class Python specific type(s) to Swift language or standard library. We rejected this option because it would be significantly more invasive in the compiler, would set the precedent for all other dynamic languages to get first class language support, and because that first class support doesn't substantially improve the experience of working with Python over existing Swift with a couple small "generally useful" extensions like this one.
>
> Naming
>
> The most fertile ground for bikeshedding is the naming of the protocol and the members. We welcome other ideas and suggestions for naming, but here are some thoughts on obvious options to consider:
>
> We considered but rejected the name CustomCallable, because the existing Custom* protocols in the standard library (CustomStringConvertible, CustomReflectable, etc) provide a way to override and custom existing builtin abilities of Swift. In contrast, this feature grants a new capability to a type.
>
> We considered but rejected a name like ExpressibleByCalling to fit with the ExpressibleBy* family of protocols (like ExpressibleByFloatLiteral, ExpressibleByStringLiteral, etc). This name family is specifically used by literal syntax, and calls are not literals. Additionally the type itself is not "expressible by calling" - instead, instances of the type may be called.
>
> On member and associated type naming, we intentionally gave these long and verbose names so they stay out of the way of user code completion. The members of this protocol are really just compiler interoperability glue. If there was a Swift attribute to disable the members from showing up in code completion, we would use it (such an attribute would also be useful for the LiteralConvertible and other compiler magic protocols).
>
>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

We already have that though, with the Objective-C bridge. How is the proposed behavior here more wildly different than the semantics of non-@objc, @objc, and @objc dynamic calls?

Charles

···

On Nov 10, 2017, at 12:04 PM, Joe Groff via swift-evolution <swift-evolution@swift.org> wrote:

I don't like the idea of some calls having wildly different semantics from others; it's difficult enough to tell what exactly a call might be doing already. Since we also lack the more obvious static "Callable" protocol idea to give even well-typed call syntax to user-defined types, this also seems like it'd be easily abused for that purpose too.

Hi Chris,

Hello all,

I have a couple of proposals cooking in a quest to make Swift interoperate with dynamically typed languages like Python better. Instead of baking in hard coded support for one language or the other, I’m preferring to add a few small but general purpose capabilities to Swift. This is the first, which allows a Swift type to become “callable”.

I’m generally in favor of adding new features if they simplify the language model or subsume existing special cases, either moving them into the standard library or simplifying their implementation.

Great!

However this proposal looks like a strictly additive feature,

It is. It is strictly sugar, as mentioned in the proposal.

which introduces yet another kind of callable thing. We already have:

- Statically dispatched functions
- VTable-dispatched class methods
- Witness table dispatched protocol methods
- ObjC methods
- Dynamic method dispatch on AnyObject
- Enum case constructors
- Curried functions and various thunks, etc

I don’t see the new dynamic callable you are proposing replacing or generalizing any of the above, it will simply be a whole new code path.

These things are completely different in character. They are not sugar: they are invasive changes that spread through the rest of the compiler. They are also (generally) not part of the user exposed semantic model for swift, they are implementation mechanics. This proposal if far less invasive, and also quite different than these.

This all comes at a great cost. If you look at the implementation of calls in lib/SILGen/SILGenApply.cpp you will see there is a great deal of complexity there to deal with all the different special cases. The type checker also has a lot of complexity related to method calls and member accesses.

I do not expect any SIL or IRGen changes associated with this proposal, just type checker changes. The type checker changes should be straight-forward, but you can evaluate that when there is a patch.

Swift is well known for being exceptional at interworking with existing C and Objective-C APIs, but its support for calling APIs written in scripting langauges like Python, Perl, and Ruby is quite lacking. These languages provide an extremely dynamic programming model where almost everything is discovered at runtime.

Most other statically compiled languages don’t attempt to solve this problem of interoperating with Python and Ruby either.

Our goal is for Swift to be awesome, not comparable to “other statically typed languages”.

I’m not sure this is a feature users expect or one that should be prioritized, given all the other work that remains in the implementation that will actually improve the day to day experience of developers.

swift-evolution isn’t about priorities. I’m not asking someone else to implement this, and you don’t tell me how I spend my engineering time :-) :-)

We propose introducing this protocol to the standard library:

protocol DynamicCallable {
  associatedtype DynamicCallableArgument
  associatedtype DynamicCallableResult

  func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws -> DynamicCallableResult
}

This is not really very general at all, because it assumes all arguments have the same type, along with all results.

Correct. This is important and common for interoperating with dynamic languages.

Why would arguments and results have different types if they’re type erased anyway?

I’m open to requiring these to be the same type if there is a benefit to doing so. What benefit do you see?

And why are string keyword names privileged in any way? What about varargs, inout parameters and other Swift-specific modifiers on calls?

Because dynamic languages support keywords, and this is all about reflecting APIs written in those languages into Swift. This API works fine for varargs. Dynamic languages do not support Swift specific keywords and (generally) do not support inout.

-Chris

···

On Nov 10, 2017, at 5:51 PM, Slava Pestov via swift-evolution <swift-evolution@swift.org> wrote:

On Nov 10, 2017, at 9:37 AM, Chris Lattner via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Leaving the "should we do it at all" question aside…

Thanks, most of the comments in this thread haven’t been constructive at all, I’m glad to get one that is :-)

We propose introducing this protocol to the standard library:

protocol DynamicCallable {
associatedtype DynamicCallableArgument
associatedtype DynamicCallableResult

func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws -> DynamicCallableResult
}

It also extends the language such that function call syntax - when applied to a value of DynamicCallable type - is accepted and transformed into a call to the dynamicCall member. The dynamicCall method takes a list of tuples: the first element is the keyword label (or an empty string if absent) and the second value is the formal parameter specified at the call site.

I don't really like passing the argument names to the parameter list. Instead, I would prefer to see them handled in the member-resolution method. You don't give a design for DynamicMemberLookupProtocol here, but assuming that you're imagining that a call like this:

  foreignObject.someValue(keyword1: 42, "foo", keyword2: 19)

Would translate to something like this:

  foreignObject.dynamicMethodLookup(name: "someValue").dynamicCall(arguments: [("keyword1", 42), ("", "foo"), ("keyword2", 19)])

Right, that was the idea.

I would instead like it to look like this:

  foreignObject.dynamicMethodLookup(name: "someValue", arguments: ["keyword1", "", "keyword2"]).dynamicCall(arguments: [42, "foo", 19])

This would better suit languages like Squeak (or Swift!) where you cannot fully look up a method without its keywords.

Good point.

(Languages like Python would simply store the keywords inside the object returned by `dynamicMethodLookup`.)

While I don’t care greatly about making Python efficient, doing this storage would be grossly inefficient and doesn’t have an obvious implementation. Python also supports currying, and global functions with keyword arguments, which means that:

  fn(keyword1: 42, "foo", keyword2: 19)

… needs to work, even if x didn’t come from Swift.

I guess we can solve both use cases by passing the keyword in both cases for a Swift style call. Your example would look like this:

foreignObject.dynamicMethodLookup(name: "someValue", arguments: ["keyword1", "", "keyword2"]).dynamicCall(arguments: [(“keyword1”, 42), (“”, “foo"), (“keyword2”: 19)])

The fn example above would look like this:

  fn.dynamicCall(arguments: [(“keyword1”, 42), (“”, “foo"), (“keyword2”: 19)]

And a member lookup with no call would simply pass no arguments.

It would also allow you to use our "get-a-closure" syntax:

  let uncalledMethod = foreignObject.someValue(keyword1:_:keyword2:)

Yep, that would pass the keywords to member lookup. They’d be ignored by Python but presumably used for Squeak.

-Chris

···

On Nov 10, 2017, at 9:10 PM, Brent Royal-Gordon <brent@architechies.com> wrote:

On Nov 10, 2017, at 9:37 AM, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

Sure, this is a great question. We could definitely introduce a new declaration kind. The tradeoff I see is that that is far more language invasive, and that none of our existing decls work this way: packaging up syntax (keyword labels) and handing it off.

One theoretical way to address this is to introduce a macro system, but that would be far more invasive and isn’t going to happen now (and perhaps never?).

Despite the claims, the impact of this proposal on the compiler is extremely narrow.

-Chris

···

On Nov 11, 2017, at 2:04 AM, Tino Heth via swift-evolution <swift-evolution@swift.org> wrote:

I generally dislike „protocol magic“, but I think in this case, there is also a precedence for an (imho) better way:
There’s no „Subscriptable“ protocol, and I would prefer callable types to be handled in the same way (afaics, subscripts are already used as „callable with different braces“ today in some cases).

You mention subscripts in „Alternatives considered“, but without an explanation why you don’t want a new declaration kind for dynamicCall — and I think it could co-exist with the orthogonal use case that you mention.

I’m not sure I understand what you’re getting at. Can you show how this would work with the example in the motivation section?

-Chris

···

On Nov 11, 2017, at 2:19 AM, Tino Heth via swift-evolution <swift-evolution@swift.org> wrote:

Example Usage

Swift is quite flexible on what can act as a closure — would it be allowed to use a dynamic callable in that context?
I guess forwarding of

let closure: ([(String, Int)]) -> Int = DynamicCallableType()

to the dynamicCall method of DynamicCallableType isn’t that hard, but wouldn’t it be odd if the value of closure changes when you leave out its type?

The obvious solution then is to make this proposal the general case and current calls a specialization of that.

Marcel

···

On Nov 10, 2017, at 19:04 , Joe Groff via swift-evolution <swift-evolution@swift.org> wrote:

I don't like the idea of some calls having wildly different semantics from others;

HI, this proposal looks really interesting!

Thanks!

Clarity on the proposal's intent
Nice cheap bridges, or lowering barriers to bridging?

The later: I’m not overly concerned about constant time performance: dynamic languages are often slow (though exceptions exist, e.g. Javascript). So if there is a beautiful approach that works great but is a bit slower, that is fine with me.

My desire is to enable access to “any” API in Python directly from Swift with natural syntax. This doesn’t mean it has to match Python of course: Python ranges and Swift ranges are syntactically very different, and Swift-in-Python should use Swift syntax. That said, there are a LOT of expression level similarities between Swift and Python, so things that can be the same should be.

This proposal isn’t the only way to achieve that, of course. I’m open to other suggestions.

I can see this providing a nice quick interface to Python from Swift, but I'm not sure if the exposed interface will be very Swifty (you probably have a much better idea of what is Swifty ;) than I do though). It seems you want it to be possible for everything to be dynamically exposed

Right: Python APIs are not “Swifty”. They do not use optionals or generics, use lowercase snake names, etc. IMO, that’s perfectly fine. Unlike the Objective-C interop in Swift - which aims to make legacy ObjC APIs feel Swifty, it is perfectly acceptable (and far less impact on the compiler) to just import Python directly into Swift. This is actually much better for the target audience anyway.

Is it common for the the argument labels in other languages to be open ended, or are labels typically finite? If the answer is finite, why not use a Swift method as the wrapper?

Python keyword arguments are open ended. They get passed to the Python API as a dictionary.

Do you want duck typing, and would it be better to expose this via a protocol?

I’m not sure how that can work, can you elaborate?

It seems like in almost every case you could do something like this:

func myMethod<X: PythonConvertible & CanQuack, Y: PythonConvertible>(a: X? = nil, b: Y) {
    pythonBridge.call("myMethod", arguments: ["a": X, "b": Y])
}

The currency type is PyVal (name TBD of course), just like AnyObject for Objective-C. I’m not sure what the code above is getting at.

It might be good to add some use-cases (a popular Python library perhaps) to the proposal where this type of bridge would be insufficient :).

There are lots of examples. In data science, NumPy and Pandas are two examples. The Python community is 1 or 2 orders of magnitude larger than Swift’s community and there is 25 years of code out there to interop with. The point of this is to make it all accessible.

It seems like this proposal pushes the responsibility of Swifty-ness and type-safety to the caller.

Yes, welcome to dynamic languages :-)

At some point you'll have to write a type-safe bridging layer, or write your entire program in non-Swifty code ("The most obvious way to write code should also behave in a safe manner"). Is the main goal to lower the barrier to Python and other dynamic languages? or is it to provide a cheap nice Swifty bridge? I have the above concerns about the latter.

It’s the later. Many Python APIs are already wrappers around C code, so if someone cares about investing a bunch of effort into making a great Swift API, it would generally make sense to wrap the C API in Swift, not wrap the Python API in Swift.

Alternative sugar

Ruby has Keyword Arguments for similar sugar:

def foo(regular, hash={})
    puts "the hash: #{hash}"

I'm sure you're aware of it, but I'll explain for completeness, any trailing argument labels are stored and passed as a hash:

foo(regular, bar: "hello", bas: 123) # outputs 'the hash: [bar: "hello", bas: 123]’

Python is similar, but allows them to be intermixed with positional arguments.

Have you considered an alternative like this? For example:

func myMethod(regular: Int, store: @argcapture [String: PythonConvertible]) -> PythonConvertible

I'm sure you have good reasons, it might make the implementation bleed out into other parts of the codebase. It would be good to include it in the proposal alternatives section though. At the moment most of the "alternatives" in the proposal just seem to be extensions to the same solution :)

I’m not sure what you’re getting at here. The system I’ve sketched out does not incorporate Python declarations, everything is dynamic. It is extremely simple. This is a huge feature :-) I think it is entirely non-scalable to do something like the ObjC importer for every language that Swift wants to interop with.

Clarity
Perhaps just that things are more clear to me now

If my extrapolation is correct a user will implement a single type that will allow a subset of a good portion of another language to be exposed (object method and property bridging). I'm guessing that the dynamic member proposal you're planning will not work with methods, it will require a property, I think this helps explain some of the motivations. It might be nice to have a more complete example that includes dynamic members. I didn't find it clear from the proposal that it would only be necessary to implement this protocol once per language.

I’m sorry, I don’t understand what you mean.

-Chris

···

On Nov 11, 2017, at 4:15 PM, Andrew Bennett <cacoyi@gmail.com> wrote:

In Java you can have annotation processors, user supplied compiler extensions, that are extensively used for making frameworks easier to use and for general interfacing to ‘foreign stuff’.

Would a limited form of user supplied compiler extension be an alternative, an import processor. Special import syntax, import beginning with ‘Foreign’, would be recognised by the compiler and a user supplied import processor would be called, in the case of the example for Python. The import processor would generate the glue code. Below is a basic example to give a feel for what I am proposing.

(Syntax, names, etc. in code below just the 1st that came into my head!)

  — Howard.

···

======================================================

import Foreign.Python.Foo // Special import tag `Foreign` calls a user supplied compiler extension that parses the imported Python and generates something along these lines (its up to the importer the exact code generated!):

protocol _ForeignPythonCallable {
    func call(method: Any, arguments: Any...) throws -> Any
}

extension _ForeignPythonCallable {
    func call(method: Any, arguments: Any...) throws -> Any {
        return "method: \(method), arguments: \(arguments)" // Method lookup and calling code for Python.
    }
}

class Foo: _ForeignPythonCallable {
    // Could override `call` if more efficient than complete general lookup was possible.
    enum _ForeignPythonMethodNames { // This might be a C struct so that it matches perfectly what the python interpreter is expecting.
        case bar
    }
    enum _ForeignPythonMethodBarArguments { // This might be a C struct so that it matches perfectly what the python interpreter is expecting.
        case x(Any)
        case noName1(Any)
        case y(Any)
    }
    func bar(x: Any, _ noName1: Any, y: Any) -> Any {
        do {
            return try call(method: _ForeignPythonMethodNames.bar, arguments: _ForeignPythonMethodBarArguments.x(x), _ForeignPythonMethodBarArguments.noName1(noName1), _ForeignPythonMethodBarArguments.y(y))
        } catch {
            fatalError("Method `bar` does not throw, therefore Python importer bug.")
        }
    }
}

// Then the consumer of `Foo` uses as per normal Swift:

let foo = Foo()
foo.bar(x: 1, 23, y: 17)

class TypedFoo: Foo { // For many dynamic languages this wouldn't be possible to automatically generate via the import, therefore would have to be hand written.
    func bar(x: Int, _ noName1: Int, y: Int) -> String {
        return super.bar(x: x, noName1, y: y) as! String
    }
}
let typedFoo = TypedFoo()
typedFoo.bar(x: 1, 23, y: 17)

On 12 Nov 2017, at 11:15 am, Andrew Bennett via swift-evolution <swift-evolution@swift.org> wrote:

HI, this proposal looks really interesting!

I have a few questions:

Clarity on the proposal's intent
Nice cheap bridges, or lowering barriers to bridging?

I can see this providing a nice quick interface to Python from Swift, but I'm not sure if the exposed interface will be very Swifty (you probably have a much better idea of what is Swifty :wink: than I do though). It seems you want it to be possible for everything to be dynamically exposed, I've used similar with Lua's meta methods, and I found it to be very powerful, you could basically implement inheritance in the language, which wasn't necessarily a good thing in retrospect.

Is it common for the the argument labels in other languages to be open ended, or are labels typically finite? If the answer is finite, why not use a Swift method as the wrapper?
Do you want duck typing, and would it be better to expose this via a protocol?

It seems like in almost every case you could do something like this:

func myMethod<X: PythonConvertible & CanQuack, Y: PythonConvertible>(a: X? = nil, b: Y) {
    pythonBridge.call("myMethod", arguments: ["a": X, "b": Y])
}

It might be good to add some use-cases (a popular Python library perhaps) to the proposal where this type of bridge would be insufficient :).

It seems like this proposal pushes the responsibility of Swifty-ness and type-safety to the caller. At some point you'll have to write a type-safe bridging layer, or write your entire program in non-Swifty code ("The most obvious way to write code should also behave in a safe manner"). Is the main goal to lower the barrier to Python and other dynamic languages? or is it to provide a cheap nice Swifty bridge? I have the above concerns about the latter.

Alternative sugar

Ruby has Keyword Arguments for similar sugar:

def foo(regular, hash={})
    puts "the hash: #{hash}"

I'm sure you're aware of it, but I'll explain for completeness, any trailing argument labels are stored and passed as a hash:

foo(regular, bar: "hello", bas: 123) # outputs 'the hash: [bar: "hello", bas: 123]'
Have you considered an alternative like this? For example:

func myMethod(regular: Int, store: @argcapture [String: PythonConvertible]) -> PythonConvertible

I'm sure you have good reasons, it might make the implementation bleed out into other parts of the codebase. It would be good to include it in the proposal alternatives section though. At the moment most of the "alternatives" in the proposal just seem to be extensions to the same solution :slight_smile:

Clarity
Perhaps just that things are more clear to me now

If my extrapolation is correct a user will implement a single type that will allow a subset of a good portion of another language to be exposed (object method and property bridging). I'm guessing that the dynamic member proposal you're planning will not work with methods, it will require a property, I think this helps explain some of the motivations. It might be nice to have a more complete example that includes dynamic members. I didn't find it clear from the proposal that it would only be necessary to implement this protocol once per language.

Thanks for considering my questions,
Andrew Bennett

On Sat, Nov 11, 2017 at 4:37 AM, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:
Hello all,

I have a couple of proposals cooking in a quest to make Swift interoperate with dynamically typed languages like Python better. Instead of baking in hard coded support for one language or the other, I’m preferring to add a few small but general purpose capabilities to Swift. This is the first, which allows a Swift type to become “callable”.

The proposal is here:
https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d

I’ve also attached a snapshot below, but it will drift out of date as the proposal is refined. Feedback and thoughts are appreciated, thanks!

-Chris

Introduce user-defined dynamically "callable" types
Proposal: SE-NNNN
Author: Chris Lattner
Review Manager: TBD
Status: Awaiting implementation
Introduction

This proposal introduces a new DynamicCallable protocol to the standard library. Types that conform to it are "callable" with the function call syntax. It is simple syntactic sugar which allows the user to write:

    a = someValue(keyword1: 42, "foo", keyword2: 19)
and have it be interpreted by the compiler as:

  a = someValue.dynamicCall(arguments: [
    ("keyword1", 42), ("", "foo"), ("keyword2", 19)
  ])
Other languages have analogous features (e.g. Python "callables"), but the primary motivation of this proposal is to allow elegant and natural interoperation with dynamic languages in Swift.

Swift-evolution thread: Discussion thread topic for that proposal

Motivation

Swift is well known for being exceptional at interworking with existing C and Objective-C APIs, but its support for calling APIs written in scripting langauges like Python, Perl, and Ruby is quite lacking. These languages provide an extremely dynamic programming model where almost everything is discovered at runtime.

Through the introduction of this proposal, and the related DynamicMemberLookupProtocol proposal, we seek to fix this problem. We believe we can make many common APIs feel very natural to use directly from Swift without all the complexity of implementing something like the Clang importer. For example, consider this Python code:

class Dog:
    def __init__(self, name):
        self.name = name
        self.tricks = [] # creates a new empty list for each dog
        
    def add_trick(self, trick):
        self.tricks.append(trick)
we would like to be able to use this from Swift like this (the comments show the corresponding syntax you would use in Python):

  // import DogModule
  // import DogModule.Dog as Dog // an alternate
  let Dog = Python.import(“DogModule.Dog")

  // dog = Dog("Brianna")
  let dog = Dog("Brianna")

  // dog.add_trick("Roll over")
  dog.add_trick("Roll over")

  // dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog("Kaylee").add_trick("snore")
Of course, this would also apply to standard Python APIs as well. Here is an example working with the Python pickleAPI and the builtin Python function open:

  // import pickle
  let pickle = Python.import("pickle")

  // file = open(filename)
  let file = Python.open(filename)

  // blob = file.read()
  let blob = file.read()

  // result = pickle.loads(blob)
  let result = pickle.loads(blob)
This can all be expressed today as library functionality written in Swift, but without this proposal, the code required is unnecessarily verbose and gross. Without it (but with the related dynamic member lookup proposal) the code would have a method name (like call) all over the code:

  // import pickle
  let pickle = Python.import("pickle") // normal method in Swift, no change.

  // file = open(filename)
  let file = Python.open.call(filename)

  // blob = file.read()
  let blob = file.read.call()

  // result = pickle.loads(blob)
  let result = pickle.loads.call(blob)

  // dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog.call("Kaylee").add_trick.call("snore")
While this is a syntactic sugar proposal, we believe that this expands Swift to be usable in important new domains. This sort of capability is also highly precedented in other languages, and is a generally useful language feature that could be used for other purposes as well.

Proposed solution

We propose introducing this protocol to the standard library:

protocol DynamicCallable {
  associatedtype DynamicCallableArgument
  associatedtype DynamicCallableResult

  func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws -> DynamicCallableResult
}
It also extends the language such that function call syntax - when applied to a value of DynamicCallable type - is accepted and transformed into a call to the dynamicCall member. The dynamicCall method takes a list of tuples: the first element is the keyword label (or an empty string if absent) and the second value is the formal parameter specified at the call site.

Before this proposal, the Swift language has two types that participate in call syntax: functions and metatypes (for initialization). Neither of those may conform to protocols at the moment, so this introduces no possible ambiguity into the language.

It is worth noting that this does not introduce the ability to provide dynamicly callable static/class members. We don't believe that this is important given the goal of supporting dynamic languages like Python, but if there is a usecase discovered in the future, it could be explored as future work. Such future work should keep in mind that call syntax on metatypes is already meaningful, and that ambiguity would have to be resolved somehow.

Discussion

While the signature for dynamicCall is highly general we expect the most common use will be clients who are programming against concrete types that implement this proposal. One very nice aspect of this is that, as a result of Swift's existing subtyping mechanics, implementations of this type can choose whether they can actually throw an error or not. For example, consider this silly implementation:

struct ParameterSummer : DynamicCallable {
  func dynamicCall(arguments: [(String, Int)]) -> Int {
    return arguments.reduce(0) { $0+$1.1 }
  }
}

let x = ParameterSummer()
print(x(1, 7, 12)) // prints 20
Because ParameterSummer's implementation of dynamicCall does not throw, the call site is known not to throw either, so the print doesn't need to be marked with try.

Example Usage

A more realistic (and motivating) example comes from a prototype Python interop layer. While the concrete details of this use case are subject to change and not important for this proposal, it is perhaps useful to have a concrete example to see how this comes together.

That prototype currently has two types which model Python values, one of which handles Python exceptions and one of which does not. Their conformances would look like this, enabling the use cases described in the Motivation section above:

extension ThrowingPyRef: DynamicCallable {
  func dynamicCall(arguments: [(String, PythonConvertible)]) throws
      -> PythonConvertible {
    // Make sure state errors are not around.
    assert(PyErr_Occurred() == nil, "Python threw an error but wasn't handled")

    // Count how many keyword arguments are in the list.
    let numKeywords = arguments.reduce(0) {
      $0 + ($1.0.isEmpty ? 0 : 1)
    }

    let kwdict = numKeywords != 0 ? PyDict_New() : nil

    // Non-keyword arguments are passed as a tuple of values.
    let argTuple = PyTuple_New(arguments.count-numKeywords)!
    var nonKeywordIndex = 0
    for (keyword, argValue) in arguments {
      if keyword.isEmpty {
        PyTuple_SetItem(argTuple, nonKeywordIndex, argValue.toPython())
        nonKeywordIndex += 1
      } else {
        PyDict_SetItem(kwdict!, keyword.toPython(), argValue.toPython())
      }
    }

    // Python calls always return a non-null value when successful. If the
    // Python function produces the equivalent of C "void", it returns the None
    // value. A null result of PyObjectCall happens when there is an error,
    // like 'self' not being a Python callable.
    guard let resultPtr = PyObject_Call(state, argTuple, kwdict) else {
      throw PythonError.invalidCall(self)
    }

    let result = PyRef(owned: resultPtr)

    // Translate a Python exception into a Swift error if one was thrown.
    if let exception = PyErr_Occurred() {
      PyErr_Clear()
      throw PythonError.exception(PyRef(borrowed: exception))
    }

    return result
  }
}

extension PyRef: DynamicCallable {
  func dynamicCall(arguments: [(String, PythonConvertible)])
      -> PythonConvertible {
    // Same as above, but internally aborts instead of throwing Swift
    // errors.
  }
}
Source compatibility

This is a strictly additive proposal with no source breaking changes.

Effect on ABI stability

This is a strictly additive proposal with no ABI breaking changes.

Effect on API resilience

This has no impact on API resilience which is not already captured by other language features.

Alternatives considered

A few alternatives were considered:

Add ability to reject parameter labels

The implementation above does not allow an implementation to staticly reject argument labels. If this was important to add, we could add another protocol to model this, along the lines of:

/// A type conforming just to this protocol would not accept parameter
/// labels in its calls.
protocol DynamicCallable {
  associatedtype DynamicCallableArgument
  associatedtype DynamicCallableResult

  func dynamicCall(arguments: [DynamicCallableArgument]) throws -> DynamicCallableResult
}

/// A type conforming to this protocol does allow optional parameter
/// labels.
protocol DynamicCallableWithKeywordsToo : DynamicCallable {
  func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws -> DynamicCallableResult
}
This would allow a type to implement one or the other based on their capabilities. This proposal is going with a very simple design, but if there is demand for this, the author is happy to switch.

Staticly checking for exact signatures

This protocol does not allow a type to specify an exact signature for the callable - a specific number of parameters with specific types. If we went down that route, the best approach would be to introduce a new declaration kind (which would end up being very similar to get-only subscripts) since, in general, a type could want multiple concrete callable signatures, and those signatures should participate in overload resolution.

While such a feature could be interesting for some use cases, it is almost entirely orthogonal from this proposal: it addresses different use cases and does not solve the needs of this proposal. It does not address our needs because even a variadic callable declaration would not provide access to the keyword argument labels we need.

Direct language support for Python

We considered implementing something analogous to the Clang importer for Python, which would add a first class Python specific type(s) to Swift language or standard library. We rejected this option because it would be significantly more invasive in the compiler, would set the precedent for all other dynamic languages to get first class language support, and because that first class support doesn't substantially improve the experience of working with Python over existing Swift with a couple small "generally useful" extensions like this one.

Naming

The most fertile ground for bikeshedding is the naming of the protocol and the members. We welcome other ideas and suggestions for naming, but here are some thoughts on obvious options to consider:

We considered but rejected the name CustomCallable, because the existing Custom* protocols in the standard library (CustomStringConvertible, CustomReflectable, etc) provide a way to override and custom existing builtin abilities of Swift. In contrast, this feature grants a new capability to a type.

We considered but rejected a name like ExpressibleByCalling to fit with the ExpressibleBy* family of protocols (like ExpressibleByFloatLiteral, ExpressibleByStringLiteral, etc). This name family is specifically used by literal syntax, and calls are not literals. Additionally the type itself is not "expressible by calling" - instead, instances of the type may be called.

On member and associated type naming, we intentionally gave these long and verbose names so they stay out of the way of user code completion. The members of this protocol are really just compiler interoperability glue. If there was a Swift attribute to disable the members from showing up in code completion, we would use it (such an attribute would also be useful for the LiteralConvertible and other compiler magic protocols).

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

I’m just asking if this should work :wink: — I didn’t see anything regarding it in the proposal.
But to take something from the document:

  // file = open(filename)
  let file = Python.open.call(filename)
let openFile: (String) -> FileHandle? = Python.open // is this be possible, or an error?

···

Am 11.11.2017 um 19:03 schrieb Chris Lattner <sabre@nondot.org>:

Swift is quite flexible on what can act as a closure — would it be allowed to use a dynamic callable in that context?
I guess forwarding of

let closure: ([(String, Int)]) -> Int = DynamicCallableType()

to the dynamicCall method of DynamicCallableType isn’t that hard, but wouldn’t it be odd if the value of closure changes when you leave out its type?

I’m not sure I understand what you’re getting at. Can you show how this would work with the example in the motivation section?

Terms of Service

Privacy Policy

Cookie Policy