Hello all,
I have a couple of proposals cooking in a quest to make Swift interoperate with dynamically typed languages like Python better. Instead of baking in hard coded support for one language or the other, I’m preferring to add a few small but general purpose capabilities to Swift. This is the first, which allows a Swift type to become “callable”.
The proposal is here:
https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d
I’ve also attached a snapshot below, but it will drift out of date as the proposal is refined. Feedback and thoughts are appreciated, thanks!
-Chris
Introduce user-defined dynamically "callable" types
Proposal: SE-NNNN <https://gist.github.com/lattner/NNNN-DynamicCallable.md>
Author: Chris Lattner <https://github.com/lattner>
Review Manager: TBD
Status: Awaiting implementation
<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#introduction>Introduction
This proposal introduces a new DynamicCallable protocol to the standard library. Types that conform to it are "callable" with the function call syntax. It is simple syntactic sugar which allows the user to write:
a = someValue(keyword1: 42, "foo", keyword2: 19)
and have it be interpreted by the compiler as:
a = someValue.dynamicCall(arguments: [
("keyword1", 42), ("", "foo"), ("keyword2", 19)
])
Other languages have analogous features (e.g. Python "callables"), but the primary motivation of this proposal is to allow elegant and natural interoperation with dynamic languages in Swift.
Swift-evolution thread: Discussion thread topic for that proposal <https://lists.swift.org/pipermail/swift-evolution/>
<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#motivation>Motivation
Swift is well known for being exceptional at interworking with existing C and Objective-C APIs, but its support for calling APIs written in scripting langauges like Python, Perl, and Ruby is quite lacking. These languages provide an extremely dynamic programming model where almost everything is discovered at runtime.
Through the introduction of this proposal, and the related DynamicMemberLookupProtocol proposal, we seek to fix this problem. We believe we can make many common APIs feel very natural to use directly from Swift without all the complexity of implementing something like the Clang importer. For example, consider this Python code:
class Dog:
def __init__(self, name):
self.name = name
self.tricks = # creates a new empty list for each dog
def add_trick(self, trick):
self.tricks.append(trick)
we would like to be able to use this from Swift like this (the comments show the corresponding syntax you would use in Python):
// import DogModule
// import DogModule.Dog as Dog // an alternate
let Dog = Python.import(“DogModule.Dog")
// dog = Dog("Brianna")
let dog = Dog("Brianna")
// dog.add_trick("Roll over")
dog.add_trick("Roll over")
// dog2 = Dog("Kaylee").add_trick("snore")
let dog2 = Dog("Kaylee").add_trick("snore")
Of course, this would also apply to standard Python APIs as well. Here is an example working with the Python pickleAPI and the builtin Python function open:
// import pickle
let pickle = Python.import("pickle")
// file = open(filename)
let file = Python.open(filename)
// blob = file.read()
let blob = file.read()
// result = pickle.loads(blob)
let result = pickle.loads(blob)
This can all be expressed today as library functionality written in Swift, but without this proposal, the code required is unnecessarily verbose and gross. Without it (but with the related dynamic member lookup proposal) the code would have a method name (like call) all over the code:
// import pickle
let pickle = Python.import("pickle") // normal method in Swift, no change.
// file = open(filename)
let file = Python.open.call(filename)
// blob = file.read()
let blob = file.read.call()
// result = pickle.loads(blob)
let result = pickle.loads.call(blob)
// dog2 = Dog("Kaylee").add_trick("snore")
let dog2 = Dog.call("Kaylee").add_trick.call("snore")
While this is a syntactic sugar proposal, we believe that this expands Swift to be usable in important new domains. This sort of capability is also highly precedented in other languages, and is a generally useful language feature that could be used for other purposes as well.
<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#proposed-solution>Proposed solution
We propose introducing this protocol to the standard library:
protocol DynamicCallable {
associatedtype DynamicCallableArgument
associatedtype DynamicCallableResult
func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws -> DynamicCallableResult
}
It also extends the language such that function call syntax - when applied to a value of DynamicCallable type - is accepted and transformed into a call to the dynamicCall member. The dynamicCall method takes a list of tuples: the first element is the keyword label (or an empty string if absent) and the second value is the formal parameter specified at the call site.
Before this proposal, the Swift language has two types that participate in call syntax: functions and metatypes (for initialization). Neither of those may conform to protocols at the moment, so this introduces no possible ambiguity into the language.
It is worth noting that this does not introduce the ability to provide dynamicly callable static/class members. We don't believe that this is important given the goal of supporting dynamic languages like Python, but if there is a usecase discovered in the future, it could be explored as future work. Such future work should keep in mind that call syntax on metatypes is already meaningful, and that ambiguity would have to be resolved somehow.
<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#discussion>Discussion
While the signature for dynamicCall is highly general we expect the most common use will be clients who are programming against concrete types that implement this proposal. One very nice aspect of this is that, as a result of Swift's existing subtyping mechanics, implementations of this type can choose whether they can actually throw an error or not. For example, consider this silly implementation:
struct ParameterSummer : DynamicCallable {
func dynamicCall(arguments: [(String, Int)]) -> Int {
return arguments.reduce(0) { $0+$1.1 }
}
}
let x = ParameterSummer()
print(x(1, 7, 12)) // prints 20
Because ParameterSummer's implementation of dynamicCall does not throw, the call site is known not to throw either, so the print doesn't need to be marked with try.
<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#example-usage>Example Usage
A more realistic (and motivating) example comes from a prototype Python interop layer. While the concrete details of this use case are subject to change and not important for this proposal, it is perhaps useful to have a concrete example to see how this comes together.
That prototype currently has two types which model Python values, one of which handles Python exceptions and one of which does not. Their conformances would look like this, enabling the use cases described in the Motivation section above:
extension ThrowingPyRef: DynamicCallable {
func dynamicCall(arguments: [(String, PythonConvertible)]) throws
-> PythonConvertible {
// Make sure state errors are not around.
assert(PyErr_Occurred() == nil, "Python threw an error but wasn't handled")
// Count how many keyword arguments are in the list.
let numKeywords = arguments.reduce(0) {
$0 + ($1.0.isEmpty ? 0 : 1)
}
let kwdict = numKeywords != 0 ? PyDict_New() : nil
// Non-keyword arguments are passed as a tuple of values.
let argTuple = PyTuple_New(arguments.count-numKeywords)!
var nonKeywordIndex = 0
for (keyword, argValue) in arguments {
if keyword.isEmpty {
PyTuple_SetItem(argTuple, nonKeywordIndex, argValue.toPython())
nonKeywordIndex += 1
} else {
PyDict_SetItem(kwdict!, keyword.toPython(), argValue.toPython())
}
}
// Python calls always return a non-null value when successful. If the
// Python function produces the equivalent of C "void", it returns the None
// value. A null result of PyObjectCall happens when there is an error,
// like 'self' not being a Python callable.
guard let resultPtr = PyObject_Call(state, argTuple, kwdict) else {
throw PythonError.invalidCall(self)
}
let result = PyRef(owned: resultPtr)
// Translate a Python exception into a Swift error if one was thrown.
if let exception = PyErr_Occurred() {
PyErr_Clear()
throw PythonError.exception(PyRef(borrowed: exception))
}
return result
}
}
extension PyRef: DynamicCallable {
func dynamicCall(arguments: [(String, PythonConvertible)])
-> PythonConvertible {
// Same as above, but internally aborts instead of throwing Swift
// errors.
}
}
<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#source-compatibility>Source compatibility
This is a strictly additive proposal with no source breaking changes.
<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#effect-on-abi-stability>Effect on ABI stability
This is a strictly additive proposal with no ABI breaking changes.
<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#effect-on-api-resilience>Effect on API resilience
This has no impact on API resilience which is not already captured by other language features.
<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#alternatives-considered>Alternatives considered
A few alternatives were considered:
<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#add-ability-to-reject-parameter-labels>Add ability to reject parameter labels
The implementation above does not allow an implementation to staticly reject argument labels. If this was important to add, we could add another protocol to model this, along the lines of:
/// A type conforming just to this protocol would not accept parameter
/// labels in its calls.
protocol DynamicCallable {
associatedtype DynamicCallableArgument
associatedtype DynamicCallableResult
func dynamicCall(arguments: [DynamicCallableArgument]) throws -> DynamicCallableResult
}
/// A type conforming to this protocol does allow optional parameter
/// labels.
protocol DynamicCallableWithKeywordsToo : DynamicCallable {
func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws -> DynamicCallableResult
}
This would allow a type to implement one or the other based on their capabilities. This proposal is going with a very simple design, but if there is demand for this, the author is happy to switch.
<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#staticly-checking-for-exact-signatures>Staticly checking for exact signatures
This protocol does not allow a type to specify an exact signature for the callable - a specific number of parameters with specific types. If we went down that route, the best approach would be to introduce a new declaration kind (which would end up being very similar to get-only subscripts) since, in general, a type could want multiple concrete callable signatures, and those signatures should participate in overload resolution.
While such a feature could be interesting for some use cases, it is almost entirely orthogonal from this proposal: it addresses different use cases and does not solve the needs of this proposal. It does not address our needs because even a variadic callable declaration would not provide access to the keyword argument labels we need.
<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#direct-language-support-for-python>Direct language support for Python
We considered implementing something analogous to the Clang importer for Python, which would add a first class Python specific type(s) to Swift language or standard library. We rejected this option because it would be significantly more invasive in the compiler, would set the precedent for all other dynamic languages to get first class language support, and because that first class support doesn't substantially improve the experience of working with Python over existing Swift with a couple small "generally useful" extensions like this one.
<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d#naming>Naming
The most fertile ground for bikeshedding is the naming of the protocol and the members. We welcome other ideas and suggestions for naming, but here are some thoughts on obvious options to consider:
We considered but rejected the name CustomCallable, because the existing Custom* protocols in the standard library (CustomStringConvertible, CustomReflectable, etc) provide a way to override and custom existing builtin abilities of Swift. In contrast, this feature grants a new capability to a type.
We considered but rejected a name like ExpressibleByCalling to fit with the ExpressibleBy* family of protocols (like ExpressibleByFloatLiteral, ExpressibleByStringLiteral, etc). This name family is specifically used by literal syntax, and calls are not literals. Additionally the type itself is not "expressible by calling" - instead, instances of the type may be called.
On member and associated type naming, we intentionally gave these long and verbose names so they stay out of the way of user code completion. The members of this protocol are really just compiler interoperability glue. If there was a Swift attribute to disable the members from showing up in code completion, we would use it (such an attribute would also be useful for the LiteralConvertible and other compiler magic protocols).