Hi everyone,
With the holidays and many other things behind us, the core team had a chance to talk about python interop + the dynamic member lookup proposal recently.
Here’s where things stand: we specifically discussed whether a counter-proposal of using “automatically generated wrappers” or “foreign classes” to solve the problem would be better. After discussion, the conclusion is no: the best approach appears to be DynamicMemberLookup/DynamicCallable or something similar in spirit to them. As such, I’ll be dusting off the proposal and we’ll eventually run it.
For transparency, I’m attaching the analysis below of what a wrapper facility could look like, and why it doesn’t work very well for Python interop. I appologize in advance that this is sort of train-of-thought and not a well written doc.
That said, it would be really great to get tighter integration between Swift and SwiftPM for other purposes! I don’t have time to push this forward in the short term though, but if someone was interested in pushing it forward, many people would love to see it discussed seriously.
-Chris
A Swift automatic wrapper facility:
Requirements:
- We want the be able to run a user defined script to generate wrappers.
- This script can have arbitrary dependencies and should get updated when one of them change.
- These dependencies won’t be visible to the Xcode build system, so the compiler will have to manage them.
- In principle, one set of wrappers should be able to depend on another set, and wants “overlays”, so we need a pretty general model.
I don’t think the clang modules based approach is a good way to go.
Proposed Approach: Tighter integration between SwiftPM and Swift
The model is that you should be able to say (strawman syntax):
import Foo from http://github.com/whatever/mypackage
import Bar from file:///some/path/on/my/machine <file:///some/path/on/my/machine>
and have the compiler ask SwiftPM to build and cache the specified module onto your local disk, then have the compiler load it like any other module. This means that “generated wrappers” is now a SwiftPM/llbuild feature, and we can use the SwiftPM “language” to describe things like:
1. Information about what command line invocation is required to generate the wrappers.
2. Dependency information so that the compiler can regenerate the wrappers when they are out of date.
3. Platform abstraction tools since things are in different locations on linux vs mac, Python 2 vs Python 3 is also something that would have to be handled somehow.
4. The directory could contain manually written .swift code, serving the function similar to “overlays” to augment the automatic wrappers generated.
We care about Playgrounds and the REPL, and they should be able to work with this model.
I think that this would be a very nice and useful feature.
Using Wrappers to implement Python Interop:
While such a thing would be generally useful, it is important to explore how well this will work to solve the actual problem at hand, since this is being pitched as an alternative to DynamicMemberLookup. Here is the example from the list:
class BankAccount:
def __init__(self, initial_balance: int = 0) -> None:
self.balance = initial_balance
def deposit(self, amount: int) -> None:
self.balance += amount
def withdraw(self, amount: int) -> None:
self.balance -= amount
def overdrawn(self) -> bool:
return self.balance < 0my_account = BankAccount(15)
my_account.withdraw(5)
print(my_account.balance)
The idea is to generate a wrapper like this (potentially including the type annotations as a refinement):
typealias BankAccount = PyVal
extension PyVal { // methods on BankAccount
init(initial_balance: PyVal) { … }
func deposit(amount: PyVal) -> PyVal { … }
func withdraw(amount: PyVal) -> PyVal { … }
func overdrawn() -> PyVal { … }
}
my_account = BankAccount(initial_balance: 15)
my_account.withdraw(amount: 5)
print(my_account.balance)
It is worth pointing out that this approach is very analogous to the “type providers” feature that Joe pushed hard for months ago, it is just a different implementation approach. The proposal specifically explains why this isn’t a great solution here:
https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#introduce-f-style-type-providers-into-swift
That said, while there are similarities, there are also differences with type providers. Here are the problems that I foresee:
1) This design still requires DynamicMemberLookup
This is because Python doesn’t have property declarations for the wrapper generator to process. The code above shows this on the last line: since there is no definition of the “balance" property, there will be no “balance member” declared in the PyVal extension. A wrapper generator can generate a decl for something it can’t “see”. You can see this in a simpler example:
class Car(object):
def __init__(self):
self.speed = 100
We really do want code like this to work:
let mini = Car()
print(mini.speed)
How will the wrapper generator produce a decl for ‘speed’ when no decl exists? Doug agreed through offline email that "we’d need to fall back to foo[dynamicMember: “speed”] or something like DynamicMemberLookup.” to handle properties.
2) Dumping properties and methods into the same scope won’t work
I didn’t expect this, and I’m not sure how this works, but apparently we accept this:
struct S {
var x: Int
func x(_ a: Int) -> Int { return a }
}
let x = S(x: 1)
x.x // returns 1, not a curried method. Bug or feature?
x.x(42) // returns 42
That said, we reject this:
struct S {
var x: Int
func x() -> Int { ... } // invalid redeclaration of x
}
which means that we’re going to have problems if we find a way to generate property decls (e.g. in response to @property in Python).
Even if we didn’t, we’d still have a problem if we wanted to incorporate types because we reject this:
struct S {
var x: Int
var x: Float // invalid redeclaration of X.
}
This means that even if we have property declarations (e.g. due to use of the Python @property marker for computed properties) we cannot actually incorporate type information into the synthesized header, because multiple classes have all their members munged together and will conflict.
Further, types in methods can lead to ambiguities too in some cases, e.g. if you have:
extension PyVal { // Class Dog
func f(a: Float) -> Float { ... }
}
extension PyVal { // Class Cat
func f(a: Int) {}
}
print(myDog.f(a: 1))
This compiles just fine, but prints out “()" instead of the result of your Dog method, because we select the second overload. In other cases, I suspect you’d fail to compile due to ambiguities. This is a case where types are really harmful for Python, and one of the reasons that the Python types do not affect runtime behavior at all.
3) It’s unclear how to incorporate initializers into this model.
The example above included this as suggested. It looks nice on the surface, but presents some problems:
typealias BankAccount = PyVal
extension PyVal { // methods on BankAccount
init(initial_balance: Int) { … }
}
This has a couple of problems. First of all, in Python classes are themselves callable values, and this break that. Second this mooshes all of the initializers for all of the Python classes onto PyVal.
While it might seem that this would make code completion for initializers ugly, this isn’t actually a problem. After all, we’re enhancing code completion to know about PyVal already, so we can clearly use this trivial local information to filter the list down.
The actual problem is that multiple Python classes can have the same initializer, and we have no way to set the ‘self’ correctly in this case. Consider:
class BankAccount:
def __init__(self, initial_balance: int = 0) -> None: …
class Gyroscope:
def __init__(self, initial_balance: int = 0) -> None: …
These will require generating one extension:
extension PyVal { // methods on BankAccount
init(initial_balance: Int) {
self.init( /* what Python class do we allocate and pass here? BankAccount or Gyroscope? */ )
}
}
I can think of a couple of solutions to this problem, I’d appreciate suggestions for other ones:
3A) Classes turn into typealiases, initializers get name mangled:
something like:
typealias BankAccount = PyVal
extension PyVal {
init(BankAccount_initial_balance: PyVal) { … }
}
...
let my_account = BankAccount(BankAccount_initial_balance: 15)
I don’t like this: this is ugly for clients, and BankAccount is still itself not a value.
3B) Classes turn into global functions:
something like:
func BankAccount(initial_balance: PyVal) -> PyVal {… }
...
let my_account = BankAccount(initial_balance: 15)
This makes the common cases work, but breaks currying and just seems weird to me. Maybe this is ok? This also opens the door for:
// type annotation for clarity only.
let BankAccount: PyVal = BankingModule.BankAccount
which makes “BankAccount” itself be a value (looked up somehow on the module it is defined in).
3C) Classes turn into types that are callable with subscripts?
We don’t actually support static subscripts right now (seems like a silly limitation…):
struct BankAccount {
// error, subscripts can’t be static.
static subscript(initial_balance initial_balance: PyVal) -> PyVal { … }
}
…
let my_account = BankAccount[initial_balance: 15]
But we could emulate them with:
struct BankAccountType {
subscript(initial_balance initial_balance: PyVal) -> PyVal { … }
}
var BankAccount: BankAccountType { return BankAccountType() }
…
let my_account = BankAccount[initial_balance: 15]
this could work, but the square brackets are “gross”, and BankAccount is not a useful Python metatype. We could improve the call-side syntax by introducing a “call decl” or a new “operator()” language feature like:
extension BankAccountType {
func () (initial_balance: PyVal) -> PyVal { … }
}
var BankAccount: BankAccountType { return BankAccountType() }
…
let my_account = BankAccount(initial_balance: 15)
but this still doesn’t solve the “BankAccount as a python value” problem.
4) Throwing / Failability of APIs
Python, like many languages, doesn’t require you to specify whether APIs can throw or not, and because it is dynamically typed, member lookup itself is failable. That said, just like in C++, it is pretty uncommon for exceptions to be thrown from APIs, and our Error Handling design wants Swift programmers to think about error handling, not just slap try on everything.
The Python interop prototype handles this by doing this:
extension PyVal {
/// Return a version of this value that may be called. It throws a Swift
/// error if the underlying Python function throws a Python exception.
public var throwing : ThrowingPyVal {
return ThrowingPyVal(self)
}
}
.. and both PyVal and ThrowingPyVal are callable. The implementation of DynamicCallable on ThrowingPyVal throws a Swift error but the implementation on PyVal does not (similar mechanic exists for DynamicMemberLookup).
This leads to a nice model: the PyVal currency type never throws a Swift error (it aborts on exception) so users don’t have to use “try” on everything. However, if they *want* to process an error, they can. Here is an example from the tutorial:
do {
let x = try Python.open.throwing("/file/that/doesnt/exist")
print(x)
} catch let err {
print("file not found, just like we expected!")
// Here is the error we got:
print(err)
}
In practice, this works really nicely with the way that “try” is required for throwing values, but produces a warning when you use it unnecessarily.
Coming back to wrappers, it isn’t clear how to handle this: a direct port of this would require synthesizing all members onto both PyVal and ThrowingPyVal. This causes tons of bloat and undermines the goal of making the generated header nice.
-Chris