[Discussion] Swift for Data Science / ML / Big Data analytics

Hey Guys,

The big data and machine learning world is dominated by Python, Scala an R.

I'm a Swifter by heart, but not so much by tools of trait.

I'd appreciate a constructive discussion on how that could be changed.

While R is a non goal for obvious reasons, i'd argue that since both Scala
and Python are general purpose languages, taking them head to head might be
a low hanging fruit.

To make the claim I'd like to reference to projects such as

- Hadoop, Spark, Hive are all huge eco-systems which are entirely JVM
based.
- Apache Parquet, a highly efficient column based storage format for big
data analytics which was implemented in Java, and C++.
- Apache Arrow, a physical memory spec that big data systems can use to
allow zero transformations on data transferred between systems. Which (for
obvious reasons) focused on JVM, to C interoperability.

Python's Buffer Protocol which ensures it's predominance (for the time
being) as a prime candidate for data science related projects
https://jeffknupp.com/blog/2017/09/15/python-is-the-
fastest-growing-programming-language-due-to-a-feature-youve-never-heard-of/

While Swift's Memory Ownership manifesto touches similar turf discussing
copy on write and optimizing memory access overhead it IMHO takes a system
level perspective targeting projects such as kernel code. I'd suggest that
viewing the problem from an efficient CPU/GPU data crunching machine
perspective might shade a different light on the requirements and use
cases.

I'd be happy to learn more, and have a constructive discussion on the
subject.

Thank you,
Max.

···

--
puıɯ ʎɯ ɯoɹɟ ʇuǝs

Hey Guys,

The big data and machine learning world is dominated by Python, Scala an R.

I'm a Swifter by heart, but not so much by tools of trait.

Hi Max,

I’m very interested in this topic, with a specific focus on Python. It isn’t the immediate thing on my priority list to deal with, but I hope that we get to push on this.

In short, I think we should build a simple Swift/Python interop story. This sort of thing has be built numerous times for many languages (owing to Python’s great support for embed-ability), including things like PyObjC, boost.python, and many others.

In Swift, it is straightforward to make this example (Python Numpy Tutorial (with Jupyter and Colab)) look something like this:

  let np = Python.import(“numpy”) // Returns a value of type Python.Object.
  let a = np.array([1, 2, 3])
  print(type(a)) // Whether we want to support type(x) or use the Swift equivalent would be up for discussion of course!
  print(a.shape)
  print(a[0], a[1], a[2])
  a[0] = 5
  print(a)

  let b = np.array([[1,2,3],[4,5,6]])
  print(b.shape)
  print(b[0, 0], b[0, 1], b[1, 0])

… which is to say, exactly identical to the Python version except that new variables need to be declared with let/var. This can be done by blessing Python.Object (which is identical to “PyObject*” at the machine level) with some special dynamic name lookup behavior: Dot syntax turns into a call to PyObject_GetAttrString, subscripts turn into PyObject_GetItem, calls turn into PyObject_Call, etc. ARC would be implemented with INCREF etc.

If we do this, the vast majority of the Python ecosystem should be directly usable from within Swift code, and the only a few major syntactic differences (e.g. ranges work differently). We would add failable inits to the primitive datatypes like Int/String/etc to convert Python.Object values into them, and add the corresponding non-failable conversions from Python.Object to those primitives.

Overall, I think it will provide a really nice experience, and allow us to leverage the vast majority of the Python ecosystem directly in Swift code. This project would also have much more narrow impact on the Swift compiler than the ObjC importer (since it works completely differently). For a first cut, I don’t think we would have to worry about Swift classes subclassing Python classes, for example.

-Chris

···

On Oct 28, 2017, at 9:45 AM, Maxim Veksler via swift-evolution <swift-evolution@swift.org> wrote:

I'd appreciate a constructive discussion on how that could be changed.

While R is a non goal for obvious reasons, i'd argue that since both Scala and Python are general purpose languages, taking them head to head might be a low hanging fruit.

To make the claim I'd like to reference to projects such as

- Hadoop, Spark, Hive are all huge eco-systems which are entirely JVM based.
- Apache Parquet, a highly efficient column based storage format for big data analytics which was implemented in Java, and C++.
- Apache Arrow, a physical memory spec that big data systems can use to allow zero transformations on data transferred between systems. Which (for obvious reasons) focused on JVM, to C interoperability.

Python's Buffer Protocol which ensures it's predominance (for the time being) as a prime candidate for data science related projects Python is the fastest growing programming language due to a feature you've never heard of

While Swift's Memory Ownership manifesto touches similar turf discussing copy on write and optimizing memory access overhead it IMHO takes a system level perspective targeting projects such as kernel code. I'd suggest that viewing the problem from an efficient CPU/GPU data crunching machine perspective might shade a different light on the requirements and use cases.

I'd be happy to learn more, and have a constructive discussion on the subject.

Thank you,
Max.

--
puıɯ ʎɯ ɯoɹɟ ʇuǝs
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

That sounds like a very interesting prospect. Do you think it would make sense to make the language features that facilitate this (dynamic dispatch of method calls, property accesses, subscript and ARC) available to Swift classes annotated in some way, so that interop like this can be implemented as a library without special treatment by the Swift compiler? This could also enable more dynamic DSL like features.

— Lukas

···

On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

… which is to say, exactly identical to the Python version except that new variables need to be declared with let/var. This can be done by blessing Python.Object (which is identical to “PyObject*” at the machine level) with some special dynamic name lookup behavior: Dot syntax turns into a call to PyObject_GetAttrString, subscripts turn into PyObject_GetItem, calls turn into PyObject_Call, etc. ARC would be implemented with INCREF etc.

I haven’t explored enough of the design space to be sure, but I’d want to make sure that a library version of this could be done without giving up ergonomics of the result. If you were interested in being able to interop with other languages that are dynamically typed and reference counted, then something like this could be possible in principle:

protocol DynamicDispatchable { // Protocol is “magic" known by the compiler.
  func retain()
  func release()
  func memberLookup(_ : String) -> Self
  func subscript<T>(_ : T) -> Self
  func call(_ args: [Self]) -> Self
}

module Python {
  struct Object : DynamicDispatchable {
    var state : UnsafePointer<PyObject>

    func retain() {
       INCREF(self)
   }

     func memberLookup(_ : String) -> Object {
        PyObject_GetAttrString(…)
     }
    etc
  }

module Perl5 {
   struct Object : DynamicDispatchable {
    var state : UnsafePointer<SV>

    func retain() {
       SvREFCNT_inc(self)
   }
….

Are there other uses for such a thing?

-Chris

···

On Oct 29, 2017, at 4:04 AM, Lukas Stabe <lukas@stabe.de> wrote:

On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

… which is to say, exactly identical to the Python version except that new variables need to be declared with let/var. This can be done by blessing Python.Object (which is identical to “PyObject*” at the machine level) with some special dynamic name lookup behavior: Dot syntax turns into a call to PyObject_GetAttrString, subscripts turn into PyObject_GetItem, calls turn into PyObject_Call, etc. ARC would be implemented with INCREF etc.

That sounds like a very interesting prospect. Do you think it would make sense to make the language features that facilitate this (dynamic dispatch of method calls, property accesses, subscript and ARC) available to Swift classes annotated in some way, so that interop like this can be implemented as a library without special treatment by the Swift compiler? This could also enable more dynamic DSL like features.

… which is to say, exactly identical to the Python version except that new variables need to be declared with let/var. This can be done by blessing Python.Object (which is identical to “PyObject*” at the machine level) with some special dynamic name lookup behavior: Dot syntax turns into a call to PyObject_GetAttrString, subscripts turn into PyObject_GetItem, calls turn into PyObject_Call, etc. ARC would be implemented with INCREF etc.

That sounds like a very interesting prospect. Do you think it would make sense to make the language features that facilitate this (dynamic dispatch of method calls, property accesses, subscript and ARC) available to Swift classes annotated in some way, so that interop like this can be implemented as a library without special treatment by the Swift compiler? This could also enable more dynamic DSL like features.

I haven’t explored enough of the design space to be sure, but I’d want to make sure that a library version of this could be done without giving up ergonomics of the result. If you were interested in being able to interop with other languages that are dynamically typed and reference counted, then something like this could be possible in principle:

Thinking about the Perl case makes it clear to me that this should not be built into the compiler as a monolithic thing. Perl supports several different types (SV/AV/HV) which represent different concepts (scalars, arrays, hashes) so baking it all together into one thing would be the wrong way to map it. In fact, the magic we need is pretty small, and seems generally useful for other things. Consider a design like this:

// not magic, things like Int, String and many other conform to this.
protocol Pythonable {
  init?(_ : PythonObject)
  func toPython() -> PythonObject
}

// Not magic.
struct PythonObject : /*protocols below*/ {
   var state : UnsafePointer<PyObject>

   subscript(_ : Pythonable…) -> PythonObject {
     ...
   }
}

// Magic, must be on the struct definition.
// Could alternatively allow custom copy/move/… ctors like C++.
protocol CustomValueWitnessTable {
  static func init(..)
  static func copy(..)
  static func move(..)
  static func destroy(..)
}

// Magic, allows anyobject-like member lookup on a type when lookup otherwise fails.
protocol DynamicMemberLookupable {
   associatedtype MemberLookupResultType
   func dynamicMemberLookup(_ : String) -> MemberLookupResultType
}

// Magic, allows “overloaded/sugared postfix ()”.
protocol CustomCallable {
  func call( …)
}

The only tricky thing about this is the call part of things. At least in the case of python, we want something like this:

   foo.bar(1, 2, a: x, b: y)

to turn into:
  foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y])

We don’t want this to be a memberlookup of a value that has “bar” as a basename and “a:” and “b:” as parameter labels.

-Chris

···

On Oct 29, 2017, at 8:23 AM, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

On Oct 29, 2017, at 4:04 AM, Lukas Stabe <lukas@stabe.de> wrote:

On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

protocol DynamicDispatchable { // Protocol is “magic" known by the compiler.
func retain()
func release()
func memberLookup(_ : String) -> Self
func subscript<T>(_ : T) -> Self
func call(_ args: [Self]) -> Self
}

module Python {
struct Object : DynamicDispatchable {
   var state : UnsafePointer<PyObject>

   func retain() {
      INCREF(self)
  }

    func memberLookup(_ : String) -> Object {
       PyObject_GetAttrString(…)
    }
   etc
}

module Perl5 {
  struct Object : DynamicDispatchable {
   var state : UnsafePointer<SV>

   func retain() {
      SvREFCNT_inc(self)
  }
….

Are there other uses for such a thing?

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

It would be really nice if Cython code could be linked into iOS and Mac apps written in Swift or Objective-C.

http://cython.org

···

--
C. Keith Ray
Senior Software Engineer / Trainer / Agile Coach
* http://www.thirdfoundationsw.com/keith_ray_resume_2014_long.pdf

On Oct 29, 2017, at 1:34 PM, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

On Oct 29, 2017, at 8:23 AM, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

On Oct 29, 2017, at 4:04 AM, Lukas Stabe <lukas@stabe.de> wrote:

On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

… which is to say, exactly identical to the Python version except that new variables need to be declared with let/var. This can be done by blessing Python.Object (which is identical to “PyObject*” at the machine level) with some special dynamic name lookup behavior: Dot syntax turns into a call to PyObject_GetAttrString, subscripts turn into PyObject_GetItem, calls turn into PyObject_Call, etc. ARC would be implemented with INCREF etc.

That sounds like a very interesting prospect. Do you think it would make sense to make the language features that facilitate this (dynamic dispatch of method calls, property accesses, subscript and ARC) available to Swift classes annotated in some way, so that interop like this can be implemented as a library without special treatment by the Swift compiler? This could also enable more dynamic DSL like features.

I haven’t explored enough of the design space to be sure, but I’d want to make sure that a library version of this could be done without giving up ergonomics of the result. If you were interested in being able to interop with other languages that are dynamically typed and reference counted, then something like this could be possible in principle:

Thinking about the Perl case makes it clear to me that this should not be built into the compiler as a monolithic thing. Perl supports several different types (SV/AV/HV) which represent different concepts (scalars, arrays, hashes) so baking it all together into one thing would be the wrong way to map it. In fact, the magic we need is pretty small, and seems generally useful for other things. Consider a design like this:

// not magic, things like Int, String and many other conform to this.
protocol Pythonable {
init?(_ : PythonObject)
func toPython() -> PythonObject
}

// Not magic.
struct PythonObject : /*protocols below*/ {
  var state : UnsafePointer<PyObject>

  subscript(_ : Pythonable…) -> PythonObject {
    ...
  }
}

// Magic, must be on the struct definition.
// Could alternatively allow custom copy/move/… ctors like C++.
protocol CustomValueWitnessTable {
static func init(..)
static func copy(..)
static func move(..)
static func destroy(..)
}

// Magic, allows anyobject-like member lookup on a type when lookup otherwise fails.
protocol DynamicMemberLookupable {
  associatedtype MemberLookupResultType
  func dynamicMemberLookup(_ : String) -> MemberLookupResultType
}

// Magic, allows “overloaded/sugared postfix ()”.
protocol CustomCallable {
func call( …)
}

The only tricky thing about this is the call part of things. At least in the case of python, we want something like this:

  foo.bar(1, 2, a: x, b: y)

to turn into:
foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y])

We don’t want this to be a memberlookup of a value that has “bar” as a basename and “a:” and “b:” as parameter labels.

-Chris

protocol DynamicDispatchable { // Protocol is “magic" known by the compiler.
func retain()
func release()
func memberLookup(_ : String) -> Self
func subscript<T>(_ : T) -> Self
func call(_ args: [Self]) -> Self
}

module Python {
struct Object : DynamicDispatchable {
  var state : UnsafePointer<PyObject>

  func retain() {
     INCREF(self)
}

   func memberLookup(_ : String) -> Object {
      PyObject_GetAttrString(…)
   }
  etc
}

module Perl5 {
struct Object : DynamicDispatchable {
  var state : UnsafePointer<SV>

  func retain() {
     SvREFCNT_inc(self)
}
….

Are there other uses for such a thing?

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

// Magic, allows “overloaded/sugared postfix ()”.
protocol CustomCallable {
func call( …)
}

The only tricky thing about this is the call part of things. At least in the case of python, we want something like this:

  foo.bar(1, 2, a: x, b: y)

to turn into:
foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y])

We don’t want this to be a memberlookup of a value that has “bar” as a basename and “a:” and “b:” as parameter labels.

I don't know much about Swift internals, but since you can already access a function member without calling it, the parsing part of this sounds like it would be straightforward.

If CustomCallable looked something like this:

protocol CustomCallable {
    associatedtype Parameter
    associatedtype Result
    func call(_ parameterList: [(label: String?, value: Parameter)]) -> Result
}

implementations could decide how they want to handle named/unnamed parameters etc themselves.

This approach would currently come with some limitations though: No throwing (this could be addressed by a separate CustomThrowingCallable protocol), and – since currently non-nominal types can't adopt protocols – an implementation of CustomCallable could not accept, for example, both normal values and closures/tuples.

Are there other uses for such a thing?

DynamicMemberLookupable could enable libraries dealing with key-value data, like SwiftyJSON for example, to let users write myJson.someKey.nested instead of myJson["someKey"]["nested"]. I'm sure there are more uses like this that are not integrations with other languages.

— Lukas

···

On 29. Oct 2017, at 21:34, Chris Lattner <clattner@nondot.org> wrote:

… which is to say, exactly identical to the Python version except that new variables need to be declared with let/var. This can be done by blessing Python.Object (which is identical to “PyObject*” at the machine level) with some special dynamic name lookup behavior: Dot syntax turns into a call to PyObject_GetAttrString, subscripts turn into PyObject_GetItem, calls turn into PyObject_Call, etc. ARC would be implemented with INCREF etc.

That sounds like a very interesting prospect. Do you think it would make sense to make the language features that facilitate this (dynamic dispatch of method calls, property accesses, subscript and ARC) available to Swift classes annotated in some way, so that interop like this can be implemented as a library without special treatment by the Swift compiler? This could also enable more dynamic DSL like features.

I haven’t explored enough of the design space to be sure, but I’d want to make sure that a library version of this could be done without giving up ergonomics of the result. If you were interested in being able to interop with other languages that are dynamically typed and reference counted, then something like this could be possible in principle:

Thinking about the Perl case makes it clear to me that this should not be built into the compiler as a monolithic thing. Perl supports several different types (SV/AV/HV) which represent different concepts (scalars, arrays, hashes) so baking it all together into one thing would be the wrong way to map it. In fact, the magic we need is pretty small, and seems generally useful for other things. Consider a design like this:

// not magic, things like Int, String and many other conform to this.
protocol Pythonable {
init?(_ : PythonObject)
func toPython() -> PythonObject
}

It’s not magic unless you expect the compiler or runtime to help with conversion between Int/String/etc. and PythonObject, as with _ObjectiveCBridgeable.

// Not magic.
struct PythonObject : /*protocols below*/ {
  var state : UnsafePointer<PyObject>

  subscript(_ : Pythonable…) -> PythonObject {
    ...
  }
}

// Magic, must be on the struct definition.
// Could alternatively allow custom copy/move/… ctors like C++.
protocol CustomValueWitnessTable {
static func init(..)
static func copy(..)
static func move(..)
static func destroy(..)
}

Swift’s implementation model supports this. As a surface-level construct it’s going to be mired in UnsafeMutablePointers, and it’s not at all clear to me that we want this level of control there. Presumably, binding to Python is going to require some compiler effort—defining how it is that Python objects are initialized/copied/moved/destroyed seems like a reasonable part of that effort.

// Magic, allows anyobject-like member lookup on a type when lookup otherwise fails.
protocol DynamicMemberLookupable {
  associatedtype MemberLookupResultType
  func dynamicMemberLookup(_ : String) -> MemberLookupResultType
}

AnyObject lookup looks for an actual declaration on any type anywhere. One could extend that mechanism to, say, return all Python methods and assume that you can call any Python method with any PythonObject instance. AnyObject lookup is fairly unprincipled as a language feature, because there’s no natural scope in which to perform name lookup, and involves hacks at many levels that don’t always work (e.g., AnyObject lookup… sometimes… fails across multiple source files for hard-to-explain reasons). You’re taking on that brokenness if you expand AnyObject lookup to another ecosystem.

Although it doesn’t really seem like AnyObject lookup is the thing you’re asking for here. It seems more like you want dynamicMemberLookup(_:) to capture “self” and the method name, and then be a callable thing as below...

// Magic, allows “overloaded/sugared postfix ()”.
protocol CustomCallable {
func call( …)
}

The only tricky thing about this is the call part of things. At least in the case of python, we want something like this:

  foo.bar(1, 2, a: x, b: y)

to turn into:
foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y])

We don’t want this to be a memberlookup of a value that has “bar” as a basename and “a:” and “b:” as parameter labels.

Well, I think the MemberLookupResult is going to get the name “bar”, argument labels “_:_:a:b:”, and arguments “1”, “2”, “x”, “y”, because that’s the Swift model of argument labels. It can then reshuffle them however it needs to for the underlying interaction with the Python interpreter.

There are definite design trade-offs here. With AnyObject lookup, it’s a known-broken feature but because it depends on synthesized Swift method declarations, it’ll behave mostly the same way as other Swift method declarations—static overloading, known (albeit weak) type signatures, etc. But, it might require more of Python’s model to be grafted onto those method declarations. With dynamic member lookup, you’re throwing away all type safety (even for a motivated Python developer who might be willing to annotate APIs with types) and creating a general language mechanism for doing that.

  - Doug

···

On Oct 29, 2017, at 1:34 PM, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

On Oct 29, 2017, at 8:23 AM, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

On Oct 29, 2017, at 4:04 AM, Lukas Stabe <lukas@stabe.de> wrote:

On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

-Chris

protocol DynamicDispatchable { // Protocol is “magic" known by the compiler.
func retain()
func release()
func memberLookup(_ : String) -> Self
func subscript<T>(_ : T) -> Self
func call(_ args: [Self]) -> Self
}

module Python {
struct Object : DynamicDispatchable {
  var state : UnsafePointer<PyObject>

  func retain() {
     INCREF(self)
}

   func memberLookup(_ : String) -> Object {
      PyObject_GetAttrString(…)
   }
  etc
}

module Perl5 {
struct Object : DynamicDispatchable {
  var state : UnsafePointer<SV>

  func retain() {
     SvREFCNT_inc(self)
}
….

Are there other uses for such a thing?

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

It would be really nice if Cython code could be linked into iOS and Mac apps written in Swift or Objective-C.

http://cython.org <http://cython.org/&gt;

Isn’t that possible today, at least on the Mac?

-Chris

···

On Oct 29, 2017, at 3:23 PM, C. Keith Ray <keithray@mac.com> wrote:

--
C. Keith Ray
Senior Software Engineer / Trainer / Agile Coach
* http://www.thirdfoundationsw.com/keith_ray_resume_2014_long.pdf

On Oct 29, 2017, at 1:34 PM, Chris Lattner via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Oct 29, 2017, at 8:23 AM, Chris Lattner via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Oct 29, 2017, at 4:04 AM, Lukas Stabe <lukas@stabe.de <mailto:lukas@stabe.de>> wrote:

On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

… which is to say, exactly identical to the Python version except that new variables need to be declared with let/var. This can be done by blessing Python.Object (which is identical to “PyObject*” at the machine level) with some special dynamic name lookup behavior: Dot syntax turns into a call to PyObject_GetAttrString, subscripts turn into PyObject_GetItem, calls turn into PyObject_Call, etc. ARC would be implemented with INCREF etc.

That sounds like a very interesting prospect. Do you think it would make sense to make the language features that facilitate this (dynamic dispatch of method calls, property accesses, subscript and ARC) available to Swift classes annotated in some way, so that interop like this can be implemented as a library without special treatment by the Swift compiler? This could also enable more dynamic DSL like features.

I haven’t explored enough of the design space to be sure, but I’d want to make sure that a library version of this could be done without giving up ergonomics of the result. If you were interested in being able to interop with other languages that are dynamically typed and reference counted, then something like this could be possible in principle:

Thinking about the Perl case makes it clear to me that this should not be built into the compiler as a monolithic thing. Perl supports several different types (SV/AV/HV) which represent different concepts (scalars, arrays, hashes) so baking it all together into one thing would be the wrong way to map it. In fact, the magic we need is pretty small, and seems generally useful for other things. Consider a design like this:

// not magic, things like Int, String and many other conform to this.
protocol Pythonable {
init?(_ : PythonObject)
func toPython() -> PythonObject
}

// Not magic.
struct PythonObject : /*protocols below*/ {
  var state : UnsafePointer<PyObject>

  subscript(_ : Pythonable…) -> PythonObject {
    ...
  }
}

// Magic, must be on the struct definition.
// Could alternatively allow custom copy/move/… ctors like C++.
protocol CustomValueWitnessTable {
static func init(..)
static func copy(..)
static func move(..)
static func destroy(..)
}

// Magic, allows anyobject-like member lookup on a type when lookup otherwise fails.
protocol DynamicMemberLookupable {
  associatedtype MemberLookupResultType
  func dynamicMemberLookup(_ : String) -> MemberLookupResultType
}

// Magic, allows “overloaded/sugared postfix ()”.
protocol CustomCallable {
func call( …)
}

The only tricky thing about this is the call part of things. At least in the case of python, we want something like this:

  foo.bar(1, 2, a: x, b: y)

to turn into:
foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y])

We don’t want this to be a memberlookup of a value that has “bar” as a basename and “a:” and “b:” as parameter labels.

-Chris

protocol DynamicDispatchable { // Protocol is “magic" known by the compiler.
func retain()
func release()
func memberLookup(_ : String) -> Self
func subscript<T>(_ : T) -> Self
func call(_ args: [Self]) -> Self
}

module Python {
struct Object : DynamicDispatchable {
  var state : UnsafePointer<PyObject>

  func retain() {
     INCREF(self)
}

   func memberLookup(_ : String) -> Object {
      PyObject_GetAttrString(…)
   }
  etc
}

module Perl5 {
struct Object : DynamicDispatchable {
  var state : UnsafePointer<SV>

  func retain() {
     SvREFCNT_inc(self)
}
….

Are there other uses for such a thing?

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

To Chris’s use case point, I see Python -> Swift interop as a very common use case in data science / ML for speeding up certain functions/modules by embedding Swift into Python, like you can with, say, ctypes or python extensions in dynamic C libraries. This is very common practice in the Python universe, it just could be a lot better with Swift instead.

Things like this *sort of* exist with Swift (https://gist.github.com/jiaaro/e111f0f64d0cdb8aca38\), but it’s not really very mature or functional.

I would LOVE to be able to speed up a lot of python code by implementing swift libraries and importing into python.

- David

···

On Oct 30, 2017, at 13:25, Douglas Gregor via swift-evolution <swift-evolution@swift.org> wrote:

On Oct 29, 2017, at 1:34 PM, Chris Lattner via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Oct 29, 2017, at 8:23 AM, Chris Lattner via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Oct 29, 2017, at 4:04 AM, Lukas Stabe <lukas@stabe.de <mailto:lukas@stabe.de>> wrote:

On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

… which is to say, exactly identical to the Python version except that new variables need to be declared with let/var. This can be done by blessing Python.Object (which is identical to “PyObject*” at the machine level) with some special dynamic name lookup behavior: Dot syntax turns into a call to PyObject_GetAttrString, subscripts turn into PyObject_GetItem, calls turn into PyObject_Call, etc. ARC would be implemented with INCREF etc.

That sounds like a very interesting prospect. Do you think it would make sense to make the language features that facilitate this (dynamic dispatch of method calls, property accesses, subscript and ARC) available to Swift classes annotated in some way, so that interop like this can be implemented as a library without special treatment by the Swift compiler? This could also enable more dynamic DSL like features.

I haven’t explored enough of the design space to be sure, but I’d want to make sure that a library version of this could be done without giving up ergonomics of the result. If you were interested in being able to interop with other languages that are dynamically typed and reference counted, then something like this could be possible in principle:

Thinking about the Perl case makes it clear to me that this should not be built into the compiler as a monolithic thing. Perl supports several different types (SV/AV/HV) which represent different concepts (scalars, arrays, hashes) so baking it all together into one thing would be the wrong way to map it. In fact, the magic we need is pretty small, and seems generally useful for other things. Consider a design like this:

// not magic, things like Int, String and many other conform to this.
protocol Pythonable {
init?(_ : PythonObject)
func toPython() -> PythonObject
}

It’s not magic unless you expect the compiler or runtime to help with conversion between Int/String/etc. and PythonObject, as with _ObjectiveCBridgeable.

// Not magic.
struct PythonObject : /*protocols below*/ {
  var state : UnsafePointer<PyObject>

  subscript(_ : Pythonable…) -> PythonObject {
    ...
  }
}

// Magic, must be on the struct definition.
// Could alternatively allow custom copy/move/… ctors like C++.
protocol CustomValueWitnessTable {
static func init(..)
static func copy(..)
static func move(..)
static func destroy(..)
}

Swift’s implementation model supports this. As a surface-level construct it’s going to be mired in UnsafeMutablePointers, and it’s not at all clear to me that we want this level of control there. Presumably, binding to Python is going to require some compiler effort—defining how it is that Python objects are initialized/copied/moved/destroyed seems like a reasonable part of that effort.

// Magic, allows anyobject-like member lookup on a type when lookup otherwise fails.
protocol DynamicMemberLookupable {
  associatedtype MemberLookupResultType
  func dynamicMemberLookup(_ : String) -> MemberLookupResultType
}

AnyObject lookup looks for an actual declaration on any type anywhere. One could extend that mechanism to, say, return all Python methods and assume that you can call any Python method with any PythonObject instance. AnyObject lookup is fairly unprincipled as a language feature, because there’s no natural scope in which to perform name lookup, and involves hacks at many levels that don’t always work (e.g., AnyObject lookup… sometimes… fails across multiple source files for hard-to-explain reasons). You’re taking on that brokenness if you expand AnyObject lookup to another ecosystem.

Although it doesn’t really seem like AnyObject lookup is the thing you’re asking for here. It seems more like you want dynamicMemberLookup(_:) to capture “self” and the method name, and then be a callable thing as below...

// Magic, allows “overloaded/sugared postfix ()”.
protocol CustomCallable {
func call( …)
}

The only tricky thing about this is the call part of things. At least in the case of python, we want something like this:

  foo.bar(1, 2, a: x, b: y)

to turn into:
foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y])

We don’t want this to be a memberlookup of a value that has “bar” as a basename and “a:” and “b:” as parameter labels.

Well, I think the MemberLookupResult is going to get the name “bar”, argument labels “_:_:a:b:”, and arguments “1”, “2”, “x”, “y”, because that’s the Swift model of argument labels. It can then reshuffle them however it needs to for the underlying interaction with the Python interpreter.

There are definite design trade-offs here. With AnyObject lookup, it’s a known-broken feature but because it depends on synthesized Swift method declarations, it’ll behave mostly the same way as other Swift method declarations—static overloading, known (albeit weak) type signatures, etc. But, it might require more of Python’s model to be grafted onto those method declarations. With dynamic member lookup, you’re throwing away all type safety (even for a motivated Python developer who might be willing to annotate APIs with types) and creating a general language mechanism for doing that.

  - Doug

-Chris

protocol DynamicDispatchable { // Protocol is “magic" known by the compiler.
func retain()
func release()
func memberLookup(_ : String) -> Self
func subscript<T>(_ : T) -> Self
func call(_ args: [Self]) -> Self
}

module Python {
struct Object : DynamicDispatchable {
  var state : UnsafePointer<PyObject>

  func retain() {
     INCREF(self)
}

   func memberLookup(_ : String) -> Object {
      PyObject_GetAttrString(…)
   }
  etc
}

module Perl5 {
struct Object : DynamicDispatchable {
  var state : UnsafePointer<SV>

  func retain() {
     SvREFCNT_inc(self)
}
….

Are there other uses for such a thing?

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

JohnMC: question for you below.

Thinking about the Perl case makes it clear to me that this should not be built into the compiler as a monolithic thing. Perl supports several different types (SV/AV/HV) which represent different concepts (scalars, arrays, hashes) so baking it all together into one thing would be the wrong way to map it. In fact, the magic we need is pretty small, and seems generally useful for other things. Consider a design like this:

// not magic, things like Int, String and many other conform to this.
protocol Pythonable {
init?(_ : PythonObject)
func toPython() -> PythonObject
}

It’s not magic unless you expect the compiler or runtime to help with conversion between Int/String/etc. and PythonObject, as with _ObjectiveCBridgeable.

Right, as I said above “not magic”. The conformances would be manually implemented in the Python overlay. This provides a free implicit conversion from "T -> Pythonable” for the T’s we care about, and a failable init from Python back to Swift types.

// Not magic.
struct PythonObject : /*protocols below*/ {
  var state : UnsafePointer<PyObject>

  subscript(_ : Pythonable…) -> PythonObject {
    ...
  }
}

// Magic, must be on the struct definition.
// Could alternatively allow custom copy/move/… ctors like C++.
protocol CustomValueWitnessTable {
static func init(..)
static func copy(..)
static func move(..)
static func destroy(..)
}

Swift’s implementation model supports this. As a surface-level construct it’s going to be mired in UnsafeMutablePointers, and it’s not at all clear to me that we want this level of control there.

There are two ways to implement it:
1) static func’s like the above, which are implemented as UnsafePointer's
2) Proper language syntax akin to the C++ “rule of 5”.

The pro’s and con’s of the first approach:

pro) full explicit control over what happens
pro) no other new language features necessary to implement this. The second approach would need something like ownership to be in place.
con) injects another avenue of unsafety (though it would be explicit, so it isn’t that bad). It isn’t obvious to me that approach #2 can be safe, but I haven’t thought about it enough.
???) discourages people from using this protocol because of its explicit unsafety.

I can think of two things that could tip the scale of the discussion:

a) The big question is whether we *want* the ability to write custom rule-of-5 style behavior for structs, or if we want it to only be used in extreme cases (like bridging interop in this proposal). If we *want* to support it someday, then adding proper “safe” support is best (if possible). If we don’t *want* people to use it, then making it Unsafe and ugly is a reasonable way to go.

b) The ownership proposal is likely to add deinit's to structs. If it also adds explicit move initializers, then it is probably the right thing to add copy initializers also (and thus go with approach #2). That said, I’m not sure how the move initializers will be spelled or if that is the likely direction. If it won’t add these, then it is probably better to go with approach #1. John, what do you think?

Presumably, binding to Python is going to require some compiler effort—defining how it is that Python objects are initialized/copied/moved/destroyed seems like a reasonable part of that effort.

Actually no. If we add these three proposals, there is no other python (or perl, etc…) specific support needed. It is all implementable in the overlay.

// Magic, allows anyobject-like member lookup on a type when lookup otherwise fails.
protocol DynamicMemberLookupable {
  associatedtype MemberLookupResultType
  func dynamicMemberLookup(_ : String) -> MemberLookupResultType
}

AnyObject lookup looks for an actual declaration on any type anywhere. One could extend that mechanism to, say, return all Python methods and assume that you can call any Python method with any PythonObject instance. AnyObject lookup is fairly unprincipled as a language feature, because there’s no natural scope in which to perform name lookup, and involves hacks at many levels that don’t always work (e.g., AnyObject lookup… sometimes… fails across multiple source files for hard-to-explain reasons). You’re taking on that brokenness if you expand AnyObject lookup to another ecosystem.

Yeah, sorry, that’s not what I meant:

Although it doesn’t really seem like AnyObject lookup is the thing you’re asking for here. It seems more like you want dynamicMemberLookup(_:) to capture “self” and the method name, and then be a callable thing as below…

That’s what I meant :-).

A type that implements this magic protocol would never fail name lookup: “foo.bar” would always fall back to calling: foo.dynamicMemberLookup(“bar")

it’s simple and more predictable than AnyObject, it also matches what dynamic languages like Python needs.

// Magic, allows “overloaded/sugared postfix ()”.
protocol CustomCallable {
func call( …)
}

The only tricky thing about this is the call part of things. At least in the case of python, we want something like this:

  foo.bar(1, 2, a: x, b: y)

to turn into:
foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y])

We don’t want this to be a memberlookup of a value that has “bar” as a basename and “a:” and “b:” as parameter labels.

Well, I think the MemberLookupResult is going to get the name “bar”, argument labels “_:_:a:b:”, and arguments “1”, “2”, “x”, “y”, because that’s the Swift model of argument labels. It can then reshuffle them however it needs to for the underlying interaction with the Python interpreter.

There are definite design trade-offs here. With AnyObject lookup, it’s a known-broken feature but because it depends on synthesized Swift method declarations, it’ll behave mostly the same way as other Swift method declarations—static overloading, known (albeit weak) type signatures, etc. But, it might require more of Python’s model to be grafted onto those method declarations. With dynamic member lookup, you’re throwing away all type safety (even for a motivated Python developer who might be willing to annotate APIs with types) and creating a general language mechanism for doing that.

Right, something like this could definitely work, but keep in mind that the Swift compiler knows nothing about Python declarations.

Perhaps the most straight-forward thing would be to support:

protocol CustomCallable {
func call(…arg list as array and kw args...)
func callMember(_ : String, …otherstuffabove...)
}

Given this, the compiler could map:

pythonThing(42) -> pythonThing.call([42])
pythonThing.method(a: 42) -> pythonThing.callMember(“method”, kwargs: [“a”: 42])

This is the simplest way to map the Swift semantics (where kw args are part of compound lookups) into the Python world.

-Chris

···

On Oct 30, 2017, at 1:25 PM, Douglas Gregor <dgregor@apple.com> wrote:

There's surprisingly little information about Cython interoperability with Cocoa/Mac applications. It's might be linkable if the app can embed Python.

···

--
C. Keith Ray

* What Every Programmer Needs To… by C. Keith Ray [PDF/iPad/Kindle] <- buy my book?
* http://www.thirdfoundationsw.com/keith_ray_resume_2014_long.pdf
* http://agilesolutionspace.blogspot.com/

On Oct 29, 2017, at 4:00 PM, Chris Lattner <clattner@nondot.org> wrote:

On Oct 29, 2017, at 3:23 PM, C. Keith Ray <keithray@mac.com> wrote:

It would be really nice if Cython code could be linked into iOS and Mac apps written in Swift or Objective-C.

http://cython.org

Isn’t that possible today, at least on the Mac?

-Chris

--
C. Keith Ray
Senior Software Engineer / Trainer / Agile Coach
* http://www.thirdfoundationsw.com/keith_ray_resume_2014_long.pdf

On Oct 29, 2017, at 1:34 PM, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

On Oct 29, 2017, at 8:23 AM, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

On Oct 29, 2017, at 4:04 AM, Lukas Stabe <lukas@stabe.de> wrote:

On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

… which is to say, exactly identical to the Python version except that new variables need to be declared with let/var. This can be done by blessing Python.Object (which is identical to “PyObject*” at the machine level) with some special dynamic name lookup behavior: Dot syntax turns into a call to PyObject_GetAttrString, subscripts turn into PyObject_GetItem, calls turn into PyObject_Call, etc. ARC would be implemented with INCREF etc.

That sounds like a very interesting prospect. Do you think it would make sense to make the language features that facilitate this (dynamic dispatch of method calls, property accesses, subscript and ARC) available to Swift classes annotated in some way, so that interop like this can be implemented as a library without special treatment by the Swift compiler? This could also enable more dynamic DSL like features.

I haven’t explored enough of the design space to be sure, but I’d want to make sure that a library version of this could be done without giving up ergonomics of the result. If you were interested in being able to interop with other languages that are dynamically typed and reference counted, then something like this could be possible in principle:

Thinking about the Perl case makes it clear to me that this should not be built into the compiler as a monolithic thing. Perl supports several different types (SV/AV/HV) which represent different concepts (scalars, arrays, hashes) so baking it all together into one thing would be the wrong way to map it. In fact, the magic we need is pretty small, and seems generally useful for other things. Consider a design like this:

// not magic, things like Int, String and many other conform to this.
protocol Pythonable {
init?(_ : PythonObject)
func toPython() -> PythonObject
}

// Not magic.
struct PythonObject : /*protocols below*/ {
  var state : UnsafePointer<PyObject>

  subscript(_ : Pythonable…) -> PythonObject {
    ...
  }
}

// Magic, must be on the struct definition.
// Could alternatively allow custom copy/move/… ctors like C++.
protocol CustomValueWitnessTable {
static func init(..)
static func copy(..)
static func move(..)
static func destroy(..)
}

// Magic, allows anyobject-like member lookup on a type when lookup otherwise fails.
protocol DynamicMemberLookupable {
  associatedtype MemberLookupResultType
  func dynamicMemberLookup(_ : String) -> MemberLookupResultType
}

// Magic, allows “overloaded/sugared postfix ()”.
protocol CustomCallable {
func call( …)
}

The only tricky thing about this is the call part of things. At least in the case of python, we want something like this:

  foo.bar(1, 2, a: x, b: y)

to turn into:
foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y])

We don’t want this to be a memberlookup of a value that has “bar” as a basename and “a:” and “b:” as parameter labels.

-Chris

protocol DynamicDispatchable { // Protocol is “magic" known by the compiler.
func retain()
func release()
func memberLookup(_ : String) -> Self
func subscript<T>(_ : T) -> Self
func call(_ args: [Self]) -> Self
}

module Python {
struct Object : DynamicDispatchable {
  var state : UnsafePointer<PyObject>

  func retain() {
     INCREF(self)
}

   func memberLookup(_ : String) -> Object {
      PyObject_GetAttrString(…)
   }
  etc
}

module Perl5 {
struct Object : DynamicDispatchable {
  var state : UnsafePointer<SV>

  func retain() {
     SvREFCNT_inc(self)
}
….

Are there other uses for such a thing?

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

To Chris’s use case point, I see Python -> Swift interop as a very common use case in data science / ML for speeding up certain functions/modules by embedding Swift into Python, like you can with, say, ctypes or python extensions in dynamic C libraries. This is very common practice in the Python universe, it just could be a lot better with Swift instead.

Things like this *sort of* exist with Swift (https://gist.github.com/jiaaro/e111f0f64d0cdb8aca38\), but it’s not really very mature or functional.

I would LOVE to be able to speed up a lot of python code by implementing swift libraries and importing into python.

There are a bunch of ways one could export Swift functionality to Python, from a library solution like the C++ Boost.Python (http://www.boost.org/doc/libs/1_65_1/libs/python/doc/html/index.html\), to code generators like SWIG or, with some future expansion of metadata, directly at runtime via reflection facilities. Python’s dynamic nature makes this direction easier.

The harder challenge (IMO!) is to get a Python library into Swift, because there isn’t a lot of type information in Python, so you have to show the Python in the much-more-strongly-typed Swift somehow (and keep it usable).

  - Doug

···

On Oct 30, 2017, at 2:54 PM, David Kopecky <dkopecky@apple.com> wrote:

- David

On Oct 30, 2017, at 13:25, Douglas Gregor via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Oct 29, 2017, at 1:34 PM, Chris Lattner via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Oct 29, 2017, at 8:23 AM, Chris Lattner via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Oct 29, 2017, at 4:04 AM, Lukas Stabe <lukas@stabe.de <mailto:lukas@stabe.de>> wrote:

On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

… which is to say, exactly identical to the Python version except that new variables need to be declared with let/var. This can be done by blessing Python.Object (which is identical to “PyObject*” at the machine level) with some special dynamic name lookup behavior: Dot syntax turns into a call to PyObject_GetAttrString, subscripts turn into PyObject_GetItem, calls turn into PyObject_Call, etc. ARC would be implemented with INCREF etc.

That sounds like a very interesting prospect. Do you think it would make sense to make the language features that facilitate this (dynamic dispatch of method calls, property accesses, subscript and ARC) available to Swift classes annotated in some way, so that interop like this can be implemented as a library without special treatment by the Swift compiler? This could also enable more dynamic DSL like features.

I haven’t explored enough of the design space to be sure, but I’d want to make sure that a library version of this could be done without giving up ergonomics of the result. If you were interested in being able to interop with other languages that are dynamically typed and reference counted, then something like this could be possible in principle:

Thinking about the Perl case makes it clear to me that this should not be built into the compiler as a monolithic thing. Perl supports several different types (SV/AV/HV) which represent different concepts (scalars, arrays, hashes) so baking it all together into one thing would be the wrong way to map it. In fact, the magic we need is pretty small, and seems generally useful for other things. Consider a design like this:

// not magic, things like Int, String and many other conform to this.
protocol Pythonable {
init?(_ : PythonObject)
func toPython() -> PythonObject
}

It’s not magic unless you expect the compiler or runtime to help with conversion between Int/String/etc. and PythonObject, as with _ObjectiveCBridgeable.

// Not magic.
struct PythonObject : /*protocols below*/ {
  var state : UnsafePointer<PyObject>

  subscript(_ : Pythonable…) -> PythonObject {
    ...
  }
}

// Magic, must be on the struct definition.
// Could alternatively allow custom copy/move/… ctors like C++.
protocol CustomValueWitnessTable {
static func init(..)
static func copy(..)
static func move(..)
static func destroy(..)
}

Swift’s implementation model supports this. As a surface-level construct it’s going to be mired in UnsafeMutablePointers, and it’s not at all clear to me that we want this level of control there. Presumably, binding to Python is going to require some compiler effort—defining how it is that Python objects are initialized/copied/moved/destroyed seems like a reasonable part of that effort.

// Magic, allows anyobject-like member lookup on a type when lookup otherwise fails.
protocol DynamicMemberLookupable {
  associatedtype MemberLookupResultType
  func dynamicMemberLookup(_ : String) -> MemberLookupResultType
}

AnyObject lookup looks for an actual declaration on any type anywhere. One could extend that mechanism to, say, return all Python methods and assume that you can call any Python method with any PythonObject instance. AnyObject lookup is fairly unprincipled as a language feature, because there’s no natural scope in which to perform name lookup, and involves hacks at many levels that don’t always work (e.g., AnyObject lookup… sometimes… fails across multiple source files for hard-to-explain reasons). You’re taking on that brokenness if you expand AnyObject lookup to another ecosystem.

Although it doesn’t really seem like AnyObject lookup is the thing you’re asking for here. It seems more like you want dynamicMemberLookup(_:) to capture “self” and the method name, and then be a callable thing as below...

// Magic, allows “overloaded/sugared postfix ()”.
protocol CustomCallable {
func call( …)
}

The only tricky thing about this is the call part of things. At least in the case of python, we want something like this:

  foo.bar(1, 2, a: x, b: y)

to turn into:
foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y])

We don’t want this to be a memberlookup of a value that has “bar” as a basename and “a:” and “b:” as parameter labels.

Well, I think the MemberLookupResult is going to get the name “bar”, argument labels “_:_:a:b:”, and arguments “1”, “2”, “x”, “y”, because that’s the Swift model of argument labels. It can then reshuffle them however it needs to for the underlying interaction with the Python interpreter.

There are definite design trade-offs here. With AnyObject lookup, it’s a known-broken feature but because it depends on synthesized Swift method declarations, it’ll behave mostly the same way as other Swift method declarations—static overloading, known (albeit weak) type signatures, etc. But, it might require more of Python’s model to be grafted onto those method declarations. With dynamic member lookup, you’re throwing away all type safety (even for a motivated Python developer who might be willing to annotate APIs with types) and creating a general language mechanism for doing that.

  - Doug

-Chris

protocol DynamicDispatchable { // Protocol is “magic" known by the compiler.
func retain()
func release()
func memberLookup(_ : String) -> Self
func subscript<T>(_ : T) -> Self
func call(_ args: [Self]) -> Self
}

module Python {
struct Object : DynamicDispatchable {
  var state : UnsafePointer<PyObject>

  func retain() {
     INCREF(self)
}

   func memberLookup(_ : String) -> Object {
      PyObject_GetAttrString(…)
   }
  etc
}

module Perl5 {
struct Object : DynamicDispatchable {
  var state : UnsafePointer<SV>

  func retain() {
     SvREFCNT_inc(self)
}
….

Are there other uses for such a thing?

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

JohnMC: question for you below.

Thinking about the Perl case makes it clear to me that this should not be built into the compiler as a monolithic thing. Perl supports several different types (SV/AV/HV) which represent different concepts (scalars, arrays, hashes) so baking it all together into one thing would be the wrong way to map it. In fact, the magic we need is pretty small, and seems generally useful for other things. Consider a design like this:

// not magic, things like Int, String and many other conform to this.
protocol Pythonable {
init?(_ : PythonObject)
func toPython() -> PythonObject
}

It’s not magic unless you expect the compiler or runtime to help with conversion between Int/String/etc. and PythonObject, as with _ObjectiveCBridgeable.

Right, as I said above “not magic”. The conformances would be manually implemented in the Python overlay. This provides a free implicit conversion from "T -> Pythonable” for the T’s we care about, and a failable init from Python back to Swift types.

Note that, under this scheme,

  let p: Pythonable = 17
  let i: Int = p as! i

will work if Int is Pythonable, but not when p comes back from Python.

// Not magic.
struct PythonObject : /*protocols below*/ {
  var state : UnsafePointer<PyObject>

  subscript(_ : Pythonable…) -> PythonObject {
    ...
  }
}

// Magic, must be on the struct definition.
// Could alternatively allow custom copy/move/… ctors like C++.
protocol CustomValueWitnessTable {
static func init(..)
static func copy(..)
static func move(..)
static func destroy(..)
}

Swift’s implementation model supports this. As a surface-level construct it’s going to be mired in UnsafeMutablePointers, and it’s not at all clear to me that we want this level of control there.

There are two ways to implement it:
1) static func’s like the above, which are implemented as UnsafePointer's
2) Proper language syntax akin to the C++ “rule of 5”.

The pro’s and con’s of the first approach:

pro) full explicit control over what happens
pro) no other new language features necessary to implement this. The second approach would need something like ownership to be in place.
con) injects another avenue of unsafety (though it would be explicit, so it isn’t that bad). It isn’t obvious to me that approach #2 can be safe, but I haven’t thought about it enough.
???) discourages people from using this protocol because of its explicit unsafety.

con) much of the UnsafePointer interface is based on the value witness table, so one has to step lightly or work with something like COpaquePointer/UnsafePointer<Void>.

I can think of two things that could tip the scale of the discussion:

a) The big question is whether we *want* the ability to write custom rule-of-5 style behavior for structs, or if we want it to only be used in extreme cases (like bridging interop in this proposal). If we *want* to support it someday, then adding proper “safe” support is best (if possible). If we don’t *want* people to use it, then making it Unsafe and ugly is a reasonable way to go.

b) The ownership proposal is likely to add deinit's to structs. If it also adds explicit move initializers, then it is probably the right thing to add copy initializers also (and thus go with approach #2). That said, I’m not sure how the move initializers will be spelled or if that is the likely direction. If it won’t add these, then it is probably better to go with approach #1. John, what do you think?

Presumably, binding to Python is going to require some compiler effort—defining how it is that Python objects are initialized/copied/moved/destroyed seems like a reasonable part of that effort.

Actually no. If we add these three proposals, there is no other python (or perl, etc…) specific support needed. It is all implementable in the overlay.

Support for working with Python objects would be implementable in the overlay, but the result isn’t necessarily ergonomic (e.g., my “as!” case from a Python-generated integer object to Int, shown above). That might be fine! More comments on this below.

// Magic, allows anyobject-like member lookup on a type when lookup otherwise fails.
protocol DynamicMemberLookupable {
  associatedtype MemberLookupResultType
  func dynamicMemberLookup(_ : String) -> MemberLookupResultType
}

AnyObject lookup looks for an actual declaration on any type anywhere. One could extend that mechanism to, say, return all Python methods and assume that you can call any Python method with any PythonObject instance. AnyObject lookup is fairly unprincipled as a language feature, because there’s no natural scope in which to perform name lookup, and involves hacks at many levels that don’t always work (e.g., AnyObject lookup… sometimes… fails across multiple source files for hard-to-explain reasons). You’re taking on that brokenness if you expand AnyObject lookup to another ecosystem.

Yeah, sorry, that’s not what I meant:

(Good)

Although it doesn’t really seem like AnyObject lookup is the thing you’re asking for here. It seems more like you want dynamicMemberLookup(_:) to capture “self” and the method name, and then be a callable thing as below…

That’s what I meant :-).

A type that implements this magic protocol would never fail name lookup: “foo.bar” would always fall back to calling: foo.dynamicMemberLookup(“bar")

it’s simple and more predictable than AnyObject, it also matches what dynamic languages like Python needs.

// Magic, allows “overloaded/sugared postfix ()”.
protocol CustomCallable {
func call( …)
}

The only tricky thing about this is the call part of things. At least in the case of python, we want something like this:

  foo.bar(1, 2, a: x, b: y)

to turn into:
foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y])

We don’t want this to be a memberlookup of a value that has “bar” as a basename and “a:” and “b:” as parameter labels.

Well, I think the MemberLookupResult is going to get the name “bar”, argument labels “_:_:a:b:”, and arguments “1”, “2”, “x”, “y”, because that’s the Swift model of argument labels. It can then reshuffle them however it needs to for the underlying interaction with the Python interpreter.

There are definite design trade-offs here. With AnyObject lookup, it’s a known-broken feature but because it depends on synthesized Swift method declarations, it’ll behave mostly the same way as other Swift method declarations—static overloading, known (albeit weak) type signatures, etc. But, it might require more of Python’s model to be grafted onto those method declarations. With dynamic member lookup, you’re throwing away all type safety (even for a motivated Python developer who might be willing to annotate APIs with types) and creating a general language mechanism for doing that.

Right, something like this could definitely work, but keep in mind that the Swift compiler knows nothing about Python declarations.

Perhaps the most straight-forward thing would be to support:

protocol CustomCallable {
func call(…arg list as array and kw args...)
func callMember(_ : String, …otherstuffabove...)
}

Given this, the compiler could map:

pythonThing(42) -> pythonThing.call([42])
pythonThing.method(a: 42) -> pythonThing.callMember(“method”, kwargs: [“a”: 42])

This is the simplest way to map the Swift semantics (where kw args are part of compound lookups) into the Python world.

Okay, I agree that this gets Swift syntax into a call to the Python interpreter fairly quickly. Over-architecting for the sake of discussion:

protocol CustomCallable {
  associatedtype CustomArgument
  associatedtype CustomNominal
  associatedtype CustomResult

  func callMember(self: CustomNominal, functionName: String, arguments: [(String, CustomArgument)]) throws -> CustomResult
  // something for class/static members
}

But this is *all* dynamic, even when one could map much of Python’s type information into Swift. For example, let’s take this:

class Dog:

    def __init__(self, name):
        self.name = name
        self.tricks = # creates a new empty list for each dog

    def add_trick(self, trick):
        self.tricks.append(trick)

With your don’t-modify-the-compiler approach, how can I create a Dog instance and add a trick? I probably need to look up the class by name, call __init__ manually, etc.

  let dogClass = python_getClassByName(“Dog”) // implemented in the Python “overlay’, I guess
  let dog = python_createInstance(dogClass) // implemented in the Python “overlay’, I guess
  dog.__init__(“Brianna”) // uses CustomCallable’s callMember
  dog.add_trick(“Roll over”) // uses CustomCallable’s callMember

With compiler integration,

  class Dog : PythonObject {
    init(_ name: Pythonable)
    func add_trick(_ trick: Pythonable)
  }

One could possibly bridge the gap with a code generator of some sort, that (for example) maps Dog’s __init__’s to global functions

  func Dog(_ name: Pythonable) -> Pythonable {
      let dogClass = python_getClassByName(“Dog”) // implemented in the Python “overlay’, I guess
      let dog = python_createInstance(dogClass) // implemented in the Python “overlay’, I guess
      dog.__init__(“Brianna”) // uses CustomCallable’s callMember
      return dog
  }

and maybe turns all Python methods into extensions on Pythonable:

  extension Pythonable {
      func add_trick
  }

With either the true “Python importer” solution or this code-generation solution, you at least get some level of code completion and basic sanity checking “for free”. In other words, you get some of the benefits of having a statically-type-checked language while still working on dynamic Pythonable types.

  - Doug

···

On Oct 30, 2017, at 9:43 PM, Chris Lattner <clattner@nondot.org> wrote:
On Oct 30, 2017, at 1:25 PM, Douglas Gregor <dgregor@apple.com <mailto:dgregor@apple.com>> wrote:

JohnMC: question for you below.

Thinking about the Perl case makes it clear to me that this should not be built into the compiler as a monolithic thing. Perl supports several different types (SV/AV/HV) which represent different concepts (scalars, arrays, hashes) so baking it all together into one thing would be the wrong way to map it. In fact, the magic we need is pretty small, and seems generally useful for other things. Consider a design like this:

// not magic, things like Int, String and many other conform to this.
protocol Pythonable {
init?(_ : PythonObject)
func toPython() -> PythonObject
}

It’s not magic unless you expect the compiler or runtime to help with conversion between Int/String/etc. and PythonObject, as with _ObjectiveCBridgeable.

Right, as I said above “not magic”. The conformances would be manually implemented in the Python overlay. This provides a free implicit conversion from "T -> Pythonable” for the T’s we care about, and a failable init from Python back to Swift types.

// Not magic.
struct PythonObject : /*protocols below*/ {
  var state : UnsafePointer<PyObject>

  subscript(_ : Pythonable…) -> PythonObject {
    ...
  }
}

// Magic, must be on the struct definition.
// Could alternatively allow custom copy/move/… ctors like C++.
protocol CustomValueWitnessTable {
static func init(..)
static func copy(..)
static func move(..)
static func destroy(..)
}

Swift’s implementation model supports this. As a surface-level construct it’s going to be mired in UnsafeMutablePointers, and it’s not at all clear to me that we want this level of control there.

There are two ways to implement it:
1) static func’s like the above, which are implemented as UnsafePointer's
2) Proper language syntax akin to the C++ “rule of 5”.

The pro’s and con’s of the first approach:

pro) full explicit control over what happens
pro) no other new language features necessary to implement this. The second approach would need something like ownership to be in place.
con) injects another avenue of unsafety (though it would be explicit, so it isn’t that bad). It isn’t obvious to me that approach #2 can be safe, but I haven’t thought about it enough.
???) discourages people from using this protocol because of its explicit unsafety.

I can think of two things that could tip the scale of the discussion:

a) The big question is whether we *want* the ability to write custom rule-of-5 style behavior for structs, or if we want it to only be used in extreme cases (like bridging interop in this proposal). If we *want* to support it someday, then adding proper “safe” support is best (if possible). If we don’t *want* people to use it, then making it Unsafe and ugly is a reasonable way to go.

b) The ownership proposal is likely to add deinit's to structs. If it also adds explicit move initializers, then it is probably the right thing to add copy initializers also (and thus go with approach #2). That said, I’m not sure how the move initializers will be spelled or if that is the likely direction. If it won’t add these, then it is probably better to go with approach #1. John, what do you think?

I was hoping not to have to add explicit move/copy initializers, perhaps ever. I would suggest one of two things:
  - We could add a Builtin type for these types in Swift. Because of our architecture, this is mostly an IRGen task.
  - We could start working on C++ import. C++ import is a huge task, but we don't really need most of it for this.

John.

···

On Oct 31, 2017, at 12:43 AM, Chris Lattner <clattner@nondot.org> wrote:
On Oct 30, 2017, at 1:25 PM, Douglas Gregor <dgregor@apple.com <mailto:dgregor@apple.com>> wrote:

Presumably, binding to Python is going to require some compiler effort—defining how it is that Python objects are initialized/copied/moved/destroyed seems like a reasonable part of that effort.

Actually no. If we add these three proposals, there is no other python (or perl, etc…) specific support needed. It is all implementable in the overlay.

// Magic, allows anyobject-like member lookup on a type when lookup otherwise fails.
protocol DynamicMemberLookupable {
  associatedtype MemberLookupResultType
  func dynamicMemberLookup(_ : String) -> MemberLookupResultType
}

AnyObject lookup looks for an actual declaration on any type anywhere. One could extend that mechanism to, say, return all Python methods and assume that you can call any Python method with any PythonObject instance. AnyObject lookup is fairly unprincipled as a language feature, because there’s no natural scope in which to perform name lookup, and involves hacks at many levels that don’t always work (e.g., AnyObject lookup… sometimes… fails across multiple source files for hard-to-explain reasons). You’re taking on that brokenness if you expand AnyObject lookup to another ecosystem.

Yeah, sorry, that’s not what I meant:

Although it doesn’t really seem like AnyObject lookup is the thing you’re asking for here. It seems more like you want dynamicMemberLookup(_:) to capture “self” and the method name, and then be a callable thing as below…

That’s what I meant :-).

A type that implements this magic protocol would never fail name lookup: “foo.bar” would always fall back to calling: foo.dynamicMemberLookup(“bar")

it’s simple and more predictable than AnyObject, it also matches what dynamic languages like Python needs.

// Magic, allows “overloaded/sugared postfix ()”.
protocol CustomCallable {
func call( …)
}

The only tricky thing about this is the call part of things. At least in the case of python, we want something like this:

  foo.bar(1, 2, a: x, b: y)

to turn into:
foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y])

We don’t want this to be a memberlookup of a value that has “bar” as a basename and “a:” and “b:” as parameter labels.

Well, I think the MemberLookupResult is going to get the name “bar”, argument labels “_:_:a:b:”, and arguments “1”, “2”, “x”, “y”, because that’s the Swift model of argument labels. It can then reshuffle them however it needs to for the underlying interaction with the Python interpreter.

There are definite design trade-offs here. With AnyObject lookup, it’s a known-broken feature but because it depends on synthesized Swift method declarations, it’ll behave mostly the same way as other Swift method declarations—static overloading, known (albeit weak) type signatures, etc. But, it might require more of Python’s model to be grafted onto those method declarations. With dynamic member lookup, you’re throwing away all type safety (even for a motivated Python developer who might be willing to annotate APIs with types) and creating a general language mechanism for doing that.

Right, something like this could definitely work, but keep in mind that the Swift compiler knows nothing about Python declarations.

Perhaps the most straight-forward thing would be to support:

protocol CustomCallable {
func call(…arg list as array and kw args...)
func callMember(_ : String, …otherstuffabove...)
}

Given this, the compiler could map:

pythonThing(42) -> pythonThing.call([42])
pythonThing.method(a: 42) -> pythonThing.callMember(“method”, kwargs: [“a”: 42])

This is the simplest way to map the Swift semantics (where kw args are part of compound lookups) into the Python world.

-Chris

I can think of two things that could tip the scale of the discussion:

a) The big question is whether we *want* the ability to write custom
rule-of-5 style behavior for structs, or if we want it to only be used in
extreme cases (like bridging interop in this proposal). If we *want* to
support it someday, then adding proper “safe” support is best (if
possible). If we don’t *want* people to use it, then making it Unsafe and
ugly is a reasonable way to go.

b) The ownership proposal is likely to add deinit's to structs. If it
also adds explicit move initializers, then it is probably the right thing
to add copy initializers also (and thus go with approach #2). That said,
I’m not sure how the move initializers will be spelled or if that is the
likely direction. If it won’t add these, then it is probably better to go
with approach #1. John, what do you think?

I was hoping not to have to add explicit move/copy initializers, perhaps
ever.

Can you elaborate more on why? I'm even more interested in your the
rationale more than the outcome :-)

I would suggest one of two things:

  - We could add a Builtin type for these types in Swift. Because of our
architecture, this is mostly an IRGen task.

Yes, this could definitely work.

  - We could start working on C++ import. C++ import is a huge task, but
we don't really need most of it for this.

This could work, but it would be unfortunate to push people to having to
write (unsafe) C++ and import it into Swift to achieve simple things like
this. I'd prefer going the direction of suggestion #1 above and allowing a
"pure swift and explicitly unsafe" solution.

-Chris

···

On Mon, Oct 30, 2017 at 10:55 PM, John McCall <rjmccall@apple.com> wrote:

I tried to send a response to this earlier today but it apparently didn’t get out. If it did, then I apologize in advance for the dupe:

// not magic, things like Int, String and many other conform to this.
protocol Pythonable {
init?(_ : PythonObject)
func toPython() -> PythonObject
}

It’s not magic unless you expect the compiler or runtime to help with conversion between Int/String/etc. and PythonObject, as with _ObjectiveCBridgeable.

Right, as I said above “not magic”. The conformances would be manually implemented in the Python overlay. This provides a free implicit conversion from "T -> Pythonable” for the T’s we care about, and a failable init from Python back to Swift types.

Note that, under this scheme,

  let p: Pythonable = 17
  let i: Int = p as! i

will work if Int is Pythonable, but not when p comes back from Python.

Right. I don't expect users to traffic in the Pythonable type, it would be something that is used internally by the Python overlay. The only time I'd expect them to see it is when they want to make one of their types Pythonable (which would be rare, but needs to be possible).

I can think of two things that could tip the scale of the discussion:

a) The big question is whether we *want* the ability to write custom rule-of-5 style behavior for structs, or if we want it to only be used in extreme cases (like bridging interop in this proposal). If we *want* to support it someday, then adding proper “safe” support is best (if possible). If we don’t *want* people to use it, then making it Unsafe and ugly is a reasonable way to go.

b) The ownership proposal is likely to add deinit's to structs. If it also adds explicit move initializers, then it is probably the right thing to add copy initializers also (and thus go with approach #2). That said, I’m not sure how the move initializers will be spelled or if that is the likely direction. If it won’t add these, then it is probably better to go with approach #1. John, what do you think?

Presumably, binding to Python is going to require some compiler effort—defining how it is that Python objects are initialized/copied/moved/destroyed seems like a reasonable part of that effort.

Actually no. If we add these three proposals, there is no other python (or perl, etc…) specific support needed. It is all implementable in the overlay.

Support for working with Python objects would be implementable in the overlay, but the result isn’t necessarily ergonomic (e.g., my “as!” case from a Python-generated integer object to Int, shown above). That might be fine! More comments on this below.

Yeah, I don't think that matters. The preferred way to do this is to use:

  Int(somePythonValue)

which would be a failable conversion. somePythonValue would be a PythonObject (the struct) not "Pythonable"

// Magic, allows anyobject-like member lookup on a type when lookup otherwise fails.
protocol DynamicMemberLookupable {
  associatedtype MemberLookupResultType
  func dynamicMemberLookup(_ : String) -> MemberLookupResultType
}

AnyObject lookup looks for an actual declaration on any type anywhere. One could extend that mechanism to, say, return all Python methods and assume that you can call any Python method with any PythonObject instance. AnyObject lookup is fairly unprincipled as a language feature, because there’s no natural scope in which to perform name lookup, and involves hacks at many levels that don’t always work (e.g., AnyObject lookup… sometimes… fails across multiple source files for hard-to-explain reasons). You’re taking on that brokenness if you expand AnyObject lookup to another ecosystem.

Yeah, sorry, that’s not what I meant:

(Good)

Also, for sake of discussion, we’d have to figure out what the type of MemberLookupResultType would be for Python. I can see several choices:

1) Make it return a "PythonObject!”
2) Make it strict, returning a PythonObject or trapping.
3) Make PythonObject itself nullable internally and return a null PythonObject.

#1 matches Python semantics directly, because it allows clients who care to check, but those who don't can ignore it. The only problem I anticipate is that it will break things like:

let x = foo.bar
let y = x.thing // fails, because x implicitly promoted to PythonObject?

#3 is gross and cuts against lots of things in Swift (recall when UnsafePointer itself was implicitly nullable, lets not go back to those bad old days). I think that #2 is the least bad tradeoff.

Right, something like this could definitely work, but keep in mind that the Swift compiler knows nothing about Python declarations.

Perhaps the most straight-forward thing would be to support:

protocol CustomCallable {
func call(…arg list as array and kw args...)
func callMember(_ : String, …otherstuffabove...)
}

Given this, the compiler could map:

pythonThing(42) -> pythonThing.call([42])
pythonThing.method(a: 42) -> pythonThing.callMember(“method”, kwargs: [“a”: 42])

This is the simplest way to map the Swift semantics (where kw args are part of compound lookups) into the Python world.

Okay, I agree that this gets Swift syntax into a call to the Python interpreter fairly quickly. Over-architecting for the sake of discussion:

protocol CustomCallable {
  associatedtype CustomArgument
  associatedtype CustomNominal
  associatedtype CustomResult

  func callMember(self: CustomNominal, functionName: String, arguments: [(String, CustomArgument)]) throws -> CustomResult
  // something for class/static members
}

But this is *all* dynamic, even when one could map much of Python’s type information into Swift. For example, let’s take this:

Yes, something like this is what I had in mind, but I think that functionName should only be required to be StringLiteralConvertible (no need to actually synthesize a real swift string).

Since you bring it up, Python exceptions will be annoying - As with other languages, Python can throw from an arbitrary expression. Modeling everything as throws in Swift would be super-annoying and unergonomic for the programmer, because we'd require 'try' everywhere. Thoughts on what to do about that are welcome!

class Dog:

    def __init__(self, name):
        self.name = name
        self.tricks = # creates a new empty list for each dog

    def add_trick(self, trick):
        self.tricks.append(trick)

With your don’t-modify-the-compiler approach, how can I create a Dog instance and add a trick? I probably need to look up the class by name, call __init__ manually, etc.

  let dogClass = python_getClassByName(“Dog”) // implemented in the Python “overlay’, I guess
  let dog = python_createInstance(dogClass) // implemented in the Python “overlay’, I guess
  dog.__init__(“Brianna”) // uses CustomCallable’s callMember
  dog.add_trick(“Roll over”) // uses CustomCallable’s callMember

I-am-not-a-python-expert, but I'd expect this to work:

  let dogModule = Python.import("DogModule")
  dogModule.Dog("jckarter").add_trick("SILGenGen”)
  let dog = dogModule.Dog(“Brianna”)
  dog.add_trick(“Roll over)

or equivalently:

  let Dog = Python.import(“DogModule.Dog")
  Dog("jckarter").add_trick("SILGenGen”)
  let dog = Dog(“Brianna”)
  dog.add_trick(“Roll over)

Seems pretty nice to me, with zero “Swift compiler knowledge of Python” required.

With compiler integration,

  class Dog : PythonObject {
    init(_ name: Pythonable)
    func add_trick(_ trick: Pythonable)
  }

Something like this is possible, but would be substantially more work, be substantially more invasive, and would set the precedent that every other dynamic language would get support hacked directly into Swift. The only reason I can see this being useful is if we wanted to support the optional typing annotations in Python. While this would be "nice to have", I think the cost/benefit tradeoff involved is totally wrong for Swift.

With either the true “Python importer” solution or this code-generation solution, you at least get some level of code completion and basic sanity checking “for free”. In other words, you get some of the benefits of having a statically-type-checked language while still working on dynamic Pythonable types.

We don't need to make Swift better at Python than Python itself is :-)

-Chris

···

On Oct 30, 2017, at 10:47 PM, Douglas Gregor <dgregor@apple.com> wrote:

Also, for sake of discussion, we’d have to figure out what the type of MemberLookupResultType would be for Python. I can see several choices:

1) Make it return a "PythonObject!”
2) Make it strict, returning a PythonObject or trapping.
3) Make PythonObject itself nullable internally and return a null PythonObject.

#1 matches Python semantics directly, because it allows clients who care to check, but those who don't can ignore it. The only problem I anticipate is that it will break things like:

let x = foo.bar
let y = x.thing // fails, because x implicitly promoted to PythonObject?

#3 is gross and cuts against lots of things in Swift (recall when UnsafePointer itself was implicitly nullable, lets not go back to those bad old days). I think that #2 is the least bad tradeoff.

I agree, PythonObject-or-die is the right trade-off.

Yes, something like this is what I had in mind, but I think that functionName should only be required to be StringLiteralConvertible (no need to actually synthesize a real swift string).

Since you bring it up, Python exceptions will be annoying - As with other languages, Python can throw from an arbitrary expression. Modeling everything as throws in Swift would be super-annoying and unergonomic for the programmer, because we'd require 'try' everywhere. Thoughts on what to do about that are welcome!

Requiring ‘try’ on every statement is annoying, but not having the ability to catch python exceptions is annoying too. We could probably make python exception handling an opt-in feature. For example:

try Python.do {
    let a = np.array([1, 2, 3])
    let b = np.array([[2], [4]])
    print(a.dot(b)) // matrix mul with incompatible shapes
}
catch let error as PythonException {
    // Handle PythonError.valueError(“objects are not aligned”)
}

Python.do enables exception handling for statements in the body, declared as
func `do`<T>(_ body: () throws -> T) throws -> T

When we execute python-throwing statements inside a Python.do{}, PythonException gets thrown. Otherwise, it traps.

The ‘Python.do’ function would ask the python overlay to enter an "error-catching" state when executing the body closure. We make PythonObjects internally nullable, but guarantee that they are non-null (or trap) when the overlay is not in the “error-caught” state. When a python exception is thrown in the error-catching state, the overlay enters the error-caught state and propagates null through any python computation in the body closure. After the body is executed, throw that python exception.

However, if there’s a throwing Swift statement after the throwing python statement in the body, the python exception won’t be caught first… So having the body as a non-throwing closure may be a better idea.

-Richard

···

On Oct 31, 2017, at 21:31, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

class Dog:

    def __init__(self, name):
        self.name = name
        self.tricks = # creates a new empty list for each dog

    def add_trick(self, trick):
        self.tricks.append(trick)

With your don’t-modify-the-compiler approach, how can I create a Dog instance and add a trick? I probably need to look up the class by name, call __init__ manually, etc.

  let dogClass = python_getClassByName(“Dog”) // implemented in the Python “overlay’, I guess
  let dog = python_createInstance(dogClass) // implemented in the Python “overlay’, I guess
  dog.__init__(“Brianna”) // uses CustomCallable’s callMember
  dog.add_trick(“Roll over”) // uses CustomCallable’s callMember

I-am-not-a-python-expert, but I'd expect this to work:

  let dogModule = Python.import("DogModule")
  dogModule.Dog("jckarter").add_trick("SILGenGen”)
  let dog = dogModule.Dog(“Brianna”)
  dog.add_trick(“Roll over)

or equivalently:

  let Dog = Python.import(“DogModule.Dog")
  Dog("jckarter").add_trick("SILGenGen”)
  let dog = Dog(“Brianna”)
  dog.add_trick(“Roll over)

Seems pretty nice to me, with zero “Swift compiler knowledge of Python” required.

With compiler integration,

  class Dog : PythonObject {
    init(_ name: Pythonable)
    func add_trick(_ trick: Pythonable)
  }

Something like this is possible, but would be substantially more work, be substantially more invasive, and would set the precedent that every other dynamic language would get support hacked directly into Swift. The only reason I can see this being useful is if we wanted to support the optional typing annotations in Python. While this would be "nice to have", I think the cost/benefit tradeoff involved is totally wrong for Swift.

With either the true “Python importer” solution or this code-generation solution, you at least get some level of code completion and basic sanity checking “for free”. In other words, you get some of the benefits of having a statically-type-checked language while still working on dynamic Pythonable types.

We don't need to make Swift better at Python than Python itself is :-)

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Add a `throws!` keyword to Swift, indicating that if a call is not covered by a `try` or `try?` keyword, a `try!` should be implicitly added.

···

On Oct 31, 2017, at 9:31 PM, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

Since you bring it up, Python exceptions will be annoying - As with other languages, Python can throw from an arbitrary expression. Modeling everything as throws in Swift would be super-annoying and unergonomic for the programmer, because we'd require 'try' everywhere. Thoughts on what to do about that are welcome!

--
Brent Royal-Gordon
Architechies

Also, for sake of discussion, we’d have to figure out what the type of MemberLookupResultType would be for Python. I can see several choices:

1) Make it return a "PythonObject!”
2) Make it strict, returning a PythonObject or trapping.
3) Make PythonObject itself nullable internally and return a null PythonObject.

#1 matches Python semantics directly, because it allows clients who care to check, but those who don't can ignore it. The only problem I anticipate is that it will break things like:

let x = foo.bar
let y = x.thing // fails, because x implicitly promoted to PythonObject?

#3 is gross and cuts against lots of things in Swift (recall when UnsafePointer itself was implicitly nullable, lets not go back to those bad old days). I think that #2 is the least bad tradeoff.

I agree, PythonObject-or-die is the right trade-off.

Yes, something like this is what I had in mind, but I think that functionName should only be required to be StringLiteralConvertible (no need to actually synthesize a real swift string).

Since you bring it up, Python exceptions will be annoying - As with other languages, Python can throw from an arbitrary expression. Modeling everything as throws in Swift would be super-annoying and unergonomic for the programmer, because we'd require 'try' everywhere. Thoughts on what to do about that are welcome!

Requiring ‘try’ on every statement is annoying, but not having the ability to catch python exceptions is annoying too. We could probably make python exception handling an opt-in feature. For example:

try Python.do {
    let a = np.array([1, 2, 3])
    let b = np.array([[2], [4]])
    print(a.dot(b)) // matrix mul with incompatible shapes
}
catch let error as PythonException {
    // Handle PythonError.valueError(“objects are not aligned”)
}

To correct my example:

do {
    try Python.do {
        let a = np.array([1, 2, 3])
        let b = np.array([[2], [4]])
        print(a.dot(b)) // matrix mul with incompatible shapes
    }
}
catch let error as PythonException {
    // Handle PythonError.valueError(“objects are not aligned”)
}

Maybe ‘Python.do {}’ should be called something like ‘Python.safely {}’.

-Richard

···

On Nov 1, 2017, at 03:14, Richard Wei via swift-evolution <swift-evolution@swift.org> wrote:

On Oct 31, 2017, at 21:31, Chris Lattner via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Python.do enables exception handling for statements in the body, declared as
func `do`<T>(_ body: () throws -> T) throws -> T

When we execute python-throwing statements inside a Python.do{}, PythonException gets thrown. Otherwise, it traps.

The ‘Python.do’ function would ask the python overlay to enter an "error-catching" state when executing the body closure. We make PythonObjects internally nullable, but guarantee that they are non-null (or trap) when the overlay is not in the “error-caught” state. When a python exception is thrown in the error-catching state, the overlay enters the error-caught state and propagates null through any python computation in the body closure. After the body is executed, throw that python exception.

However, if there’s a throwing Swift statement after the throwing python statement in the body, the python exception won’t be caught first… So having the body as a non-throwing closure may be a better idea.

-Richard

class Dog:

    def __init__(self, name):
        self.name = name
        self.tricks = # creates a new empty list for each dog

    def add_trick(self, trick):
        self.tricks.append(trick)

With your don’t-modify-the-compiler approach, how can I create a Dog instance and add a trick? I probably need to look up the class by name, call __init__ manually, etc.

  let dogClass = python_getClassByName(“Dog”) // implemented in the Python “overlay’, I guess
  let dog = python_createInstance(dogClass) // implemented in the Python “overlay’, I guess
  dog.__init__(“Brianna”) // uses CustomCallable’s callMember
  dog.add_trick(“Roll over”) // uses CustomCallable’s callMember

I-am-not-a-python-expert, but I'd expect this to work:

  let dogModule = Python.import("DogModule")
  dogModule.Dog("jckarter").add_trick("SILGenGen”)
  let dog = dogModule.Dog(“Brianna”)
  dog.add_trick(“Roll over)

or equivalently:

  let Dog = Python.import(“DogModule.Dog")
  Dog("jckarter").add_trick("SILGenGen”)
  let dog = Dog(“Brianna”)
  dog.add_trick(“Roll over)

Seems pretty nice to me, with zero “Swift compiler knowledge of Python” required.

With compiler integration,

  class Dog : PythonObject {
    init(_ name: Pythonable)
    func add_trick(_ trick: Pythonable)
  }

Something like this is possible, but would be substantially more work, be substantially more invasive, and would set the precedent that every other dynamic language would get support hacked directly into Swift. The only reason I can see this being useful is if we wanted to support the optional typing annotations in Python. While this would be "nice to have", I think the cost/benefit tradeoff involved is totally wrong for Swift.

With either the true “Python importer” solution or this code-generation solution, you at least get some level of code completion and basic sanity checking “for free”. In other words, you get some of the benefits of having a statically-type-checked language while still working on dynamic Pythonable types.

We don't need to make Swift better at Python than Python itself is :-)

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution