# \[Discussion\] Swift for Data Science / ML / Big Data analytics **URL:** \ **Category:** Discussion\ **Created:** [October 28, 2017, 4:44pm UTC](https://forums.swift.org/t/discussion-swift-for-data-science-ml-big-data-analytics/6905 "2017-10-28T16:44:51Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Maxim\_Veksler](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/maxim_veksler/32/1147_2.png) [@Maxim\_Veksler](https://forums.swift.org/u/Maxim_Veksler)\ **Post date:** [October 28, 2017, 4:44pm UTC](https://forums.swift.org/t/discussion-swift-for-data-science-ml-big-data-analytics/6905/1 "2017-10-28T16:44:51Z")
Hey Guys, The big data and machine learning world is dominated by Python, Scala an R. I'm a Swifter by heart, but not so much by tools of trait. I'd appreciate a constructive discussion on how that could be changed. While R is a non goal for obvious reasons, i'd argue that since both Scala and Python are general purpose languages, taking them head to head might be a low hanging fruit. To make the claim I'd like to reference to projects such as - Hadoop, Spark, Hive are all huge eco-systems which are entirely JVM based. - Apache Parquet, a highly efficient column based storage format for big data analytics which was implemented in Java, and C++. - Apache Arrow, a physical memory spec that big data systems can use to allow zero transformations on data transferred between systems. Which (for obvious reasons) focused on JVM, to C interoperability. Python's Buffer Protocol which ensures it's predominance (for the time being) as a prime candidate for data science related projects [https://jeffknupp.com/blog/2017/09/15/python-is-the-](https://jeffknupp.com/blog/2017/09/15/python-is-the-) fastest-growing-programming-language-due-to-a-feature-youve-never-heard-of/ While Swift's Memory Ownership manifesto touches similar turf discussing copy on write and optimizing memory access overhead it IMHO takes a system level perspective targeting projects such as kernel code. I'd suggest that viewing the problem from an efficient CPU/GPU data crunching machine perspective might shade a different light on the requirements and use cases. I'd be happy to learn more, and have a constructive discussion on the subject. Thank you, Max. > **···** > > -- > puıɯ ʎɯ ɯoɹɟ ʇuǝs ---
**Author:** ![Chris\_Lattner](https://avatars.discourse-cdn.com/v4/letter/c/bc8723/32.png) [@Chris\_Lattner](https://forums.swift.org/u/Chris_Lattner)\ **Post date:** [October 28, 2017, 9:10pm UTC](https://forums.swift.org/t/discussion-swift-for-data-science-ml-big-data-analytics/6905/2 "2017-10-28T21:10:40Z")
> Hey Guys, > > The big data and machine learning world is dominated by Python, Scala an R. > > I'm a Swifter by heart, but not so much by tools of trait. Hi Max, I’m very interested in this topic, with a specific focus on Python. It isn’t the immediate thing on my priority list to deal with, but I hope that we get to push on this. In short, I think we should build a simple Swift/Python interop story. This sort of thing has be built numerous times for many languages (owing to Python’s great support for embed-ability), including things like PyObjC, boost.python, and many others. In Swift, it is straightforward to make this example ([Python Numpy Tutorial (with Jupyter and Colab)](http://cs231n.github.io/python-numpy-tutorial/#numpy-arrays%5C)) look something like this:   let np = Python.import(“numpy”) // Returns a value of type Python.Object.   let a = np.array([1, 2, 3])   print(type(a)) // Whether we want to support type(x) or use the Swift equivalent would be up for discussion of course!   print(a.shape)   print(a[0], a[1], a[2])   a[0] = 5   print(a)   let b = np.array([[1,2,3],[4,5,6]])   print(b.shape)   print(b[0, 0], b[0, 1], b[1, 0]) … which is to say, exactly identical to the Python version except that new variables need to be declared with let/var. This can be done by blessing Python.Object (which is identical to “PyObject\*” at the machine level) with some special dynamic name lookup behavior: Dot syntax turns into a call to PyObject\_GetAttrString, subscripts turn into PyObject\_GetItem, calls turn into PyObject\_Call, etc. ARC would be implemented with INCREF etc. If we do this, the vast majority of the Python ecosystem should be directly usable from within Swift code, and the only a few major syntactic differences (e.g. ranges work differently). We would add failable inits to the primitive datatypes like Int/String/etc to convert Python.Object values into them, and add the corresponding non-failable conversions from Python.Object to those primitives. Overall, I think it will provide a really nice experience, and allow us to leverage the vast majority of the Python ecosystem directly in Swift code. This project would also have much more narrow impact on the Swift compiler than the ObjC importer (since it works completely differently). For a first cut, I don’t think we would have to worry about Swift classes subclassing Python classes, for example. -Chris > **···** > > On Oct 28, 2017, at 9:45 AM, Maxim Veksler via swift-evolution \ wrote: > > > I'd appreciate a constructive discussion on how that could be changed. > > > > While R is a non goal for obvious reasons, i'd argue that since both Scala and Python are general purpose languages, taking them head to head might be a low hanging fruit. > > > > To make the claim I'd like to reference to projects such as > > > > - Hadoop, Spark, Hive are all huge eco-systems which are entirely JVM based. > > - Apache Parquet, a highly efficient column based storage format for big data analytics which was implemented in Java, and C++. > > - Apache Arrow, a physical memory spec that big data systems can use to allow zero transformations on data transferred between systems. Which (for obvious reasons) focused on JVM, to C interoperability. > > > > Python's Buffer Protocol which ensures it's predominance (for the time being) as a prime candidate for data science related projects [Python is the fastest growing programming language due to a feature you've never heard of](https://jeffknupp.com/blog/2017/09/15/python-is-the-fastest-growing-programming-language-due-to-a-feature-youve-never-heard-of/) > > > > While Swift's Memory Ownership manifesto touches similar turf discussing copy on write and optimizing memory access overhead it IMHO takes a system level perspective targeting projects such as kernel code. I'd suggest that viewing the problem from an efficient CPU/GPU data crunching machine perspective might shade a different light on the requirements and use cases. > > > > I'd be happy to learn more, and have a constructive discussion on the subject. > > > > Thank you, > > Max. > > > > -- > > puıɯ ʎɯ ɯoɹɟ ʇuǝs > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > swift-evolution mailing list > > swift-evolution@swift.org > > [https://lists.swift.org/mailman/listinfo/swift-evolution](https://lists.swift.org/mailman/listinfo/swift-evolution) ---
**Author:** ![ahti](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ahti/32/604_2.png) [@ahti](https://forums.swift.org/u/ahti)\ **Post date:** [October 29, 2017, 11:04am UTC](https://forums.swift.org/t/discussion-swift-for-data-science-ml-big-data-analytics/6905/3 "2017-10-29T11:04:05Z")
That sounds like a very interesting prospect. Do you think it would make sense to make the language features that facilitate this (dynamic dispatch of method calls, property accesses, subscript and ARC) available to Swift classes annotated in some way, so that interop like this can be implemented as a library without special treatment by the Swift compiler? This could also enable more dynamic DSL like features. — Lukas > **···** > > > On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution \ wrote: > > > > … which is to say, exactly identical to the Python version except that new variables need to be declared with let/var. This can be done by blessing Python.Object (which is identical to “PyObject\*” at the machine level) with some special dynamic name lookup behavior: Dot syntax turns into a call to PyObject\_GetAttrString, subscripts turn into PyObject\_GetItem, calls turn into PyObject\_Call, etc. ARC would be implemented with INCREF etc. ---
**Author:** ![Chris\_Lattner](https://avatars.discourse-cdn.com/v4/letter/c/bc8723/32.png) [@Chris\_Lattner](https://forums.swift.org/u/Chris_Lattner)\ **Post date:** [October 29, 2017, 3:23pm UTC](https://forums.swift.org/t/discussion-swift-for-data-science-ml-big-data-analytics/6905/4 "2017-10-29T15:23:41Z")
I haven’t explored enough of the design space to be sure, but I’d want to make sure that a library version of this could be done without giving up ergonomics of the result. If you were interested in being able to interop with other languages that are dynamically typed and reference counted, then something like this could be possible in principle: protocol DynamicDispatchable { // Protocol is “magic" known by the compiler.   func retain()   func release()   func memberLookup(\_ : String) -\> Self   func subscript\(\_ : T) -\> Self   func call(\_ args: [Self]) -\> Self } module Python {   struct Object : DynamicDispatchable {     var state : UnsafePointer\     func retain() {        INCREF(self)    }      func memberLookup(\_ : String) -\> Object {         PyObject\_GetAttrString(…)      }     etc   } module Perl5 {    struct Object : DynamicDispatchable {     var state : UnsafePointer\     func retain() {        SvREFCNT\_inc(self)    } …. Are there other uses for such a thing? -Chris > **···** > > > On Oct 29, 2017, at 4:04 AM, Lukas Stabe \ wrote: > > > > > On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution \ wrote: > > > > > > … which is to say, exactly identical to the Python version except that new variables need to be declared with let/var. This can be done by blessing Python.Object (which is identical to “PyObject\*” at the machine level) with some special dynamic name lookup behavior: Dot syntax turns into a call to PyObject\_GetAttrString, subscripts turn into PyObject\_GetItem, calls turn into PyObject\_Call, etc. ARC would be implemented with INCREF etc. > > > > That sounds like a very interesting prospect. Do you think it would make sense to make the language features that facilitate this (dynamic dispatch of method calls, property accesses, subscript and ARC) available to Swift classes annotated in some way, so that interop like this can be implemented as a library without special treatment by the Swift compiler? This could also enable more dynamic DSL like features. ---
**Author:** ![Chris\_Lattner](https://avatars.discourse-cdn.com/v4/letter/c/bc8723/32.png) [@Chris\_Lattner](https://forums.swift.org/u/Chris_Lattner)\ **Post date:** [October 29, 2017, 8:34pm UTC](https://forums.swift.org/t/discussion-swift-for-data-science-ml-big-data-analytics/6905/5 "2017-10-29T20:34:05Z")
> > > … which is to say, exactly identical to the Python version except that new variables need to be declared with let/var. This can be done by blessing Python.Object (which is identical to “PyObject\*” at the machine level) with some special dynamic name lookup behavior: Dot syntax turns into a call to PyObject\_GetAttrString, subscripts turn into PyObject\_GetItem, calls turn into PyObject\_Call, etc. ARC would be implemented with INCREF etc. > > > > That sounds like a very interesting prospect. Do you think it would make sense to make the language features that facilitate this (dynamic dispatch of method calls, property accesses, subscript and ARC) available to Swift classes annotated in some way, so that interop like this can be implemented as a library without special treatment by the Swift compiler? This could also enable more dynamic DSL like features. > > I haven’t explored enough of the design space to be sure, but I’d want to make sure that a library version of this could be done without giving up ergonomics of the result. If you were interested in being able to interop with other languages that are dynamically typed and reference counted, then something like this could be possible in principle: Thinking about the Perl case makes it clear to me that this should not be built into the compiler as a monolithic thing. Perl supports several different types (SV/AV/HV) which represent different concepts (scalars, arrays, hashes) so baking it all together into one thing would be the wrong way to map it. In fact, the magic we need is pretty small, and seems generally useful for other things. Consider a design like this: // not magic, things like Int, String and many other conform to this. protocol Pythonable {   init?(\_ : PythonObject)   func toPython() -\> PythonObject } // Not magic. struct PythonObject : /\*protocols below\*/ {    var state : UnsafePointer\    subscript(\_ : Pythonable…) -\> PythonObject {      ...    } } // Magic, must be on the struct definition. // Could alternatively allow custom copy/move/… ctors like C++. protocol CustomValueWitnessTable {   static func init(..)   static func copy(..)   static func move(..)   static func destroy(..) } // Magic, allows anyobject-like member lookup on a type when lookup otherwise fails. protocol DynamicMemberLookupable {    associatedtype MemberLookupResultType    func dynamicMemberLookup(\_ : String) -\> MemberLookupResultType } // Magic, allows “overloaded/sugared postfix ()”. protocol CustomCallable {   func call( …) } The only tricky thing about this is the call part of things. At least in the case of python, we want something like this:    foo.bar(1, 2, a: x, b: y) to turn into:   foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y]) We don’t want this to be a memberlookup of a value that has “bar” as a basename and “a:” and “b:” as parameter labels. -Chris > **···** > > > On Oct 29, 2017, at 8:23 AM, Chris Lattner via swift-evolution \ wrote: > > > > > On Oct 29, 2017, at 4:04 AM, Lukas Stabe \ wrote: > > > > > > > On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution \ wrote: > > > protocol DynamicDispatchable { // Protocol is “magic" known by the compiler. > > func retain() > > func release() > > func memberLookup(\_ : String) -\> Self > > func subscript\(\_ : T) -\> Self > > func call(\_ args: [Self]) -\> Self > > } > > > > module Python { > > struct Object : DynamicDispatchable { > >    var state : UnsafePointer\ > > > >    func retain() { > >       INCREF(self) > >   } > > > >     func memberLookup(\_ : String) -\> Object { > >        PyObject\_GetAttrString(…) > >     } > >    etc > > } > > > > module Perl5 { > >   struct Object : DynamicDispatchable { > >    var state : UnsafePointer\ > > > >    func retain() { > >       SvREFCNT\_inc(self) > >   } > > …. > > > > Are there other uses for such a thing? > > > > -Chris > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > swift-evolution mailing list > > swift-evolution@swift.org > > [https://lists.swift.org/mailman/listinfo/swift-evolution](https://lists.swift.org/mailman/listinfo/swift-evolution) ---
**Author:** ![ckeithray](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ckeithray/32/2317_2.png) [@ckeithray](https://forums.swift.org/u/ckeithray)\ **Post date:** [October 29, 2017, 10:23pm UTC](https://forums.swift.org/t/discussion-swift-for-data-science-ml-big-data-analytics/6905/6 "2017-10-29T22:23:57Z")
It would be really nice if Cython code could be linked into iOS and Mac apps written in Swift or Objective-C. [http://cython.org](http://cython.org) > **···** > > -- > C. Keith Ray > Senior Software Engineer / Trainer / Agile Coach > \* [http://www.thirdfoundationsw.com/keith\_ray\_resume\_2014\_long.pdf](http://www.thirdfoundationsw.com/keith_ray_resume_2014_long.pdf) > > > On Oct 29, 2017, at 1:34 PM, Chris Lattner via swift-evolution \ wrote: > > > > > On Oct 29, 2017, at 8:23 AM, Chris Lattner via swift-evolution \ wrote: > > > > > > > On Oct 29, 2017, at 4:04 AM, Lukas Stabe \ wrote: > > > > > > > > > On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution \ wrote: > > > > > > > > > > … which is to say, exactly identical to the Python version except that new variables need to be declared with let/var. This can be done by blessing Python.Object (which is identical to “PyObject\*” at the machine level) with some special dynamic name lookup behavior: Dot syntax turns into a call to PyObject\_GetAttrString, subscripts turn into PyObject\_GetItem, calls turn into PyObject\_Call, etc. ARC would be implemented with INCREF etc. > > > > > > > > That sounds like a very interesting prospect. Do you think it would make sense to make the language features that facilitate this (dynamic dispatch of method calls, property accesses, subscript and ARC) available to Swift classes annotated in some way, so that interop like this can be implemented as a library without special treatment by the Swift compiler? This could also enable more dynamic DSL like features. > > > > > > I haven’t explored enough of the design space to be sure, but I’d want to make sure that a library version of this could be done without giving up ergonomics of the result. If you were interested in being able to interop with other languages that are dynamically typed and reference counted, then something like this could be possible in principle: > > > > Thinking about the Perl case makes it clear to me that this should not be built into the compiler as a monolithic thing. Perl supports several different types (SV/AV/HV) which represent different concepts (scalars, arrays, hashes) so baking it all together into one thing would be the wrong way to map it. In fact, the magic we need is pretty small, and seems generally useful for other things. Consider a design like this: > > > > // not magic, things like Int, String and many other conform to this. > > protocol Pythonable { > > init?(\_ : PythonObject) > > func toPython() -\> PythonObject > > } > > > > // Not magic. > > struct PythonObject : /\*protocols below\*/ { > >   var state : UnsafePointer\ > > > >   subscript(\_ : Pythonable…) -\> PythonObject { > >     ... > >   } > > } > > > > // Magic, must be on the struct definition. > > // Could alternatively allow custom copy/move/… ctors like C++. > > protocol CustomValueWitnessTable { > > static func init(..) > > static func copy(..) > > static func move(..) > > static func destroy(..) > > } > > > > // Magic, allows anyobject-like member lookup on a type when lookup otherwise fails. > > protocol DynamicMemberLookupable { > >   associatedtype MemberLookupResultType > >   func dynamicMemberLookup(\_ : String) -\> MemberLookupResultType > > } > > > > // Magic, allows “overloaded/sugared postfix ()”. > > protocol CustomCallable { > > func call( …) > > } > > > > The only tricky thing about this is the call part of things. At least in the case of python, we want something like this: > > > >   foo.bar(1, 2, a: x, b: y) > > > > to turn into: > > foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y]) > > > > We don’t want this to be a memberlookup of a value that has “bar” as a basename and “a:” and “b:” as parameter labels. > > > > -Chris > > > > > protocol DynamicDispatchable { // Protocol is “magic" known by the compiler. > > > func retain() > > > func release() > > > func memberLookup(\_ : String) -\> Self > > > func subscript\(\_ : T) -\> Self > > > func call(\_ args: [Self]) -\> Self > > > } > > > > > > module Python { > > > struct Object : DynamicDispatchable { > > >   var state : UnsafePointer\ > > > > > >   func retain() { > > >      INCREF(self) > > > } > > > > > >    func memberLookup(\_ : String) -\> Object { > > >       PyObject\_GetAttrString(…) > > >    } > > >   etc > > > } > > > > > > module Perl5 { > > > struct Object : DynamicDispatchable { > > >   var state : UnsafePointer\ > > > > > >   func retain() { > > >      SvREFCNT\_inc(self) > > > } > > > …. > > > > > > Are there other uses for such a thing? > > > > > > -Chris > > > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > swift-evolution mailing list > > > swift-evolution@swift.org > > > [https://lists.swift.org/mailman/listinfo/swift-evolution](https://lists.swift.org/mailman/listinfo/swift-evolution) > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > swift-evolution mailing list > > swift-evolution@swift.org \ > > [https://lists.swift.org/mailman/listinfo/swift-evolution](https://lists.swift.org/mailman/listinfo/swift-evolution) ---
**Author:** ![ahti](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ahti/32/604_2.png) [@ahti](https://forums.swift.org/u/ahti)\ **Post date:** [October 30, 2017, 5:04am UTC](https://forums.swift.org/t/discussion-swift-for-data-science-ml-big-data-analytics/6905/7 "2017-10-30T05:04:15Z")
> // Magic, allows “overloaded/sugared postfix ()”. > protocol CustomCallable { > func call( …) > } > > The only tricky thing about this is the call part of things. At least in the case of python, we want something like this: > >   foo.bar(1, 2, a: x, b: y) > > to turn into: > foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y]) > > We don’t want this to be a memberlookup of a value that has “bar” as a basename and “a:” and “b:” as parameter labels. I don't know much about Swift internals, but since you can already access a function member without calling it, the parsing part of this sounds like it would be straightforward. If CustomCallable looked something like this: protocol CustomCallable {     associatedtype Parameter     associatedtype Result     func call(\_ parameterList: [(label: String?, value: Parameter)]) -\> Result } implementations could decide how they want to handle named/unnamed parameters etc themselves. This approach would currently come with some limitations though: No throwing (this could be addressed by a separate CustomThrowingCallable protocol), and – since currently non-nominal types can't adopt protocols – an implementation of CustomCallable could not accept, for example, both normal values and closures/tuples. > > Are there other uses for such a thing? DynamicMemberLookupable could enable libraries dealing with key-value data, like SwiftyJSON for example, to let users write myJson.someKey.nested instead of myJson["someKey"]["nested"]. I'm sure there are more uses like this that are not integrations with other languages. — Lukas > **···** > > > On 29. Oct 2017, at 21:34, Chris Lattner \ wrote: ---
**Author:** ![Douglas\_Gregor](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/douglas_gregor/32/1362_2.png) [@Douglas\_Gregor](https://forums.swift.org/u/Douglas_Gregor)\ **Post date:** [October 30, 2017, 8:25pm UTC](https://forums.swift.org/t/discussion-swift-for-data-science-ml-big-data-analytics/6905/8 "2017-10-30T20:25:24Z")
> > > > … which is to say, exactly identical to the Python version except that new variables need to be declared with let/var. This can be done by blessing Python.Object (which is identical to “PyObject\*” at the machine level) with some special dynamic name lookup behavior: Dot syntax turns into a call to PyObject\_GetAttrString, subscripts turn into PyObject\_GetItem, calls turn into PyObject\_Call, etc. ARC would be implemented with INCREF etc. > > > > > > That sounds like a very interesting prospect. Do you think it would make sense to make the language features that facilitate this (dynamic dispatch of method calls, property accesses, subscript and ARC) available to Swift classes annotated in some way, so that interop like this can be implemented as a library without special treatment by the Swift compiler? This could also enable more dynamic DSL like features. > > > > I haven’t explored enough of the design space to be sure, but I’d want to make sure that a library version of this could be done without giving up ergonomics of the result. If you were interested in being able to interop with other languages that are dynamically typed and reference counted, then something like this could be possible in principle: > > Thinking about the Perl case makes it clear to me that this should not be built into the compiler as a monolithic thing. Perl supports several different types (SV/AV/HV) which represent different concepts (scalars, arrays, hashes) so baking it all together into one thing would be the wrong way to map it. In fact, the magic we need is pretty small, and seems generally useful for other things. Consider a design like this: > > // not magic, things like Int, String and many other conform to this. > protocol Pythonable { > init?(\_ : PythonObject) > func toPython() -\> PythonObject > } It’s not magic unless you expect the compiler or runtime to help with conversion between Int/String/etc. and PythonObject, as with \_ObjectiveCBridgeable. > // Not magic. > struct PythonObject : /\*protocols below\*/ { >   var state : UnsafePointer\ > >   subscript(\_ : Pythonable…) -\> PythonObject { >     ... >   } > } > > // Magic, must be on the struct definition. > // Could alternatively allow custom copy/move/… ctors like C++. > protocol CustomValueWitnessTable { > static func init(..) > static func copy(..) > static func move(..) > static func destroy(..) > } Swift’s implementation model supports this. As a surface-level construct it’s going to be mired in UnsafeMutablePointers, and it’s not at all clear to me that we want this level of control there. Presumably, binding to Python is going to require some compiler effort—defining how it is that Python objects are initialized/copied/moved/destroyed seems like a reasonable part of that effort. > // Magic, allows anyobject-like member lookup on a type when lookup otherwise fails. > protocol DynamicMemberLookupable { >   associatedtype MemberLookupResultType >   func dynamicMemberLookup(\_ : String) -\> MemberLookupResultType > } AnyObject lookup looks for an actual declaration on any type anywhere. One could extend that mechanism to, say, return all Python methods and assume that you can call any Python method with any PythonObject instance. AnyObject lookup is fairly unprincipled as a language feature, because there’s no natural scope in which to perform name lookup, and involves hacks at many levels that don’t always work (e.g., AnyObject lookup… sometimes… fails across multiple source files for hard-to-explain reasons). You’re taking on that brokenness if you expand AnyObject lookup to another ecosystem. Although it doesn’t really seem like AnyObject lookup is the thing you’re asking for here. It seems more like you want dynamicMemberLookup(\_:) to capture “self” and the method name, and then be a callable thing as below... > // Magic, allows “overloaded/sugared postfix ()”. > protocol CustomCallable { > func call( …) > } > > The only tricky thing about this is the call part of things. At least in the case of python, we want something like this: > >   foo.bar(1, 2, a: x, b: y) > > to turn into: > foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y]) > > We don’t want this to be a memberlookup of a value that has “bar” as a basename and “a:” and “b:” as parameter labels. Well, I think the MemberLookupResult is going to get the name “bar”, argument labels “\_:\_🅰b:”, and arguments “1”, “2”, “x”, “y”, because that’s the Swift model of argument labels. It can then reshuffle them however it needs to for the underlying interaction with the Python interpreter. There are definite design trade-offs here. With AnyObject lookup, it’s a known-broken feature but because it depends on synthesized Swift method declarations, it’ll behave mostly the same way as other Swift method declarations—static overloading, known (albeit weak) type signatures, etc. But, it might require more of Python’s model to be grafted onto those method declarations. With dynamic member lookup, you’re throwing away all type safety (even for a motivated Python developer who might be willing to annotate APIs with types) and creating a general language mechanism for doing that.   - Doug > **···** > > > On Oct 29, 2017, at 1:34 PM, Chris Lattner via swift-evolution \ wrote: > > > > > On Oct 29, 2017, at 8:23 AM, Chris Lattner via swift-evolution \ wrote: > > > > > > > On Oct 29, 2017, at 4:04 AM, Lukas Stabe \ wrote: > > > > > > > > > On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution \ wrote: > > > -Chris > > > > > protocol DynamicDispatchable { // Protocol is “magic" known by the compiler. > > > func retain() > > > func release() > > > func memberLookup(\_ : String) -\> Self > > > func subscript\(\_ : T) -\> Self > > > func call(\_ args: [Self]) -\> Self > > > } > > > > > > module Python { > > > struct Object : DynamicDispatchable { > > >   var state : UnsafePointer\ > > > > > >   func retain() { > > >      INCREF(self) > > > } > > > > > >    func memberLookup(\_ : String) -\> Object { > > >       PyObject\_GetAttrString(…) > > >    } > > >   etc > > > } > > > > > > module Perl5 { > > > struct Object : DynamicDispatchable { > > >   var state : UnsafePointer\ > > > > > >   func retain() { > > >      SvREFCNT\_inc(self) > > > } > > > …. > > > > > > Are there other uses for such a thing? > > > > > > -Chris > > > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > swift-evolution mailing list > > > swift-evolution@swift.org > > > [https://lists.swift.org/mailman/listinfo/swift-evolution](https://lists.swift.org/mailman/listinfo/swift-evolution) > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > swift-evolution mailing list > > swift-evolution@swift.org \ > > [https://lists.swift.org/mailman/listinfo/swift-evolution](https://lists.swift.org/mailman/listinfo/swift-evolution) ---
**Author:** ![Chris\_Lattner](https://avatars.discourse-cdn.com/v4/letter/c/bc8723/32.png) [@Chris\_Lattner](https://forums.swift.org/u/Chris_Lattner)\ **Post date:** [October 29, 2017, 11:00pm UTC](https://forums.swift.org/t/discussion-swift-for-data-science-ml-big-data-analytics/6905/9 "2017-10-29T23:00:25Z")
> It would be really nice if Cython code could be linked into iOS and Mac apps written in Swift or Objective-C. > > [http://cython.org](http://cython.org) \<[http://cython.org/>](http://cython.org/>); Isn’t that possible today, at least on the Mac? -Chris > **···** > > > On Oct 29, 2017, at 3:23 PM, C. Keith Ray \ wrote: > > > -- > > C. Keith Ray > > Senior Software Engineer / Trainer / Agile Coach > > \* [http://www.thirdfoundationsw.com/keith\_ray\_resume\_2014\_long.pdf](http://www.thirdfoundationsw.com/keith_ray_resume_2014_long.pdf) > > > > > On Oct 29, 2017, at 1:34 PM, Chris Lattner via swift-evolution \\> wrote: > > > > > > > On Oct 29, 2017, at 8:23 AM, Chris Lattner via swift-evolution \\> wrote: > > > > > > > > > On Oct 29, 2017, at 4:04 AM, Lukas Stabe \\> wrote: > > > > > > > > > > > On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution \\> wrote: > > > > > > > > > > > > … which is to say, exactly identical to the Python version except that new variables need to be declared with let/var. This can be done by blessing Python.Object (which is identical to “PyObject\*” at the machine level) with some special dynamic name lookup behavior: Dot syntax turns into a call to PyObject\_GetAttrString, subscripts turn into PyObject\_GetItem, calls turn into PyObject\_Call, etc. ARC would be implemented with INCREF etc. > > > > > > > > > > That sounds like a very interesting prospect. Do you think it would make sense to make the language features that facilitate this (dynamic dispatch of method calls, property accesses, subscript and ARC) available to Swift classes annotated in some way, so that interop like this can be implemented as a library without special treatment by the Swift compiler? This could also enable more dynamic DSL like features. > > > > > > > > I haven’t explored enough of the design space to be sure, but I’d want to make sure that a library version of this could be done without giving up ergonomics of the result. If you were interested in being able to interop with other languages that are dynamically typed and reference counted, then something like this could be possible in principle: > > > > > > Thinking about the Perl case makes it clear to me that this should not be built into the compiler as a monolithic thing. Perl supports several different types (SV/AV/HV) which represent different concepts (scalars, arrays, hashes) so baking it all together into one thing would be the wrong way to map it. In fact, the magic we need is pretty small, and seems generally useful for other things. Consider a design like this: > > > > > > // not magic, things like Int, String and many other conform to this. > > > protocol Pythonable { > > > init?(\_ : PythonObject) > > > func toPython() -\> PythonObject > > > } > > > > > > // Not magic. > > > struct PythonObject : /\*protocols below\*/ { > > >   var state : UnsafePointer\ > > > > > >   subscript(\_ : Pythonable…) -\> PythonObject { > > >     ... > > >   } > > > } > > > > > > // Magic, must be on the struct definition. > > > // Could alternatively allow custom copy/move/… ctors like C++. > > > protocol CustomValueWitnessTable { > > > static func init(..) > > > static func copy(..) > > > static func move(..) > > > static func destroy(..) > > > } > > > > > > // Magic, allows anyobject-like member lookup on a type when lookup otherwise fails. > > > protocol DynamicMemberLookupable { > > >   associatedtype MemberLookupResultType > > >   func dynamicMemberLookup(\_ : String) -\> MemberLookupResultType > > > } > > > > > > // Magic, allows “overloaded/sugared postfix ()”. > > > protocol CustomCallable { > > > func call( …) > > > } > > > > > > The only tricky thing about this is the call part of things. At least in the case of python, we want something like this: > > > > > >   foo.bar(1, 2, a: x, b: y) > > > > > > to turn into: > > > foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y]) > > > > > > We don’t want this to be a memberlookup of a value that has “bar” as a basename and “a:” and “b:” as parameter labels. > > > > > > -Chris > > > > > > > protocol DynamicDispatchable { // Protocol is “magic" known by the compiler. > > > > func retain() > > > > func release() > > > > func memberLookup(\_ : String) -\> Self > > > > func subscript\(\_ : T) -\> Self > > > > func call(\_ args: [Self]) -\> Self > > > > } > > > > > > > > module Python { > > > > struct Object : DynamicDispatchable { > > > >   var state : UnsafePointer\ > > > > > > > >   func retain() { > > > >      INCREF(self) > > > > } > > > > > > > >    func memberLookup(\_ : String) -\> Object { > > > >       PyObject\_GetAttrString(…) > > > >    } > > > >   etc > > > > } > > > > > > > > module Perl5 { > > > > struct Object : DynamicDispatchable { > > > >   var state : UnsafePointer\ > > > > > > > >   func retain() { > > > >      SvREFCNT\_inc(self) > > > > } > > > > …. > > > > > > > > Are there other uses for such a thing? > > > > > > > > -Chris > > > > > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > > swift-evolution mailing list > > > > swift-evolution@swift.org \ > > > > [https://lists.swift.org/mailman/listinfo/swift-evolution](https://lists.swift.org/mailman/listinfo/swift-evolution) > > > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > swift-evolution mailing list > > > swift-evolution@swift.org \ > > > [https://lists.swift.org/mailman/listinfo/swift-evolution](https://lists.swift.org/mailman/listinfo/swift-evolution) ---
**Author:** ![David\_Kopecky](https://avatars.discourse-cdn.com/v4/letter/d/a88e4f/32.png) [@David\_Kopecky](https://forums.swift.org/u/David_Kopecky)\ **Post date:** [October 30, 2017, 9:54pm UTC](https://forums.swift.org/t/discussion-swift-for-data-science-ml-big-data-analytics/6905/10 "2017-10-30T21:54:05Z")
To Chris’s use case point, I see Python -\> Swift interop as a very common use case in data science / ML for speeding up certain functions/modules by embedding Swift into Python, like you can with, say, ctypes or python extensions in dynamic C libraries. This is very common practice in the Python universe, it just could be a lot better with Swift instead. Things like this \*sort of\* exist with Swift ([https://gist.github.com/jiaaro/e111f0f64d0cdb8aca38\](https://gist.github.com/jiaaro/e111f0f64d0cdb8aca38%5C)), but it’s not really very mature or functional. I would LOVE to be able to speed up a lot of python code by implementing swift libraries and importing into python. - David > **···** > > > On Oct 30, 2017, at 13:25, Douglas Gregor via swift-evolution \ wrote: > > > > > On Oct 29, 2017, at 1:34 PM, Chris Lattner via swift-evolution \\> wrote: > > > > > > > On Oct 29, 2017, at 8:23 AM, Chris Lattner via swift-evolution \\> wrote: > > > > > > > > > On Oct 29, 2017, at 4:04 AM, Lukas Stabe \\> wrote: > > > > > > > > > > > On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution \\> wrote: > > > > > > > > > > > > … which is to say, exactly identical to the Python version except that new variables need to be declared with let/var. This can be done by blessing Python.Object (which is identical to “PyObject\*” at the machine level) with some special dynamic name lookup behavior: Dot syntax turns into a call to PyObject\_GetAttrString, subscripts turn into PyObject\_GetItem, calls turn into PyObject\_Call, etc. ARC would be implemented with INCREF etc. > > > > > > > > > > That sounds like a very interesting prospect. Do you think it would make sense to make the language features that facilitate this (dynamic dispatch of method calls, property accesses, subscript and ARC) available to Swift classes annotated in some way, so that interop like this can be implemented as a library without special treatment by the Swift compiler? This could also enable more dynamic DSL like features. > > > > > > > > I haven’t explored enough of the design space to be sure, but I’d want to make sure that a library version of this could be done without giving up ergonomics of the result. If you were interested in being able to interop with other languages that are dynamically typed and reference counted, then something like this could be possible in principle: > > > > > > Thinking about the Perl case makes it clear to me that this should not be built into the compiler as a monolithic thing. Perl supports several different types (SV/AV/HV) which represent different concepts (scalars, arrays, hashes) so baking it all together into one thing would be the wrong way to map it. In fact, the magic we need is pretty small, and seems generally useful for other things. Consider a design like this: > > > > > > // not magic, things like Int, String and many other conform to this. > > > protocol Pythonable { > > > init?(\_ : PythonObject) > > > func toPython() -\> PythonObject > > > } > > > > It’s not magic unless you expect the compiler or runtime to help with conversion between Int/String/etc. and PythonObject, as with \_ObjectiveCBridgeable. > > > > > // Not magic. > > > struct PythonObject : /\*protocols below\*/ { > > >   var state : UnsafePointer\ > > > > > >   subscript(\_ : Pythonable…) -\> PythonObject { > > >     ... > > >   } > > > } > > > > > > // Magic, must be on the struct definition. > > > // Could alternatively allow custom copy/move/… ctors like C++. > > > protocol CustomValueWitnessTable { > > > static func init(..) > > > static func copy(..) > > > static func move(..) > > > static func destroy(..) > > > } > > > > Swift’s implementation model supports this. As a surface-level construct it’s going to be mired in UnsafeMutablePointers, and it’s not at all clear to me that we want this level of control there. Presumably, binding to Python is going to require some compiler effort—defining how it is that Python objects are initialized/copied/moved/destroyed seems like a reasonable part of that effort. > > > > > // Magic, allows anyobject-like member lookup on a type when lookup otherwise fails. > > > protocol DynamicMemberLookupable { > > >   associatedtype MemberLookupResultType > > >   func dynamicMemberLookup(\_ : String) -\> MemberLookupResultType > > > } > > > > AnyObject lookup looks for an actual declaration on any type anywhere. One could extend that mechanism to, say, return all Python methods and assume that you can call any Python method with any PythonObject instance. AnyObject lookup is fairly unprincipled as a language feature, because there’s no natural scope in which to perform name lookup, and involves hacks at many levels that don’t always work (e.g., AnyObject lookup… sometimes… fails across multiple source files for hard-to-explain reasons). You’re taking on that brokenness if you expand AnyObject lookup to another ecosystem. > > > > Although it doesn’t really seem like AnyObject lookup is the thing you’re asking for here. It seems more like you want dynamicMemberLookup(\_:) to capture “self” and the method name, and then be a callable thing as below... > > > > > // Magic, allows “overloaded/sugared postfix ()”. > > > protocol CustomCallable { > > > func call( …) > > > } > > > > > > The only tricky thing about this is the call part of things. At least in the case of python, we want something like this: > > > > > >   foo.bar(1, 2, a: x, b: y) > > > > > > to turn into: > > > foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y]) > > > > > > We don’t want this to be a memberlookup of a value that has “bar” as a basename and “a:” and “b:” as parameter labels. > > > > Well, I think the MemberLookupResult is going to get the name “bar”, argument labels “\_:\_🅰b:”, and arguments “1”, “2”, “x”, “y”, because that’s the Swift model of argument labels. It can then reshuffle them however it needs to for the underlying interaction with the Python interpreter. > > > > There are definite design trade-offs here. With AnyObject lookup, it’s a known-broken feature but because it depends on synthesized Swift method declarations, it’ll behave mostly the same way as other Swift method declarations—static overloading, known (albeit weak) type signatures, etc. But, it might require more of Python’s model to be grafted onto those method declarations. With dynamic member lookup, you’re throwing away all type safety (even for a motivated Python developer who might be willing to annotate APIs with types) and creating a general language mechanism for doing that. > > > >   - Doug > > > > > -Chris > > > > > > > protocol DynamicDispatchable { // Protocol is “magic" known by the compiler. > > > > func retain() > > > > func release() > > > > func memberLookup(\_ : String) -\> Self > > > > func subscript\(\_ : T) -\> Self > > > > func call(\_ args: [Self]) -\> Self > > > > } > > > > > > > > module Python { > > > > struct Object : DynamicDispatchable { > > > >   var state : UnsafePointer\ > > > > > > > >   func retain() { > > > >      INCREF(self) > > > > } > > > > > > > >    func memberLookup(\_ : String) -\> Object { > > > >       PyObject\_GetAttrString(…) > > > >    } > > > >   etc > > > > } > > > > > > > > module Perl5 { > > > > struct Object : DynamicDispatchable { > > > >   var state : UnsafePointer\ > > > > > > > >   func retain() { > > > >      SvREFCNT\_inc(self) > > > > } > > > > …. > > > > > > > > Are there other uses for such a thing? > > > > > > > > -Chris > > > > > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > > swift-evolution mailing list > > > > swift-evolution@swift.org \ > > > > [https://lists.swift.org/mailman/listinfo/swift-evolution](https://lists.swift.org/mailman/listinfo/swift-evolution) > > > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > swift-evolution mailing list > > > swift-evolution@swift.org \ > > > [https://lists.swift.org/mailman/listinfo/swift-evolution](https://lists.swift.org/mailman/listinfo/swift-evolution) > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > swift-evolution mailing list > > swift-evolution@swift.org \ > > [https://lists.swift.org/mailman/listinfo/swift-evolution](https://lists.swift.org/mailman/listinfo/swift-evolution) ---
**Author:** ![Chris\_Lattner](https://avatars.discourse-cdn.com/v4/letter/c/bc8723/32.png) [@Chris\_Lattner](https://forums.swift.org/u/Chris_Lattner)\ **Post date:** [October 31, 2017, 4:43am UTC](https://forums.swift.org/t/discussion-swift-for-data-science-ml-big-data-analytics/6905/11 "2017-10-31T04:43:39Z")
JohnMC: question for you below. > > Thinking about the Perl case makes it clear to me that this should not be built into the compiler as a monolithic thing. Perl supports several different types (SV/AV/HV) which represent different concepts (scalars, arrays, hashes) so baking it all together into one thing would be the wrong way to map it. In fact, the magic we need is pretty small, and seems generally useful for other things. Consider a design like this: > > > > // not magic, things like Int, String and many other conform to this. > > protocol Pythonable { > > init?(\_ : PythonObject) > > func toPython() -\> PythonObject > > } > > It’s not magic unless you expect the compiler or runtime to help with conversion between Int/String/etc. and PythonObject, as with \_ObjectiveCBridgeable. Right, as I said above “not magic”. The conformances would be manually implemented in the Python overlay. This provides a free implicit conversion from "T -\> Pythonable” for the T’s we care about, and a failable init from Python back to Swift types. > > // Not magic. > > struct PythonObject : /\*protocols below\*/ { > >   var state : UnsafePointer\ > > > >   subscript(\_ : Pythonable…) -\> PythonObject { > >     ... > >   } > > } > > > > // Magic, must be on the struct definition. > > // Could alternatively allow custom copy/move/… ctors like C++. > > protocol CustomValueWitnessTable { > > static func init(..) > > static func copy(..) > > static func move(..) > > static func destroy(..) > > } > > Swift’s implementation model supports this. As a surface-level construct it’s going to be mired in UnsafeMutablePointers, and it’s not at all clear to me that we want this level of control there. There are two ways to implement it: 1) static func’s like the above, which are implemented as UnsafePointer's 2) Proper language syntax akin to the C++ “rule of 5”. The pro’s and con’s of the first approach: pro) full explicit control over what happens pro) no other new language features necessary to implement this. The second approach would need something like ownership to be in place. con) injects another avenue of unsafety (though it would be explicit, so it isn’t that bad). It isn’t obvious to me that approach #2 can be safe, but I haven’t thought about it enough. ???) discourages people from using this protocol because of its explicit unsafety. I can think of two things that could tip the scale of the discussion: a) The big question is whether we \*want\* the ability to write custom rule-of-5 style behavior for structs, or if we want it to only be used in extreme cases (like bridging interop in this proposal). If we \*want\* to support it someday, then adding proper “safe” support is best (if possible). If we don’t \*want\* people to use it, then making it Unsafe and ugly is a reasonable way to go. b) The ownership proposal is likely to add deinit's to structs. If it also adds explicit move initializers, then it is probably the right thing to add copy initializers also (and thus go with approach #2). That said, I’m not sure how the move initializers will be spelled or if that is the likely direction. If it won’t add these, then it is probably better to go with approach #1. John, what do you think? > Presumably, binding to Python is going to require some compiler effort—defining how it is that Python objects are initialized/copied/moved/destroyed seems like a reasonable part of that effort. Actually no. If we add these three proposals, there is no other python (or perl, etc…) specific support needed. It is all implementable in the overlay. > > // Magic, allows anyobject-like member lookup on a type when lookup otherwise fails. > > protocol DynamicMemberLookupable { > >   associatedtype MemberLookupResultType > >   func dynamicMemberLookup(\_ : String) -\> MemberLookupResultType > > } > > AnyObject lookup looks for an actual declaration on any type anywhere. One could extend that mechanism to, say, return all Python methods and assume that you can call any Python method with any PythonObject instance. AnyObject lookup is fairly unprincipled as a language feature, because there’s no natural scope in which to perform name lookup, and involves hacks at many levels that don’t always work (e.g., AnyObject lookup… sometimes… fails across multiple source files for hard-to-explain reasons). You’re taking on that brokenness if you expand AnyObject lookup to another ecosystem. Yeah, sorry, that’s not what I meant: > Although it doesn’t really seem like AnyObject lookup is the thing you’re asking for here. It seems more like you want dynamicMemberLookup(\_:) to capture “self” and the method name, and then be a callable thing as below… That’s what I meant :-). A type that implements this magic protocol would never fail name lookup: “foo.bar” would always fall back to calling: foo.dynamicMemberLookup(“bar") it’s simple and more predictable than AnyObject, it also matches what dynamic languages like Python needs. > > // Magic, allows “overloaded/sugared postfix ()”. > > protocol CustomCallable { > > func call( …) > > } > > > > The only tricky thing about this is the call part of things. At least in the case of python, we want something like this: > > > >   foo.bar(1, 2, a: x, b: y) > > > > to turn into: > > foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y]) > > > > We don’t want this to be a memberlookup of a value that has “bar” as a basename and “a:” and “b:” as parameter labels. > > Well, I think the MemberLookupResult is going to get the name “bar”, argument labels “\_:\_🅰b:”, and arguments “1”, “2”, “x”, “y”, because that’s the Swift model of argument labels. It can then reshuffle them however it needs to for the underlying interaction with the Python interpreter. > > There are definite design trade-offs here. With AnyObject lookup, it’s a known-broken feature but because it depends on synthesized Swift method declarations, it’ll behave mostly the same way as other Swift method declarations—static overloading, known (albeit weak) type signatures, etc. But, it might require more of Python’s model to be grafted onto those method declarations. With dynamic member lookup, you’re throwing away all type safety (even for a motivated Python developer who might be willing to annotate APIs with types) and creating a general language mechanism for doing that. Right, something like this could definitely work, but keep in mind that the Swift compiler knows nothing about Python declarations. Perhaps the most straight-forward thing would be to support: protocol CustomCallable { func call(…arg list as array and kw args...) func callMember(\_ : String, …otherstuffabove...) } Given this, the compiler could map: pythonThing(42) -\> pythonThing.call([42]) pythonThing.method(a: 42) -\> pythonThing.callMember(“method”, kwargs: [“a”: 42]) This is the simplest way to map the Swift semantics (where kw args are part of compound lookups) into the Python world. -Chris > **···** > > On Oct 30, 2017, at 1:25 PM, Douglas Gregor \ wrote: ---
**Author:** ![ckeithray](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ckeithray/32/2317_2.png) [@ckeithray](https://forums.swift.org/u/ckeithray)\ **Post date:** [October 29, 2017, 11:09pm UTC](https://forums.swift.org/t/discussion-swift-for-data-science-ml-big-data-analytics/6905/12 "2017-10-29T23:09:45Z")
There's surprisingly little information about Cython interoperability with Cocoa/Mac applications. It's might be linkable if the app can embed Python. > **···** > > -- > C. Keith Ray > > \* [What Every Programmer Needs To… by C. Keith Ray [PDF/iPad/Kindle]](https://leanpub.com/wepntk) \<- buy my book? > \* [http://www.thirdfoundationsw.com/keith\_ray\_resume\_2014\_long.pdf](http://www.thirdfoundationsw.com/keith_ray_resume_2014_long.pdf) > \* [http://agilesolutionspace.blogspot.com/](http://agilesolutionspace.blogspot.com/) > > > On Oct 29, 2017, at 4:00 PM, Chris Lattner \ wrote: > > > > > On Oct 29, 2017, at 3:23 PM, C. Keith Ray \ wrote: > > > > > > It would be really nice if Cython code could be linked into iOS and Mac apps written in Swift or Objective-C. > > > > > > [http://cython.org](http://cython.org) > > > > Isn’t that possible today, at least on the Mac? > > > > -Chris > > > > > -- > > > C. Keith Ray > > > Senior Software Engineer / Trainer / Agile Coach > > > \* [http://www.thirdfoundationsw.com/keith\_ray\_resume\_2014\_long.pdf](http://www.thirdfoundationsw.com/keith_ray_resume_2014_long.pdf) > > > > > > > > On Oct 29, 2017, at 1:34 PM, Chris Lattner via swift-evolution \ wrote: > > > > > > > > > > > On Oct 29, 2017, at 8:23 AM, Chris Lattner via swift-evolution \ wrote: > > > > > > > > > > > > > On Oct 29, 2017, at 4:04 AM, Lukas Stabe \ wrote: > > > > > > > > > > > > > > On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution \ wrote: > > > > > > > > > > > > > > … which is to say, exactly identical to the Python version except that new variables need to be declared with let/var. This can be done by blessing Python.Object (which is identical to “PyObject\*” at the machine level) with some special dynamic name lookup behavior: Dot syntax turns into a call to PyObject\_GetAttrString, subscripts turn into PyObject\_GetItem, calls turn into PyObject\_Call, etc. ARC would be implemented with INCREF etc. > > > > > > > > > > > > That sounds like a very interesting prospect. Do you think it would make sense to make the language features that facilitate this (dynamic dispatch of method calls, property accesses, subscript and ARC) available to Swift classes annotated in some way, so that interop like this can be implemented as a library without special treatment by the Swift compiler? This could also enable more dynamic DSL like features. > > > > > > > > > > I haven’t explored enough of the design space to be sure, but I’d want to make sure that a library version of this could be done without giving up ergonomics of the result. If you were interested in being able to interop with other languages that are dynamically typed and reference counted, then something like this could be possible in principle: > > > > > > > > Thinking about the Perl case makes it clear to me that this should not be built into the compiler as a monolithic thing. Perl supports several different types (SV/AV/HV) which represent different concepts (scalars, arrays, hashes) so baking it all together into one thing would be the wrong way to map it. In fact, the magic we need is pretty small, and seems generally useful for other things. Consider a design like this: > > > > > > > > // not magic, things like Int, String and many other conform to this. > > > > protocol Pythonable { > > > > init?(\_ : PythonObject) > > > > func toPython() -\> PythonObject > > > > } > > > > > > > > // Not magic. > > > > struct PythonObject : /\*protocols below\*/ { > > > >   var state : UnsafePointer\ > > > > > > > >   subscript(\_ : Pythonable…) -\> PythonObject { > > > >     ... > > > >   } > > > > } > > > > > > > > // Magic, must be on the struct definition. > > > > // Could alternatively allow custom copy/move/… ctors like C++. > > > > protocol CustomValueWitnessTable { > > > > static func init(..) > > > > static func copy(..) > > > > static func move(..) > > > > static func destroy(..) > > > > } > > > > > > > > // Magic, allows anyobject-like member lookup on a type when lookup otherwise fails. > > > > protocol DynamicMemberLookupable { > > > >   associatedtype MemberLookupResultType > > > >   func dynamicMemberLookup(\_ : String) -\> MemberLookupResultType > > > > } > > > > > > > > // Magic, allows “overloaded/sugared postfix ()”. > > > > protocol CustomCallable { > > > > func call( …) > > > > } > > > > > > > > The only tricky thing about this is the call part of things. At least in the case of python, we want something like this: > > > > > > > >   foo.bar(1, 2, a: x, b: y) > > > > > > > > to turn into: > > > > foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y]) > > > > > > > > We don’t want this to be a memberlookup of a value that has “bar” as a basename and “a:” and “b:” as parameter labels. > > > > > > > > -Chris > > > > > > > > > protocol DynamicDispatchable { // Protocol is “magic" known by the compiler. > > > > > func retain() > > > > > func release() > > > > > func memberLookup(\_ : String) -\> Self > > > > > func subscript\(\_ : T) -\> Self > > > > > func call(\_ args: [Self]) -\> Self > > > > > } > > > > > > > > > > module Python { > > > > > struct Object : DynamicDispatchable { > > > > >   var state : UnsafePointer\ > > > > > > > > > >   func retain() { > > > > >      INCREF(self) > > > > > } > > > > > > > > > >    func memberLookup(\_ : String) -\> Object { > > > > >       PyObject\_GetAttrString(…) > > > > >    } > > > > >   etc > > > > > } > > > > > > > > > > module Perl5 { > > > > > struct Object : DynamicDispatchable { > > > > >   var state : UnsafePointer\ > > > > > > > > > >   func retain() { > > > > >      SvREFCNT\_inc(self) > > > > > } > > > > > …. > > > > > > > > > > Are there other uses for such a thing? > > > > > > > > > > -Chris > > > > > > > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > > > swift-evolution mailing list > > > > > swift-evolution@swift.org > > > > > [https://lists.swift.org/mailman/listinfo/swift-evolution](https://lists.swift.org/mailman/listinfo/swift-evolution) > > > > > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > > swift-evolution mailing list > > > > swift-evolution@swift.org > > > > [https://lists.swift.org/mailman/listinfo/swift-evolution](https://lists.swift.org/mailman/listinfo/swift-evolution) ---
**Author:** ![Douglas\_Gregor](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/douglas_gregor/32/1362_2.png) [@Douglas\_Gregor](https://forums.swift.org/u/Douglas_Gregor)\ **Post date:** [October 31, 2017, 5:24am UTC](https://forums.swift.org/t/discussion-swift-for-data-science-ml-big-data-analytics/6905/13 "2017-10-31T05:24:54Z")
> To Chris’s use case point, I see Python -\> Swift interop as a very common use case in data science / ML for speeding up certain functions/modules by embedding Swift into Python, like you can with, say, ctypes or python extensions in dynamic C libraries. This is very common practice in the Python universe, it just could be a lot better with Swift instead. > > Things like this \*sort of\* exist with Swift ([https://gist.github.com/jiaaro/e111f0f64d0cdb8aca38\](https://gist.github.com/jiaaro/e111f0f64d0cdb8aca38%5C)), but it’s not really very mature or functional. > > I would LOVE to be able to speed up a lot of python code by implementing swift libraries and importing into python. There are a bunch of ways one could export Swift functionality to Python, from a library solution like the C++ Boost.Python ([http://www.boost.org/doc/libs/1\_65\_1/libs/python/doc/html/index.html\](http://www.boost.org/doc/libs/1_65_1/libs/python/doc/html/index.html%5C)), to code generators like SWIG or, with some future expansion of metadata, directly at runtime via reflection facilities. Python’s dynamic nature makes this direction easier. The harder challenge (IMO!) is to get a Python library into Swift, because there isn’t a lot of type information in Python, so you have to show the Python in the much-more-strongly-typed Swift somehow (and keep it usable).   - Doug > **···** > > > On Oct 30, 2017, at 2:54 PM, David Kopecky \ wrote: > > > - David > > > > > On Oct 30, 2017, at 13:25, Douglas Gregor via swift-evolution \\> wrote: > > > > > > > On Oct 29, 2017, at 1:34 PM, Chris Lattner via swift-evolution \\> wrote: > > > > > > > > > On Oct 29, 2017, at 8:23 AM, Chris Lattner via swift-evolution \\> wrote: > > > > > > > > > > > On Oct 29, 2017, at 4:04 AM, Lukas Stabe \\> wrote: > > > > > > > > > > > > > On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution \\> wrote: > > > > > > > > > > > > > > … which is to say, exactly identical to the Python version except that new variables need to be declared with let/var. This can be done by blessing Python.Object (which is identical to “PyObject\*” at the machine level) with some special dynamic name lookup behavior: Dot syntax turns into a call to PyObject\_GetAttrString, subscripts turn into PyObject\_GetItem, calls turn into PyObject\_Call, etc. ARC would be implemented with INCREF etc. > > > > > > > > > > > > That sounds like a very interesting prospect. Do you think it would make sense to make the language features that facilitate this (dynamic dispatch of method calls, property accesses, subscript and ARC) available to Swift classes annotated in some way, so that interop like this can be implemented as a library without special treatment by the Swift compiler? This could also enable more dynamic DSL like features. > > > > > > > > > > I haven’t explored enough of the design space to be sure, but I’d want to make sure that a library version of this could be done without giving up ergonomics of the result. If you were interested in being able to interop with other languages that are dynamically typed and reference counted, then something like this could be possible in principle: > > > > > > > > Thinking about the Perl case makes it clear to me that this should not be built into the compiler as a monolithic thing. Perl supports several different types (SV/AV/HV) which represent different concepts (scalars, arrays, hashes) so baking it all together into one thing would be the wrong way to map it. In fact, the magic we need is pretty small, and seems generally useful for other things. Consider a design like this: > > > > > > > > // not magic, things like Int, String and many other conform to this. > > > > protocol Pythonable { > > > > init?(\_ : PythonObject) > > > > func toPython() -\> PythonObject > > > > } > > > > > > It’s not magic unless you expect the compiler or runtime to help with conversion between Int/String/etc. and PythonObject, as with \_ObjectiveCBridgeable. > > > > > > > // Not magic. > > > > struct PythonObject : /\*protocols below\*/ { > > > >   var state : UnsafePointer\ > > > > > > > >   subscript(\_ : Pythonable…) -\> PythonObject { > > > >     ... > > > >   } > > > > } > > > > > > > > // Magic, must be on the struct definition. > > > > // Could alternatively allow custom copy/move/… ctors like C++. > > > > protocol CustomValueWitnessTable { > > > > static func init(..) > > > > static func copy(..) > > > > static func move(..) > > > > static func destroy(..) > > > > } > > > > > > Swift’s implementation model supports this. As a surface-level construct it’s going to be mired in UnsafeMutablePointers, and it’s not at all clear to me that we want this level of control there. Presumably, binding to Python is going to require some compiler effort—defining how it is that Python objects are initialized/copied/moved/destroyed seems like a reasonable part of that effort. > > > > > > > // Magic, allows anyobject-like member lookup on a type when lookup otherwise fails. > > > > protocol DynamicMemberLookupable { > > > >   associatedtype MemberLookupResultType > > > >   func dynamicMemberLookup(\_ : String) -\> MemberLookupResultType > > > > } > > > > > > AnyObject lookup looks for an actual declaration on any type anywhere. One could extend that mechanism to, say, return all Python methods and assume that you can call any Python method with any PythonObject instance. AnyObject lookup is fairly unprincipled as a language feature, because there’s no natural scope in which to perform name lookup, and involves hacks at many levels that don’t always work (e.g., AnyObject lookup… sometimes… fails across multiple source files for hard-to-explain reasons). You’re taking on that brokenness if you expand AnyObject lookup to another ecosystem. > > > > > > Although it doesn’t really seem like AnyObject lookup is the thing you’re asking for here. It seems more like you want dynamicMemberLookup(\_:) to capture “self” and the method name, and then be a callable thing as below... > > > > > > > // Magic, allows “overloaded/sugared postfix ()”. > > > > protocol CustomCallable { > > > > func call( …) > > > > } > > > > > > > > The only tricky thing about this is the call part of things. At least in the case of python, we want something like this: > > > > > > > >   foo.bar(1, 2, a: x, b: y) > > > > > > > > to turn into: > > > > foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y]) > > > > > > > > We don’t want this to be a memberlookup of a value that has “bar” as a basename and “a:” and “b:” as parameter labels. > > > > > > Well, I think the MemberLookupResult is going to get the name “bar”, argument labels “\_:\_🅰b:”, and arguments “1”, “2”, “x”, “y”, because that’s the Swift model of argument labels. It can then reshuffle them however it needs to for the underlying interaction with the Python interpreter. > > > > > > There are definite design trade-offs here. With AnyObject lookup, it’s a known-broken feature but because it depends on synthesized Swift method declarations, it’ll behave mostly the same way as other Swift method declarations—static overloading, known (albeit weak) type signatures, etc. But, it might require more of Python’s model to be grafted onto those method declarations. With dynamic member lookup, you’re throwing away all type safety (even for a motivated Python developer who might be willing to annotate APIs with types) and creating a general language mechanism for doing that. > > > > > >   - Doug > > > > > > > -Chris > > > > > > > > > protocol DynamicDispatchable { // Protocol is “magic" known by the compiler. > > > > > func retain() > > > > > func release() > > > > > func memberLookup(\_ : String) -\> Self > > > > > func subscript\(\_ : T) -\> Self > > > > > func call(\_ args: [Self]) -\> Self > > > > > } > > > > > > > > > > module Python { > > > > > struct Object : DynamicDispatchable { > > > > >   var state : UnsafePointer\ > > > > > > > > > >   func retain() { > > > > >      INCREF(self) > > > > > } > > > > > > > > > >    func memberLookup(\_ : String) -\> Object { > > > > >       PyObject\_GetAttrString(…) > > > > >    } > > > > >   etc > > > > > } > > > > > > > > > > module Perl5 { > > > > > struct Object : DynamicDispatchable { > > > > >   var state : UnsafePointer\ > > > > > > > > > >   func retain() { > > > > >      SvREFCNT\_inc(self) > > > > > } > > > > > …. > > > > > > > > > > Are there other uses for such a thing? > > > > > > > > > > -Chris > > > > > > > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > > > swift-evolution mailing list > > > > > swift-evolution@swift.org \ > > > > > [https://lists.swift.org/mailman/listinfo/swift-evolution](https://lists.swift.org/mailman/listinfo/swift-evolution) > > > > > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > > swift-evolution mailing list > > > > swift-evolution@swift.org \ > > > > [https://lists.swift.org/mailman/listinfo/swift-evolution](https://lists.swift.org/mailman/listinfo/swift-evolution) > > > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > swift-evolution mailing list > > > swift-evolution@swift.org \ > > > [https://lists.swift.org/mailman/listinfo/swift-evolution](https://lists.swift.org/mailman/listinfo/swift-evolution) ---
**Author:** ![Douglas\_Gregor](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/douglas_gregor/32/1362_2.png) [@Douglas\_Gregor](https://forums.swift.org/u/Douglas_Gregor)\ **Post date:** [October 31, 2017, 5:47am UTC](https://forums.swift.org/t/discussion-swift-for-data-science-ml-big-data-analytics/6905/14 "2017-10-31T05:47:00Z")
> JohnMC: question for you below. > > > > Thinking about the Perl case makes it clear to me that this should not be built into the compiler as a monolithic thing. Perl supports several different types (SV/AV/HV) which represent different concepts (scalars, arrays, hashes) so baking it all together into one thing would be the wrong way to map it. In fact, the magic we need is pretty small, and seems generally useful for other things. Consider a design like this: > > > > > > // not magic, things like Int, String and many other conform to this. > > > protocol Pythonable { > > > init?(\_ : PythonObject) > > > func toPython() -\> PythonObject > > > } > > > > It’s not magic unless you expect the compiler or runtime to help with conversion between Int/String/etc. and PythonObject, as with \_ObjectiveCBridgeable. > > Right, as I said above “not magic”. The conformances would be manually implemented in the Python overlay. This provides a free implicit conversion from "T -\> Pythonable” for the T’s we care about, and a failable init from Python back to Swift types. Note that, under this scheme,   let p: Pythonable = 17   let i: Int = p as! i will work if Int is Pythonable, but not when p comes back from Python. > > > // Not magic. > > > struct PythonObject : /\*protocols below\*/ { > > >   var state : UnsafePointer\ > > > > > >   subscript(\_ : Pythonable…) -\> PythonObject { > > >     ... > > >   } > > > } > > > > > > // Magic, must be on the struct definition. > > > // Could alternatively allow custom copy/move/… ctors like C++. > > > protocol CustomValueWitnessTable { > > > static func init(..) > > > static func copy(..) > > > static func move(..) > > > static func destroy(..) > > > } > > > > Swift’s implementation model supports this. As a surface-level construct it’s going to be mired in UnsafeMutablePointers, and it’s not at all clear to me that we want this level of control there. > > There are two ways to implement it: > 1) static func’s like the above, which are implemented as UnsafePointer's > 2) Proper language syntax akin to the C++ “rule of 5”. > > The pro’s and con’s of the first approach: > > pro) full explicit control over what happens > pro) no other new language features necessary to implement this. The second approach would need something like ownership to be in place. > con) injects another avenue of unsafety (though it would be explicit, so it isn’t that bad). It isn’t obvious to me that approach #2 can be safe, but I haven’t thought about it enough. > ???) discourages people from using this protocol because of its explicit unsafety. con) much of the UnsafePointer interface is based on the value witness table, so one has to step lightly or work with something like COpaquePointer/UnsafePointer\. > I can think of two things that could tip the scale of the discussion: > > a) The big question is whether we \*want\* the ability to write custom rule-of-5 style behavior for structs, or if we want it to only be used in extreme cases (like bridging interop in this proposal). If we \*want\* to support it someday, then adding proper “safe” support is best (if possible). If we don’t \*want\* people to use it, then making it Unsafe and ugly is a reasonable way to go. > > b) The ownership proposal is likely to add deinit's to structs. If it also adds explicit move initializers, then it is probably the right thing to add copy initializers also (and thus go with approach #2). That said, I’m not sure how the move initializers will be spelled or if that is the likely direction. If it won’t add these, then it is probably better to go with approach #1. John, what do you think? > > > Presumably, binding to Python is going to require some compiler effort—defining how it is that Python objects are initialized/copied/moved/destroyed seems like a reasonable part of that effort. > > Actually no. If we add these three proposals, there is no other python (or perl, etc…) specific support needed. It is all implementable in the overlay. Support for working with Python objects would be implementable in the overlay, but the result isn’t necessarily ergonomic (e.g., my “as!” case from a Python-generated integer object to Int, shown above). That might be fine! More comments on this below. > > > // Magic, allows anyobject-like member lookup on a type when lookup otherwise fails. > > > protocol DynamicMemberLookupable { > > >   associatedtype MemberLookupResultType > > >   func dynamicMemberLookup(\_ : String) -\> MemberLookupResultType > > > } > > > > AnyObject lookup looks for an actual declaration on any type anywhere. One could extend that mechanism to, say, return all Python methods and assume that you can call any Python method with any PythonObject instance. AnyObject lookup is fairly unprincipled as a language feature, because there’s no natural scope in which to perform name lookup, and involves hacks at many levels that don’t always work (e.g., AnyObject lookup… sometimes… fails across multiple source files for hard-to-explain reasons). You’re taking on that brokenness if you expand AnyObject lookup to another ecosystem. > > Yeah, sorry, that’s not what I meant: (Good) > > Although it doesn’t really seem like AnyObject lookup is the thing you’re asking for here. It seems more like you want dynamicMemberLookup(\_:) to capture “self” and the method name, and then be a callable thing as below… > > That’s what I meant :-). > > A type that implements this magic protocol would never fail name lookup: “foo.bar” would always fall back to calling: foo.dynamicMemberLookup(“bar") > > it’s simple and more predictable than AnyObject, it also matches what dynamic languages like Python needs. > > > > // Magic, allows “overloaded/sugared postfix ()”. > > > protocol CustomCallable { > > > func call( …) > > > } > > > > > > The only tricky thing about this is the call part of things. At least in the case of python, we want something like this: > > > > > >   foo.bar(1, 2, a: x, b: y) > > > > > > to turn into: > > > foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y]) > > > > > > We don’t want this to be a memberlookup of a value that has “bar” as a basename and “a:” and “b:” as parameter labels. > > > > Well, I think the MemberLookupResult is going to get the name “bar”, argument labels “\_:\_🅰b:”, and arguments “1”, “2”, “x”, “y”, because that’s the Swift model of argument labels. It can then reshuffle them however it needs to for the underlying interaction with the Python interpreter. > > > > There are definite design trade-offs here. With AnyObject lookup, it’s a known-broken feature but because it depends on synthesized Swift method declarations, it’ll behave mostly the same way as other Swift method declarations—static overloading, known (albeit weak) type signatures, etc. But, it might require more of Python’s model to be grafted onto those method declarations. With dynamic member lookup, you’re throwing away all type safety (even for a motivated Python developer who might be willing to annotate APIs with types) and creating a general language mechanism for doing that. > > Right, something like this could definitely work, but keep in mind that the Swift compiler knows nothing about Python declarations. > > Perhaps the most straight-forward thing would be to support: > > protocol CustomCallable { > func call(…arg list as array and kw args...) > func callMember(\_ : String, …otherstuffabove...) > } > > Given this, the compiler could map: > > pythonThing(42) -\> pythonThing.call([42]) > pythonThing.method(a: 42) -\> pythonThing.callMember(“method”, kwargs: [“a”: 42]) > > This is the simplest way to map the Swift semantics (where kw args are part of compound lookups) into the Python world. Okay, I agree that this gets Swift syntax into a call to the Python interpreter fairly quickly. Over-architecting for the sake of discussion: protocol CustomCallable {   associatedtype CustomArgument   associatedtype CustomNominal   associatedtype CustomResult   func callMember(self: CustomNominal, functionName: String, arguments: [(String, CustomArgument)]) throws -\> CustomResult   // something for class/static members } But this is \*all\* dynamic, even when one could map much of Python’s type information into Swift. For example, let’s take this: class Dog:     def \_\_init\_\_(self, name):         self.name = name         self.tricks = # creates a new empty list for each dog     def add\_trick(self, trick):         self.tricks.append(trick) With your don’t-modify-the-compiler approach, how can I create a Dog instance and add a trick? I probably need to look up the class by name, call \_\_init\_\_ manually, etc.   let dogClass = python\_getClassByName(“Dog”) // implemented in the Python “overlay’, I guess   let dog = python\_createInstance(dogClass) // implemented in the Python “overlay’, I guess   dog.\_\_init\_\_(“Brianna”) // uses CustomCallable’s callMember   dog.add\_trick(“Roll over”) // uses CustomCallable’s callMember With compiler integration,   class Dog : PythonObject {     init(\_ name: Pythonable)     func add\_trick(\_ trick: Pythonable)   } One could possibly bridge the gap with a code generator of some sort, that (for example) maps Dog’s \_\_init\_\_’s to global functions   func Dog(\_ name: Pythonable) -\> Pythonable {       let dogClass = python\_getClassByName(“Dog”) // implemented in the Python “overlay’, I guess       let dog = python\_createInstance(dogClass) // implemented in the Python “overlay’, I guess       dog.\_\_init\_\_(“Brianna”) // uses CustomCallable’s callMember       return dog   } and maybe turns all Python methods into extensions on Pythonable:   extension Pythonable {       func add\_trick   } With either the true “Python importer” solution or this code-generation solution, you at least get some level of code completion and basic sanity checking “for free”. In other words, you get some of the benefits of having a statically-type-checked language while still working on dynamic Pythonable types.   - Doug > **···** > > > On Oct 30, 2017, at 9:43 PM, Chris Lattner \ wrote: > > On Oct 30, 2017, at 1:25 PM, Douglas Gregor \\> wrote: ---
**Author:** ![John\_McCall](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/john_mccall/32/1351_2.png) [@John\_McCall](https://forums.swift.org/u/John_McCall)\ **Post date:** [October 31, 2017, 5:55am UTC](https://forums.swift.org/t/discussion-swift-for-data-science-ml-big-data-analytics/6905/15 "2017-10-31T05:55:44Z")
> JohnMC: question for you below. > > > > Thinking about the Perl case makes it clear to me that this should not be built into the compiler as a monolithic thing. Perl supports several different types (SV/AV/HV) which represent different concepts (scalars, arrays, hashes) so baking it all together into one thing would be the wrong way to map it. In fact, the magic we need is pretty small, and seems generally useful for other things. Consider a design like this: > > > > > > // not magic, things like Int, String and many other conform to this. > > > protocol Pythonable { > > > init?(\_ : PythonObject) > > > func toPython() -\> PythonObject > > > } > > > > It’s not magic unless you expect the compiler or runtime to help with conversion between Int/String/etc. and PythonObject, as with \_ObjectiveCBridgeable. > > Right, as I said above “not magic”. The conformances would be manually implemented in the Python overlay. This provides a free implicit conversion from "T -\> Pythonable” for the T’s we care about, and a failable init from Python back to Swift types. > > > > // Not magic. > > > struct PythonObject : /\*protocols below\*/ { > > >   var state : UnsafePointer\ > > > > > >   subscript(\_ : Pythonable…) -\> PythonObject { > > >     ... > > >   } > > > } > > > > > > // Magic, must be on the struct definition. > > > // Could alternatively allow custom copy/move/… ctors like C++. > > > protocol CustomValueWitnessTable { > > > static func init(..) > > > static func copy(..) > > > static func move(..) > > > static func destroy(..) > > > } > > > > Swift’s implementation model supports this. As a surface-level construct it’s going to be mired in UnsafeMutablePointers, and it’s not at all clear to me that we want this level of control there. > > There are two ways to implement it: > 1) static func’s like the above, which are implemented as UnsafePointer's > 2) Proper language syntax akin to the C++ “rule of 5”. > > The pro’s and con’s of the first approach: > > pro) full explicit control over what happens > pro) no other new language features necessary to implement this. The second approach would need something like ownership to be in place. > con) injects another avenue of unsafety (though it would be explicit, so it isn’t that bad). It isn’t obvious to me that approach #2 can be safe, but I haven’t thought about it enough. > ???) discourages people from using this protocol because of its explicit unsafety. > > I can think of two things that could tip the scale of the discussion: > > a) The big question is whether we \*want\* the ability to write custom rule-of-5 style behavior for structs, or if we want it to only be used in extreme cases (like bridging interop in this proposal). If we \*want\* to support it someday, then adding proper “safe” support is best (if possible). If we don’t \*want\* people to use it, then making it Unsafe and ugly is a reasonable way to go. > > b) The ownership proposal is likely to add deinit's to structs. If it also adds explicit move initializers, then it is probably the right thing to add copy initializers also (and thus go with approach #2). That said, I’m not sure how the move initializers will be spelled or if that is the likely direction. If it won’t add these, then it is probably better to go with approach #1. John, what do you think? I was hoping not to have to add explicit move/copy initializers, perhaps ever. I would suggest one of two things:   - We could add a Builtin type for these types in Swift. Because of our architecture, this is mostly an IRGen task.   - We could start working on C++ import. C++ import is a huge task, but we don't really need most of it for this. John. > **···** > > > On Oct 31, 2017, at 12:43 AM, Chris Lattner \ wrote: > > On Oct 30, 2017, at 1:25 PM, Douglas Gregor \\> wrote: > > > > Presumably, binding to Python is going to require some compiler effort—defining how it is that Python objects are initialized/copied/moved/destroyed seems like a reasonable part of that effort. > > > > Actually no. If we add these three proposals, there is no other python (or perl, etc…) specific support needed. It is all implementable in the overlay. > > > > > > // Magic, allows anyobject-like member lookup on a type when lookup otherwise fails. > > > > protocol DynamicMemberLookupable { > > > >   associatedtype MemberLookupResultType > > > >   func dynamicMemberLookup(\_ : String) -\> MemberLookupResultType > > > > } > > > > > > AnyObject lookup looks for an actual declaration on any type anywhere. One could extend that mechanism to, say, return all Python methods and assume that you can call any Python method with any PythonObject instance. AnyObject lookup is fairly unprincipled as a language feature, because there’s no natural scope in which to perform name lookup, and involves hacks at many levels that don’t always work (e.g., AnyObject lookup… sometimes… fails across multiple source files for hard-to-explain reasons). You’re taking on that brokenness if you expand AnyObject lookup to another ecosystem. > > > > Yeah, sorry, that’s not what I meant: > > > > > Although it doesn’t really seem like AnyObject lookup is the thing you’re asking for here. It seems more like you want dynamicMemberLookup(\_:) to capture “self” and the method name, and then be a callable thing as below… > > > > That’s what I meant :-). > > > > A type that implements this magic protocol would never fail name lookup: “foo.bar” would always fall back to calling: foo.dynamicMemberLookup(“bar") > > > > it’s simple and more predictable than AnyObject, it also matches what dynamic languages like Python needs. > > > > > > // Magic, allows “overloaded/sugared postfix ()”. > > > > protocol CustomCallable { > > > > func call( …) > > > > } > > > > > > > > The only tricky thing about this is the call part of things. At least in the case of python, we want something like this: > > > > > > > >   foo.bar(1, 2, a: x, b: y) > > > > > > > > to turn into: > > > > foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y]) > > > > > > > > We don’t want this to be a memberlookup of a value that has “bar” as a basename and “a:” and “b:” as parameter labels. > > > > > > Well, I think the MemberLookupResult is going to get the name “bar”, argument labels “\_:\_🅰b:”, and arguments “1”, “2”, “x”, “y”, because that’s the Swift model of argument labels. It can then reshuffle them however it needs to for the underlying interaction with the Python interpreter. > > > > > > There are definite design trade-offs here. With AnyObject lookup, it’s a known-broken feature but because it depends on synthesized Swift method declarations, it’ll behave mostly the same way as other Swift method declarations—static overloading, known (albeit weak) type signatures, etc. But, it might require more of Python’s model to be grafted onto those method declarations. With dynamic member lookup, you’re throwing away all type safety (even for a motivated Python developer who might be willing to annotate APIs with types) and creating a general language mechanism for doing that. > > > > Right, something like this could definitely work, but keep in mind that the Swift compiler knows nothing about Python declarations. > > > > Perhaps the most straight-forward thing would be to support: > > > > protocol CustomCallable { > > func call(…arg list as array and kw args...) > > func callMember(\_ : String, …otherstuffabove...) > > } > > > > Given this, the compiler could map: > > > > pythonThing(42) -\> pythonThing.call([42]) > > pythonThing.method(a: 42) -\> pythonThing.callMember(“method”, kwargs: [“a”: 42]) > > > > This is the simplest way to map the Swift semantics (where kw args are part of compound lookups) into the Python world. > > > > -Chris ---
**Author:** ![Chris\_Lattner](https://avatars.discourse-cdn.com/v4/letter/c/bc8723/32.png) [@Chris\_Lattner](https://forums.swift.org/u/Chris_Lattner)\ **Post date:** [October 31, 2017, 4:48pm UTC](https://forums.swift.org/t/discussion-swift-for-data-science-ml-big-data-analytics/6905/16 "2017-10-31T16:48:08Z")
> I can think of two things that could tip the scale of the discussion: > > a) The big question is whether we \*want\* the ability to write custom > rule-of-5 style behavior for structs, or if we want it to only be used in > extreme cases (like bridging interop in this proposal). If we \*want\* to > support it someday, then adding proper “safe” support is best (if > possible). If we don’t \*want\* people to use it, then making it Unsafe and > ugly is a reasonable way to go. > > b) The ownership proposal is likely to add deinit's to structs. If it > also adds explicit move initializers, then it is probably the right thing > to add copy initializers also (and thus go with approach #2). That said, > I’m not sure how the move initializers will be spelled or if that is the > likely direction. If it won’t add these, then it is probably better to go > with approach #1. John, what do you think? > > I was hoping not to have to add explicit move/copy initializers, perhaps > ever. Can you elaborate more on why? I'm even more interested in your the rationale more than the outcome :-) I would suggest one of two things: >   - We could add a Builtin type for these types in Swift. Because of our > architecture, this is mostly an IRGen task. Yes, this could definitely work. >   - We could start working on C++ import. C++ import is a huge task, but > we don't really need most of it for this. This could work, but it would be unfortunate to push people to having to write (unsafe) C++ and import it into Swift to achieve simple things like this. I'd prefer going the direction of suggestion #1 above and allowing a "pure swift and explicitly unsafe" solution. -Chris > **···** > > On Mon, Oct 30, 2017 at 10:55 PM, John McCall \ wrote: ---
**Author:** ![Chris\_Lattner](https://avatars.discourse-cdn.com/v4/letter/c/bc8723/32.png) [@Chris\_Lattner](https://forums.swift.org/u/Chris_Lattner)\ **Post date:** [November 1, 2017, 4:31am UTC](https://forums.swift.org/t/discussion-swift-for-data-science-ml-big-data-analytics/6905/17 "2017-11-01T04:31:45Z")
I tried to send a response to this earlier today but it apparently didn’t get out. If it did, then I apologize in advance for the dupe: > > > > // not magic, things like Int, String and many other conform to this. > > > > protocol Pythonable { > > > > init?(\_ : PythonObject) > > > > func toPython() -\> PythonObject > > > > } > > > > > > It’s not magic unless you expect the compiler or runtime to help with conversion between Int/String/etc. and PythonObject, as with \_ObjectiveCBridgeable. > > > > Right, as I said above “not magic”. The conformances would be manually implemented in the Python overlay. This provides a free implicit conversion from "T -\> Pythonable” for the T’s we care about, and a failable init from Python back to Swift types. > > Note that, under this scheme, > >   let p: Pythonable = 17 >   let i: Int = p as! i > > will work if Int is Pythonable, but not when p comes back from Python. Right. I don't expect users to traffic in the Pythonable type, it would be something that is used internally by the Python overlay. The only time I'd expect them to see it is when they want to make one of their types Pythonable (which would be rare, but needs to be possible). > > I can think of two things that could tip the scale of the discussion: > > > > a) The big question is whether we \*want\* the ability to write custom rule-of-5 style behavior for structs, or if we want it to only be used in extreme cases (like bridging interop in this proposal). If we \*want\* to support it someday, then adding proper “safe” support is best (if possible). If we don’t \*want\* people to use it, then making it Unsafe and ugly is a reasonable way to go. > > > > b) The ownership proposal is likely to add deinit's to structs. If it also adds explicit move initializers, then it is probably the right thing to add copy initializers also (and thus go with approach #2). That said, I’m not sure how the move initializers will be spelled or if that is the likely direction. If it won’t add these, then it is probably better to go with approach #1. John, what do you think? > > > > > Presumably, binding to Python is going to require some compiler effort—defining how it is that Python objects are initialized/copied/moved/destroyed seems like a reasonable part of that effort. > > > > Actually no. If we add these three proposals, there is no other python (or perl, etc…) specific support needed. It is all implementable in the overlay. > > Support for working with Python objects would be implementable in the overlay, but the result isn’t necessarily ergonomic (e.g., my “as!” case from a Python-generated integer object to Int, shown above). That might be fine! More comments on this below. Yeah, I don't think that matters. The preferred way to do this is to use:   Int(somePythonValue) which would be a failable conversion. somePythonValue would be a PythonObject (the struct) not "Pythonable" > > > > // Magic, allows anyobject-like member lookup on a type when lookup otherwise fails. > > > > protocol DynamicMemberLookupable { > > > >   associatedtype MemberLookupResultType > > > >   func dynamicMemberLookup(\_ : String) -\> MemberLookupResultType > > > > } > > > > > > AnyObject lookup looks for an actual declaration on any type anywhere. One could extend that mechanism to, say, return all Python methods and assume that you can call any Python method with any PythonObject instance. AnyObject lookup is fairly unprincipled as a language feature, because there’s no natural scope in which to perform name lookup, and involves hacks at many levels that don’t always work (e.g., AnyObject lookup… sometimes… fails across multiple source files for hard-to-explain reasons). You’re taking on that brokenness if you expand AnyObject lookup to another ecosystem. > > > > Yeah, sorry, that’s not what I meant: > > (Good) Also, for sake of discussion, we’d have to figure out what the type of MemberLookupResultType would be for Python. I can see several choices: 1) Make it return a "PythonObject!” 2) Make it strict, returning a PythonObject or trapping. 3) Make PythonObject itself nullable internally and return a null PythonObject. #1 matches Python semantics directly, because it allows clients who care to check, but those who don't can ignore it. The only problem I anticipate is that it will break things like: let x = foo.bar let y = x.thing // fails, because x implicitly promoted to PythonObject? #3 is gross and cuts against lots of things in Swift (recall when UnsafePointer itself was implicitly nullable, lets not go back to those bad old days). I think that #2 is the least bad tradeoff. > > Right, something like this could definitely work, but keep in mind that the Swift compiler knows nothing about Python declarations. > > > > Perhaps the most straight-forward thing would be to support: > > > > protocol CustomCallable { > > func call(…arg list as array and kw args...) > > func callMember(\_ : String, …otherstuffabove...) > > } > > > > Given this, the compiler could map: > > > > pythonThing(42) -\> pythonThing.call([42]) > > pythonThing.method(a: 42) -\> pythonThing.callMember(“method”, kwargs: [“a”: 42]) > > > > This is the simplest way to map the Swift semantics (where kw args are part of compound lookups) into the Python world. > > Okay, I agree that this gets Swift syntax into a call to the Python interpreter fairly quickly. Over-architecting for the sake of discussion: > > protocol CustomCallable { >   associatedtype CustomArgument >   associatedtype CustomNominal >   associatedtype CustomResult > >   func callMember(self: CustomNominal, functionName: String, arguments: [(String, CustomArgument)]) throws -\> CustomResult >   // something for class/static members > } > > But this is \*all\* dynamic, even when one could map much of Python’s type information into Swift. For example, let’s take this: Yes, something like this is what I had in mind, but I think that functionName should only be required to be StringLiteralConvertible (no need to actually synthesize a real swift string). Since you bring it up, Python exceptions will be annoying - As with other languages, Python can throw from an arbitrary expression. Modeling everything as throws in Swift would be super-annoying and unergonomic for the programmer, because we'd require 'try' everywhere. Thoughts on what to do about that are welcome! > class Dog: > >     def \_\_init\_\_(self, name): >         self.name = name >         self.tricks = # creates a new empty list for each dog > >     def add\_trick(self, trick): >         self.tricks.append(trick) > > With your don’t-modify-the-compiler approach, how can I create a Dog instance and add a trick? I probably need to look up the class by name, call \_\_init\_\_ manually, etc. > >   let dogClass = python\_getClassByName(“Dog”) // implemented in the Python “overlay’, I guess >   let dog = python\_createInstance(dogClass) // implemented in the Python “overlay’, I guess >   dog.\_\_init\_\_(“Brianna”) // uses CustomCallable’s callMember >   dog.add\_trick(“Roll over”) // uses CustomCallable’s callMember I-am-not-a-python-expert, but I'd expect this to work:   let dogModule = Python.import("DogModule")   dogModule.Dog("jckarter").add\_trick("SILGenGen”)   let dog = dogModule.Dog(“Brianna”)   dog.add\_trick(“Roll over) or equivalently:   let Dog = Python.import(“DogModule.Dog")   Dog("jckarter").add\_trick("SILGenGen”)   let dog = Dog(“Brianna”)   dog.add\_trick(“Roll over) Seems pretty nice to me, with zero “Swift compiler knowledge of Python” required. > With compiler integration, > >   class Dog : PythonObject { >     init(\_ name: Pythonable) >     func add\_trick(\_ trick: Pythonable) >   } Something like this is possible, but would be substantially more work, be substantially more invasive, and would set the precedent that every other dynamic language would get support hacked directly into Swift. The only reason I can see this being useful is if we wanted to support the optional typing annotations in Python. While this would be "nice to have", I think the cost/benefit tradeoff involved is totally wrong for Swift. > With either the true “Python importer” solution or this code-generation solution, you at least get some level of code completion and basic sanity checking “for free”. In other words, you get some of the benefits of having a statically-type-checked language while still working on dynamic Pythonable types. We don't need to make Swift better at Python than Python itself is :-) -Chris > **···** > > > On Oct 30, 2017, at 10:47 PM, Douglas Gregor \ wrote: ---
**Author:** ![rxwei](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/rxwei/32/664_2.png) [@rxwei](https://forums.swift.org/u/rxwei)\ **Post date:** [November 1, 2017, 10:14am UTC](https://forums.swift.org/t/discussion-swift-for-data-science-ml-big-data-analytics/6905/18 "2017-11-01T10:14:43Z")
> Also, for sake of discussion, we’d have to figure out what the type of MemberLookupResultType would be for Python. I can see several choices: > > 1) Make it return a "PythonObject!” > 2) Make it strict, returning a PythonObject or trapping. > 3) Make PythonObject itself nullable internally and return a null PythonObject. > > #1 matches Python semantics directly, because it allows clients who care to check, but those who don't can ignore it. The only problem I anticipate is that it will break things like: > > let x = foo.bar > let y = x.thing // fails, because x implicitly promoted to PythonObject? > > #3 is gross and cuts against lots of things in Swift (recall when UnsafePointer itself was implicitly nullable, lets not go back to those bad old days). I think that #2 is the least bad tradeoff. I agree, PythonObject-or-die is the right trade-off. > Yes, something like this is what I had in mind, but I think that functionName should only be required to be StringLiteralConvertible (no need to actually synthesize a real swift string). > > Since you bring it up, Python exceptions will be annoying - As with other languages, Python can throw from an arbitrary expression. Modeling everything as throws in Swift would be super-annoying and unergonomic for the programmer, because we'd require 'try' everywhere. Thoughts on what to do about that are welcome! Requiring ‘try’ on every statement is annoying, but not having the ability to catch python exceptions is annoying too. We could probably make python exception handling an opt-in feature. For example: try Python.do {     let a = np.array([1, 2, 3])     let b = np.array([[2], [4]])     print(a.dot(b)) // matrix mul with incompatible shapes } catch let error as PythonException {     // Handle PythonError.valueError(“objects are not aligned”) } Python.do enables exception handling for statements in the body, declared as func `do`\(\_ body: () throws -\> T) throws -\> T When we execute python-throwing statements inside a Python.do{}, PythonException gets thrown. Otherwise, it traps. The ‘Python.do’ function would ask the python overlay to enter an "error-catching" state when executing the body closure. We make PythonObjects internally nullable, but guarantee that they are non-null (or trap) when the overlay is not in the “error-caught” state. When a python exception is thrown in the error-catching state, the overlay enters the error-caught state and propagates null through any python computation in the body closure. After the body is executed, throw that python exception. However, if there’s a throwing Swift statement after the throwing python statement in the body, the python exception won’t be caught first… So having the body as a non-throwing closure may be a better idea. -Richard > **···** > > > On Oct 31, 2017, at 21:31, Chris Lattner via swift-evolution \ wrote: > > > > class Dog: > > > > > >     def \_\_init\_\_(self, name): > > >         self.name = name > > >         self.tricks = # creates a new empty list for each dog > > > > > >     def add\_trick(self, trick): > > >         self.tricks.append(trick) > > > > > > With your don’t-modify-the-compiler approach, how can I create a Dog instance and add a trick? I probably need to look up the class by name, call \_\_init\_\_ manually, etc. > > > > > >   let dogClass = python\_getClassByName(“Dog”) // implemented in the Python “overlay’, I guess > > >   let dog = python\_createInstance(dogClass) // implemented in the Python “overlay’, I guess > > >   dog.\_\_init\_\_(“Brianna”) // uses CustomCallable’s callMember > > >   dog.add\_trick(“Roll over”) // uses CustomCallable’s callMember > > > > I-am-not-a-python-expert, but I'd expect this to work: > > > >   let dogModule = Python.import("DogModule") > >   dogModule.Dog("jckarter").add\_trick("SILGenGen”) > >   let dog = dogModule.Dog(“Brianna”) > >   dog.add\_trick(“Roll over) > > > > or equivalently: > > > >   let Dog = Python.import(“DogModule.Dog") > >   Dog("jckarter").add\_trick("SILGenGen”) > >   let dog = Dog(“Brianna”) > >   dog.add\_trick(“Roll over) > > > > Seems pretty nice to me, with zero “Swift compiler knowledge of Python” required. > > > > > With compiler integration, > > > > > >   class Dog : PythonObject { > > >     init(\_ name: Pythonable) > > >     func add\_trick(\_ trick: Pythonable) > > >   } > > > > Something like this is possible, but would be substantially more work, be substantially more invasive, and would set the precedent that every other dynamic language would get support hacked directly into Swift. The only reason I can see this being useful is if we wanted to support the optional typing annotations in Python. While this would be "nice to have", I think the cost/benefit tradeoff involved is totally wrong for Swift. > > > > > With either the true “Python importer” solution or this code-generation solution, you at least get some level of code completion and basic sanity checking “for free”. In other words, you get some of the benefits of having a statically-type-checked language while still working on dynamic Pythonable types. > > > > We don't need to make Swift better at Python than Python itself is :-) > > > > -Chris > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > swift-evolution mailing list > > swift-evolution@swift.org > > [https://lists.swift.org/mailman/listinfo/swift-evolution](https://lists.swift.org/mailman/listinfo/swift-evolution) ---
**Author:** ![Brent\_Royal-Gordon](https://avatars.discourse-cdn.com/v4/letter/b/53a042/32.png) [@Brent\_Royal-Gordon](https://forums.swift.org/u/Brent_Royal-Gordon)\ **Post date:** [November 9, 2017, 1:31pm UTC](https://forums.swift.org/t/discussion-swift-for-data-science-ml-big-data-analytics/6905/19 "2017-11-09T13:31:46Z")
Add a `throws!` keyword to Swift, indicating that if a call is not covered by a `try` or `try?` keyword, a `try!` should be implicitly added. > **···** > > > On Oct 31, 2017, at 9:31 PM, Chris Lattner via swift-evolution \ wrote: > > > > Since you bring it up, Python exceptions will be annoying - As with other languages, Python can throw from an arbitrary expression. Modeling everything as throws in Swift would be super-annoying and unergonomic for the programmer, because we'd require 'try' everywhere. Thoughts on what to do about that are welcome! > > -- > Brent Royal-Gordon > Architechies ---
**Author:** ![rxwei](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/rxwei/32/664_2.png) [@rxwei](https://forums.swift.org/u/rxwei)\ **Post date:** [November 1, 2017, 10:20am UTC](https://forums.swift.org/t/discussion-swift-for-data-science-ml-big-data-analytics/6905/20 "2017-11-01T10:20:29Z")
> > Also, for sake of discussion, we’d have to figure out what the type of MemberLookupResultType would be for Python. I can see several choices: > > > > 1) Make it return a "PythonObject!” > > 2) Make it strict, returning a PythonObject or trapping. > > 3) Make PythonObject itself nullable internally and return a null PythonObject. > > > > #1 matches Python semantics directly, because it allows clients who care to check, but those who don't can ignore it. The only problem I anticipate is that it will break things like: > > > > let x = foo.bar > > let y = x.thing // fails, because x implicitly promoted to PythonObject? > > > > #3 is gross and cuts against lots of things in Swift (recall when UnsafePointer itself was implicitly nullable, lets not go back to those bad old days). I think that #2 is the least bad tradeoff. > > I agree, PythonObject-or-die is the right trade-off. > > > Yes, something like this is what I had in mind, but I think that functionName should only be required to be StringLiteralConvertible (no need to actually synthesize a real swift string). > > > > Since you bring it up, Python exceptions will be annoying - As with other languages, Python can throw from an arbitrary expression. Modeling everything as throws in Swift would be super-annoying and unergonomic for the programmer, because we'd require 'try' everywhere. Thoughts on what to do about that are welcome! > > Requiring ‘try’ on every statement is annoying, but not having the ability to catch python exceptions is annoying too. We could probably make python exception handling an opt-in feature. For example: > > try Python.do { >     let a = np.array([1, 2, 3]) >     let b = np.array([[2], [4]]) >     print(a.dot(b)) // matrix mul with incompatible shapes > } > catch let error as PythonException { >     // Handle PythonError.valueError(“objects are not aligned”) > } To correct my example: do {     try Python.do {         let a = np.array([1, 2, 3])         let b = np.array([[2], [4]])         print(a.dot(b)) // matrix mul with incompatible shapes     } } catch let error as PythonException {     // Handle PythonError.valueError(“objects are not aligned”) } Maybe ‘Python.do {}’ should be called something like ‘Python.safely {}’. -Richard > **···** > > > On Nov 1, 2017, at 03:14, Richard Wei via swift-evolution \ wrote: > > > > > On Oct 31, 2017, at 21:31, Chris Lattner via swift-evolution \\> wrote: > > > Python.do enables exception handling for statements in the body, declared as > > func `do`\(\_ body: () throws -\> T) throws -\> T > > > > When we execute python-throwing statements inside a Python.do{}, PythonException gets thrown. Otherwise, it traps. > > > > The ‘Python.do’ function would ask the python overlay to enter an "error-catching" state when executing the body closure. We make PythonObjects internally nullable, but guarantee that they are non-null (or trap) when the overlay is not in the “error-caught” state. When a python exception is thrown in the error-catching state, the overlay enters the error-caught state and propagates null through any python computation in the body closure. After the body is executed, throw that python exception. > > > > However, if there’s a throwing Swift statement after the throwing python statement in the body, the python exception won’t be caught first… So having the body as a non-throwing closure may be a better idea. > > > > -Richard > > > > > > class Dog: > > > > > > > >     def \_\_init\_\_(self, name): > > > >         self.name = name > > > >         self.tricks = # creates a new empty list for each dog > > > > > > > >     def add\_trick(self, trick): > > > >         self.tricks.append(trick) > > > > > > > > With your don’t-modify-the-compiler approach, how can I create a Dog instance and add a trick? I probably need to look up the class by name, call \_\_init\_\_ manually, etc. > > > > > > > >   let dogClass = python\_getClassByName(“Dog”) // implemented in the Python “overlay’, I guess > > > >   let dog = python\_createInstance(dogClass) // implemented in the Python “overlay’, I guess > > > >   dog.\_\_init\_\_(“Brianna”) // uses CustomCallable’s callMember > > > >   dog.add\_trick(“Roll over”) // uses CustomCallable’s callMember > > > > > > I-am-not-a-python-expert, but I'd expect this to work: > > > > > >   let dogModule = Python.import("DogModule") > > >   dogModule.Dog("jckarter").add\_trick("SILGenGen”) > > >   let dog = dogModule.Dog(“Brianna”) > > >   dog.add\_trick(“Roll over) > > > > > > or equivalently: > > > > > >   let Dog = Python.import(“DogModule.Dog") > > >   Dog("jckarter").add\_trick("SILGenGen”) > > >   let dog = Dog(“Brianna”) > > >   dog.add\_trick(“Roll over) > > > > > > Seems pretty nice to me, with zero “Swift compiler knowledge of Python” required. > > > > > > > With compiler integration, > > > > > > > >   class Dog : PythonObject { > > > >     init(\_ name: Pythonable) > > > >     func add\_trick(\_ trick: Pythonable) > > > >   } > > > > > > Something like this is possible, but would be substantially more work, be substantially more invasive, and would set the precedent that every other dynamic language would get support hacked directly into Swift. The only reason I can see this being useful is if we wanted to support the optional typing annotations in Python. While this would be "nice to have", I think the cost/benefit tradeoff involved is totally wrong for Swift. > > > > > > > With either the true “Python importer” solution or this code-generation solution, you at least get some level of code completion and basic sanity checking “for free”. In other words, you get some of the benefits of having a statically-type-checked language while still working on dynamic Pythonable types. > > > > > > We don't need to make Swift better at Python than Python itself is :-) > > > > > > -Chris > > > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > swift-evolution mailing list > > > swift-evolution@swift.org \ > > > [https://lists.swift.org/mailman/listinfo/swift-evolution](https://lists.swift.org/mailman/listinfo/swift-evolution) > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > swift-evolution mailing list > > swift-evolution@swift.org \ > > [https://lists.swift.org/mailman/listinfo/swift-evolution](https://lists.swift.org/mailman/listinfo/swift-evolution) [Next page](https://forums.swift.org/t/discussion-swift-for-data-science-ml-big-data-analytics/6905.md?page=2)