Proposal: Introduce User-defined "Dynamic Member Lookup" Types

[benjamin.g]

Doug and others have brought up some great points, and I think Doug’s idea
of a common infrastructure for importing declarations from other languages
is _extremely_ attractive for the long-term future of Swift.

However, unlike this proposal, that will (I imagine as a non-compiler
engineer) be a colossal undertaking, and as such it’s not going to make it
into Swift 5, or possibly even 6 or 7. I understand, then, Chris’s (and
other’s) desire to start interfacing with Python code now, not later. For
me, far and away the biggest problem with this proposal (and the only
outright deal-breaker) is that dynamic member lookups do not differentiate
themselves in any way from statically-checked member lookups syntactically.
I don’t object as strongly as others to the idea of adding this kind of
dynamism to the language, but if it’s going to be there, it should not be
possible to slightly misspell a static member name and end up with an
unexpectedly dynamic member that may or may not fail at compile-time.

As noted at the end of my message, I think one can write a Swift library
to make working with the Python runtime much easier, and write a wrapper
generator that produces Swift APIs from Python code that use said Swift
library.

That might be true for Python; I’m quite certain it’s not the case for
Ruby or JS as they exist in the wild. However…

At the risk of stating a universally unpopular opinion, beyond interop
with dynamic languages such as Python, there’s also value in dynamic member
lookup for •purely Swift-centric• reasons.

I say this with mixed feelings. I share Douglas’s concerns about breaking
Swift’s safety bubble. I like it that the language generally makes it clear
•at the point of use• which language features have runtime pitfalls that
the compiler cannot check. The points where this is not true of the
language (e.g. array subscripts) always feel a bit gritty between the teeth.

There can, however, be great value in constructing type membership at
runtime. The ergonomic access to JSON that’s been discussed in this
proposal is nothing to sneeze at. Dynamic languages make for lovely DSLs in
a way that fully statically type checked languages never do. Things like
Rails’s programmatically generated named routes (e.g. “user_post_path(id)”
because routing configuration created a user resource with nested posts)
are possible to approximate with statically typed approaches — but only
clumsily, never with the same expressiveness and concision.

I'm sorry but, no. Please, don't bring ruby magic and "lovely DSL" found
in dynamic languages into swift right now. I've had my share of dynamic
programming as well, mostly python and javascript, and i think it's now
pretty clear they make wonderful 20 lines scripts, but horrible 20 folders
projects languages. Typescript and python type annotations aren't taking
off for no reason (and if you look even further, in the case of client-side
javascript, the "hype" is going to language even stricter like elm).

The "JSON parsing" example given in the proposal is actually a perfect
example of what could go wrong if we were to add that to swift : people
will have the choice of declaring an actual struct as well as parsing
annotations (and by doing so, create an actual *documentation* and
"contract", somewhere, of what to expect). Or they'll just use dynamic
magic, and spread field accesss looking like real properties a little
everywhere in the code. And i know very well what choice they'll take.

On the other hand, if we wanted to add some "magic like" behavior, i have
an intuition (but that's just what it is, since i'm not coding compilers
for a living), that having a proper macro systems, or higher order types,
as well as good enough tooling to generate bridges to other languages, we
could be both type safe, and terse.

Actually, if people really feel the urge the interface with dynamic
language with a "dynamic feel", i'm not even sure something like a
"SwiftScript" fork wouldn't be a better idea than just try to cram python
behaviors into a statically typed language.

···

On Thu, Nov 30, 2017 at 10:48 PM, Paul Cantrell via swift-evolution < swift-evolution@swift.org> wrote:

On Nov 30, 2017, at 3:40 PM, Douglas Gregor via swift-evolution < > swift-evolution@swift.org> wrote:
On Nov 30, 2017, at 1:01 PM, Zach Wolfe <zacharyreidwolfe@gmail.com> > wrote:

I don’t mean to sneeze at the downside of dynamic typing either. They too
are nothing to sneeze at. I’ve spent many years on both sides of the
dynamic divide, and all I see is difficult tradeoffs.

Swift so far has said that dynamism isn’t worth the tradeoffs; it’s
aggressively avoided even robust runtime reflection, much less dynamic
dispatch. That may indeed be the right call for the language — but I’m not
easily convinced there isn’t room to open the door wider.

I think of Chris’s post back in the early days of this list about the
“programmer model” the language creates, and wonder if there’s a way we
can’t make a model that leaves the door open to this kind of dynamic
dispatch where it makes sense for the people and situation at hand, while
still staying true to the language’s spirit of “compile-time safety by
default, exceptions to that by intent at point of use.”

Opening the door to dynamism might take some of the flavor of the way the
language currently handles unsafe memory access: it’s possible and even
reasonably ergonomic, but you always know when you’ve crossed the Rubicon
into Dangerland.

Cheers,

Paul

- Doug

On Nov 30, 2017, at 2:24 AM, Douglas Gregor via swift-evolution < > swift-evolution@swift.org> wrote:

On Nov 26, 2017, at 10:04 PM, Chris Lattner via swift-evolution < > swift-evolution@swift.org> wrote:

I’d like to formally propose the inclusion of user-defined dynamic member
lookup types.

Here is my latest draft of the proposal:
https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438
DynamicMemberLookup proposal by lattner · Pull Request #768 · apple/swift-evolution · GitHub

An implementation of this design is available here:
https://github.com/apple/swift/pull/13076

The implementation is straight-forward and (IMO) non-invasive in the
compiler.

I think better interoperability with Python (and other OO languages in
widespread use) is a good goal, and I agree that the implementation of the
feature described is straight-forward and not terribly invasive in the
compiler.

However, I do not think this proposal is going in the right direction for
Swift. I have objections on several different grounds.

*Philosophy*
Swift is, unabashedly, a strong statically-typed language. We don’t allow
implicit down casting, we require “as?” so you have to cope with the
possibility of failure (or use “as!” and think hard about the “!”). Even
the gaping hole that is AnyObject dispatch still requires the existence of
an @objc declaration and produces an optional lookup result, so the user
must contend with the potential for dynamic failure. Whenever we discuss
adding more dynamic features to Swift, there’s a strong focus on
maintaining that strong static type system.

IMO, this proposal is a significant departure from the fundamental
character of Swift, because it allows access to possibly-nonexistent
members (as well as calls with incorrect arguments, in the related
proposal) without any indication that the operation might fail. It’s easy
to fall through these cracks for any type that supports
DynamicMemberLookupProtocol—a single-character typo when using a
DynamicMemberLookupProtocol-capable type means you’ve fallen out of the
safety that Swift provides. I think that’s a poor experience for the Python
interoperability case, but more on that in the Tooling section below.

While we shouldn’t necessarily avoid a feature simply because it can be
used distastefully, consider something like this:

public extension NSObject : DynamicMemberLookupProtocol,
DynamicCallableProtocol { … }

that goes directly to the Objective-C runtime to resolve member lookups
and calls—avoiding @objc, bridging headers, and so on. It’s almost
frighteningly convenient, and one could imagine some mixed
Objective-C/Swift code bases where this would save a lot of typing (of
code)… at the cost of losing static typing in the language. The presence of
that one extension means I can no longer rely on the safety guarantees
Swift normally provides, for any project that imports that extension and
uses a subclass of NSObject. At best, we as a community decide “don’t do
that”; at worse, some nontrivial fraction of the community decides that the
benefits outweigh the costs (for this type or some other), and we can no
longer say that Swift is a strong statically-typed language without adding
“unless you’re using something that adopts DynamicMemberLookupProtocol”.

*Tooling*
The Python interoperability enabled by this proposal *does* look fairly
nice when you look at a small, correctly-written example. However,
absolutely none of the tooling assistance we rely on when writing such code
will work for Python interoperability. Examples:

* As noted earlier, if you typo’d a name of a Python entity or passed the
wrong number of arguments to it, the compiler will not tell you: it’ll be a
runtime failure in the Python interpreter. I guess that’s what you’d get if
you were writing the code in Python, but Swift is supposed to be *better*
than Python if we’re to convince a community to use Swift instead.
* Code completion won’t work, because Swift has no visibility into
declarations written in Python
* Indexing/jump-to-definition/lookup documentation/generated interface
won’t ever work. None of the IDE features supported by SourceKit will work,
which will be a significant regression for users coming from a
Python-capable IDE.

Statically-typed languages should be a boon for tooling, but if a user
coming from Python to Swift *because* it’s supposed to be a better
development experience actually sees a significantly worse development
experience, we’re not going to win them over. It’ll just feel inconsistent.

*Dynamic Typing Features*
It’s possible that the right evolutionary path for Swift involves some
notion of dynamic typing, which would have a lot of the properties sought
by this proposal (and the DynamicCallableProtocol one). If that is true—and
I’m not at all convinced that it is—we shouldn’t accidentally fall into a
suboptimal design by taking small, easy, steps. If we’re to include
dynamic-typing facilities, we should look at more existing practice—C#
‘dynamic' is one such approach, but more promising would be some form of
gradual typing a la TypeScript that let’s one more smoothly (and probably
explicitly) shift between strong and weak typing.

*How Should Python Interoperability Work?*
Going back to the central motivator for this proposal, I think that
providing something akin to the Clang Importer provides the best
interoperability experience: it would turn Python declarations into *real*
Swift declarations, so that we get the various tooling benefits of having a
strong statically-typed language. Sure, the argument types will all by
PyObject or PyVal, but the names are there for code completion (and
indexing, etc.) to work, and one could certainly imagine growing the
importer to support Python’s typing annotations
<https://docs.python.org/3/library/typing.html&gt;\. But the important part
here is that it doesn’t change the language model at all—it’s a compiler
feature, separate from the language. Yes, the Clang importer is a big
gnarly beast—but if the goal is to support N such importers, we can
refactor and share common infrastructure to make them similar, perhaps
introducing some kind of type provider infrastructure to allow one to write
new importers as Swift modules.

In truth, you don’t even need the compiler to be involved. The dynamic
“subscript” operation could be implemented in a Swift library, and one
could write a Python program to process a Python module and emit Swift
wrappers that call into that subscript operation. You’ll get all of the
tooling benefits with no compiler changes, and can tweak the wrapper
generation however much you want, using typing annotations or other
Python-specific information to create better wrappers over time.

- Doug

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Chris_Lattner]

Philosophy
Swift is, unabashedly, a strong statically-typed language.

That’s factually incorrect.

You’re going to have to explain that statement without reference to AnyObject (we’ll discuss that case below).

Perhaps it depends on what you mean by “strong”: I interpreted that as meaning that it provides type safety, with a level of strength akin to what Java provides: a level that could support mobile code deployment.

Swift certainly does not provide that strong of a static type system, because it gives people explicit ways to opt out of that. UnsafeMutableRawPointer, unsafe bitcast, and many other facilities support this. It also allows calling into non-type safe code, so it isn’t very strong that way. There are also race conditions and other holes in the type system.

All that said, I think it is correct that a subset of Swift exists that does provide strong type safety, but particularly when bridging to C/ObjC is involved, that quickly goes away.

More problematically for your argument: your preferred approach requires the introduction of (something like) DynamicMemberLookupProtocol or something like AnyObject-For-Python, so your proposal would be additive on top of solving the core problem I’m trying to solve. It isn’t an alternative approach at all.

I wouldn’t say that’s my preferred approach. My preferred approach involves taking the method/property/etc. declarations that already exist in Python and mapping them into corresponding Swift declarations so we have something to find with name lookup. One could put all of these declarations on some PyVal struct or PythonObject and there would be no need for AnyObject-for-Python or DynamicMemberLookupProtocol.

You’re suggesting that the transitive closure of all Python methods and properties be preprocessed into a single gigantic Swift PyVal type? I guess something like that could be done.

I would be concerned because there are many N^2 or worse algorithms in the Swift compiler would probably explode. It also doesn’t provide the great tooling experience that you’re seeking, given that code completion would show everything in the Python universe, which is not helpful.

Further, it doesn’t provide a *better* experience than what I’m suggesting, it seems strictly worse. A preprocessing step prevents users from playfully importing random Python modules into the Swift repl and playgrounds. It also seems worse for implementors (who will get a stream of new bugs about compiler scalability).

Whenever we discuss adding more dynamic features to Swift, there’s a strong focus on maintaining that strong static type system.

Which this does, by providing full type safety - unlike AnyObject lookup.

You get dynamic safety because it goes into the Python interpreter; fair enough. You get no help from your tools to form a correct invocation of any method provided by Python.

Sure, that’s status quo for Python APIs.

IMO, this proposal is a significant departure from the fundamental character of Swift, because it allows access to possibly-nonexistent members (as well as calls with incorrect arguments, in the related proposal) without any indication that the operation might fail.

The only way your claim is correct is if someone implements the protocol wrong. What you describe is true of AnyObject lookup though, so I understand how you could be confused by that.

AnyObject lookup still requires you to find an actual declaration with a type signature. Yes, there are still failure cases. The dynamic type might be totally unrelated to the class in which you found the declaration you’re supposedly calling, which is a typical “unrecognized selector” failure and will always be an issue with dynamic typing. The actual type safety hole you’re presumably referring to is that the selector could be overloaded with a different type signature, and we don’t proactively check that the signature we type-checked against matches the signature found at runtime. It’s doable with the Objective-C method encodings, but has never been considered worthwhile.

To be clear, I’m not suggesting a change to AnyObject lookup. I don’t think that it is worth changing at this point in time. My point was to observe that the proposed DynamicMemberLookupProtocol proposal does not suffer from these problems. It really is type / memory safe, assuming a sane implementation.

It’s easy to fall through these cracks for any type that supports DynamicMemberLookupProtocol—a single-character typo when using a DynamicMemberLookupProtocol-capable type means you’ve fallen out of the safety that Swift provides.

Since you seem to be latching on to this, I’ll say it again: the proposal is fully type safe :-)

That said, the “simple typo” problem is fundamental to the problem of working with a dynamically typed language: unless you can eradicate all “fundamental dynamism” from the language, you cannot prevent this.

That said, since this is a problem inherent with these languages, it is something people are already very familiar with, and something that everyone using those APIs has had to deal with. This is also not our problem to solve, and people in the Python community have been discussing and working on it over a large part of its 25 year history. I find your belief that we could solve this for Python better than the Python community itself has both idealistic and a bit naive.

I’m not pretending we can fully solve the problem. I’m pointing out that depending entirely on DynamicMemberLookupProtocol throws away information about method declarations that is present in Python and used by Python tooling to good effect.

I’m not opposed to going further over time, but I’d like to get started at some point :-). I’m not in a super urgent hurry to get this in in the next week or month or anything like that, but I also don’t want to wait until Swift 10.

Tooling
The Python interoperability enabled by this proposal *does* look fairly nice when you look at a small, correctly-written example. However, absolutely none of the tooling assistance we rely on when writing such code will work for Python interoperability. Examples:

* As noted earlier, if you typo’d a name of a Python entity or passed the wrong number of arguments to it, the compiler will not tell you: it’ll be a runtime failure in the Python interpreter. I guess that’s what you’d get if you were writing the code in Python, but Swift is supposed to be *better* than Python if we’re to convince a community to use Swift instead.
* Code completion won’t work, because Swift has no visibility into declarations written in Python
* Indexing/jump-to-definition/lookup documentation/generated interface won’t ever work. None of the IDE features supported by SourceKit will work, which will be a significant regression for users coming from a Python-capable IDE.

Yes, welcome to the realities of modern Python development!

Python plugins for IDEs (e.g., for Atom) provide code completion, goto definition, and other related features. None of the Swift tooling will work if Swift’s interoperability with Python is entirely based on DynamicMemberLookupProtocol.

I don’t understand your rationale here. I think you agree that we need to support the fully dynamic case (which is what I’m proposing). That said, this step does not preclude introducing importer magic (either through compiler hackery or a theoretical "type providers” style of feature). Doing so would provide the functionality you’re describing.

Statically-typed languages should be a boon for tooling, but if a user coming from Python to Swift *because* it’s supposed to be a better development experience actually sees a significantly worse development experience, we’re not going to win them over. It’ll just feel inconsistent.

By your argument we should ban AnyObject lookup as well, given its inconsistency with the rest of the language.

By my argument, we should at least replace the ImplicitlyUnwrappedOptional result with a true Optional (so one has to acknowledge that the method shouldn’t be there). We’ve seriously considered it, but it’s a source-breaking change, and it hasn’t seemed worth the engineering effort to pursue it.

Ok, that would be a nice step, but doesn’t fix the type safety hole.

In any case, my proposal allows the use of strong optional results as well, so the fate of AnyObject isn’t really bound up with it.

I don’t think that removing AnyObject dispatch entirely is possible at this point in Swift’s lifetime. While AnyObject has become much less prominent than it was in the Swift 1.0 days ([AnyObject] and [NSObject : AnyObject], oh my!), there is still a significant amount of code using it in the wild.

Completely agreed. The major advantage I see of changing it now is if there is some small mostly-user-invisible-change that allows a dramatic simplification to the compiler implementation.

Dynamic Typing Features
It’s possible that the right evolutionary path for Swift involves some notion of dynamic typing, which would have a lot of the properties sought by this proposal (and the DynamicCallableProtocol one). If that is true—and I’m not at all convinced that it is—we shouldn’t accidentally fall into a suboptimal design by taking small, easy, steps.

Given that you haven’t followed the discussion on the many threads we’ve had on this, and haven’t proposed a workable approach to this problem, I’m not sure upon what basis your fears and uncertainty and doubt are founded.

A few meta-comments here. First of all, following all previous threads on a discussion is not realistic.

I understand that, but you’re also accusing the proposal of being a suboptimal design made by looking at a series of small easy steps, instead of the right design for the long term. I’m pointing out that it is hard to see the rationale for that sort of claim.

This is part of the reason why we have different stages in a proposal’s lifetime, and is the responsibility of the proposal’s authors to capture alternatives and rationale in the proposal to make it self-contained.

Agreed. As I mentioned in my previous email, I definitely screwed that up by not capturing this discussion in the proposal. Thank you again for pulling this perspective to the front of the discussion so I could fix that oversight.

This proposal went through rapid iteration in the pitch phase and has now been elevated to a pull request to ask for formal review—you should expect more people to come on board having not read those threads. I appreciate that you have now captured more alternatives and rationale in the proposal.

Agreed. This is why I’ve been proactive about starting threads and trying to keep visibility on the proposal each time there is a significant change. I really do value the discussion and feedback (both on the proposed direction but also the writing itself).

Second, it is absolutely reasonable to disagree with the technical direction of a proposal without providing a complete solution to the problem that the proposal is attempting to solve. Some problems aren’t worth solving at all, or fully.

Ok, but at some point, if there is no alternative proposed, then a strong opposition has the appearance of saying “we shouldn’t solve this problem”. It was my understanding that thought that this was a worthwhile problem to solve.

How Should Python Interoperability Work?
Going back to the central motivator for this proposal, I think that providing something akin to the Clang Importer provides the best interoperability experience: it would turn Python declarations into *real* Swift declarations, so that we get the various tooling benefits of having a strong statically-typed language.

This could be an theoretically interesting refinement to this proposal but I’m personally very skeptical that this is every going to happen. I’ve put the rationale into the alternatives section of the proposal. I don’t explain it in the proposal in this way directly, but I believe it is far more likely for a Pythonista transplant into Swift to rewrite their code in Swift than it is to use Python type annotations.

I assume that this belief is based on type annotations lack of traction in the Python community thus far?

There are many parts to this, which have to do with the ObjC<->Swift situation being very different than the Python<->Swift situation:

1) The annotations don’t have significant traction in the Python community.
2) The Python annotations are not as powerful as ObjC generics are, and thus lack important expressive capability.
3) Many Python APIs are wrappers for C APIs. “Swiftizing” a Python API in this case means writing a new Swift wrapper for the API, not adding type annotations.
4) The Python community doesn’t care about Swift, and are not motivated to do things to make Swift succeed.
5) There is no “clang equivalent” for Python (that I’m aware of) which close enough to the way Clang does for us to directly use. The owners of the existing Python compiler/interpreter implementations are not going to be strongly motivated to change their stuff for us.

Finally, just MHO, but I don’t expect a lot of “mix and match" Python/Swift apps to exist (where the developer owns both the Python and the Swift code), which is one case where type annotations are super awesome in ObjC. IMO, the most important use-case is a Swift program that uses some Python APIs.

Sure, the argument types will all by PyObject or PyVal,

That’s the root of the problem. Python has the “fully dynamic” equivalent of “id” in Objective-C, so we need to represent that somehow. Even if we followed the implementation approach of the Clang importer, we would need some way to represent this dynamic case. That type needs features like DynamicMemberLookup or AnyObject. In my opinion, the DynamicMemberLookup approach is better in every way than AnyObject is.

The AnyObject approach has the advantage of knowing the set of declared, reachable APIs:

* Code completion shows all of the APIs that are possible to use via dynamic dispatch, with their signatures so can fill in the right # of arguments, see the names of the parameters, see documentation, etc.
* Indexing/refactoring/goto definition all point you to the declarations that could be the targets of dynamic dispatch

The DynamicMemberLookup approach is better for cases where you don’t have a declaration of the member you want to access. I suspect that’s not the common case.

Your points are valid, but the advantages for Objective-C don’t obviously translate to Swift. Note that ObjC (due to its heritage) has very long method names that are perhaps arguably designed to not conflict with each other often. Python doesn’t have this heritage, and it has much shorter names, which means that we’ll get a lot more conflicts and a lot less “safety" out of this.

AnyObject lookup also depends on a strange set of scoping heuristics that was designed to be similar to Clang’s “header import” scope. It isn’t clear that this approach will work in Python, given that it doesn’t have an analogue of umbrella headers that import things that cross frameworks.

Which approach do you think is the best way to handle the untyped “actually dynamic” case?

AnyObject already exists in the language, and it fits the untyped “actually dynamic” case well. It does require having a declaration for the thing you want to reference, which I consider to be important: we can code-complete those declarations, goto-definition to see those declarations, index/refactor/look up documentation based on those declarations.

I’d be more inclined to push for the ImplicitlyUnwrappedOptional -> Optional change if we did something to make AnyObject more prominent in Swift.

I didn’t realize that you were thinking we would literally use AnyObject itself. I haven’t thought fully through it, but I think this will provide several problems:

1) You’re mushing all of the ObjC and Python world’s together, making the ObjC interop worse just because you’re doing some Python stuff too.
2) You’re introducing ambiguity: does “ao = [1,2,3]” create an NSArray or a Python array? How do string literals work? (The answer is obvious, Python loses). Maybe there is some really complicated bridging solution to these problems, but that causes its own massive complexity spiral.
3) You can’t realistically overload the Python operator set on AnyObject, which means you get a worse python experience.
4) AnyObject magic is currently limited to Apple platforms. This would bring its problems to other platforms like Linux.

There are probably other issues, but I haven’t thought through it.

but the names are there for code completion (and indexing, etc.) to work, and one could certainly imagine growing the importer to support Python’s typing annotations <https://docs.python.org/3/library/typing.html&gt;\.

You’re basing this on the flawed assumption that local variables will pervasively have types, which I can’t imagine being the case. Even on "typable” API, I wouldn’t expect people to commonly get code completion results for reasons now explained in the proposal.

Remember that one *does* get code completion results for member access into an AnyObject… lots of them… but the list filters down pretty fast when you type a few characters, and then you get a member access that’ll fill in stubs for (say) the arguments to the method you were trying to call. But you can’t get those code completion results without having declarations to complete to.

Fair point, it’s unclear to me how useful this would be with python’s style of naming, but it could work.

In truth, you don’t even need the compiler to be involved. The dynamic “subscript” operation could be implemented in a Swift library, and one could write a Python program to process a Python module and emit Swift wrappers that call into that subscript operation. You’ll get all of the tooling benefits with no compiler changes, and can tweak the wrapper generation however much you want, using typing annotations or other Python-specific information to create better wrappers over time.

I’d love for you to sketch out how any of this works with an acceptable user experience, because I don't see anything concrete here.

We don’t need the basic dynamic case in the language to do this experiment. Take the PyVal struct from the proposal. Now, write a Python script that loads some module Foo and uses Python’s inspect <https://docs.python.org/3/library/inspect.html&gt; module to go find the classes, methods, etc., and pretty-print Swift code that uses PyVal. So this:

  def add_trick(self, trick):

turns into

  extension PyVal {
    func add_trick(_ trick: PyVal) -> PyVal {
      /* do the magic to call into Python */
    }
  }

Using the inspect module, you can extract parameter names, default arguments, docstrings, and more to reflect the existing Python API as Swift API, packed into a bridging module.

Note that we have a “flat” namespace of all Python methods on PyVal, which is basically what you get with AnyObject today. Swift tooling will provide code completion for member accesses into PyVal. Goto definition will jump to the pretty-printed declarations, which could have the docstrings formatted in comments and would show up in QuickHelp. The types are weak (everything is PyVal), but that’s what we expect from importing a dynamically-typed language.

As I mention above, I expect this to expose significant scalability problems in the Swift compiler and it also defeats REPL/Playgrounds. Being able to use the Swift REPL is really important for Python programmers.

-Chris

···

On Dec 1, 2017, at 12:26 AM, Douglas Gregor <dgregor@apple.com> wrote:

[xwu (Xiaodi Wu)]

It's not about getting "Python developers" or "Ruby developers" to switch
to Swift. Yes, if you believe the objective is to attract people who like
Python to Swift, then you must offer something "better than Python." Now,
given that some of the things that attract people to Python have to do with
its dynamic behavior, you've defined an impossible task right off the bat.
But that's not the objective here and it misses the point entirely. No, the
motivation instead is this:

Why would someone--someone who might not even know Python or Ruby--choose
to use those languages? Why does some bother to learn Python or Ruby other
than curiosity or love of the language itself? The answer is that there are
some tasks that are best accomplished using those languages because of
libraries and their user communities--so much better than one's other
favorite languages that those other languages are not even in contention.
In my case, many excellent computational biology tools were only available
(nowadays, are most mature and have the biggest ecosystem of users) in
languages such as Python or Perl. So the use case here is, how do we make
Swift a viable candidate for doing those things which today drive users to
Python? The answer here is _not_: build a better Python. Nor does it
require, out of the gate, even being as good as Python. The solution is to
provide a _reasonably_ ergonomic to _interoperate with libraries available
in Python_, with the benefit that those parts that you can write in native
Swift will make the overall result safer and faster, etc.

Again, it's _not_ about "scooping up developers of other languages"--it's
about expanding the universe of computational tasks for which Swift is a
viable candidate. This is why, also, Chris's insistence that the solution
be equally available in the REPL and in Playgrounds is so important.

···

On Fri, Dec 1, 2017 at 4:17 AM, Karl Wagner via swift-evolution < swift-evolution@swift.org> wrote:

> On 1. Dec 2017, at 07:05, Chris Lattner via swift-evolution < > swift-evolution@swift.org> wrote:
>
> Hi Doug,
>
> Thank you for the detailed email. I have been traveling today, so I
haven’t had a chance to respond until now. I haven’t read the down-thread
emails, so I apologize if any of this was already discussed:
>
>> I think better interoperability with Python (and other OO languages in
widespread use) is a good goal, and I agree that the implementation of the
feature described is straight-forward and not terribly invasive in the
compiler.
>
> Fantastic, I’m really pleased to hear that! I only care about solving
the problem, so if we can find a good technical solution to the problems
than I’ll be happy.
>
> A funny thing about swift-evolution is that it is populated with lots of
people who already use Swift and want to see incremental improvements, but
the people who *aren’t* using Swift yet (but should be) aren’t represented
here. As you know, I’m perhaps the biggest proponent of Swift spreading
into new domains and earning the love of new users.

I don’t really appreciate that argument. You also need to consider: what
improvements are we offering these people who don’t yet use Swift?

For Objective-C developers, the static typing and resulting performance
and safety improvements made this a worthwhile switch. Plus, we could
interoperate very well with existing Objective-C codebases and libraries
(e.g. we can have generic subclasses of NSObject!), so we could actually
use a lot of these nice features when writing our mac/iOS applications.

For Python developers, I gather the pitch would be similar: Swift has all
kinds of nice features which will make your code safer and faster. We’re
unlikely to achieve the same tight integration with Python (or any other
language) that we have with Objective-C, though. We used to rely a lot on
AnyObject, but since Objective-C made the importing experience so much
better, you rarely (if ever) see it in practice. Objective-C can also be
more dynamic than we show it as - you can have methods which are not
declared anywhere, and which the object synthesises during its
message-handling routines. We don’t support calling these non-existing
methods on Objective-C objects, so why should we allow it for these objects?

My worry is that in eagerness to try and scoop up developers of other
languages, that we diminish the Swift language to the point where it
doesn’t really provide a benefit for them. Python developers will be
writing the same unsafe, non-checked code using Swift that they do in
Python; where’s the win?

[Jon_Gilbert1 (Jon Gilbert)]

I think we would be better served by a transpiler that translates Python (etc.) into Swift at compile time.

Look what Google did with j2objc (https://github.com/google/j2objc\). It translates Java right into Objective C. You can even put your Java code right in XCode and it auto-translates at build time.

Clearly, this is no small feat, and j2objc is a technical marvel that took world-class engineers years to perfect.

My point is, “Dynamic Member Lookup” is not the only solution, and it’s not the ideal solution if indeed it compromises the static guarantees of Swift.

Therefore, we should consider what other approaches might entail. The main players in Swift have lots of money and technical resources they could pour into a set of revolutionary transpilers.

Surely we don’t want to allow Goole to be the only company to provide a library that translates other codebases directly to a primarily Apple language, do we?

That being said, I am still interested to hear Chris’s response to these concerns, and if they were already addressed on a previous message and I missed that, then please forgive me.

- Jon

···

On Dec 1, 2017, at 02:44, Xiaodi Wu via swift-evolution <swift-evolution@swift.org> wrote:

So the use case here is, how do we make Swift a viable candidate for doing those things which today drive users to Python? The answer here is _not_: build a better Python. Nor does it require, out of the gate, even being as good as Python. The solution is to provide a _reasonably_ ergonomic to _interoperate with libraries available in Python_, with the benefit that those parts that you can write in native Swift will make the overall result safer and faster, etc.

[davedelong]

+

Dave

···

On Dec 1, 2017, at 8:35 AM, Jon Gilbert via swift-evolution <swift-evolution@swift.org> wrote:

On Nov 30, 2017, at 08:10, Karl Wagner via swift-evolution <swift-evolution@swift.org> wrote:

Personally, I feel this system is designed to let you write Python, using Swift as a wrapper language - except unlike with Objective-C,

Let me paraphrase the proposal—it basically says this system is designed to make it easier to “fake import” existing Python/Ruby/JS libs to make it easier for people who depend on them to adopt Swift. Especially in the world of server-side Swift, I can see how this could help speed Swift’s adoption.

However, I believe that you bring up an extremely valid concern. To extrapolate from your points, this proposal seems to have a high potential to erode what makes Swift special, and it provides a way to get around Swift’s safe-guards, avoid writing Swift versions of libraries, and ultimately, avoid hiring real Swift developers.

it doesn’t have any static information about what you can do or not. It can’t provide the “safe by default” guarantees are other core principles and set Swift apart from other languages.

Exactly.

Look how JSON parsing is done in Swift 3.2 and 4.0—we make static types. The 1’s and 0’s from a web response go through layers of proper error handling before they become proper types Swift.

So why does this proposal keeps mentioning JSON parsing as an excuse for making a dynamic ghetto within the language? Because some people are parsing untyped JSON? Well, the solution for that is, type your frickin’ JSON.

Why compromise Swift’s static-ness and roll out the red carpet for the dynamic, unwashed masses?

Wouldn’t it be better to create an AI that quickly translates Pyton, Ruby, or JS into Swift? Augustus built huge public bathhouses for the unwashed Romans.

When we consider interoperability with other languages, it should be from the perspective of mapping their features to Swift’s philosophies.

Amen.

This proposal takes the reverse approach, and makes Swift like Python, so I’m against it.

This is my concern as well.

Perhaps there is a way to make the proposal require that you declare Swift types and protocols that map to these other languages, with verbose error handling for when this glue breaks down (just like we have for typed JSON).

Then, we could also make scripts that auto-generate the necessary Swift typing boilerplate interface code from the Python/Ruby/JS source, kinda like we get for Obj.-C bridging headers. This could be a good step towards the full AI translation that I mentioned.

Of course, maybe I am missing something here, and my opinion is wrong; Chris will let us know if these concerns are valid.

However, I hope we don’t hear how we are just being paranoid, because I assure you, this is not paranoia. It’s just about maintaining the static-ness and purity of Swift to help save the world from bad code.

Everything in life and engineering is a trade-off. Swift’s trademark is that it consistently sacrifices the convenience of dynamic-ness for the safety and performance of static-ness. This proposal does seem to do the opposite, and we’d be naive not to fear the effect this will have.

Ask yourself... what if Product knows about it? The last conversation a tech lead or engineering manager wants to have, I would imagine, is to explain on deaf ears why it would be bad to cut costs by farming out the iOS work to a cheap Python shop overseas, then just wrap it in sugary compiler hooks. Or why it would be bad for a Swift project to depend upon a stack of third-language dependencies that CocoaPods/Carthage can’t manage, XCode can’t analyze, stack traces can’t delve into (or can it?), lldb can’t step through, etc.

I do not believe this is unreasonable paranoia, I just believe it looks that way to people who work with an elite squad of engineers at a top company, who have perhaps not seen what goes on at most places, where budgets are tight and lots of people are not the biggest fans of Apple. It’s already with much grinding of teeth that non-Windows, non-Linux devs are necessary.

Given Swift’s emphasis on safety, this push to make danger easier really does surprise me, unless I just don’t understand this correctly. Have we re-evaluated the importance of having the language itself prohibit bad practices? Or is there some reason why this proposal does not actually make it easier to write code that looks like good, type-safe Swift code, but actually isn’t?

Finally, to really be honest, may I ask why web devs, many of whom never touched a line of Swift, deserve all this dynamic sugar, while on the other hand, we SWIFT devs don’t get any? How will people feel who were told that dynamic is so unsafe that Swift was designed to prevent it altogether? People who have been through all the migrations and member-wise trenches might chafe a bit. Is that unreasonable?

Lastly, if you can “fake import” Javascript or Python to Swift via this proposal, is there anything that would prevent the “fake import” of masquerading Swift modules? In other words, could lazy devs who want hacky dynamic-ness abuse this? If so then maybe I’m in favor after all ;P (Just kidding.)

- Jon

PS—I am just trying to think critically here. Ultimately this proposal would not hurt me, but I am just worried about the big picture. I can see why it’s a tough call and I am sure that some pressures exist that might make even this compromise to be worth it. I just feel like we should not underestimate how much bad will come along with the good, nor should we assume, “Well that kind of bad would never happen where I work, and only idiots would abuse this.”
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[gwendal.roue (Gwendal Roué)]

In Chris's playground, it is very clear that you have to import PythonGlue in order to activate the retroactive conformance on Int, Double, etc.

This is very fine to me. Sorry for expressing a useless concern.

Gwendal

···

Le 1 déc. 2017 à 08:38, Gwendal Roué <gwendal.roue@gmail.com> a écrit :

In this context, I too am concerned with retroactive conformance. If it were forbidden, it would make it very clear where the dynamic scope starts and ends:

[Chris_Lattner]

After thinking about your email a bit more, I think I might understand the disconnect we’re having. I think we have different versions in mind of what “success” looks like:

I believe your view is that people start using Python APIs from their Swift code, but gradually add type annotations (either inline or in a sidecar database) to make those APIs progressively more Swifty. New features will be added to Python (its type system, compilers, databases for common APIs, etc) to improve this interoperability, along the lines of what we’ve done for Objective-C over the years. Fast forward several years, and large Python libraries would be nice to use from Swift - perhaps nicer than they are to use from Python itself. This view aligns with what happened with Objective-C <-> Swift interoperability.

In contrast, I’m specifically interested in developers in certain large domains (e.g. data science and ML) which are “forced” to use Python because that is where all the libraries are. I have spoken to many of these sorts of people, and a large number of them really *dislike* using Python for all the obvious reasons (including the tooling issues you point out). My view of success is that we allow them to write all of *their code* in Swift, which will lead to a massive quality of life benefit for these frustrated people. You’re right that they will still chaff when using imported Python APIs (which don’t feel “swifty” for LOTS of reasons - e.g. method naming and design patterns), but my view is that this will provide incentive for these people to provide a proper Swift implementation for these libraries over time.

In short, your end game is a pervasively intertwined Swift/Python world (like ObjC and Swift are). My view is that Swift holds Python at arm's length, and wins over the hearts and minds of developers, leading to new Swift APIs designed for Swift.

I’m not sure if you agree with that portrayal of your position, and if not, I’m sorry and will try to understand another way. However, if I’m close, then I have several concerns about that vision and don’t believe the end-game is achievable or better than what I’m proposing. Consider:

- I don’t think there will be a lot of success getting people who *actually love* Python to use Swift, unless there is already an extrinsic reason for them to use it.
- Type hints are not widely used in Python, and there are believable reasons that they won’t ever be. Because they are specifically poorly suited for libraries, their use seems to be in “user’s own code” - but my proposal solves this already! See below for examples.
- Even if type hints were widely adopted, it would require massive extensions to them and to Python to make them be "good enough” to provide value for the things you’re envisioning. Analogs to instancetype, objc generics, lots of the C macros, and many of the other things we’ve added to ObjC would have to be added.
- The Python community has no reason to do this work for the Swift community, or accept changes to Python that are required to make this actually great.
- Beyond the core type system, the Python and Objective-C languages work extremely differently in other ways (e.g. lack of umbrella headers making AnyObject-style dispatch questionable).
- Even with those annotations and all the work, the APIs we’d end up with are not going to be good Swift APIs. I don’t think that “renamification” and "IUO audits" would ever actually happen in practice, for example.
- I believe the engineering effort required to implement this vision is so massive (including the changes to Swift, Python, and Python libraries) that it simply will never actually happen.

More concerning to me is that your design of using the existing AnyObject type presents a really concerning technical problem, scalability: It is not simply a Swift/ObjC/Python world, Javascript is also super important. There are also a number of other less-widely used dynamic languages that are interesting. Your design leads to them all being mashed together into a common runtime system. I cannot imagine how this would end up being a good thing for system complexity, and would lead to each of them being jeopardized in different ways. As I have mentioned before numerous times, it also opens the door for enormous Swift compiler complexity, as each of the object models need to be supported in the compiler (so you can subclass each languages’ types), and many other complexities.

If you are serious about improving the tooling situation for people using Python APIs in Swift, the most natural way to do so is to follow the approach that the MyPy community (they are the ones who have thought about this the most) is using: provide progressive typing extensions, use data flow analysis to propagate around types, and enhance the tooling to have support this. I’m skeptical that this will ever be worthwhile, but a sketch could look like this:

Consider the first example from: mypy - Examples

With my proposals you could write the dictionary part very similar to the Python part (other pieces of the example, e.g. list comprehensions are uninteresting to me):

let d: PyVal = {:}
…
d[word] = d.get(word, 0) + 1

We can teach the compiler simple data flow analysis, to know that ‘d’ is a Python dictionary. We could then allow the user to go further, with some new syntax along the lines of:

let d: @type(Dict<String, Int>) PyVal = {:} // Lots of other syntactic choices possible.
…
d[word] = d.get(word, 0) + 1

However, the really important thing to recognize about these examples it that the type annotations are really poor at representing “generic” code like common Python libraries. They are designed for “user code” - like that on the mypy web page - which use concrete types.

As it turns out, my proposal provides a solution for this part of the problem, because we’re allowing user code to be written in Swift! A Swift programmer working with Python APIs would actually write this as:

let d: Dictionary<String,Int> = {:}
…
d[word, default: 0] += 1

Similarly, there is no reason to do anything to make the second and third examples work nicely in Swift: you’d just define a Swift class, and a Swift function.

MyPy:

class BankAccount:
    def __init__(self, initial_balance: int = 0) -> None:
        self.balance = initial_balance
    def deposit(self, amount: int) -> None:
        self.balance += amount
    def withdraw(self, amount: int) -> None:
        self.balance -= amount
    def overdrawn(self) -> bool:
        return self.balance < 0

my_account = BankAccount(15)
my_account.withdraw(5)
print(my_account.balance)
Swift:

class BankAccount {
   var balance : Int
   init(initialBalance: Int) { … }
  func deposit(….
}

Even the proposals that I’m making - simple and isolated though they are - are enough to provide a major quality of life improvement for people writing large amounts of code against Python APIs. Beyond that, they are small extensions with low complexity, scale to supporting many different dynamic languages over time, require a level of engineering effort that is plausible to be built, and do not require some sort of "executive buy in” from the Python community.

-Chris

···

On Dec 1, 2017, at 12:26 AM, Douglas Gregor <dgregor@apple.com> wrote:

On Nov 30, 2017, at 10:05 PM, Chris Lattner <clattner@nondot.org <mailto:clattner@nondot.org>> wrote:
Hi Doug,

Thank you for the detailed email. I have been traveling today, so I haven’t had a chance to respond until now. I haven’t read the down-thread emails, so I apologize if any of this was already discussed:

I think better interoperability with Python (and other OO languages in widespread use) is a good goal, and I agree that the implementation of the feature described is straight-forward and not terribly invasive in the compiler.

Fantastic, I’m really pleased to hear that! I only care about solving the problem, so if we can find a good technical solution to the problems than I’ll be happy.

[bjhomer]

Responses inline.

Personally, I feel this system is designed to let you write Python, using Swift as a wrapper language - except unlike with Objective-C,

Let me paraphrase the proposal—it basically says this system is designed to make it easier to “fake import” existing Python/Ruby/JS libs to make it easier for people who depend on them to adopt Swift. Especially in the world of server-side Swift, I can see how this could help speed Swift’s adoption.

However, I believe that you bring up an extremely valid concern. To extrapolate from your points, this proposal seems to have a high potential to erode what makes Swift special, and it provides a way to get around Swift’s safe-guards, avoid writing Swift versions of libraries, and ultimately, avoid hiring real Swift developers.

The point is not to let you write Python in Swift. If you want to write Python, just write Python. The point is to let you call Python libraries (or any dynamic language) from Swift, in the same way that we can already call C libraries from Swift. The ability to call C libraries does not make people “avoid hiring real Swift developers”; it just means that Swift developers can take advantage of C libraries. The goal of this proposal is to make Swift able to use more existing libraries. It sounds like your argument is “we want people to rewrite those libraries in Swift instead”. That’s a great goal, but it will only be achieved if Swift gains the kind of widespread cross-platform adoption that would be facilitated by this very proposal.

it doesn’t have any static information about what you can do or not. It can’t provide the “safe by default” guarantees are other core principles and set Swift apart from other languages.

Exactly.

Look how JSON parsing is done in Swift 3.2 and 4.0—we make static types. The 1’s and 0’s from a web response go through layers of proper error handling before they become proper types Swift.

So why does this proposal keeps mentioning JSON parsing as an excuse for making a dynamic ghetto within the language? Because some people are parsing untyped JSON? Well, the solution for that is, type your frickin’ JSON.

It doesn’t. There was one brief mention of a JSON example, to show how this proposal is useful for more than just Python interop. But most of the proposal talks about Python, not JSON.

Why compromise Swift’s static-ness and roll out the red carpet for the dynamic, unwashed masses?

Wouldn’t it be better to create an AI that quickly translates Pyton, Ruby, or JS into Swift? Augustus built huge public bathhouses for the unwashed Romans.

If you can create an AI that quickly and accurately translates Python, Ruby, and JS into Swift, go right ahead. To the extent that those projects are not taking advantage of the dynamic features of their language, that’s likely feasible. It’s going to be a lot more difficult for the kind of code that does take advantage of the dynamic nature of the language. For example, dynamic languages can inspect the function name to vary its behavior. You can write one “fetchAll____FromDatabase()” function, and then call it as fetchAllEntiresFromDatabase(), fetchAllUsersFromDatabase(), or fetchAllCommentsFromDatabase(), and automatically get the right behavior. I don’t think you’re going to be able to automatically create an AI that mimics that behavior on Swift; it fundamentally relies on the dynamic binding present in those languages.

It’s clear that you think dynamic languages are inferior to static languages. That’s fine, but the fact is that there are existing useful libraries in those languages which do not currently exist in Swift. Is it your argument that Swift should not be able to make use of such libraries because of their “unclean-ness”? I disagree.

When we consider interoperability with other languages, it should be from the perspective of mapping their features to Swift’s philosophies.

Amen.

This proposal takes the reverse approach, and makes Swift like Python, so I’m against it.

This is my concern as well.

Perhaps there is a way to make the proposal require that you declare Swift types and protocols that map to these other languages, with verbose error handling for when this glue breaks down (just like we have for typed JSON).

Then, we could also make scripts that auto-generate the necessary Swift typing boilerplate interface code from the Python/Ruby/JS source, kinda like we get for Obj.-C bridging headers. This could be a good step towards the full AI translation that I mentioned.

Of course, maybe I am missing something here, and my opinion is wrong; Chris will let us know if these concerns are valid.

However, I hope we don’t hear how we are just being paranoid, because I assure you, this is not paranoia. It’s just about maintaining the static-ness and purity of Swift to help save the world from bad code.

You seem to conflate “libraries written in a dynamic language” with “bad code”. That’s an unjustified assumption that ignores the large quantity of existing very useful libraries written in those languages. Or maybe you’re assuming that if this proposal goes through, people will choose to write dynamic code in their native Swift projects? It’s certainly possible that some people will do that, but the way this proposal is written makes it unlikely for dynamic behavior to spread throughout the project itself, because you still need an underlying source of dynamism. This proposal only makes it possible to call into an existing dynamic system; it does not introduce a new dynamic system directly into Swift.

Everything in life and engineering is a trade-off. Swift’s trademark is that it consistently sacrifices the convenience of dynamic-ness for the safety and performance of static-ness. This proposal does seem to do the opposite, and we’d be naive not to fear the effect this will have.

Ask yourself... what if Product knows about it? The last conversation a tech lead or engineering manager wants to have, I would imagine, is to explain on deaf ears why it would be bad to cut costs by farming out the iOS work to a cheap Python shop overseas, then just wrap it in sugary compiler hooks. Or why it would be bad for a Swift project to depend upon a stack of third-language dependencies that CocoaPods/Carthage can’t manage, XCode can’t analyze, stack traces can’t delve into (or can it?), lldb can’t step through, etc.

I do not believe this is unreasonable paranoia, I just believe it looks that way to people who work with an elite squad of engineers at a top company, who have perhaps not seen what goes on at most places, where budgets are tight and lots of people are not the biggest fans of Apple. It’s already with much grinding of teeth that non-Windows, non-Linux devs are necessary.

If someone’s wanting to farm out their iOS app to a 3rd party, they can already do that, whether or not Swift supports calling into dynamic libraries. They can even already do it from Swift with the existing C API for calling Python, in fact; this proposal just makes it a little easier. But if the entire app is really written in Python, a few C shims to get into Python-land aren’t a large burden. And they can already do it via ObjC as well. I think it’s unlikely that writing an entire iOS app in Python would be an efficient way to do things, but if that’s actually a better way to write the project, why are we stopping them? Is it the goal of the core team to encourage people to use Swift even when it’s not the best choice? I would suggest rather that the goal should be to improve Swift to make it the best choice. If that means being able to call into some existing business logic written in dyanmic_language_here, why is that problematic? If an engineering manager cannot make a convincing case that writing their app in Swift or ObjC is a better idea, then perhaps there’s not much of a case to be made. I find this scenario implausible, though, for a variety of reasons. For one, Python code can’t take advantage of the various system frameworks like Swift or ObjC can.

Given Swift’s emphasis on safety, this push to make danger easier really does surprise me, unless I just don’t understand this correctly. Have we re-evaluated the importance of having the language itself prohibit bad practices? Or is there some reason why this proposal does not actually make it easier to write code that looks like good, type-safe Swift code, but actually isn’t?

Finally, to really be honest, may I ask why web devs, many of whom never touched a line of Swift, deserve all this dynamic sugar, while on the other hand, we SWIFT devs don’t get any? How will people feel who were told that dynamic is so unsafe that Swift was designed to prevent it altogether? People who have been through all the migrations and member-wise trenches might chafe a bit. Is that unreasonable?

It seems odd that you talk about “deserving” dynamic sugar. The goal of this proposal is to make it easier to use more existing libraries, via a little bit of sugar goes a lot way toward that goal. All Swift users can take advantage of that. If we can make Swift a more viable language for web development (by enabling them to use existing dynamic libraries) without harming the language, why shouldn’t we? Of course, I think your argument is that this proposal does harm the language, but I fail to see how that’s the case. This proposal does precisely nothing to the language unless you’re calling methods on a type that conforms to DynamicMemberLookupProtocol. And when you’re doing that, the dynamism is well contained; it only exists those types. None of your other code is affected in any way.

Lastly, if you can “fake import” Javascript or Python to Swift via this proposal, is there anything that would prevent the “fake import” of masquerading Swift modules? In other words, could lazy devs who want hacky dynamic-ness abuse this? If so then maybe I’m in favor after all ;P (Just kidding.)

You can’t “fake import” Swift modules using this, because there’s no good way to look up Swift classes or methods by name, so you would have no way to make it work. This is about leveraging existing dynamic systems, not introducing new ones.

- Jon

PS—I am just trying to think critically here. Ultimately this proposal would not hurt me, but I am just worried about the big picture. I can see why it’s a tough call and I am sure that some pressures exist that might make even this compromise to be worth it. I just feel like we should not underestimate how much bad will come along with the good, nor should we assume, “Well that kind of bad would never happen where I work, and only idiots would abuse this.”

-BJ

···

On Dec 1, 2017, at 8:35 AM, Jon Gilbert via swift-evolution <swift-evolution@swift.org> wrote:

On Nov 30, 2017, at 08:10, Karl Wagner via swift-evolution <swift-evolution@swift.org> wrote:

[benjamin.g]

>
> Personally, I feel this system is designed to let you write Python,
using Swift as a wrapper language - except unlike with Objective-C,

Let me paraphrase the proposal—it basically says this system is designed
to make it easier to “fake import” existing Python/Ruby/JS libs to make it
easier for people who depend on them to adopt Swift. Especially in the
world of server-side Swift, I can see how this could help speed Swift’s
adoption.

As a sidenote, the weakest points of python on the server side is
compile-time type safety (none) and concurrency (almost none either). If we
want to convince anyone at the moment to switch to swift on the backend,
i'd say the urgent part isn't to make one or two function calls to a python
library more "swifty" (but without any guarantee that we didn't mistype a
function name). I'd say we should better focus on having a great agent
concurrency model, as well as even better generics and faster compile time.

From what i've observed around me, people seem to migrate python code to

golang. Not ruby or lisp.

···

On Fri, Dec 1, 2017 at 4:35 PM, Jon Gilbert via swift-evolution < swift-evolution@swift.org> wrote:

> On Nov 30, 2017, at 08:10, Karl Wagner via swift-evolution < > swift-evolution@swift.org> wrote:

However, I believe that you bring up an extremely valid concern. To
extrapolate from your points, this proposal seems to have a high potential
to erode what makes Swift special, and it provides a way to get around
Swift’s safe-guards, avoid writing Swift versions of libraries, and
ultimately, avoid hiring real Swift developers.

> it doesn’t have any static information about what you can do or not. It
can’t provide the “safe by default” guarantees are other core principles
and set Swift apart from other languages.

Exactly.

Look how JSON parsing is done in Swift 3.2 and 4.0—we make static types.
The 1’s and 0’s from a web response go through layers of proper error
handling before they become proper types Swift.

So why does this proposal keeps mentioning JSON parsing as an excuse for
making a dynamic ghetto within the language? Because some people are
parsing untyped JSON? Well, the solution for that is, type your frickin’
JSON.

Why compromise Swift’s static-ness and roll out the red carpet for the
dynamic, unwashed masses?

Wouldn’t it be better to create an AI that quickly translates Pyton, Ruby,
or JS into Swift? Augustus built huge public bathhouses for the unwashed
Romans.

> When we consider interoperability with other languages, it should be
from the perspective of mapping their features to Swift’s philosophies.

Amen.

> This proposal takes the reverse approach, and makes Swift like Python,
so I’m against it.

This is my concern as well.

Perhaps there is a way to make the proposal require that you declare Swift
types and protocols that map to these other languages, with verbose error
handling for when this glue breaks down (just like we have for typed JSON).

Then, we could also make scripts that auto-generate the necessary Swift
typing boilerplate interface code from the Python/Ruby/JS source, kinda
like we get for Obj.-C bridging headers. This could be a good step towards
the full AI translation that I mentioned.

Of course, maybe I am missing something here, and my opinion is wrong;
Chris will let us know if these concerns are valid.

However, I hope we don’t hear how we are just being paranoid, because I
assure you, this is not paranoia. It’s just about maintaining the
static-ness and purity of Swift to help save the world from bad code.

Everything in life and engineering is a trade-off. Swift’s trademark is
that it consistently sacrifices the convenience of dynamic-ness for the
safety and performance of static-ness. This proposal does seem to do the
opposite, and we’d be naive not to fear the effect this will have.

Ask yourself... what if Product knows about it? The last conversation a
tech lead or engineering manager wants to have, I would imagine, is to
explain on deaf ears why it would be bad to cut costs by farming out the
iOS work to a cheap Python shop overseas, then just wrap it in sugary
compiler hooks. Or why it would be bad for a Swift project to depend upon a
stack of third-language dependencies that CocoaPods/Carthage can’t manage,
XCode can’t analyze, stack traces can’t delve into (or can it?), lldb can’t
step through, etc.

I do not believe this is unreasonable paranoia, I just believe it looks
that way to people who work with an elite squad of engineers at a top
company, who have perhaps not seen what goes on at most places, where
budgets are tight and lots of people are not the biggest fans of Apple.
It’s already with much grinding of teeth that non-Windows, non-Linux devs
are necessary.

Given Swift’s emphasis on safety, this push to make danger easier really
does surprise me, unless I just don’t understand this correctly. Have we
re-evaluated the importance of having the language itself prohibit bad
practices? Or is there some reason why this proposal does not actually make
it easier to write code that looks like good, type-safe Swift code, but
actually isn’t?

Finally, to really be honest, may I ask why web devs, many of whom never
touched a line of Swift, deserve all this dynamic sugar, while on the other
hand, we SWIFT devs don’t get any? How will people feel who were told that
dynamic is so unsafe that Swift was designed to prevent it altogether?
People who have been through all the migrations and member-wise trenches
might chafe a bit. Is that unreasonable?

Lastly, if you can “fake import” Javascript or Python to Swift via this
proposal, is there anything that would prevent the “fake import” of
masquerading Swift modules? In other words, could lazy devs who want hacky
dynamic-ness abuse this? If so then maybe I’m in favor after all ;P (Just
kidding.)

- Jon

PS—I am just trying to think critically here. Ultimately this proposal
would not hurt me, but I am just worried about the big picture. I can see
why it’s a tough call and I am sure that some pressures exist that might
make even this compromise to be worth it. I just feel like we should not
underestimate how much bad will come along with the good, nor should we
assume, “Well that kind of bad would never happen where I work, and only
idiots would abuse this.”
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[anandabits (Matthew Johnson)]

I think better interoperability with Python (and other OO languages in widespread use) is a good goal, and I agree that the implementation of the feature described is straight-forward and not terribly invasive in the compiler.

However, I do not think this proposal is going in the right direction for Swift. I have objections on several different grounds.

+1 to everything Doug says here. He articulates the concerns I tried to voice in an earlier thread more clearly and more thoroughly.

Doug’s email was pretty confusing, so I’ll try to clear up some of the misconceptions here. I also updated the proposal with a new section at the end, please check it out.

Thank you for your latest reply and the latest reply to Doug. I can only speak for myself, but it helped me to better understand the motivation you have for the proposal. I really do appreciate your view and the motivation behind it.

This design introduces a significant hole in the type system which is contrary to the spirit of Swift.

There is no “hole” in the type system. The proposal is fully type safe.

Further, the addition is absolutely in the spirit of Swift, because it is directly analogous to an existing feature: AnyObject lookup. The difference between it and AnyObject lookup is that AnyObject lookup is:

a) specific to Objective-C interop
b) type unsafe
c) massively invasive on the rest of the compiler.

We’ve made many attempts to narrow the scope of AnyObject lookup, but it is difficult to do so given backwards compatibility constraints and the limited nature of Objective-C metadata.

I may have spoken incorrectly but I do still believe it is contrary to what I (and apparently many others) feel is the spirit of Swift.

As I understand it, AnyObject lookup was a necessity during the historical development of Swift. The attempts to narrow its scope have merit regardless of how far they can go. They indicate that perhaps it is a feature we would remove if that were possible. I would love to see that happen someday (although I know it is unlikely), or at least see it be made explicit at the call site. I suspect if this feature did not exist and it was proposed today it would be met with at least as much resistance as your proposals have.

It doesn’t just make dynamic language interop easy, it also changes the semantics of any type which conforms to DynamicMemberLookupProtocol. That is not something that is locally visible when looking at a piece of code. Worse, it does so in a way that trivial mistakes such as a typo can turn into runtime errors. Worst of all, as Doug points out it is possible to use retroactive conformance to change the semantics of vast quantities of widely used types in one fell swoop.

This is a feature, like many others, which can be misused. This has not been the metric we have used to judge what should go into Swift. I can elaborate if my response to Doug wasn’t clear.

The primary problem I have is that misuse is too easy and too subtle. It is at least as easy to misuse as other features in Swift which have been carefully designed to be obvious at usage sites.

I was happy to see that you are willing to consider Xiaodi’s suggestion to require conformance to be stated as part of the original type declaration. That would be a significant improvement to the proposal IMO. I strongly urge you to reconsider the decision of that dynamic members must be made available with no indication at usage sites. An indication of dynamic lookup at usage sites aligns very well (IMO) with the rest of Swift (AnyObject lookup aside) by calling attention to code that requires extra care to get right.

One additional concern that I don’t believe has been mentioned is that of evolution. When a library is imported from a dynamic language and the library makes breaking changes Swift code written using this feature will break without notice until a runtime error occurs. Is that acceptable? That may be what users of dynamic languages are accustomed to but can we do better in Swift? If we can make the process of upgrading dependencies less a less harrowing experience perhaps that could be a major selling point in moving them to Swift. But we won’t accomplish that with this design.

How would you this to work with *any* interoperability approach? Dynamic languages are in fact dynamic, and need the ability to do dynamic lookups somehow. Those dynamic lookups will all have the problem you observe.

I’ll mostly let Doug continue the discussion of the approach he is suggesting. It looks to me like that is an approach that might help in this area. For example, if a symbol is removed it may be possible to produce a compiler error in at least some cases. I don’t think Doug’s suggestion precludes full dynamism but would offer a much better experience in at least some cases. Perhaps it would even be enough to call into question whether full dynamism is really necessary or not (as Doug seems to think). We won’t know for sure without exploring this further.

I understand that you believe the direction he has advocated is not realistic. Perhaps, but I do appreciate that the conversation is happening before a final decision is made. I support further exploration of this at least as long as Doug believes it is a good idea to do so.

Interop with dynamic languages is a worthy goal, but we should do it in a way that has no potential impact on code that does not need the feature. A semantic variant of “don’t pay for what you don’t use” should apply here. If we add this feature everyone will pay for it every time they read unfamiliar Swift code. They will have to contend with the potential for conformances to this protocol, even on standard library types. That feels like a pretty high cost to me, especially for programmers who aren’t using the dynamic language interop features themselves.

You seem to fear that people will pervasively adopt this and use it for weird things. While that is possible, that would be simply one way to misuse Swift. What makes it more likely for someone to do this than to use unsafe features incorrectly, providing an apparently type safe API that actually isn’t?

It’s not the potential that I fear. As mentioned above, it is that the presence or absence of this potential in a given piece of code is too hard to detect, especially for a reader who is unfamiliar with the code in question. It is pretty clear that I am not the only Swift developer with significant concerns about this.

Further, a consistent theme has been the desire to avoid things like compiler flags that could fragment the community. I fear this feature has the potential to do that, especially if it is really successful at attracting people from the dynamic language community.

I really fail to see how this concern relates here. This has nothing to do with compiler flags. This has the potential to expand the community, not fragment it.

One of the stated aims of this proposal is to bring many new programmers into the Swift community. That is a wonderful goal!

Of course that may include programmers with a different set of technical values than are currently predominant in the Swift community. If we make it really easy to use Swift as a highly dynamic language we may soon see a large sub-community form that uses Swift in this way. Regardless of how you would feel about this I think it’s an important possible outcome to consider.

Your followup to Doug helped to clarify that many programmers in the data science and ML spaces you are hoping to bring into the Swift community are actually frustrated with having to use Python. That is really interesting and should influence the design. Perhaps these people would appreciate having dynamic member lookup be apparent at usage sites. Have you asked them?

You mentioned that many of them may prefer proper Swift APIs for the libraries they use and are hoping that the issues with dynamism will incentive them to create these in time. Perhaps requiring some kind of call-site annotation would increase this incentive without being significant enough to disincentive using these libraries in Swift.

Changing the proposal to restrict conformance to the initial type declaration and requiring usage site annotation of some kind for dynamic member lookup would address my most significant concerns about your proposal. I hope you will reconsider these points of design. I don’t think we need something heavy-handed, but something similar in weight to optional chaining or force unwrapping. If you modify this proposal to include such syntax perhaps it would eventually be appropriate to consider requiring the same syntax for AnyObject lookup as well (I would really like that).

In particular, if you continue to insist on standard dot syntax for member lookup I would appreciate hearing more about why that is a sticking point for you, especially in light of your recent comment about people looking to move away from Python.

I have conflicted feelings about this proposal regardless of whether the above mentioned changes are made or not. The value of interoperability with established libraries in other languages is significant in some domains. On the other hand, I question whether now is the time to introduce more dynamic features into Swift and whether this is the right first step. It is a feature that may have a significant impact on how some people choose to use the language. As others have pointed out, Swift is still lacking features that could solve many of the use cases for which dynamic features like this could be used (or abused).

Both perspectives have merit and I really don’t know which path would lead to a better outcome. In any case, I would like to see this proposal strengthened so that should it be introduced the benefits are maximized and the costs are minimized. With all due respect, I don’t think there has been enough consideration given to the costs that may be incurred should the proposal be accepted as written.

Matthew

···

On Dec 1, 2017, at 12:23 AM, Chris Lattner <clattner@nondot.org> wrote:

On Nov 30, 2017, at 6:15 AM, Matthew Johnson <matthew@anandabits.com> wrote:

-Chris

[Marc_Schlichte]

A somewhat extreme alternative could be the use of IDLs and "libs as services“:

A Python lib could aditionally expose its APIs (or a relevant subset of it) via an IDL. Generators would help with the necessary stub and skeleton code on both Python and Swift sides.

My impression is that only big libs like TensorFlow are relevant to be bridged anyhow so that the effort to create an IDLized API for that community is acceptable.

IDLs are an old idea but they seem to get reinvented every now and then. Google has a new one called FIDL for its Fuchsia project: (https://fuchsia.googlesource.com/fidl/\)

Cheers
Marc

···

Am 01.12.2017 um 17:07 schrieb Jon Gilbert via swift-evolution <swift-evolution@swift.org>:

On Dec 1, 2017, at 02:44, Xiaodi Wu via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

So the use case here is, how do we make Swift a viable candidate for doing those things which today drive users to Python? The answer here is _not_: build a better Python. Nor does it require, out of the gate, even being as good as Python. The solution is to provide a _reasonably_ ergonomic to _interoperate with libraries available in Python_, with the benefit that those parts that you can write in native Swift will make the overall result safer and faster, etc.

I think we would be better served by a transpiler that translates Python (etc.) into Swift at compile time.

Look what Google did with j2objc (https://github.com/google/j2objc\). It translates Java right into Objective C. You can even put your Java code right in XCode and it auto-translates at build time.

Clearly, this is no small feat, and j2objc is a technical marvel that took world-class engineers years to perfect.

My point is, “Dynamic Member Lookup” is not the only solution, and it’s not the ideal solution if indeed it compromises the static guarantees of Swift.

Therefore, we should consider what other approaches might entail. The main players in Swift have lots of money and technical resources they could pour into a set of revolutionary transpilers.

Surely we don’t want to allow Goole to be the only company to provide a library that translates other codebases directly to a primarily Apple language, do we?

That being said, I am still interested to hear Chris’s response to these concerns, and if they were already addressed on a previous message and I missed that, then please forgive me.

- Jon
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[ckeithray]

I *don't* want dynamic calls to stand out. I want seamless interoperability.

let arrayOfDogs = [ Python.Module(dog).Dog("Brianna"), Python.Module(dog).Dog("Kevin") ] // Swift array of Python objects
let b = arrayOfDogs.sorted() // use Swift 'sort' function on Python objects (invoking Python '<' and '==' operators)
let c = arrayOfDogs.forEach { $0.bark() } // Swift 'forEach' calling Python 'bark'
etc.

···

--
C. Keith Ray

* What Every Programmer Needs To… by C. Keith Ray [PDF/iPad/Kindle] <- buy my book?
* http://www.thirdfoundationsw.com/keith_ray_resume_2014_long.pdf
* http://agilesolutionspace.blogspot.com/

On Dec 3, 2017, at 6:57 AM, Tino Heth via swift-evolution <swift-evolution@swift.org> wrote:

What do you think about having a special method ‘dynamic' defined in this protocol, such that the compiler magic lookup behavior is only available when chained after that method?

I guess „dynamic“ isn’t a common property name, but imho such magic doesn’t feel right, even when there are no collisions in real-world code.

  let y = np^.arange?^(24)?^.reshape^?(2, 3, 4)!

That looks quite ugly… but imho that’s at least partially because of the choice of character
let y = np:arange?:reshape?:(2, 3, 4)!
doesn’t seem to be as bad as the use of carets.

But still, you would have to jump through some hoops to use both kinds of dispatch side by side (because you always have to think about how to call a method).
On the other hand, we also have &inoutParameter and try, which makes some things less convenient — and I’d say that dynamic dispatch is somehow comparable to inout-parameters (both shouldn’t be used if you can get around it).

So, why do people want differentiate?
I can see two major motivations for a different syntax:

a) you want to dynamic calls to stand out — which also could be done by tools:

let y = np.arange?(24)?.reshape?(2, 3, 4)! // let the editor use a different font/color for non-static stuff

b) you want to be sure that in a given situation, no dynamic dispatch is used — which could be solved by keeping the dot as general way to call something, and add an alternative that will only use static lookup

No matter what direction is taken here, imho this whole story has a really gigantic impact on the shape of Swift, and I have strong concerns if the two changes are reviewed in a normal way and added this year… there’s no experience with the proposed changes yet, and afaics, it has become quite hard to correct additions that turn out to be not as positive as expected.

- Tino

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Jon_Hull]

Hi Chris,

I am definitely in favor of providing dynamic features in Swift, and of being able to interoperate easily with dynamic languages. I really like the idea overall.

Great!

I was about to write up a different idea I had for partially mitigating some of the issues around being able to mistype method names, etc…, but then I remembered a usability principle that I first heard from members of the Lisa team (discovered the hard way): Things which behave the same should look the same, and things which behave differently need to look different.

That’s a good principle. However, a dynamic member lookup is just a member lookup. By that principle, it should look like a member lookup :-)

The behavior is different though (the invisible guard rails which we normally rely on to prevent mistakes are missing), so it should look similar without being identical.

If Swift were entirely dynamic, there wouldn’t be an issue here, because developers would have strategies for dealing with it. It is the rarity of use that causes the issue here. Swift developers have built up trust in the system to catch their typing mistakes, so if we make it so that the system can silently fail, there will be an increased error rate (above what you would have in a purely dynamic system).

Let me give a non-computer analogy. There are lots of arguments about whether the toilet seat should always be put down for the next user or just be left in it’s current position. Some households have them randomly up or down based on who used them last. This equates to the dynamic case because no trust can be built that the seat will be down, so everyone learns to check before sitting. Let’s say we add a foot pedal though, which raises the seat when you step on it, and lowers it when it is not pressed (like a garbage can lid). Now everyone can reason about the state of the system and can trust that it will be down unless they step on the pedal. As a result, over time, people will be able to just sit without checking. This is great, but if we re-introduce a little bit of dynamism (let’s say someone can lift the seat without using the pedal and it will stay up)… now the people who have built trust in the system, and just sit, will fall in.

The end result is that either some trust is lost in the system as a whole (people use the dynamic way of thinking when they don’t need to) or they continue to have errors when dealing with the rare dynamic cases.

I also like Paul’s suggestion of different syntax coloring for the dynamic calls. In the toilet seat analogy, it is like one of those little lights you can buy that turns green or red depending on whether the seat is up or down.

Further, I incorporated some of the conversation with Matthew into the proposal, showing how adding even a single sigil to dynamic member lookup to distinguish it is problematic:
https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#increasing-visibility-of-dynamic-member-lookups

Further, adding something like .dynamic would completely undermind the proposal. You can already write:

  x.get(“foo”).get(“bar”)

having to write:

  x.dynamic.foo.dynamic.bar

has no point.

Oh no, I meant a single ‘dynamic’ would enable dynamic lookup for the entire chain, so it would be:

  x.dynamic.foo.bar

This is different than requiring a sigil for each lookup.

What this means is that it is easy to wrap commonly used calls in a normal swift method:

  func addTrick(_ name:String) {
    self.dynamic.add_trick(name)
  }

This would require wrapping all calls for them to be usable.

I don’t think that chaining with ‘dynamic' is too large of a burden (when it works for the entire chain), so that would be the fallback for calls that haven’t been wrapped. Even if calls are never wrapped, I think this is workable.

My expectation though is that frequently used calls would end up getting wrapped eventually, and the wrappers would covert to/from Swift types. Basically, I want a system which encourages people to thoughtfully Swift-ify™ the interface for working with python code incrementally over time. Python coders may have no incentive to do this, but Swift users of that code sure do.

I guess what I am arguing is that we should prioritize/optimize the ease of creating that translation layer at the cost of making pure/direct calls into Python require an extra word of typing. Two birds, one stone.

Thanks,
Jon

···

On Dec 3, 2017, at 9:39 AM, Chris Lattner <clattner@nondot.org> wrote:
On Dec 3, 2017, at 5:45 AM, Jonathan Hull <jhull@gbis.com <mailto:jhull@gbis.com>> wrote:

[ahltorp (Magnus Ahltorp)]

This example shows what many on this list don't believe: that any Swift method or member access can fail. If the return value of this "get" method is an IUO, or not an Optional at all, and doesn't throw, then the expression would have to fail hard if "foo" didn't resolve to something meaningful.

The most common argument against this proposal is that someone could make an API using Dynamic Member Lookup that could fail even though it is not apparent to the caller. But, as we see in the example, this is just as possible today.

Another example where potential failure is not apparent to the caller:

struct Foo {
    var bar: Int
}

struct Example {
    var n: Int
    var foo: Foo { return Foo(bar: n + 1) }
}

let a = Example(n: Int.max)
a.foo.bar

Swift doesn't save the caller in this case, so together with Chris' example, I don't really understand the "not apparent to the caller" argument. Crashability is totally in the hands of the implementor of a method or member access, now and with the proposal.

/Magnus

···

4 Dec. 2017 02:40 Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

That’s a good principle. However, a dynamic member lookup is just a member lookup. By that principle, it should look like a member lookup :-)

Further, I incorporated some of the conversation with Matthew into the proposal, showing how adding even a single sigil to dynamic member lookup to distinguish it is problematic:
https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#increasing-visibility-of-dynamic-member-lookups

Further, adding something like .dynamic would completely undermind the proposal. You can already write:

  x.get(“foo”).get(“bar”)

having to write:

  x.dynamic.foo.dynamic.bar

has no point.

[Chris_Lattner]

The Objective-C runtime does not track the type of members or methods. The Clang importer uses the same heuristic approach that Objective-C/Clang/GCC do. If you have something of type “id” named “x”, when you write “x.foo” if there is a declaration of “foo” in some class in some imported header, it will assume it has the type of that declaration. While this is an important heuristic, this is not always the case, and does in fact cause bugs in users code.

In contrast, the DynamicMemberLookup proposal itself is fully type safe (three fully type safe implementation approaches are explained in it) and the Python interop layer I’m prototyping uses it.

-Chris

···

On Nov 30, 2017, at 10:26 PM, Slava Pestov <spestov@apple.com> wrote:

On Nov 30, 2017, at 10:23 PM, Chris Lattner via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

The difference between it and AnyObject lookup is that AnyObject lookup is:

b) type unsafe

Can you explain in what sense AnyObject lookup is type unsafe where your proposal is type safe? Both seem type safe in the sense that your program will not hit undefined behavior at runtime and instead trap if you access a non-existent member; both are type unsafe in the sense that you cannot reason statically about whether any specific member lookup will succeed. What is the distinguishing characteristic between the two that makes one safe and the other not?

[ckeithray]

Should we compare dynamic member lookup to Microsoft's DLR ?

Dynamic Language Runtime Overview - .NET Framework

Quote:
The dynamic language runtime (DLR) is a runtime environment that adds a set of services for dynamic languages to the common language runtime (CLR). The DLR makes it easier to develop dynamic languages to run on the .NET Framework and to add dynamic features to statically typed languages.
Dynamic languages can identify the type of an object at run time, whereas in statically typed languages such as C# and Visual Basic (when you use Option Explicit On) you must specify object types at design time. Examples of dynamic languages are Lisp, Smalltalk, JavaScript, PHP, Ruby, Python, ColdFusion, Lua, Cobra, and Groovy.
Most dynamic languages provide the following advantages for developers:
The ability to use a rapid feedback loop (REPL, or read-evaluate-print loop). This lets you enter several statements and immediately execute them to see the results.
Support for both top-down development and more traditional bottom-up development. For example, when you use a top-down approach, you can call functions that are not yet implemented and then add underlying implementations when you need them.
Easier refactoring and code modifications, because you do not have to change static type declarations throughout the code.
Dynamic languages make excellent scripting languages. Customers can easily extend applications created by using dynamic languages with new commands and functionality. Dynamic languages are also frequently used for creating Web sites and test harnesses, maintaining server farms, developing various utilities, and performing data transformations. +
The purpose of the DLR is to enable a system of dynamic languages to run on the .NET Framework and give them .NET interoperability. The DLR introduces dynamic objects to C# and Visual Basic in Visual Studio 2010 to support dynamic behavior in these languages and enable their interoperation with dynamic languages.

···

--
C. Keith Ray

* What Every Programmer Needs To… by C. Keith Ray [PDF/iPad/Kindle] <- buy my book?
* http://www.thirdfoundationsw.com/keith_ray_resume_2014_long.pdf
* http://agilesolutionspace.blogspot.com/

On Dec 1, 2017, at 8:07 AM, Benjamin G via swift-evolution <swift-evolution@swift.org> wrote:

On Fri, Dec 1, 2017 at 4:35 PM, Jon Gilbert via swift-evolution <swift-evolution@swift.org> wrote:
> On Nov 30, 2017, at 08:10, Karl Wagner via swift-evolution <swift-evolution@swift.org> wrote:
>
> Personally, I feel this system is designed to let you write Python, using Swift as a wrapper language - except unlike with Objective-C,

Let me paraphrase the proposal—it basically says this system is designed to make it easier to “fake import” existing Python/Ruby/JS libs to make it easier for people who depend on them to adopt Swift. Especially in the world of server-side Swift, I can see how this could help speed Swift’s adoption.

As a sidenote, the weakest points of python on the server side is compile-time type safety (none) and concurrency (almost none either). If we want to convince anyone at the moment to switch to swift on the backend, i'd say the urgent part isn't to make one or two function calls to a python library more "swifty" (but without any guarantee that we didn't mistype a function name). I'd say we should better focus on having a great agent concurrency model, as well as even better generics and faster compile time.
From what i've observed around me, people seem to migrate python code to golang. Not ruby or lisp.

However, I believe that you bring up an extremely valid concern. To extrapolate from your points, this proposal seems to have a high potential to erode what makes Swift special, and it provides a way to get around Swift’s safe-guards, avoid writing Swift versions of libraries, and ultimately, avoid hiring real Swift developers.

> it doesn’t have any static information about what you can do or not. It can’t provide the “safe by default” guarantees are other core principles and set Swift apart from other languages.

Exactly.

Look how JSON parsing is done in Swift 3.2 and 4.0—we make static types. The 1’s and 0’s from a web response go through layers of proper error handling before they become proper types Swift.

So why does this proposal keeps mentioning JSON parsing as an excuse for making a dynamic ghetto within the language? Because some people are parsing untyped JSON? Well, the solution for that is, type your frickin’ JSON.

Why compromise Swift’s static-ness and roll out the red carpet for the dynamic, unwashed masses?

Wouldn’t it be better to create an AI that quickly translates Pyton, Ruby, or JS into Swift? Augustus built huge public bathhouses for the unwashed Romans.

> When we consider interoperability with other languages, it should be from the perspective of mapping their features to Swift’s philosophies.

Amen.

> This proposal takes the reverse approach, and makes Swift like Python, so I’m against it.

This is my concern as well.

Perhaps there is a way to make the proposal require that you declare Swift types and protocols that map to these other languages, with verbose error handling for when this glue breaks down (just like we have for typed JSON).

Then, we could also make scripts that auto-generate the necessary Swift typing boilerplate interface code from the Python/Ruby/JS source, kinda like we get for Obj.-C bridging headers. This could be a good step towards the full AI translation that I mentioned.

Of course, maybe I am missing something here, and my opinion is wrong; Chris will let us know if these concerns are valid.

However, I hope we don’t hear how we are just being paranoid, because I assure you, this is not paranoia. It’s just about maintaining the static-ness and purity of Swift to help save the world from bad code.

Everything in life and engineering is a trade-off. Swift’s trademark is that it consistently sacrifices the convenience of dynamic-ness for the safety and performance of static-ness. This proposal does seem to do the opposite, and we’d be naive not to fear the effect this will have.

Ask yourself... what if Product knows about it? The last conversation a tech lead or engineering manager wants to have, I would imagine, is to explain on deaf ears why it would be bad to cut costs by farming out the iOS work to a cheap Python shop overseas, then just wrap it in sugary compiler hooks. Or why it would be bad for a Swift project to depend upon a stack of third-language dependencies that CocoaPods/Carthage can’t manage, XCode can’t analyze, stack traces can’t delve into (or can it?), lldb can’t step through, etc.

I do not believe this is unreasonable paranoia, I just believe it looks that way to people who work with an elite squad of engineers at a top company, who have perhaps not seen what goes on at most places, where budgets are tight and lots of people are not the biggest fans of Apple. It’s already with much grinding of teeth that non-Windows, non-Linux devs are necessary.

Given Swift’s emphasis on safety, this push to make danger easier really does surprise me, unless I just don’t understand this correctly. Have we re-evaluated the importance of having the language itself prohibit bad practices? Or is there some reason why this proposal does not actually make it easier to write code that looks like good, type-safe Swift code, but actually isn’t?

Finally, to really be honest, may I ask why web devs, many of whom never touched a line of Swift, deserve all this dynamic sugar, while on the other hand, we SWIFT devs don’t get any? How will people feel who were told that dynamic is so unsafe that Swift was designed to prevent it altogether? People who have been through all the migrations and member-wise trenches might chafe a bit. Is that unreasonable?

Lastly, if you can “fake import” Javascript or Python to Swift via this proposal, is there anything that would prevent the “fake import” of masquerading Swift modules? In other words, could lazy devs who want hacky dynamic-ness abuse this? If so then maybe I’m in favor after all ;P (Just kidding.)

- Jon

PS—I am just trying to think critically here. Ultimately this proposal would not hurt me, but I am just worried about the big picture. I can see why it’s a tough call and I am sure that some pressures exist that might make even this compromise to be worth it. I just feel like we should not underestimate how much bad will come along with the good, nor should we assume, “Well that kind of bad would never happen where I work, and only idiots would abuse this.”
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Douglas_Gregor]

Philosophy

More problematically for your argument: your preferred approach requires the introduction of (something like) DynamicMemberLookupProtocol or something like AnyObject-For-Python, so your proposal would be additive on top of solving the core problem I’m trying to solve. It isn’t an alternative approach at all.

I wouldn’t say that’s my preferred approach. My preferred approach involves taking the method/property/etc. declarations that already exist in Python and mapping them into corresponding Swift declarations so we have something to find with name lookup. One could put all of these declarations on some PyVal struct or PythonObject and there would be no need for AnyObject-for-Python or DynamicMemberLookupProtocol.

You’re suggesting that the transitive closure of all Python methods and properties be preprocessed into a single gigantic Swift PyVal type? I guess something like that could be done.

That’s effectively what Swift is already doing with AnyObject.

I would be concerned because there are many N^2 or worse algorithms in the Swift compiler would probably explode.

As noted above, we already do this for AnyObject with every @objc entity everywhere. The # of overloads for a given name is usually not so high that it’s a problem for the compiler.

It also doesn’t provide the great tooling experience that you’re seeking, given that code completion would show everything in the Python universe, which is not helpful.

It’s better for code completion to provide too much than to provide nothing at all. If code completion provides too much, typing a small number of characters will reduce the completion set down to something manageable quite fast.

Further, it doesn’t provide a *better* experience than what I’m suggesting, it seems strictly worse.

For the same working Swift code

  dog.add_trick(rollOver)

that calls into Python, the solution I’m proposing:

* Gave you the add_trick(<#trick#>) code completion with the number of parameters and their names
* Supports goto definition to jump to the wrapper method on PyVal
* Supports “quick help” to show the declaration of that wrapper method on PyVal, showing the documentation (docstring) and in which Python class it was declared.
* Supports indexing functionality so we can find the likely uses of “add_trick"

DynamicMemberLookup doesn’t gave any of those, because “add_trick” is just a string that looks like an identifier. Only the Python runtime can resolve.

A preprocessing step prevents users from playfully importing random Python modules into the Swift repl and playgrounds.

*At worst*, you invoke the tool from the command line to build the Swift module that corresponds to a given Python module.

  py2swift <pythonmodulename>

We could absolutely introduce tooling hooks to make the compiler initiate that step.

It also seems worse for implementors (who will get a stream of new bugs about compiler scalability).

To my knowledge, AnyObject lookup has not been a serious source of performance problems for the compiler. The global lookup table it requires was annoying to implement, but it’s just a symbol table.

Whenever we discuss adding more dynamic features to Swift, there’s a strong focus on maintaining that strong static type system.

Which this does, by providing full type safety - unlike AnyObject lookup.

You get dynamic safety because it goes into the Python interpreter; fair enough. You get no help from your tools to form a correct invocation of any method provided by Python.

Sure, that’s status quo for Python APIs.

That’s not the status quo for Python IDEs. They will give you the basic shape of method calls.

Tooling
The Python interoperability enabled by this proposal *does* look fairly nice when you look at a small, correctly-written example. However, absolutely none of the tooling assistance we rely on when writing such code will work for Python interoperability. Examples:

* As noted earlier, if you typo’d a name of a Python entity or passed the wrong number of arguments to it, the compiler will not tell you: it’ll be a runtime failure in the Python interpreter. I guess that’s what you’d get if you were writing the code in Python, but Swift is supposed to be *better* than Python if we’re to convince a community to use Swift instead.
* Code completion won’t work, because Swift has no visibility into declarations written in Python
* Indexing/jump-to-definition/lookup documentation/generated interface won’t ever work. None of the IDE features supported by SourceKit will work, which will be a significant regression for users coming from a Python-capable IDE.

Yes, welcome to the realities of modern Python development!

Python plugins for IDEs (e.g., for Atom) provide code completion, goto definition, and other related features. None of the Swift tooling will work if Swift’s interoperability with Python is entirely based on DynamicMemberLookupProtocol.

I don’t understand your rationale here. I think you agree that we need to support the fully dynamic case (which is what I’m proposing). That said, this step does not preclude introducing importer magic (either through compiler hackery or a theoretical "type providers” style of feature). Doing so would provide the functionality you’re describing.

I don’t agree that we need language support for the fully-dynamic case. There has to be a way to handle the fully-dynamic case, but it can be string-based APIs vended by an Python interoperability library.

I believe that the majority of property and method accesses in Python programs will be resolved to a definition in the current model or an imported module, i.e., a definition that is visible to the compiler at compile-time. There are exceptional cases involving programmatically creating properties from JSON, a database, etc., but while prominent I suspect they account for a relatively small fraction of use sites. Do you agree with this statement?

How Should Python Interoperability Work?
Going back to the central motivator for this proposal, I think that providing something akin to the Clang Importer provides the best interoperability experience: it would turn Python declarations into *real* Swift declarations, so that we get the various tooling benefits of having a strong statically-typed language.

This could be an theoretically interesting refinement to this proposal but I’m personally very skeptical that this is every going to happen. I’ve put the rationale into the alternatives section of the proposal. I don’t explain it in the proposal in this way directly, but I believe it is far more likely for a Pythonista transplant into Swift to rewrite their code in Swift than it is to use Python type annotations.

I assume that this belief is based on type annotations lack of traction in the Python community thus far?

There are many parts to this, which have to do with the ObjC<->Swift situation being very different than the Python<->Swift situation:

1) The annotations don’t have significant traction in the Python community.
2) The Python annotations are not as powerful as ObjC generics are, and thus lack important expressive capability.
3) Many Python APIs are wrappers for C APIs. “Swiftizing” a Python API in this case means writing a new Swift wrapper for the API, not adding type annotations.
4) The Python community doesn’t care about Swift, and are not motivated to do things to make Swift succeed.
5) There is no “clang equivalent” for Python (that I’m aware of) which close enough to the way Clang does for us to directly use. The owners of the existing Python compiler/interpreter implementations are not going to be strongly motivated to change their stuff for us.

Finally, just MHO, but I don’t expect a lot of “mix and match" Python/Swift apps to exist (where the developer owns both the Python and the Swift code), which is one case where type annotations are super awesome in ObjC. IMO, the most important use-case is a Swift program that uses some Python APIs.

Let’s consider Python type annotations to be useless for our purposes. None of my argument hinges on it.

Sure, the argument types will all by PyObject or PyVal,

That’s the root of the problem. Python has the “fully dynamic” equivalent of “id” in Objective-C, so we need to represent that somehow. Even if we followed the implementation approach of the Clang importer, we would need some way to represent this dynamic case. That type needs features like DynamicMemberLookup or AnyObject. In my opinion, the DynamicMemberLookup approach is better in every way than AnyObject is.

The AnyObject approach has the advantage of knowing the set of declared, reachable APIs:

* Code completion shows all of the APIs that are possible to use via dynamic dispatch, with their signatures so can fill in the right # of arguments, see the names of the parameters, see documentation, etc.
* Indexing/refactoring/goto definition all point you to the declarations that could be the targets of dynamic dispatch

The DynamicMemberLookup approach is better for cases where you don’t have a declaration of the member you want to access. I suspect that’s not the common case.

Your points are valid, but the advantages for Objective-C don’t obviously translate to Swift. Note that ObjC (due to its heritage) has very long method names that are perhaps arguably designed to not conflict with each other often. Python doesn’t have this heritage, and it has much shorter names, which means that we’ll get a lot more conflicts and a lot less “safety" out of this.

AnyObject lookup resolves conflicts by collapsing similar overloads, and that’s okay: we don’t really care which class contained the declaration we found; we just care about the shape of the declaration. The lack of this magic will be a problem for the wrapper approach I’m describing, but that’s fixable.

AnyObject lookup also depends on a strange set of scoping heuristics that was designed to be similar to Clang’s “header import” scope. It isn’t clear that this approach will work in Python, given that it doesn’t have an analogue of umbrella headers that import things that cross frameworks.

Not sure what you think the scoping heuristics are. AnyObject lookup is fairly simple to define: it finds all @objc members of every class and protocol in your current module and any modules you imported.

Which approach do you think is the best way to handle the untyped “actually dynamic” case?

AnyObject already exists in the language, and it fits the untyped “actually dynamic” case well. It does require having a declaration for the thing you want to reference, which I consider to be important: we can code-complete those declarations, goto-definition to see those declarations, index/refactor/look up documentation based on those declarations.

I’d be more inclined to push for the ImplicitlyUnwrappedOptional -> Optional change if we did something to make AnyObject more prominent in Swift.

I didn’t realize that you were thinking we would literally use AnyObject itself. I haven’t thought fully through it, but I think this will provide several problems:

1) You’re mushing all of the ObjC and Python world’s together, making the ObjC interop worse just because you’re doing some Python stuff too.
2) You’re introducing ambiguity: does “ao = [1,2,3]” create an NSArray or a Python array? How do string literals work? (The answer is obvious, Python loses). Maybe there is some really complicated bridging solution to these problems, but that causes its own massive complexity spiral.
3) You can’t realistically overload the Python operator set on AnyObject, which means you get a worse python experience.
4) AnyObject magic is currently limited to Apple platforms. This would bring its problems to other platforms like Linux.

There are probably other issues, but I haven’t thought through it.

You are correct that literally using AnyObject has these issues. There appears to be a lot of confusion about what the AnyObject model *is*, so let’s sort through that. I think it is reasonable to take the AnyObject model and replicate it for PythonObject.

In truth, you don’t even need the compiler to be involved. The dynamic “subscript” operation could be implemented in a Swift library, and one could write a Python program to process a Python module and emit Swift wrappers that call into that subscript operation. You’ll get all of the tooling benefits with no compiler changes, and can tweak the wrapper generation however much you want, using typing annotations or other Python-specific information to create better wrappers over time.

I’d love for you to sketch out how any of this works with an acceptable user experience, because I don't see anything concrete here.

We don’t need the basic dynamic case in the language to do this experiment. Take the PyVal struct from the proposal. Now, write a Python script that loads some module Foo and uses Python’s inspect <https://docs.python.org/3/library/inspect.html&gt; module to go find the classes, methods, etc., and pretty-print Swift code that uses PyVal. So this:

  def add_trick(self, trick):

turns into

  extension PyVal {
    func add_trick(_ trick: PyVal) -> PyVal {
      /* do the magic to call into Python */
    }
  }

Using the inspect module, you can extract parameter names, default arguments, docstrings, and more to reflect the existing Python API as Swift API, packed into a bridging module.

Note that we have a “flat” namespace of all Python methods on PyVal, which is basically what you get with AnyObject today. Swift tooling will provide code completion for member accesses into PyVal. Goto definition will jump to the pretty-printed declarations, which could have the docstrings formatted in comments and would show up in QuickHelp. The types are weak (everything is PyVal), but that’s what we expect from importing a dynamically-typed language.

As I mention above, I expect this to expose significant scalability problems in the Swift compiler and it also defeats REPL/Playgrounds. Being able to use the Swift REPL is really important for Python programmers.

I’ve addressed both of these issues above. The approach I am proposing requires no language or compiler support, works with existing Swift tooling, fits with an existing notion in the language (AnyObject), and can be implemented via a Python script.

  - Doug

···

On Dec 1, 2017, at 1:30 AM, Chris Lattner <clattner@nondot.org> wrote:

On Dec 1, 2017, at 12:26 AM, Douglas Gregor <dgregor@apple.com <mailto:dgregor@apple.com>> wrote:

From my prior message, having the declarations of all Python methods and properties available on PyVal makes Swift tooling work:

[benjamin.g]

I think the fear most of us , poor developers working with developers of
various skills, is the potential for abuse. I've heard many times that it
isn't a guiding principle for swift, so i'll just reformulate the concern
in a more acceptable way :
Could your proposal explain how the swift language will still keep
encouraging developers to use static / compile-time type checking *when
it's possible* ? (because i think we can all argue that a compile-time
checks is better than a runtime one). Obviously, python interop isn't my
concern here. I'm talking about the impact of your proposal on pure swift
code.

As example, calling some feature "unsafe", or "force", makes every
developer pause and wonder if there isn't a better alternative. I'd like to
see the same kind of things for dynamic calls.

···

On Fri, Dec 1, 2017 at 10:30 AM, Chris Lattner via swift-evolution < swift-evolution@swift.org> wrote:

On Dec 1, 2017, at 12:26 AM, Douglas Gregor <dgregor@apple.com> wrote:

*Philosophy*
Swift is, unabashedly, a strong statically-typed language.

That’s factually incorrect.

You’re going to have to explain that statement without reference to
AnyObject (we’ll discuss that case below).

Perhaps it depends on what you mean by “strong”: I interpreted that as
meaning that it provides type safety, with a level of strength akin to what
Java provides: a level that could support mobile code deployment.

Swift certainly does not provide that strong of a static type system,
because it gives people explicit ways to opt out of that.
UnsafeMutableRawPointer, unsafe bitcast, and many other facilities support
this. It also allows calling into non-type safe code, so it isn’t very
strong that way. There are also race conditions and other holes in the
type system.

All that said, I think it is correct that a subset of Swift exists that
does provide strong type safety, but particularly when bridging to C/ObjC
is involved, that quickly goes away.

More problematically for your argument: your preferred approach requires
the introduction of (something like) DynamicMemberLookupProtocol or
something like AnyObject-For-Python, so your proposal would be additive on
top of solving the core problem I’m trying to solve. It isn’t an
alternative approach at all.

I wouldn’t say that’s my preferred approach. My preferred approach
involves taking the method/property/etc. declarations that already exist in
Python and mapping them into corresponding Swift declarations so we have
something to find with name lookup. One could put all of these declarations
on some PyVal struct or PythonObject and there would be no need for
AnyObject-for-Python or DynamicMemberLookupProtocol.

You’re suggesting that the transitive closure of all Python methods and
properties be preprocessed into a single gigantic Swift PyVal type? I
guess something like that could be done.

I would be concerned because there are many N^2 or worse algorithms in the
Swift compiler would probably explode. It also doesn’t provide the great
tooling experience that you’re seeking, given that code completion would
show everything in the Python universe, which is not helpful.

Further, it doesn’t provide a *better* experience than what I’m
suggesting, it seems strictly worse. A preprocessing step prevents users
from playfully importing random Python modules into the Swift repl and
playgrounds. It also seems worse for implementors (who will get a stream
of new bugs about compiler scalability).

Whenever we discuss adding more dynamic features to Swift, there’s a
strong focus on maintaining that strong static type system.

Which this does, by providing full type safety - unlike AnyObject lookup.

You get dynamic safety because it goes into the Python interpreter; fair
enough. You get no help from your tools to form a correct invocation of any
method provided by Python.

Sure, that’s status quo for Python APIs.

IMO, this proposal is a significant departure from the fundamental
character of Swift, because it allows access to possibly-nonexistent
members (as well as calls with incorrect arguments, in the related
proposal) without any indication that the operation might fail.

The only way your claim is correct is if someone implements the protocol
wrong. What you describe is true of AnyObject lookup though, so I
understand how you could be confused by that.

AnyObject lookup still requires you to find an actual declaration with a
type signature. Yes, there are still failure cases. The dynamic type might
be totally unrelated to the class in which you found the declaration you’re
supposedly calling, which is a typical “unrecognized selector” failure and
will always be an issue with dynamic typing. The actual type safety hole
you’re presumably referring to is that the selector could be overloaded
with a different type signature, and we don’t proactively check that the
signature we type-checked against matches the signature found at runtime.
It’s doable with the Objective-C method encodings, but has never been
considered worthwhile.

To be clear, I’m not suggesting a change to AnyObject lookup. I don’t
think that it is worth changing at this point in time. My point was to
observe that the proposed DynamicMemberLookupProtocol proposal does not
suffer from these problems. It really is type / memory safe, assuming a
sane implementation.

It’s easy to fall through these cracks for any type that supports
DynamicMemberLookupProtocol—a single-character typo when using a
DynamicMemberLookupProtocol-capable type means you’ve fallen out of the
safety that Swift provides.

Since you seem to be latching on to this, I’ll say it again: the proposal
is fully type safe :-)

That said, the “simple typo” problem is fundamental to the problem of
working with a dynamically typed language: unless you can eradicate all
“fundamental dynamism” from the language, you cannot prevent this.

That said, since this is a problem inherent with these languages, it is
something people are already very familiar with, and something that
everyone using those APIs has had to deal with. This is also not our
problem to solve, and people in the Python community have been discussing
and working on it over a large part of its 25 year history. I find your
belief that we could solve this for Python better than the Python community
itself has both idealistic and a bit naive.

I’m not pretending we can fully solve the problem. I’m pointing out that
depending entirely on DynamicMemberLookupProtocol throws away information
about method declarations that is present in Python and used by Python
tooling to good effect.

I’m not opposed to going further over time, but I’d like to get started at
some point :-). I’m not in a super urgent hurry to get this in in the next
week or month or anything like that, but I also don’t want to wait until
Swift 10.

*Tooling*
The Python interoperability enabled by this proposal *does* look fairly
nice when you look at a small, correctly-written example. However,
absolutely none of the tooling assistance we rely on when writing such code
will work for Python interoperability. Examples:

* As noted earlier, if you typo’d a name of a Python entity or passed the
wrong number of arguments to it, the compiler will not tell you: it’ll be a
runtime failure in the Python interpreter. I guess that’s what you’d get if
you were writing the code in Python, but Swift is supposed to be *better*
than Python if we’re to convince a community to use Swift instead.
* Code completion won’t work, because Swift has no visibility into
declarations written in Python
* Indexing/jump-to-definition/lookup documentation/generated interface
won’t ever work. None of the IDE features supported by SourceKit will work,
which will be a significant regression for users coming from a
Python-capable IDE.

Yes, welcome to the realities of modern Python development!

Python plugins for IDEs (e.g., for Atom) provide code completion, goto
definition, and other related features. None of the Swift tooling will work
if Swift’s interoperability with Python is entirely based on
DynamicMemberLookupProtocol.

I don’t understand your rationale here. I think you agree that we need to
support the fully dynamic case (which is what I’m proposing). That said,
this step does not preclude introducing importer magic (either through
compiler hackery or a theoretical "type providers” style of feature).
Doing so would provide the functionality you’re describing.

Statically-typed languages should be a boon for tooling, but if a user
coming from Python to Swift *because* it’s supposed to be a better
development experience actually sees a significantly worse development
experience, we’re not going to win them over. It’ll just feel inconsistent.

By your argument we should ban AnyObject lookup as well, given its
inconsistency with the rest of the language.

By my argument, we should at least replace the ImplicitlyUnwrappedOptional
result with a true Optional (so one has to acknowledge that the method
shouldn’t be there). We’ve seriously considered it, but it’s a
source-breaking change, and it hasn’t seemed worth the engineering effort
to pursue it.

Ok, that would be a nice step, but doesn’t fix the type safety hole.

In any case, my proposal allows the use of strong optional results as
well, so the fate of AnyObject isn’t really bound up with it.

I don’t think that removing AnyObject dispatch entirely is possible at
this point in Swift’s lifetime. While AnyObject has become much less
prominent than it was in the Swift 1.0 days ([AnyObject] and [NSObject :
AnyObject], oh my!), there is still a significant amount of code using it
in the wild.

Completely agreed. The major advantage I see of changing it now is if
there is some small mostly-user-invisible-change that allows a dramatic
simplification to the compiler implementation.

*Dynamic Typing Features*
It’s possible that the right evolutionary path for Swift involves some
notion of dynamic typing, which would have a lot of the properties sought
by this proposal (and the DynamicCallableProtocol one). If that is true—and
I’m not at all convinced that it is—we shouldn’t accidentally fall into a
suboptimal design by taking small, easy, steps.

Given that you haven’t followed the discussion on the many threads we’ve
had on this, and haven’t proposed a workable approach to this problem, I’m
not sure upon what basis your fears and uncertainty and doubt are founded.

A few meta-comments here. First of all, following all previous threads on
a discussion is not realistic.

I understand that, but you’re also accusing the proposal of being a
suboptimal design made by looking at a series of small easy steps, instead
of the right design for the long term. I’m pointing out that it is hard to
see the rationale for that sort of claim.

This is part of the reason why we have different stages in a proposal’s
lifetime, and is the responsibility of the proposal’s authors to capture
alternatives and rationale in the proposal to make it self-contained.

Agreed. As I mentioned in my previous email, I definitely screwed that up
by not capturing this discussion in the proposal. Thank you again for
pulling this perspective to the front of the discussion so I could fix that
oversight.

This proposal went through rapid iteration in the pitch phase and has now
been elevated to a pull request to ask for formal review—you should expect
more people to come on board having not read those threads. I appreciate
that you have now captured more alternatives and rationale in the proposal.

Agreed. This is why I’ve been proactive about starting threads and trying
to keep visibility on the proposal each time there is a significant
change. I really do value the discussion and feedback (both on the
proposed direction but also the writing itself).

Second, it is absolutely reasonable to disagree with the technical
direction of a proposal without providing a complete solution to the
problem that the proposal is attempting to solve. Some problems aren’t
worth solving at all, or fully.

Ok, but at some point, if there is no alternative proposed, then a strong
opposition has the appearance of saying “we shouldn’t solve this problem”.
It was my understanding that thought that this was a worthwhile problem to
solve.

*How Should Python Interoperability Work?*
Going back to the central motivator for this proposal, I think that
providing something akin to the Clang Importer provides the best
interoperability experience: it would turn Python declarations into *real*
Swift declarations, so that we get the various tooling benefits of having a
strong statically-typed language.

This could be an theoretically interesting refinement to this proposal but
I’m personally very skeptical that this is every going to happen. I’ve put
the rationale into the alternatives section of the proposal. I don’t
explain it in the proposal in this way directly, but I believe it is far
more likely for a Pythonista transplant into Swift to rewrite their code in
Swift than it is to use Python type annotations.

I assume that this belief is based on type annotations lack of traction in
the Python community thus far?

There are many parts to this, which have to do with the ObjC<->Swift
situation being very different than the Python<->Swift situation:

1) The annotations don’t have significant traction in the Python
community.
2) The Python annotations are not as powerful as ObjC generics are, and
thus lack important expressive capability.
3) Many Python APIs are wrappers for C APIs. “Swiftizing” a Python API in
this case means writing a new Swift wrapper for the API, not adding type
annotations.
4) The Python community doesn’t care about Swift, and are not motivated to
do things to make Swift succeed.
5) There is no “clang equivalent” for Python (that I’m aware of) which
close enough to the way Clang does for us to directly use. The owners of
the existing Python compiler/interpreter implementations are not going to
be strongly motivated to change their stuff for us.

Finally, just MHO, but I don’t expect a lot of “mix and match"
Python/Swift apps to exist (where the developer owns both the Python and
the Swift code), which is one case where type annotations are super awesome
in ObjC. IMO, the most important use-case is a Swift program that uses
some Python APIs.

Sure, the argument types will all by PyObject or PyVal,

That’s the root of the problem. Python has the “fully dynamic” equivalent
of “id” in Objective-C, so we need to represent that somehow. Even if we
followed the implementation approach of the Clang importer, we would need
some way to represent this dynamic case. That type needs features like
DynamicMemberLookup or AnyObject. In my opinion, the DynamicMemberLookup
approach is better in every way than AnyObject is.

The AnyObject approach has the advantage of knowing the set of declared,
reachable APIs:

* Code completion shows all of the APIs that are possible to use via
dynamic dispatch, with their signatures so can fill in the right # of
arguments, see the names of the parameters, see documentation, etc.
* Indexing/refactoring/goto definition all point you to the declarations
that could be the targets of dynamic dispatch

The DynamicMemberLookup approach is better for cases where you don’t have
a declaration of the member you want to access. I suspect that’s not the
common case.

Your points are valid, but the advantages for Objective-C don’t obviously
translate to Swift. Note that ObjC (due to its heritage) has very long
method names that are perhaps arguably designed to not conflict with each
other often. Python doesn’t have this heritage, and it has much shorter
names, which means that we’ll get a lot more conflicts and a lot less
“safety" out of this.

AnyObject lookup also depends on a strange set of scoping heuristics that
was designed to be similar to Clang’s “header import” scope. It isn’t
clear that this approach will work in Python, given that it doesn’t have an
analogue of umbrella headers that import things that cross frameworks.

Which approach do you think is the best way to handle the untyped
“actually dynamic” case?

AnyObject already exists in the language, and it fits the untyped
“actually dynamic” case well. It does require having a declaration for the
thing you want to reference, which I consider to be important: we can
code-complete those declarations, goto-definition to see those
declarations, index/refactor/look up documentation based on those
declarations.

I’d be more inclined to push for the ImplicitlyUnwrappedOptional ->
Optional change if we did something to make AnyObject more prominent in
Swift.

I didn’t realize that you were thinking we would literally use AnyObject
itself. I haven’t thought fully through it, but I think this will provide
several problems:

1) You’re mushing all of the ObjC and Python world’s together, making the
ObjC interop worse just because you’re doing some Python stuff too.
2) You’re introducing ambiguity: does “ao = [1,2,3]” create an NSArray or
a Python array? How do string literals work? (The answer is obvious,
Python loses). Maybe there is some really complicated bridging solution to
these problems, but that causes its own massive complexity spiral.
3) You can’t realistically overload the Python operator set on AnyObject,
which means you get a worse python experience.
4) AnyObject magic is currently limited to Apple platforms. This would
bring its problems to other platforms like Linux.

There are probably other issues, but I haven’t thought through it.

but the names are there for code completion (and indexing, etc.) to work,
and one could certainly imagine growing the importer to support Python’s typing
annotations <https://docs.python.org/3/library/typing.html&gt;\.

You’re basing this on the flawed assumption that local variables will
pervasively have types, which I can’t imagine being the case. Even on
"typable” API, I wouldn’t expect people to commonly get code completion
results for reasons now explained in the proposal.

Remember that one *does* get code completion results for member access
into an AnyObject… lots of them… but the list filters down pretty fast when
you type a few characters, and then you get a member access that’ll fill in
stubs for (say) the arguments to the method you were trying to call. But
you can’t get those code completion results without having declarations to
complete to.

Fair point, it’s unclear to me how useful this would be with python’s
style of naming, but it could work.

In truth, you don’t even need the compiler to be involved. The dynamic
“subscript” operation could be implemented in a Swift library, and one
could write a Python program to process a Python module and emit Swift
wrappers that call into that subscript operation. You’ll get all of the
tooling benefits with no compiler changes, and can tweak the wrapper
generation however much you want, using typing annotations or other
Python-specific information to create better wrappers over time.

I’d love for you to sketch out how any of this works with an acceptable
user experience, because I don't see anything concrete here.

We don’t need the basic dynamic case in the language to do this
experiment. Take the PyVal struct from the proposal. Now, write a Python
script that loads some module Foo and uses Python’s inspect
<https://docs.python.org/3/library/inspect.html&gt; module to go find the
classes, methods, etc., and pretty-print Swift code that uses PyVal. So
this:

def add_trick(self, trick):

turns into

extension PyVal {
  func add_trick(_ trick: PyVal) -> PyVal {
    /* do the magic to call into Python */
  }
}

Using the inspect module, you can extract parameter names, default
arguments, docstrings, and more to reflect the existing Python API as Swift
API, packed into a bridging module.

Note that we have a “flat” namespace of all Python methods on PyVal, which
is basically what you get with AnyObject today. Swift tooling will provide
code completion for member accesses into PyVal. Goto definition will jump
to the pretty-printed declarations, which could have the docstrings
formatted in comments and would show up in QuickHelp. The types are weak
(everything is PyVal), but that’s what we expect from importing a
dynamically-typed language.

As I mention above, I expect this to expose significant scalability
problems in the Swift compiler and it also defeats REPL/Playgrounds. Being
able to use the Swift REPL is really important for Python programmers.

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Ponyboy47 (Jacob Williams)]

I’ve been going back and forth on whether I’m for or against the current revision of the proposal and I’d have to say that I am in favor of it as is for the following reasons (many of which have already been stated):

It was stated early on in this proposal that Python is not going to be given the same first class support that C/Objective-C have in the compiler. It would be unreasonable and would set a precedent that all languages would get the same treatment when the community wants to add interoperability. Many arguments have been made that we NEED to have static typing for dynamic languages. I don’t see static typing coming without first class compiler support.

Also, the very nature of these dynamic languages means static typing would violate basic principles of the dynamic language. I think it would be much worse and scare away python developers who are suddenly forced to statically type things that they may know to be dynamic types in their python APIs. Of course, the other side of the coin is that swift developers would be scared away by the lack of static typing when using Python interop. I see the latter as a non-issue though because anyone using python knows to expect things without static types.

Forcing static types where there are no static types sounds like a really bad idea to me. So I’m ok with the proposed PyVal type for all python types. Extensions can be written to make casting a PyVal object to the various swift types easy (if/when desired).

There has also been much discussion on the potential for abuse. There was an example given of adding conformance to NSObject I believe. Chris and Xiaodi both proposed solutions to this that would minimize (and possibly eliminate) the potential for abuse.

However, there will always be potential for a feature to be misused. The potential does not mean it is an absolute given that it will be misused (I’m an optimist if you can’t tell ;).

I don’t view this protocol as a beginner user feature. This is not a protocol that will be commonly used or even commonly desired. It has a very specific set of uses and I strongly believe that the community will continue to do a good job at suggesting the right tool for the right job. The advanced users that are aware of this protocol will most likely properly recommend this protocol when applicable. Your average Joe Shmoe isn’t going to even be aware of this. Those who are curious, will probably ask about it. There will be StackOverflow responses describing it and what it does and when/why to use it. (This is just my two cents based on what I’ve witnessed in the swift community and on StackOverflow in the past). I’m sure that if the documentation on this protocol says “Used to provide interoperability with dynamic languages” then most users are going to steer clear of it unless they are writing an interop layer with a dynamic language.

I don’t think the goal here is to write Python (or ruby or javascript or [dynamic language]) in swift. The aim of this proposal it to make it easy and possible to use python from swift. I don’t expect to write a full blown python project in swift. I expect to call the few python APIs from module xyz and be done with python. I’m trying to write a swift project not a python one. If I wanted to use python extensively then I would just stick with python (I work as a python developer and Python is definitely not my language of choice, but it does have its upsides). I don’t want to write my own swift implementation for something where a python module already exists so I import the python module and do the few things with it that I actually need. I don’t need or expect IDE integration with the language I’m using at that moment. I can read docs for my 2-3 python API calls while I’m developing it.

Even with this proposal, people could write their own swift libraries that are just statically typed wrappers around a python library. This proposal does not limit people in any way. It does however, open the door for interoperability with a large set of dynamic languages.

···

On Dec 1, 2017, at 10:37 AM, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

On Dec 1, 2017, at 12:26 AM, Douglas Gregor <dgregor@apple.com <mailto:dgregor@apple.com>> wrote:

On Nov 30, 2017, at 10:05 PM, Chris Lattner <clattner@nondot.org <mailto:clattner@nondot.org>> wrote:
Hi Doug,

Thank you for the detailed email. I have been traveling today, so I haven’t had a chance to respond until now. I haven’t read the down-thread emails, so I apologize if any of this was already discussed:

I think better interoperability with Python (and other OO languages in widespread use) is a good goal, and I agree that the implementation of the feature described is straight-forward and not terribly invasive in the compiler.

Fantastic, I’m really pleased to hear that! I only care about solving the problem, so if we can find a good technical solution to the problems than I’ll be happy.

After thinking about your email a bit more, I think I might understand the disconnect we’re having. I think we have different versions in mind of what “success” looks like:

I believe your view is that people start using Python APIs from their Swift code, but gradually add type annotations (either inline or in a sidecar database) to make those APIs progressively more Swifty. New features will be added to Python (its type system, compilers, databases for common APIs, etc) to improve this interoperability, along the lines of what we’ve done for Objective-C over the years. Fast forward several years, and large Python libraries would be nice to use from Swift - perhaps nicer than they are to use from Python itself. This view aligns with what happened with Objective-C <-> Swift interoperability.

In contrast, I’m specifically interested in developers in certain large domains (e.g. data science and ML) which are “forced” to use Python because that is where all the libraries are. I have spoken to many of these sorts of people, and a large number of them really *dislike* using Python for all the obvious reasons (including the tooling issues you point out). My view of success is that we allow them to write all of *their code* in Swift, which will lead to a massive quality of life benefit for these frustrated people. You’re right that they will still chaff when using imported Python APIs (which don’t feel “swifty” for LOTS of reasons - e.g. method naming and design patterns), but my view is that this will provide incentive for these people to provide a proper Swift implementation for these libraries over time.

In short, your end game is a pervasively intertwined Swift/Python world (like ObjC and Swift are). My view is that Swift holds Python at arm's length, and wins over the hearts and minds of developers, leading to new Swift APIs designed for Swift.

I’m not sure if you agree with that portrayal of your position, and if not, I’m sorry and will try to understand another way. However, if I’m close, then I have several concerns about that vision and don’t believe the end-game is achievable or better than what I’m proposing. Consider:

- I don’t think there will be a lot of success getting people who *actually love* Python to use Swift, unless there is already an extrinsic reason for them to use it.
- Type hints are not widely used in Python, and there are believable reasons that they won’t ever be. Because they are specifically poorly suited for libraries, their use seems to be in “user’s own code” - but my proposal solves this already! See below for examples.
- Even if type hints were widely adopted, it would require massive extensions to them and to Python to make them be "good enough” to provide value for the things you’re envisioning. Analogs to instancetype, objc generics, lots of the C macros, and many of the other things we’ve added to ObjC would have to be added.
- The Python community has no reason to do this work for the Swift community, or accept changes to Python that are required to make this actually great.
- Beyond the core type system, the Python and Objective-C languages work extremely differently in other ways (e.g. lack of umbrella headers making AnyObject-style dispatch questionable).
- Even with those annotations and all the work, the APIs we’d end up with are not going to be good Swift APIs. I don’t think that “renamification” and "IUO audits" would ever actually happen in practice, for example.
- I believe the engineering effort required to implement this vision is so massive (including the changes to Swift, Python, and Python libraries) that it simply will never actually happen.

More concerning to me is that your design of using the existing AnyObject type presents a really concerning technical problem, scalability: It is not simply a Swift/ObjC/Python world, Javascript is also super important. There are also a number of other less-widely used dynamic languages that are interesting. Your design leads to them all being mashed together into a common runtime system. I cannot imagine how this would end up being a good thing for system complexity, and would lead to each of them being jeopardized in different ways. As I have mentioned before numerous times, it also opens the door for enormous Swift compiler complexity, as each of the object models need to be supported in the compiler (so you can subclass each languages’ types), and many other complexities.

If you are serious about improving the tooling situation for people using Python APIs in Swift, the most natural way to do so is to follow the approach that the MyPy community (they are the ones who have thought about this the most) is using: provide progressive typing extensions, use data flow analysis to propagate around types, and enhance the tooling to have support this. I’m skeptical that this will ever be worthwhile, but a sketch could look like this:

Consider the first example from: mypy - Examples

With my proposals you could write the dictionary part very similar to the Python part (other pieces of the example, e.g. list comprehensions are uninteresting to me):

let d: PyVal = {:}
…
d[word] = d.get(word, 0) + 1

We can teach the compiler simple data flow analysis, to know that ‘d’ is a Python dictionary. We could then allow the user to go further, with some new syntax along the lines of:

let d: @type(Dict<String, Int>) PyVal = {:} // Lots of other syntactic choices possible.
…
d[word] = d.get(word, 0) + 1

However, the really important thing to recognize about these examples it that the type annotations are really poor at representing “generic” code like common Python libraries. They are designed for “user code” - like that on the mypy web page - which use concrete types.

As it turns out, my proposal provides a solution for this part of the problem, because we’re allowing user code to be written in Swift! A Swift programmer working with Python APIs would actually write this as:

let d: Dictionary<String,Int> = {:}
…
d[word, default: 0] += 1

Similarly, there is no reason to do anything to make the second and third examples work nicely in Swift: you’d just define a Swift class, and a Swift function.

MyPy:

class BankAccount:
    def __init__(self, initial_balance: int = 0) -> None:
        self.balance = initial_balance
    def deposit(self, amount: int) -> None:
        self.balance += amount
    def withdraw(self, amount: int) -> None:
        self.balance -= amount
    def overdrawn(self) -> bool:
        return self.balance < 0

my_account = BankAccount(15)
my_account.withdraw(5)
print(my_account.balance)
Swift:

class BankAccount {
   var balance : Int
   init(initialBalance: Int) { … }
  func deposit(….
}

Even the proposals that I’m making - simple and isolated though they are - are enough to provide a major quality of life improvement for people writing large amounts of code against Python APIs. Beyond that, they are small extensions with low complexity, scale to supporting many different dynamic languages over time, require a level of engineering effort that is plausible to be built, and do not require some sort of "executive buy in” from the Python community.

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[davedelong]

This email reminded me of something…

One of the beautiful things about the announcement of Swift in 2014 was that no one saw it coming, even though a whole bunch of signs were in place. One of those signs was Objective-C Modules, which *essentially* describe an API in an abstract syntax, so that it can be used from another language. (It was billed as <wavy hands> making compilation faster </wavy hands>, but IIRC this was one of its trojan purposes).

Can that same tactic be used here? Can we somehow generate modules for python code, and then import those modules as the basis for cross-library invocations?

Dave

···

On Dec 1, 2017, at 1:49 PM, Marc Schlichte via swift-evolution <swift-evolution@swift.org> wrote:

A somewhat extreme alternative could be the use of IDLs and "libs as services“:

A Python lib could aditionally expose its APIs (or a relevant subset of it) via an IDL. Generators would help with the necessary stub and skeleton code on both Python and Swift sides.

My impression is that only big libs like TensorFlow are relevant to be bridged anyhow so that the effort to create an IDLized API for that community is acceptable.

IDLs are an old idea but they seem to get reinvented every now and then. Google has a new one called FIDL for its Fuchsia project: (https://fuchsia.googlesource.com/fidl/\)

Cheers
Marc

Am 01.12.2017 um 17:07 schrieb Jon Gilbert via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>>:

On Dec 1, 2017, at 02:44, Xiaodi Wu via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

So the use case here is, how do we make Swift a viable candidate for doing those things which today drive users to Python? The answer here is _not_: build a better Python. Nor does it require, out of the gate, even being as good as Python. The solution is to provide a _reasonably_ ergonomic to _interoperate with libraries available in Python_, with the benefit that those parts that you can write in native Swift will make the overall result safer and faster, etc.

I think we would be better served by a transpiler that translates Python (etc.) into Swift at compile time.

Look what Google did with j2objc (https://github.com/google/j2objc\). It translates Java right into Objective C. You can even put your Java code right in XCode and it auto-translates at build time.

Clearly, this is no small feat, and j2objc is a technical marvel that took world-class engineers years to perfect.

My point is, “Dynamic Member Lookup” is not the only solution, and it’s not the ideal solution if indeed it compromises the static guarantees of Swift.

Therefore, we should consider what other approaches might entail. The main players in Swift have lots of money and technical resources they could pour into a set of revolutionary transpilers.

Surely we don’t want to allow Goole to be the only company to provide a library that translates other codebases directly to a primarily Apple language, do we?

That being said, I am still interested to hear Chris’s response to these concerns, and if they were already addressed on a previous message and I missed that, then please forgive me.

- Jon
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution