A dynamicFeaturesHere {} block would not tell you any of these things either; if a project relies on interop with Python, you'd find these blocks surrounding the entirety of the code.
1 Like
The constants would need to be created locally, so that duplicates can be filtered out.
This is only true if the whole project relies on interop with Python (or dynamic member lookup).
If only part of the project uses dynamic member lookup this would allow to clearly see where those parts begin while not being too intrusive.
@anandabits sums up my reservations very nicely in his review here
Nonetheless I'll reiterate some issues myself:
Losing type safety of member names without being made visible at the point of use is something that does not fit with the feel and direction of Swift IMHO.
While Swift does have calls which may dynamically fail such as calls on IOUs or integer overflow it does not have calls which may fail due to a typo in the member name, i.e. all calls on IOUs will be successful if the receiver is present, because the members being accessed or the methods being called will have been type checked to exist.
In summary all current dynamic failures will be because of the value of an argument to the call (and I do count the receiver as argument here) which is inherently dynamic in itself but never because of a typo in the member being accessed or method being called which are inherently static in a statically typed language.
For that reason I would strongly suggest to make the use of dynamic member lookup visible at the point of use. The preferred method would by something similar to "try" (e.g. "dynamic" or "dyn"), maybe with the possibility to mark whole blocks.
But as Matthew pointed out, this has not yet been discussed enough.
I have worked extensively with Smalltalk for about 15 years in large projects and have used Ruby, Python and JavaScript to a lesser degree as well, and therefore know about the ease with which mistyped member or method names can cause errors, especially with respect to refactoring and program and library evolution.
In-depth consideration and participation in discussions on the list.
No, you made that perfectly clear. I was referring to a hypothetical 3rd party JSON library that implements the API in your example. This was in reply to the claim by @xwu that every Swift file using dynamic lookup will contain an import statement that would make it clear that the file uses dynamic lookup. That is clearly not the case, wouldn’t you agree?
Thank you for clarifying that point Chris. I haven’t thought about throwing subscripts for a while and had forgotten that they aren’t supported yet. I’m glad to hear that the proposal intends to work with them when they are supported. I hope those are supported soon, especially if this proposal is accepted. It seems like a choice that implementers of this protocol should be considering.
You’re right that duplicates could be an issue but that does not mean the constants would have to be declared locally. If this approach became common I think the community would find ways to avoid this problem without everyone having to declare constants themselves.
Thorsten provided the obvious rebuttal to this straw man. This is only true for projects that pervasively and directly use Python interop. This is not true for projects where Python interop is used to support a specific feature. It is also not true for projects that adopt the arguably better approach of designing an API that exposes the necessary capabilities provided by the Python library in a safer, Swifitier API.
The motivation of interop is so that Swift programs can use Python libraries. It has been clearly articulated that the goal is not to make Swift a better Python than Python. A Swift program larger than a script should not be spending most of its time working directly with PyVal even if a Python library is behind its core features.
4 Likes
Summary: I'm in total support of the use case addressed by this proposal, which is interoperability with dynamic languages. I'm in support of having a dynamic lookup mechanism. But I'd rather have the dynamic lookup capability tailored to the needs of Swift first, and then have accommodations made for interoperability with other languages than the reverse. This proposal does not try to address any of Swift needs for dynamism and focusses first and foremost on integration other languages, and this approach makes me uneasy.
What is your evaluation of the proposal?
I'm a bit confused by the whole process. The usefulness of this proposal relies in most part on another unreviewed proposal for calling functions dynamically. This proposal alone clearly does not constitute an adequate solution to the problem of interacting with Python, or most other dynamic language. Why aren't we reviewing the other proposal first, or the two together?
Is the problem being addressed significant enough to warrant a change to Swift?
I think the problem is significant enough. Accessing APIs from other language is a good way to make Swift more useful, and this could warrant additions to Swift.
Does this proposal fit well with the feel and direction of Swift?
I think using a protocol is fine. But if we use a protocol it should actually behaves like one and allow you to cast to the protocol and then dynamically call properties:
let a = (someValue as? DynamicMemberLookupProtocol)?.someDynamicProperty
Also, it should work with generics:
func extractName<T: DynamicMemberLookupProtocol>(_ value: T) -> String {
return value.name.stringValue // `name` is a dynamic lookup
}
I think key-paths should be supported. Maybe they are, but it's not mentioned in the proposal.
I don't have anything against the idea of a type allowing dynamic members. But this proposal is very centered on foreign languages and I think there are loose ends about how this attach to existing and potential future dynamic features of Swift. Here's a few exploratory ideas I could come with that would require further thought:
-
In Objective-C land, some member functions can be declared statically without an implementation. The implementation is filled in at runtime through a runtime dispatch mechanism similar to the one proposed here. Is this something we want in Swift?
To implement this, we could need two protocols: one for the dynamic implementation of methods/properties (where some methods are implemented dynamically even though they are declared statically), and a separate one for dynamic calling syntax (which would require the other one).
-
There's also a
dynamicattribute you can currently apply to@objcfunctions so they are always dispatched through the Objective-C runtime. Is there any plan to implement this kind of dynamic dispatch for Swift objects, the kind that can work with overrides in class extensions?It seems to me like the name
dynamicalready in use for@objcmight have future uses in Swift, and that this is a bit confusing with the name of the protocol because both mechanisms would implement a dynamic member lookup system. I'd tend to name the protocolFallbackMemberLookupProtocol(nodynamic) to emphasize that it handles cases the language itself cannot resolve and that it is not the same thing thandynamickeyword is about. -
We have yet to craft a good reflection system for Swift, but to me it looks sensible that we could build on top of that reflection system a dynamic wrapper that would resolve any Swift call dynamically. Is this proposal good enough for implementing such a wrapper?
And for a random question: is this protocol good enough to forward method calls through the Objective-C runtime?
I know these are all questions that are not linked to integrating foreign languages into Swift. But to me it seems like Swift should have one dynamic dispatch system capable of handling both integration with foreign languages and its internal use cases. I think the proposal is a little too shy of exploring the possibilities and tradeoffs of using it inside of Swift, and I fear we'll discover shortcomings when we do.
If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
There's opDispatch in D which is used as a fallback handler for calling methods. It's a template function (instantiated at compile time) that receives the argument names and types and does something with it. It being a template, it can statically choose its return type based on the name and can emit a compile-time error when the input types are not what it expects. More powerful generics in Swift could perhaps allow some aspects of this to be implemented, but the general approach does not really work for Swift where specialized versions of generic methods are optional.
There are multiple ways to handle unknown method calls in Objective-C. They generally lack type-safety, but otherwise enable many patterns that are difficult to implement otherwise. Think of Core Data. I think it'd be nice to have something similar in Swift.
In both cases, the method name and the arguments are provided together, unlike in this proposal. I'm not too sure what this implies though.
How much effort did you put into your review? A glance, a quick reading, or an in-depth study?
Read an early draft of this proposal, read the proposal and most of the comments here. Suggested using a system for auto-creating static bindings earlier (which I recognize has shortcomings). Took some time to think about it and form an opinion, which is difficult because I'm torn between allowing the use case now and waiting for Swift to have less nebulous plans about its own dynamic stuff, including reflection.
1 Like
Yes. I agree with you that the proposal provides no way to provide a syntactic "guarantee" that you can tell at a glance whether dynamic lookup is happening in some unknown code, but that is already true of Swift. Even without these proposals, the Python interop layer overloads the + operator to do completely dynamically-dispatched Python addition. Furthermore, there is no way to "know at a glance" whether some random Swift method is implemented in terms of Python dynamic calls, so any method or computed property could exhibit the same behavior.
In my personal opinion, there is no advantage to making Swift inconsistent by introducing a syntactic distinguisher for this one class of dynamicism. Furthermore, I have yet to understand why dynamic behavior is so bad that it needs to be called out in the first place.
That said, this is just my opinion, and I respect that you have a different one.
-Chris
3 Likes
Remember that this proposal centers around member access - it should be perfectly capable of forwarding property accesses through the ObjC runtime. For method calls, you'll need the DynamicCallable proposal, it should be able to do forwarding through the ObjC runtime for them.
-Chris
If only a specific feature requires Python interop, then only that file would import Python. Given the great flexibility in Swift to how files are organized, the specific feature can be implemented in an extension in its own file.
It doesn't matter whether the project uses Python pervasively or very little: there is no more information to be gained from some dynamicFeaturesHere {} blocks because whenever portion of the code that would be surrounded by such a block can also be put in separate files that uniquely import Python. (And anyone who wouldn't take the time to organize their code wouldn't be any more likely to meticulously fence only the interop portions of their code with dynamicFeaturesHere.)
The conclusion here doesn't follow from the premise. The distinction between "a Swift program that uses Python libraries" and "a better Python" would come down to this: is the program using features implemented in a library which happens to be written in Python, or is the program using the interop facilities to enable new dynamic features in Swift?
I agree with you that this proposal is about the former. However, just as you might call functions in C (e.g., sin) pervasively throughout your code without being said to be using Swift as "a better C," you might call library functions written in Python pervasively without using Swift as a better Python.
There is this refrain, it seems, that nothing from another language should be exposed in Swift unless it's made Swifty. But that is utterly inconsistent with how Swift has supported existing interop. One of the key pillars of Swift is memory safety, and simultaneously another tentpole feature is seamless, overhead-free access to unsafe C APIs, pointers and all. The same can be said for how this proposal would enable bridging to dynamic languages.
2 Likes
+1 to this, I think this is key to understanding the approach to Python interop. The proposal is to handle interoperability with Python the same way we handle interoperability with C: working with a C API almost always means writing a Swift wrapper for it, one that is swifty and memory safe.
-Chris
2 Likes
What is your evaluation of the proposal?
It's good. +1 from me.
Is the problem being addressed significant enough to warrant a change to Swift?
Absolutely. Interop with other languages should not be restricted to a clunky foreign function interface full of stringly-typed programming constructs.
Does this proposal fit well with the feel and direction of Swift?
Yes. Swift is about bringing static checking wins to bear on real-world problems.
If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
GHC 7 introduced rebindable syntax for Haskell -- this goes above and beyond the famous do notation. In OCaml there are many forms of metaprogramming, including full-on MetaOCaml.
How much effort did you put into your review? A glance, a quick reading, or an in-depth study?
I've skimmed it a few times.
My only complaint is that this doesn't go far enough. In addition to DynamicCallable, which was proposed elsewhere, we should have a way generate a function dynamically. That is to say, we should be able to receive the captured variables and the formal parameters to an anonymous function, as well as recursively translating the body of the function according to these or similar dynamic protocols.
Look at Racket for an example of an ecosystem that is betting heavily on "language-oriented programming," that is, on the metasyntactic interpretation style famously explored in Structure and Interpretation of Computer Programs (1986).
2 Likes
Which shows that we can't really understand the current proposal without considering the various aspects of that other one. For instance, Swift is using -- or is going in the direction of -- name of properties/methods being compound names such as foo(bar:baz:). What Swift does statically goes a bit against the current combination of the two dynamic proposals. Which mean you can't even mimic "static Swift" with dynamic dispatch.
Take this example:
let call = myObject.perform(_:with:)
call(someSelector, someObject) // can't add labels here!
If you wanted to do the same dynamically by combining the two proposals, the syntax would suddenly have to change:
let call = myWrapper.perform
call(someSelector, with: someObject) // need the labels here!
This is all fine if you're interfacing with Python because it is to be expected that Python code doesn't behave like Swift code. But I think it ought to be possible to make dynamic code behave like Swift code when desirable (such as when wrapping Swift or Objectice-C code).
In this particular case, I'd recommend adding compound names to the subscript in DynamicMemberLookupProtocol, even if they are redundant with those in DynamicCallable and left unused for the Python wrappers. In general, I'd recommend paying more attention to make sure you at least can represent the various Swift semantics within this dynamic system.
Edit: Looks like I missed that the other proposal describes a DynamicCallableKeywordedMethod protocol that does the a lookup+call operation and could, perhaps, in theory allow this to work.
4 Likes
In my personal opinion, there is no advantage to making Swift inconsistent by introducing a syntactic distinguisher for this one class of dynamicism. Furthermore, I have yet to understand why dynamic behavior is so bad that it needs to be called out in the first place.
FWIW, I don't think dynamic behavior is bad. I have tried to make that clear throughout my posts. Swift is a very pragmatic language. The motivation and general design of the proposal are very well aligned with that spirit of pragmatism. That is why I don't oppose the proposal in principle, I only oppose a specific detail of the design.
The specific point that I do not like is allowing a string literal to look like a symbol. This is not dynamic behavior at all, it is syntactic sugar. Repetition of literals is widely considered a bad practice. The downsides of doing this are not mitigated by syntactic sugar that makes them appear is if they are symbols. In fact, the downsides are magnified, especially in a language like Swift where symbols are usually statically checked and most programmers learn to rely on that verification.
I don't think it's a radical position to think that in a language with a strong static type system things that look like symbols should actually be symbols that are statically checked (at least unless there is some other indication that this verification is disabled in a given context).
Perhaps the strongest way I can state my position is that if I were writing Swift code that interoperated with a library written in Python I would certainly not want to repeat member name literals throughout my code (either disguised as symbols or not). I would appreciate a way to ensure I (or another team member) did not accidentally use the dynamic lookup sugar which by design involves repeated literals. Unfortunately if this proposal is accepted as-is I would not have a way to turn off dynamic member lookup for PyVal.
In short, my position is that syntactic sugar which by design reduces static verification of correctness is a feature that should adopt a "don't pay for what you don't use" philosophy. Making this feature visible at usage sites eliminates cognitive burden for readers where it is absent and alerts them to the increased potential for error where it is present. It also allows people to make a different choice, even when they are working with a type that supports dynamic member lookup.
That said, this is just my opinion, and I respect that you have a different one.
I respect yours as well and appreciate the discussion. I know we are both motivated by wanting to make Swift the best language it can be.
2 Likes
It doesn’t matter whether the project uses Python pervasively or very little: there is no more information to be gained from some dynamicFeaturesHere {} blocks because whenever portion of the code that would be surrounded by such a block can also be put in separate files that uniquely import Python. (And anyone who wouldn’t take the time to organize their code wouldn’t be any more likely to meticulously fence only the interop portions of their code with dynamicFeaturesHere.)
You keep emphasizing Python despite the fact that I am emphasizing the fact that the proposal allows any type in Swift to support dynamic member lookup. All code within a module is implicitly imported into every file. There will be projects which use this proposal for purposes far removed from dynamic language interop. The proposal itself includes the JSON example. I would be surprised if nobody takes that as inspiration for use in their own projects. We have also seen several respected members of the community mention use cases they have for the feature. It will be used for many purposes. This breadth of use should be fair game for discussion during this review despite it not being the motivator for the proposal.
1 Like
I think it's fair to think of JSON and the like as dynamic languages for the purposes of this discussion. Nothing about what I said is specific to Python. If you import a library that vends a dynamic type, then you may be using its dynamic features; likewise, if you import a library that provides unsafe C APIs, then you may be using its unsafe features.
To ensure you're not using dynamic sugar, wrap the type in the same way you would for unsafe C APIs. There shouldn't be a way to turn off dynamic member lookup for PyVal: whether or not dynamic member lookup is supported is a fundamental property of a type--which is the very justification for disabling retroactive conformance. In fact, there shouldn't be much you can do with a PyVal that isn't dynamic.
You argue that we shouldn't attempt to make Swift a "better" Python: I agree. This proposal allows dynamic types to be bridged to Swift and its members invoked in the same way that it would be in the original language. If there isn't static checking for members in the original language, then the type bridged to Swift doesn't need to offer static checking either; a wrapper can provide that functionality if so desired.
I think that's not the right way to look at it. "Reduces" implies there was something there before to be reduced; but a bridged dynamic library never had static guarantees that you seek. There's nothing to be reduced. If you want to enhance the library with Swifty guarantees, then wrap the library. But, as you say, the aim here isn't to build a better Python.
To ensure you’re not using dynamic sugar, wrap the type in the same way you would for unsafe C APIs. There shouldn’t be a way to turn off dynamic member lookup for PyVal: whether or not dynamic member lookup is supported is a fundamental property of a type–which is the very justification for disabling retroactive conformance. In fact, there shouldn’t be much you can do with a PyVal that isn’t dynamic.
Of course. But maybe I would prefer to write this wrapper using constants for the dynamic member names rather then using a string literal every time I invoke a member. If I choose to do this I would prefer that dynamic member lookup to not be available to my wrapper code.
I think that’s not the right way to look at it. “Reduces” implies there was something there before to be reduced; but a bridged dynamic library never had static guarantees that you seek. There’s nothing to be reduced. If you want to enhance the library with Swifty guarantees, then wrap the library. But, as you say, the aim here isn’t to build a better Python.
I am referring to the check that usually happens for Swift code which is written with the proposed syntax.
2 Likes
If your type wraps PyVal, then dynamic member lookup would not be available to your users. But it's true that if you extend PyVal, there's nothing to ensure that someone is invoking your extension method and not some other method because of a typo. Is this what you're concerned about?
I suppose we could enable importing a library with a choice as to whether or not dynamic features are imported. It'd be extra syntax but it would address some of these concerns. For instance, suppose you had to write
import dynamic Python
[Strawman syntax chosen deliberately because it's not the act of importing that's dynamic but the features of the library being imported.]
Then, suppose you vend some library which wraps a Python library, but your downstream users may either use it with dynamic lookup or can choose to use only the wrapped, static stuff. Then, you could choose:
import dynamic NumPy
or instead you could write
import NumPy // Only the static methods, please!