Proposal: Introduce User-defined "Dynamic Member Lookup" Types

[Chris_Lattner]

Until, and if, the “resistance” presents some conceptual explanation of how this could cause harm (I’m not asking for anything concrete, just a logical series of events that causes a plausible problem in practice), my belief is that the Swift community will see this as unwarranted fear.

This is particularly true given that the proposal is completely consistent with previous design decisions in Swift, decisions which have served us well.

-Chris

···

On Dec 6, 2017, at 8:11 PM, Joe DeCapo via swift-evolution <swift-evolution@swift.org> wrote:

Has all of this ruined the language thus far? No. Because the Swift core team doesn’t design, and the Swift community doesn’t adopt, ill-designed APIs that turn these facts into problems.

Yeah, I think I'd prefer this to stay as a normal protocol conformance. But if the proportion of resistance is high enough,…

[Paul_Cantrell]

But IUO’s are marked with an “!” to differentiate it from a normal type, where as DynamicMemberLookup is just a normal protocol conformance. I would be curious as to what you think of this idea [1]? Would this still be too much of a constraint just to make sure that when people use this they’re explicitly aware of what they’re doing?

I think that sort of approach seems like an acceptable middle ground. There's probably some bike-shedding to be done about the terminology and such. I just think it's overboard to require explicit annotations at the call site (especially for every single call site invocation). I would totally be fine with some sort of extra annotation at the declaration site beyond a normal protocol conformance. I do think this jibes well with C#'s `dynamic` approach from what I've gathered so far, and which other people seem to think is a worthy standard we should look toward for inspiration.

This seems marginally tolerable, but excessive.

Do we mark every usage of a type that can generate precondition failures or fatal errors for reasons other than “no such method?” No, we don’t.

Do we use a syntactically privileged marker instead of just using words like “unsafe” for types that expose direct access to raw memory? No, we don’t.

Has all of this ruined the language thus far? No. Because the Swift core team doesn’t design, and the Swift community doesn’t adopt, ill-designed APIs that turn these facts into problems.

My main objection to many of the critical responses to the proposal is that the examples all seem to ignore where the variables that are being invoked come from. I think we all tend to have a pretty good idea of what types we're dealing with, and what their behaviors are. And when we run into issues, we inspect those types. If this makes it more apparent on inspection that it has some special behavior, then I'm all for that. It would be similar to inspecting a type that crashed which turned out to be an IUO.

My main objection to the critical responses is that most of the objections are fundamentally cultural, not technical, and are fear-based, not evidence-based.

If a little extra language ceremony helps assuage those fears, I guess I’d consider it. I still think static analysis — starting and mostly ending with boring old syntax coloring — answers most all the concerns people have raised, and this debate is a tempest in a teapot.

Cheers, P

···

On Dec 6, 2017, at 8:52 PM, Joe DeCapo via swift-evolution <swift-evolution@swift.org> wrote:

On Dec 6, 2017, at 5:46 PM, Letanyan Arumugam <letanyan.a@gmail.com <mailto:letanyan.a@gmail.com>> wrote:

[jrgoodle (Joe DeCapo)]

This seems marginally tolerable, but excessive.

Do we mark every usage of a type that can generate precondition failures or fatal errors for reasons other than “no such method?” No, we don’t.

Do we use a syntactically privileged marker instead of just using words like “unsafe” for types that expose direct access to raw memory? No, we don’t.

Has all of this ruined the language thus far? No. Because the Swift core team doesn’t design, and the Swift community doesn’t adopt, ill-designed APIs that turn these facts into problems.

Yeah, I think I'd prefer this to stay as a normal protocol conformance. But if the proportion of resistance is high enough, I think the adoption of some annotation above and beyond the norm would be ok (again, at the declaration site, not the call site). Especially since this is a somewhat privileged protocol if any of the restrictions suggested in the proposal go forward, since they don't really apply to normal protocols.

My main objection to the critical responses is that most of the objections are fundamentally cultural, not technical, and are fear-based, not evidence-based.

If a little extra language ceremony helps assuage those fears, I guess I’d consider it. I still think static analysis — starting and mostly ending with boring old syntax coloring — answers most all the concerns people have raised, and this debate is a tempest in a teapot.

I agree wholeheartedly. I was just trying to bring some specifics to the examples given so far.

···

On Dec 6, 2017, at 9:45 PM, Paul Cantrell <cantrell@pobox.com> wrote:

[Letan (Letanyan Arumugam)]

This seems marginally tolerable, but excessive.

Do we mark every usage of a type that can generate precondition failures or fatal errors for reasons other than “no such method?” No, we don’t.

fatalError shouldn’t be used excessively. API surface areas for these types are going to be massive (infinite technically). I assume many people are going to be writing a lot of code would these types and calling many methods and properties which would all essentially have a fatalError. Would you consider it good code if the majority of all your types had methods defined with fatalError calls. What’s so wrong with adding another layer of protection on top to bypass if you want to do this.

Do we use a syntactically privileged marker instead of just using words like “unsafe” for types that expose direct access to raw memory? No, we don’t.

Okay we use Unsafe. Then should we have something similar for this. A UncheckedDynamicMemberLookup [1] and a DynamicMemberLookup [2]? my one objection to this would be that I would like to convert a UncheckedDynamicMemberLookup to a DynamicMemberLookup.

[1] would work like the current proposal for DynamicMemberLookup
[2] would only allow returning an optional

My main objection to the critical responses is that most of the objections are fundamentally cultural, not technical, and are fear-based, not evidence-based.

The goal of this proposal is to bring people from a culture where excessive use of this would be the norm for them. Why would it be so hard to imagine that people would adopt bad principles, knowing or unknowing, because this is what they’ve always done?

Evidence is going to be hard to get since I don’t know any other language like Swift that has done this for the same reasons before. As far as C# goes were they trying to get people from a heavily based dynamic community or help those already in the community?

If a little extra language ceremony helps assuage those fears, I guess I’d consider it. I still think static analysis — starting and mostly ending with boring old syntax coloring — answers most all the concerns people have raised, and this debate is a tempest in a teapot.

I'm unsure of this, but as far as I’m aware Swift language proposals shouldn’t rely on editor features. But like I said I’m unsure of this and if someone could clarify this that would great.

[Letan (Letanyan Arumugam)]

Until, and if, the “resistance” presents some conceptual explanation of how this could cause harm (I’m not asking for anything concrete, just a logical series of events that causes a plausible problem in practice), my belief is that the Swift community will see this as unwarranted fear.

My fear is that a design pattern around dynamic members and calls arise that is used to bypass the perceived initial annoyance of Swifts type system from new developers that come over to Swift and are now starting to try and go native. They have no reason to think about their conforming types as something that might fail because they’re using it to model behaviour that they’re used to (good or bad). I don’t see why it’s so bad to remind people that these conformances should be failing and only in rare cases should you ever have a dynamic member lookup that is fine to ignore all failing lookups.

People coming from JavaScript could perceivably make dictionaries conform. And later JSON, database, file and basically all resource API’s would follow.

Why would all of this happen rather than people behaving the way current Swift community members behave?
Because I worry that by bringing in people from other languages that a new learning path is created. One where you start by learning your language interoperating with Swift. And then pick up other Swift features as you go along using your Python API for example. This would create a disparate Swift community.

[benjamin.g]

Has all of this ruined the language thus far? No. Because the Swift core
team doesn’t design, and the Swift community doesn’t adopt, ill-designed
APIs that turn these facts into problems.

Yeah, I think I'd prefer this to stay as a normal protocol conformance.
But if the proportion of resistance is high enough,…

Until, and if, the “resistance” presents some conceptual explanation of
how this could cause harm (I’m not asking for anything concrete, just a
logical series of events that causes a plausible problem in practice), my
belief is that the Swift community will see this as unwarranted fear.

On the server side :
automatically generate an administration api for your model based on
introspection. Since swift doesn't provide anything convenient, and people
may simply try to "port" approach from python framework (like django),
they'll resort on recreating some kind of BaseDynamicObject that you'll
have to extend for all your base classe, using some "properties()" and
"methods()" functions to define your properties and methods for your model.

Or :
Automatically generate a database schema based on your model. Same idea.

···

On Thu, Dec 7, 2017 at 6:41 AM, Chris Lattner via swift-evolution < swift-evolution@swift.org> wrote:

On Dec 6, 2017, at 8:11 PM, Joe DeCapo via swift-evolution < > swift-evolution@swift.org> wrote:

This is particularly true given that the proposal is completely consistent
with previous design decisions in Swift, decisions which have served us
well.

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Nevin]

I have no input on whether or not this dynamism should be added to Swift.

However, *if* we add it, then I strongly prefer that dynamic member lookup
should use the exact same syntax as static member lookup, namely a single
dot. Member lookup is member lookup, the user-facing behavior is the same
is both cases, and we should present a single coherent experience with
maximum elegance and simplicity.

Nevin

[Chris_Lattner]

This does not improve clarity of code, it merely serves to obfuscate logic. It is immediately apparent from the APIs being used, the API style, and the static types (in Xcode or through static declarations) that this is all Python stuff.

It may be immediately apparent when the types involved are obviously dynamic, such as in this example where Python.import is explicitly used. However, my concern is less about the intended use case of dynamic language interop than I am that this feature will be generally available to all types in Swift.

This is big change from AnyObject dispatch. It opens up the dynamism to types and contexts that are not necessarily obviously using dynamic lookup, callable, etc. Maybe this won’t turn out to be a problem in practice but I still think it’s a legitimate concern.

Sure, it is a legit concern, but it is also nothing new. This is the standard concern with type inference.

While there are weird cases, in practice, values do not get magicked out of no-where. They most commonly are either root values like:

  let np = Python.import(“foo”)
  let pyInt = PyVal(42)

or they come for parameters:

  func f(x : PyVal) {

The place that is potentially surprising is when the type gets buried because you’re working with some API that returns a [String, PyVal] dictionary or something:

  let x = foo()[“someKey”]

and you don’t realize that PyVal’s are involved. However, if you are actively writing the code, you have access to code completion and other things that tell you these types, and if it is important for the clarity of the code, you write this instead:

  let x :PyVal = foo()[“someKey”]

There is nothing specific to this proposal about this issue.

I’m uncertain what the right answer is. I’m still not really comfortable with opening up dynamic lookup to any user-defined type without some way to indicate to readers that dynamic lookup is happening in a piece of code. Maybe there is a less localized annotation that would indicate dynamic lookup is in effect for a larger chunk of code.

You seem to be extremely concerned that people will adopt DynamicMemberLookup for types where it doesn’t make sense and abuse the feature. I am having a real problem understanding what your concern is, so I’d really love for you to explain some theoretical examples of the bad thing that happens: why someone (non-maliciously) adopts the protocol, what code gets written, and what harm actually comes from it.

Let me use a made up tale from a parallel universe to illustrate why I don’t understand your concern. Imagine if Swift didn’t already interoperate with C, and did not already have IUOs. Someone who cared about C language interoperability would quickly realize that the ergonomics of importing everything as strong optionals is a non-starter, jeopardizing the usability of C interop, and would propose IUOs as a feature.

We’d then have a long and drawn out conversation about the various options on how to model this, the pros and cons of each, and would settle on IUO as the least bad design (as an aside, in our universe, when we went through the design process that led to IUOs, this is exactly what happened, we even considered syntaxing them as interobangs :-).

At that point, there would be a general uproar because IUOs have high potential for abuse: Swift is “all about” strong types and safety, which IUOs undermine. Strong optionals are considered a pain to use by some people and widely misunderstood (I think they are the biggest challenge in learning Swift in practice), and so it is a reasonable feature that people could pervasively adopt IUOs, leading to a much worse world all around.

This made up parallel universe is exactly analogous to what is happening now. DynamicMemberLookup is no more dangerous and harmful than IUOs are. They will be one more tool in the toolbox. While it is possible that someone will abuse it, this will not be widespread. People who are particularly worried will build a single new rule into their linters (which already flag uses of x!), and the world will keep revolving.

Even the behavior of AnyObject was carefully designed and considered, and were really really good reasons for it returning IUO.

I am not trying to call into question the choices made in the past. Swift wouldn’t be the great language with a bright future that it is today without an incredibly successful migration of a large user base from Objective-C to Swift. This is a huge accomplishment and couldn’t have happened without making really good decisions about some really hard tradeoffs.

You miss my point. My point is that AnyObject lookup was carefully considered, has stood the test of time, and is the *right* answer. Swift 1 would not have been nearly as successful without it.

-Chris

···

On Dec 2, 2017, at 7:11 PM, Matthew Johnson <matthew@anandabits.com> wrote:

[xwu (Xiaodi Wu)]

I believe that adding explicit syntax would be counterproductive to your
goals, and would not make dynamic lookup syntax more clear. I assume that
you would also want the same thing for DynamicCallable too, and operator
overloads, subscripts, and every other operation you perform on these
values, since they all have the exact same behavior.

If we required some syntax even as minimal as “foo.^bar” and "baz^(42)”,
that change would turn this (which uses runtime failing or IUO return
values like AnyObject):

let np = Python.import("numpy")
let x = np.array([6, 7, 8])
let y = np.arange(24).reshape(2, 3, 4)
let a = np.ones(3, dtype: np.int32)
let b = np.linspace(0, pi, 3)
let c = a+b
let d = np.exp(c)
print(d)

into:

let np = Python.import("numpy")
let b = np^.array^([6, 7, 8])
let y = np^.arange^(24)^.reshape^(2, 3, 4)

let a = np^.ones^(3, dtype: np^.int32)
let b = np^.linspace^(0, pi, 3)
let c = a+^b
let d = np^.exp^(c)

This does not improve clarity of code, it merely serves to obfuscate
logic. It is immediately apparent from the APIs being used, the API style,
and the static types (in Xcode or through static declarations) that this is
all Python stuff. When you start mixing in use of native Swift types like
dictionaries (something we want to encourage because they are typed!) you
end up with an inconsistent mismash where people would just try adding
syntax or applying fixits continuously until the code builds.

That’s not Swift. You just wrote a bunch of Python. For example, Swift has
a native Int32.+ operator which fails on overflow - does your example also
do that? Anybody’s guess! Does your numpy array conform to Collection? I
guess not, because it’s an opaque Python value.

That’s exactly the kind of stuff I, as a user of the language, really
don't want to see mixed together with real Swift. I appreciate the need to
use functionality from libraries written in Python, but I don’t appreciate
it being so invisible and pervasive throughout the language. If you have a
bunch of Python logic, I’d prefer you wrote as much of it as possible in
Python, with as few bridging points to Swift as you can get away with. I
remain convinced that this design encourages the opposite - because, as you
said earlier, it’s “too good”.

The use case that Chris is solving here is precisely how to enable the
writing of this type of code in Swift. Put another way, how do I interface
with libraries written in Python without writing my own code in Python?
"That's exactly the kind of stuff I really don't want to see mixed together
with real Swift" and "I'd prefer you wrote as much of it as possible in
Python" is not an answer; it's rejecting the legitimacy of the use case in
the first place.

As for the point about Swift already including non-marked,

potentially-crashing operations (like the + operator, or Array
subscripting): nobody likes that behaviour!

I, for one, very much like that behavior. Swift has many non-marked,
potentially-crashing operations because that is both performant and safe.
There is a common misconception that crashing is unsafe, with corresponding
misconceptions such as avoiding "!". This is simply incorrect.

···

On Sun, Dec 3, 2017 at 2:20 PM, Karl Wagner via swift-evolution < swift-evolution@swift.org> wrote:

Whenever I come to a new Swift codebase, I almost universally find that
people have written their own “safe” Array accessor which integrates
bounds-checking and returns an Optional. The issue has come up here many,
many times for inclusion in the standard library. I certainly would not use
it as justification for adding more of those kinds of unmarked,
potentially-unsafe operations. Also, enough Swift developers know about the
Array subscript behaviour that the square brackets almost become a marker,
like “!”, of a potentially-failing operation. The same is not true of the
dot operator, in general.

I also don’t agree with the comparisons to Objective-C/AnyObject dispatch.
It’s true that it’s unsafe to an extent, but it’s also orders of magnitude
safer than this sort of dispatch. Clang is integrated in to the compiler,
and can at least perform some rudimentary checking of method
signatures/selectors. This sort of dispatch provides absolutely no
protections whatsoever — is “arange” really a function? Is it not really a
typo for “arrange”? That’s something I need to Google. With regular Swift I
can assume that if the compiler allows it, there is a function called
“arange” somewhere, and all I need to worry about is whether the erased
AnyObject is of the correct type to respond to that message. And as I said
earlier, AnyObject is incredibly rare in practice anyway. So no, I don’t
agree that we should just keep lowering the safeguards; it’s like
demolishing your house because of one draughty door.

What I *could* support, would be some kind of optional syntax, possibly
with some kind of associated scope which allows you omit it. Something like:

// Normally, optionals are required.
let result: PythonObject? = pythonObj.someProperty?.someFunction(1, 2, 3)

// Within a special scope, you can omit them. The scope will bail at the
first lookup failure and return nil.
let result: PythonObject? = Python {
    return pythonObj.someProperty.someFunction(1, 2, 3)
}

Perhaps the “Python” object could conform to a protocol with an associated
type for the objects it can implicitly unwrap. There would be some
additional compiler work, for sure, but that’s secondary to a good language
model IMO (easy for me to say, I know).

- Karl

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[hartbit (David Hart)]

Sent from my iPad

For all those reasons, we really do need something like AnyObject dispatch if we care about working with dynamically typed languages. The design I’m suggesting carefully cordons this off into its own struct type, so it doesn’t infect the rest of the type system, and is non-invasive in the compiler.

I am quite familiar with dynamic languages and agree that this is necessary if we are going to fully open up access to these languages from Swift.

Ok, then it appears you agree that something like anyobject dispatch is necessary for effective dynamic language interop.

I strongly urge you to reconsider the decision of that dynamic members must be made available with no indication at usage sites. An indication of dynamic lookup at usage sites aligns very well (IMO) with the rest of Swift (AnyObject lookup aside) by calling attention to code that requires extra care to get right.

I don’t understand this. The proposal is fully type safe, and this approach is completely precedented by AnyObject. Swift’s type system supports many ways to express fallibility, and keeping those decisions orthogonal to this proposal is the right thing to do, because it allows the author of the type to decide what model makes sense for them.

Allowing the author of the type to choose whether the mechanism is hidden or visible is exactly what I don’t want to allow. I think you have the right design regarding types and semantics - the author chooses. But I don’t want these calls to look like ordinary member lookup when I’m reading code.

They inherently have a much greater chance of failure than ordinary member lookup. Further, authors are likely to choose immediate traps or nil IUO as failure modes as forcing users to deal with Optional on every call is likely to be untenable. I believe this behavior should be represented by some kind of syntax at the usage site. I don’t believe it is an undue burden. It would make the dynamic lookup semantic clear to all readers and would help to discourage abuse.

I believe that adding explicit syntax would be counterproductive to your goals, and would not make dynamic lookup syntax more clear. I assume that you would also want the same thing for DynamicCallable too, and operator overloads, subscripts, and every other operation you perform on these values, since they all have the exact same behavior.

If we required some syntax even as minimal as “foo.^bar” and "baz^(42)”, that change would turn this (which uses runtime failing or IUO return values like AnyObject):

  let np = Python.import("numpy")
  let x = np.array([6, 7, 8])
  let y = np.arange(24).reshape(2, 3, 4)
  
  let a = np.ones(3, dtype: np.int32)
  let b = np.linspace(0, pi, 3)
  let c = a+b
  let d = np.exp(c)
  print(d)

into:

  let np = Python.import("numpy")
  let b = np^.array^([6, 7, 8])
  let y = np^.arange^(24)^.reshape^(2, 3, 4)
  
  let a = np^.ones^(3, dtype: np^.int32)
  let b = np^.linspace^(0, pi, 3)
  let c = a+^b
  let d = np^.exp^(c)

This does not improve clarity of code, it merely serves to obfuscate logic. It is immediately apparent from the APIs being used, the API style, and the static types (in Xcode or through static declarations) that this is all Python stuff.

It may be immediately apparent when the types involved are obviously dynamic, such as in this example where Python.import is explicitly used. However, my concern is less about the intended use case of dynamic language interop than I am that this feature will be generally available to all types in Swift.

This is big change from AnyObject dispatch. It opens up the dynamism to types and contexts that are not necessarily obviously using dynamic lookup, callable, etc. Maybe this won’t turn out to be a problem in practice but I still think it’s a legitimate concern.

If dynamism if restricted to subclasses of a DynamicObject type, like Xiaodi suggested earlier, then we can protect ourselves from this dynamic dispatch being generally available to all types in Swift.

When you start mixing in use of native Swift types like dictionaries (something we want to encourage because they are typed!) you end up with an inconsistent mismash where people would just try adding syntax or applying fixits continuously until the code builds.

Beyond that, it is counterproductive to your goals, because it means that people are far less likely to use to use optional returns. Doing so (which produces a safer result) would cause a double tax in syntax, and would be a confusing jumble. I can’t bring myself to do the whole example above, one line - just converting member lookup syntax but not callable syntax - would end up:

  let y = np^.arange?^(24)^.reshape^?(2, 3, 4)

If you made DynamicCallable also return optional it would be:

  let y = np^.arange?^(24)?^.reshape^?(2, 3, 4)!

or something. This is such madness that no one would do that.

Yes, I agree. The interaction with optional chaining makes it unworkable. I hadn’t thought that all the way through. Thank you for indulging in the discussion about this idea.

I’m uncertain what the right answer is. I’m still not really comfortable with opening up dynamic lookup to any user-defined type without some way to indicate to readers that dynamic lookup is happening in a piece of code. Maybe there is a less localized annotation that would indicate dynamic lookup is in effect for a larger chunk of code.

I think that making dynamic calls syntactically different to readers is going too far in the direction of safety. Plus, it makes the language inconsistent as we already have AnyObject dispatch with exactly the same syntax. But I understand why you would want it if *any* type could end up being conformed to a dynamic lookupable/callable protocol. Like said above, I think that a DynamicObject type is enough protection to not bother making the syntax heavier at the point of use.

···

On 3 Dec 2017, at 04:11, Matthew Johnson via swift-evolution <swift-evolution@swift.org> wrote:
On Dec 2, 2017, at 7:40 PM, Chris Lattner <clattner@nondot.org <mailto:clattner@nondot.org>> wrote:

On Dec 2, 2017, at 2:13 PM, Matthew Johnson <matthew@anandabits.com <mailto:matthew@anandabits.com>> wrote:

One idea that hasn’t been explored yet is introducing a dynamic lookup effect. That would provide clean syntax at the expression level while still making it clear that dynamic lookup is happening. This probably isn’t the right approach as it would be viral and wouldn’t even indicate that the code in a specific function body even contains a dynamic lookup. I mention it mostly to broaden the conversation and thought space about what kind of approach might address my concerns without cluttering up expressions.

Another, potentially more viable approach would be to use an approach similar to try and the proposed async of a statement modifier (possibly also allowed at the expression level as with try and async). This could be used in conjunction with an effect as mentioned above but could also be used independently even though there isn’t a current precedent for that. Your example written with this approach might be:

  let np = Python.import("numpy")
  let b = dynamic np.array([6, 7, 8])
  let y = dynamic np.arange(24).reshape(2, 3, 4)
  
  let a = dynamic np.ones(3, dtype: np.int32)
  let b = dynamic np.linspace(0, pi, 3)
  let c = dynamic a + b
  let d = dynamic np.exp(c)

Note: `dynamic` is just a straw man here. This is obviously a lot of boilerplate repetition of the same modifier. I would also be perfectly happy allowing this annotation to apply to a block of code or even a whole function body. Then it might be:

dynamic {
  let np = Python.import("numpy")
  let b = np.array([6, 7, 8])
  let y = np.arange(24).reshape(2, 3, 4)
  
  let a = np.ones(3, dtype: np.int32)
  let b = np.linspace(0, pi, 3)
  let c = a + b
  let d = np.exp(c)
}

The most obvious objection to this is that it introduces nesting and does so in a way that only very indirectly influences control flow (through the potential for dynamic failure).

My objective certainly isn’t to make code ugly and obfuscate logic. It is simply to make the usage of dynamic features that are prone to failure at runtime immediately clear to a reader. This should be as lightweight as possible while still providing valuable information to the reader.

Swift already has a dynamic member lookup feature, "AnyObject dispatch" which does not use additional punctuation, so this would break precedent.

I would prefer if dynamic lookup were visible with AnyObject as well. For that reason I don’t believe it makes a good precedent to follow. In fact, I would prefer to see us go the other direction and perhaps even consider revising dynamic lookup syntax for AnyObject in the future.

This is definitely not going to happen. The change Doug mentioned is to have AnyObject lookup return optional instead of IUO, which forces ? on the clients. Adding other syntax (like you’re suggesting) is certainly not going to happen.

The entire point of AnyObject dispatch is to improve syntactic elegance and clarity of code using it. There is no other reason to exist. Making code that uses it syntactically onerous completely defeats the point of having it in the first place, as I’ve mentioned before.

I agree. The interaction with optional chaining is significantly more onerous than I had considered. I should have worked through an example on my own. I do hope you will consider a less localized approach to usage-site annotation though.

Furthermore, your premise that Swift does not have invisibly failable operations is plainly wrong. Array subscript and even integer addition can fail.

I am well aware of these behaviors. The difference IMO is that programmers tend to be well aware of these preconditions even if they also often choose to ignore them. Dynamic lookup will not be so clear. This is especially true if people use it with types that also have an API available through static lookup.

Even the behavior of AnyObject was carefully designed and considered, and were really really good reasons for it returning IUO.

I am not trying to call into question the choices made in the past. Swift wouldn’t be the great language with a bright future that it is today without an incredibly successful migration of a large user base from Objective-C to Swift. This is a huge accomplishment and couldn’t have happened without making really good decisions about some really hard tradeoffs.

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

[trs (Thorsten Seitz)]

+1

-Thorsten

···

Am 03.12.2017 um 21:20 schrieb Karl Wagner via swift-evolution <swift-evolution@swift.org>:

I believe that adding explicit syntax would be counterproductive to your goals, and would not make dynamic lookup syntax more clear. I assume that you would also want the same thing for DynamicCallable too, and operator overloads, subscripts, and every other operation you perform on these values, since they all have the exact same behavior.

If we required some syntax even as minimal as “foo.^bar” and "baz^(42)”, that change would turn this (which uses runtime failing or IUO return values like AnyObject):

  let np = Python.import("numpy")
  let x = np.array([6, 7, 8])
  let y = np.arange(24).reshape(2, 3, 4)
  
  let a = np.ones(3, dtype: np.int32)
  let b = np.linspace(0, pi, 3)
  let c = a+b
  let d = np.exp(c)
  print(d)

into:

  let np = Python.import("numpy")
  let b = np^.array^([6, 7, 8])
  let y = np^.arange^(24)^.reshape^(2, 3, 4)
  
  let a = np^.ones^(3, dtype: np^.int32)
  let b = np^.linspace^(0, pi, 3)
  let c = a+^b
  let d = np^.exp^(c)

This does not improve clarity of code, it merely serves to obfuscate logic. It is immediately apparent from the APIs being used, the API style, and the static types (in Xcode or through static declarations) that this is all Python stuff. When you start mixing in use of native Swift types like dictionaries (something we want to encourage because they are typed!) you end up with an inconsistent mismash where people would just try adding syntax or applying fixits continuously until the code builds.

That’s not Swift. You just wrote a bunch of Python. For example, Swift has a native Int32.+ operator which fails on overflow - does your example also do that? Anybody’s guess! Does your numpy array conform to Collection? I guess not, because it’s an opaque Python value.

That’s exactly the kind of stuff I, as a user of the language, really don't want to see mixed together with real Swift. I appreciate the need to use functionality from libraries written in Python, but I don’t appreciate it being so invisible and pervasive throughout the language. If you have a bunch of Python logic, I’d prefer you wrote as much of it as possible in Python, with as few bridging points to Swift as you can get away with. I remain convinced that this design encourages the opposite - because, as you said earlier, it’s “too good”.

As for the point about Swift already including non-marked, potentially-crashing operations (like the + operator, or Array subscripting): nobody likes that behaviour! Whenever I come to a new Swift codebase, I almost universally find that people have written their own “safe” Array accessor which integrates bounds-checking and returns an Optional. The issue has come up here many, many times for inclusion in the standard library. I certainly would not use it as justification for adding more of those kinds of unmarked, potentially-unsafe operations. Also, enough Swift developers know about the Array subscript behaviour that the square brackets almost become a marker, like “!”, of a potentially-failing operation. The same is not true of the dot operator, in general.

I also don’t agree with the comparisons to Objective-C/AnyObject dispatch. It’s true that it’s unsafe to an extent, but it’s also orders of magnitude safer than this sort of dispatch. Clang is integrated in to the compiler, and can at least perform some rudimentary checking of method signatures/selectors. This sort of dispatch provides absolutely no protections whatsoever — is “arange” really a function? Is it not really a typo for “arrange”? That’s something I need to Google. With regular Swift I can assume that if the compiler allows it, there is a function called “arange” somewhere, and all I need to worry about is whether the erased AnyObject is of the correct type to respond to that message. And as I said earlier, AnyObject is incredibly rare in practice anyway. So no, I don’t agree that we should just keep lowering the safeguards; it’s like demolishing your house because of one draughty door.

What I could support, would be some kind of optional syntax, possibly with some kind of associated scope which allows you omit it. Something like:

// Normally, optionals are required.
let result: PythonObject? = pythonObj.someProperty?.someFunction(1, 2, 3)

// Within a special scope, you can omit them. The scope will bail at the first lookup failure and return nil.
let result: PythonObject? = Python {
    return pythonObj.someProperty.someFunction(1, 2, 3)
}

Perhaps the “Python” object could conform to a protocol with an associated type for the objects it can implicitly unwrap. There would be some additional compiler work, for sure, but that’s secondary to a good language model IMO (easy for me to say, I know).

- Karl
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[jrgoodle (Joe DeCapo)]

When I began writing Python, I initially named all my variables in camel case. And when I used the Subprocess module, I was passing return codes throughout my code base like it was bash. Even though I didn't know what was idiomatic in Python at that time, I could still sense a code smell in what I was writing. I soon learned that snake case was the standard, so I updated my code. And eventually I learned how to use exceptions and I refactored my code to use them instead of return codes. I doubt that this is unusual when coming to a new language with previous experience in other languages. It seems inevitable, and a normal process for becoming familiar with the idioms of a certain language community. I don't think we need to be so fearful of the community fracturing if people from a dynamic language background come to Swift and initially use those idioms. It will always be more burdensome to do things in a non-idiomatic way, and over time people will learn how to write idiomatic Swift.

Also, since this is an expert-level feature, I highly doubt that newcomers to Swift will be writing their own types that adopt the DynamicMemberLookup or DynamicCallable protocols. They'll simply be using existing bridges to the dynamic language they want to interact with. And if expert Swift users find a legitimate use case for this beyond dynamic language interop, I don't think that's necessarily a bad thing if it helps them solve their problem.

···

On Dec 7, 2017, at 12:38 AM, Letanyan Arumugam via swift-evolution <swift-evolution@swift.org> wrote:

My fear is that a design pattern around dynamic members and calls arise that is used to bypass the perceived initial annoyance of Swifts type system from new developers that come over to Swift and are now starting to try and go native. They have no reason to think about their conforming types as something that might fail because they’re using it to model behaviour that they’re used to (good or bad). I don’t see why it’s so bad to remind people that these conformances should be failing and only in rare cases should you ever have a dynamic member lookup that is fine to ignore all failing lookups.

People coming from JavaScript could perceivably make dictionaries conform. And later JSON, database, file and basically all resource API’s would follow.

Why would all of this happen rather than people behaving the way current Swift community members behave?
Because I worry that by bringing in people from other languages that a new learning path is created. One where you start by learning your language interoperating with Swift. And then pick up other Swift features as you go along using your Python API for example. This would create a disparate Swift community.

[gwendal.roue (Gwendal Roué)]

The explicit harm that Chris is looking for is yet to be shown.

You should not assume that everybody feels an horror thrill by reading such applications of dynamism. As a matter of fact, users of dynamic languages *live* and *enjoy* this. Python and Ruby users, obviously (Rails + Django), but also ours close cousins, the Objective-C developers, who rely on Key-Value coding or validation methods. Those use cases are, I'm sorry to say it, *compelling* use cases for dynamism.

Ironically, "Explicit is better than implicit" is a... Python motto ;-)

Gwendal

···

Le 7 déc. 2017 à 11:00, Benjamin G via swift-evolution <swift-evolution@swift.org> a écrit :

Until, and if, the “resistance” presents some conceptual explanation of how this could cause harm (I’m not asking for anything concrete, just a logical series of events that causes a plausible problem in practice), my belief is that the Swift community will see this as unwarranted fear.

On the server side :
automatically generate an administration api for your model based on introspection. Since swift doesn't provide anything convenient, and people may simply try to "port" approach from python framework (like django), they'll resort on recreating some kind of BaseDynamicObject that you'll have to extend for all your base classe, using some "properties()" and "methods()" functions to define your properties and methods for your model.

Or :
Automatically generate a database schema based on your model. Same idea.

[xwu (Xiaodi Wu)]

This seems marginally tolerable, but excessive.

Do we mark every usage of a type that can generate precondition failures
or fatal errors for reasons other than “no such method?” No, we don’t.

fatalError shouldn’t be used excessively. API surface areas for these
types are going to be massive (infinite technically). I assume many people
are going to be writing a lot of code would these types and calling many
methods and properties which would all essentially have a fatalError. Would
you consider it good code if the majority of all your types had methods
defined with fatalError calls.

What is the basis for this claim? Probably the majority of standard library
methods check preconditions and trap on failure. That is how I write my
code as well.

What’s so wrong with adding another layer of protection on top to bypass if

···

On Thu, Dec 7, 2017 at 00:37 Letanyan Arumugam via swift-evolution < swift-evolution@swift.org> wrote:

you want to do this.

Do we use a syntactically privileged marker instead of just using words
like “unsafe” for types that expose direct access to raw memory? No, we
don’t.

Okay we use Unsafe. Then should we have something similar for this. A
UncheckedDynamicMemberLookup [1] and a DynamicMemberLookup [2]? my one
objection to this would be that I would like to convert a
UncheckedDynamicMemberLookup to a DynamicMemberLookup.

[1] would work like the current proposal for DynamicMemberLookup
[2] would only allow returning an optional

My main objection to the critical responses is that most of the objections
are fundamentally cultural, not technical, and are fear-based, not
evidence-based.

The goal of this proposal is to bring people from a culture where
excessive use of this would be the norm for them. Why would it be so hard
to imagine that people would adopt bad principles, knowing or unknowing,
because this is what they’ve always done?

Evidence is going to be hard to get since I don’t know any other language
like Swift that has done this for the same reasons before. As far as C#
goes were they trying to get people from a heavily based dynamic community
or help those already in the community?

If a little extra language ceremony helps assuage those fears, I guess I’d
consider it. I still think static analysis — starting and mostly ending
with boring old syntax coloring — answers most all the concerns people have
raised, and this debate is a tempest in a teapot.

I'm unsure of this, but as far as I’m aware Swift language proposals
shouldn’t rely on editor features. But like I said I’m unsure of this and
if someone could clarify this that would great.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Vincent_Esche]

> Alas while Swift's generics provide a nice basic foundation, they still
have big gaps in what can be expressed through them.
> And i'm not talking about HKT or any such more advanced use of types).
I'm talking about the basic stuff. Generic abstractions over type
conversion. Generic abstractions over arithmetic operators. This kind of
stuff. Stuff, that ironically would be necessary for a generic yet
idiomatic implementation of Linear Algebra algorithms in Swift.

This is all orthogonal to the proposal. I’m certainly not opposed to the
generics system getting better :-), […]

Exactly. That’s my whole point. How do we make sure these are understood to
be orthogonal? We’re talking about the average Joe here.
For someone used to stringly-typed dictionaries (Python e.g.) and
stringly-typed reflection (many message-passing/dynamic languages) it will
not be clear that 9/10 times dynamic stringly-typed member lookup is not
the right tool of choice for modeling type relations in a statically typed
language, which is exactly how these things are done in many scripting
languages.

And I think that we have more urgent topics at hand that need to get
fixed.
> Swift's generics are very limited. We hardly have any tooling.
> Diagnostics are just shy of migrating from "utter garbage" to "getting
usable", by modern standards.
> Integration with Xcode still to this day is an utter clusterfuck. With
errors pointing to the void, regularly.
> I'd prefer the team to focus on these, solidifying the foundation and
once that's done
> and the result have been proven to be sound by time I'd love to see
Swift get some more dynamism.

I’m not disagreeing with your point about other things needing to be
improved, but the only question is whether this proposal fits with the
right long term direction for Swift. It is not a prioritization question.

I’m not so much thinking of it as a prioritization issue (as you already
did the implementation and made clear from the beginning that you would do
the heavy lifting, not the core team), but more of a timing issue. It
absolutely is a timing issue from my point of view, for the very reasons
that I gave previously.

A frequent argument against any of the reasonable doubts expressed in this
thread is the notion that the community “would figure something out” and
“prevent any anti-patterns and misuses from gaining a foothold”.

I have doubts about this. There is plenty of evidence for the contrary:

- We don't have a strong unit testing culture in the swift community.
Hardly any Swift project on Github has proper unit test coverage. Most have
none at all.
  And I can’t blame them/us, as XCTest is more of a burden to use than a
help. XCTest on Linux is even worse if not border-line offensive to work
with.
- We don't have a strong culture of “avoid IUOs at all costs” either. I see
them all over the place when going through random Github projects.
  They are everywhere. I would give you numbers, but “!” is kinda hard to
search for (i.e. impossible) using Github’s source code search feature.
- There is still plenty of use of AnyObject and its cousins where for a
long time we have had better solutions.
- There is still incredible amounts of non-typesafe APIs (and I’m talking
about fresh 3rd-party stuff here, not legacy frameworks).
- …

At this point in time it's like opening Pandora's box.

I, still, have yet to see any evidence of harm that this proposal can
cause.

The harm from my perspective is not so much of technical nature than more
of a cultural one.
And plenty of “evidence” has been provided in this thread for why this
proposal is a risk for the language.

If not then Letanyan put it perfectly:

My main objection to the critical responses is that most of the objections
are fundamentally cultural, not technical, and are fear-based, not
evidence-based.

The goal of this proposal is to *bring people from a culture where
excessive use of this would be the norm for them*. *Why would it be so
hard to imagine that people would adopt bad principles, knowing or
unknowing, because this is what they’ve always done?*

Evidence is going to be hard to get since I don’t know any other language
like Swift that has done this for the same reasons before. As far as C#
goes were they trying to get people from a heavily based dynamic community
or help those already in the community?

[…]

My fear is that a design pattern around dynamic members and calls arise

that is *used to bypass the perceived initial annoyance of Swifts type
system from new developers that come over to Swift* and are now starting
to try and go native. They *have no reason to think* about their
conforming types as *something that might fail* because *they’re using it*
to model behaviour that they’re used to (good or bad). I don’t see why it’s
so bad to remind people that these conformances should be failing and only
in rare cases should you ever have a dynamic member lookup that is fine to
ignore all failing lookups.

*People coming from JavaScript could perceivably make dictionaries conform.

And later JSON, database, file and basically all resource API’s would
follow.*

Why would all of this happen rather than people behaving the way current

Swift community members behave?

Because I worry that by bringing in people from other languages that *a new

learning path is created*. One where you start by learning your language
interoperating with Swift. And then pick up other Swift features as you go
along using your Python API for example. *This would create a disparate
Swift community.*

"This would create a disparate Swift community” is the money quote here and
the elephant in the room.
I fully agree with Letanyan. Couldn’t have said it any better.
What more evidence do we need that there _is_ potential for harm? This is
no blind and reactionary fear-mongering.

- How do you plan to teach the proper use of this feature? How would the
documentation introduce it
> and how would it ensure that people fully understand its purpose and
what it decidedly is _not_ for?
> - How would the diagnostics (in particular in code mixing static and
dynamic Swift) look like?
> - How would the debugging of dynamic APIs look like?

This is an expert-only feature like the ExpressibleBy protocols. This is
not something that will be introduced or taught in chapter one of the Swift
book, or in the Swift Tour.

So expert-only features don’t need to be documented/taught or get
human-friendly diagnostics or debugging support?

Regardless of whether something is an expert-only feature it will come into
contact with users of all levels of expertise.
And regardless of whether something is an expert-only feature it begs the
question of …

- How we will introduce & teach this feature to people coming from static
languages? How for those coming from dynamic languages?
- How we will make sure that users of all experience levels don’t get lost
when interfacing with an API that makes use of it?
- How we will make sure that we don’t create a blackboxed sub-language that
is void of any support for debugging and/or introspection.

I don’t understand this complete neglect of the community/educational
aspect of adding a language feature
with the potential to completely changing the shape of large parts of the
still young Swift ecosystem.

Vincent

···

On Wed, Dec 6, 2017 at 6:14 PM, Chris Lattner <clattner@nondot.org> wrote:

On Dec 6, 2017, at 5:41 AM, Vincent Esche via swift-evolution < > swift-evolution@swift.org> wrote:

On Thu, Dec 7, 2017 at 7:37 AM, Letanyan Arumugam via swift-evolution < swift-evolution@swift.org> wrote:

On Thu, Dec 7, 2017 at 7:38 AM, Letanyan Arumugam via swift-evolution < swift-evolution@swift.org> wrote:

> Until, and if, the “resistance” presents some conceptual explanation of
how this could cause harm (I’m not asking for anything concrete, just a
logical series of events that causes a plausible problem in practice), my
belief is that the Swift community will see this as unwarranted fear.
>

My fear is that a design pattern around dynamic members and calls arise
that is used to bypass the perceived initial annoyance of Swifts type
system from new developers that come over to Swift and are now starting to
try and go native. They have no reason to think about their conforming
types as something that might fail because they’re using it to model
behaviour that they’re used to (good or bad). I don’t see why it’s so bad
to remind people that these conformances should be failing and only in rare
cases should you ever have a dynamic member lookup that is fine to ignore
all failing lookups.

People coming from JavaScript could perceivably make dictionaries conform.
And later JSON, database, file and basically all resource API’s would
follow.

Why would all of this happen rather than people behaving the way current
Swift community members behave?
Because I worry that by bringing in people from other languages that a new
learning path is created. One where you start by learning your language
interoperating with Swift. And then pick up other Swift features as you go
along using your Python API for example. This would create a disparate
Swift community.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Paul_Cantrell]

My main objection to the critical responses is that most of the objections are fundamentally cultural, not technical, and are fear-based, not evidence-based.

The goal of this proposal is to bring people from a culture where excessive use of this would be the norm for them. Why would it be so hard to imagine that people would adopt bad principles, knowing or unknowing, because this is what they’ve always done?

Languages bring in new developers who misapply their experience all the time. Look at the early days of Ruby when it was actively working to pull new developers away from Java, developers who were in the habit of using names like AbstractItemManagerFactoryImpl. Look at beginner Swift code posted in Stack Overflow questions, littered with careless force unwraps that are clearly there because that’s the fixit and sort of smells like C or Java, not because they actually thought about what they were doing.

Note in both cases how the center of gravity of the language did •not• move. Note how poorly designed libraries that didn’t fit the language did •not• see adoption. Note how the core community acted as good teachers to the newcomers.

My fear is that a design pattern around dynamic members and calls arise that is used to bypass the perceived initial annoyance of Swifts type system from new developers that come over to Swift and are now starting to try and go native.

That contradicts the historical experience of other languages that have seen “crossover” adoption from developers bringing very different mental models with them.

A Swift library released today that senselessly used dynamic member lookup without good engineering reason would go over just about as well as an AbstractItemManagerFactoryImpl would have gone over with the Ruby community in 2007.

Again, a language feature alone will not erase the good sense of the language’s stewards, or poison its community.

People come to Swift to write Swift. People trying to write Python or Ruby or C in Swift will feel the friction, learn from that, adjust their habits taking a cue from the code they find in the surrounding ecosystem, and learn to think “Swiftly.” This has been the case in every language before Swift; it will not stop being the case now.

Joe hit this nail on the head:

When I began writing Python, I initially named all my variables in camel case. And when I used the Subprocess module, I was passing return codes throughout my code base like it was bash. Even though I didn't know what was idiomatic in Python at that time, I could still sense a code smell in what I was writing. I soon learned that snake case was the standard, so I updated my code. And eventually I learned how to use exceptions and I refactored my code to use them instead of return codes. I doubt that this is unusual when coming to a new language with previous experience in other languages. It seems inevitable, and a normal process for becoming familiar with the idioms of a certain language community. I don't think we need to be so fearful of the community fracturing if people from a dynamic language background come to Swift and initially use those idioms. It will always be more burdensome to do things in a non-idiomatic way, and over time people will learn how to write idiomatic Swift.

Exactly.

Evidence is going to be hard to get since I don’t know any other language like Swift that has done this for the same reasons before. As far as C# goes were they trying to get people from a heavily based dynamic community or help those already in the community?

If a little extra language ceremony helps assuage those fears, I guess I’d consider it. I still think static analysis — starting and mostly ending with boring old syntax coloring — answers most all the concerns people have raised, and this debate is a tempest in a teapot.

I'm unsure of this, but as far as I’m aware Swift language proposals shouldn’t rely on editor features. But like I said I’m unsure of this and if someone could clarify this that would great.

Type inference effectively does, to at least the same degree as this proposal. See my previous post reflecting on why Java chose not to have any sort of type inference:

https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20171127/041926.html

• • •

It’s wise to remember Flon’s Law:

“There is not now, nor has there ever been, nor will there ever be, any programming language in which it is the least bit difficult to write bad code.”

We should design languages to make writing excellent code as pleasant and rewarding as possible. Or more accurately, we should design them so that there exists a way of writing excellent code that is pleasingly well suited to the language and the ecosystem that surrounds it.

Designing languages instead to forcibly prevent or punish bad programming is not merely futile, but in fact actively thwarts that first goal.

Now, how can we use dynamic member lookup in Swift to make its way of writing excellent code more excellent still?

Cheers,

Paul

···

On Dec 7, 2017, at 12:37 AM, Letanyan Arumugam <letanyan.a@gmail.com> wrote:
On Dec 7, 2017, at 10:08 AM, Joe DeCapo via swift-evolution <swift-evolution@swift.org> wrote:
On Dec 7, 2017, at 12:37 AM, Letanyan Arumugam <letanyan.a@gmail.com> wrote:

[masters3d]

Sent from my iPad

For all those reasons, we really do need something like AnyObject dispatch if we care about working with dynamically typed languages. The design I’m suggesting carefully cordons this off into its own struct type, so it doesn’t infect the rest of the type system, and is non-invasive in the compiler.

I am quite familiar with dynamic languages and agree that this is necessary if we are going to fully open up access to these languages from Swift.

Ok, then it appears you agree that something like anyobject dispatch is necessary for effective dynamic language interop.

I strongly urge you to reconsider the decision of that dynamic members must be made available with no indication at usage sites. An indication of dynamic lookup at usage sites aligns very well (IMO) with the rest of Swift (AnyObject lookup aside) by calling attention to code that requires extra care to get right.

I don’t understand this. The proposal is fully type safe, and this approach is completely precedented by AnyObject. Swift’s type system supports many ways to express fallibility, and keeping those decisions orthogonal to this proposal is the right thing to do, because it allows the author of the type to decide what model makes sense for them.

Allowing the author of the type to choose whether the mechanism is hidden or visible is exactly what I don’t want to allow. I think you have the right design regarding types and semantics - the author chooses. But I don’t want these calls to look like ordinary member lookup when I’m reading code.

They inherently have a much greater chance of failure than ordinary member lookup. Further, authors are likely to choose immediate traps or nil IUO as failure modes as forcing users to deal with Optional on every call is likely to be untenable. I believe this behavior should be represented by some kind of syntax at the usage site. I don’t believe it is an undue burden. It would make the dynamic lookup semantic clear to all readers and would help to discourage abuse.

I believe that adding explicit syntax would be counterproductive to your goals, and would not make dynamic lookup syntax more clear. I assume that you would also want the same thing for DynamicCallable too, and operator overloads, subscripts, and every other operation you perform on these values, since they all have the exact same behavior.

If we required some syntax even as minimal as “foo.^bar” and "baz^(42)”, that change would turn this (which uses runtime failing or IUO return values like AnyObject):

  let np = Python.import("numpy")
  let x = np.array([6, 7, 8])
  let y = np.arange(24).reshape(2, 3, 4)
  
  let a = np.ones(3, dtype: np.int32)
  let b = np.linspace(0, pi, 3)
  let c = a+b
  let d = np.exp(c)
  print(d)

into:

  let np = Python.import("numpy")
  let b = np^.array^([6, 7, 8])
  let y = np^.arange^(24)^.reshape^(2, 3, 4)
  
  let a = np^.ones^(3, dtype: np^.int32)
  let b = np^.linspace^(0, pi, 3)
  let c = a+^b
  let d = np^.exp^(c)

This does not improve clarity of code, it merely serves to obfuscate logic. It is immediately apparent from the APIs being used, the API style, and the static types (in Xcode or through static declarations) that this is all Python stuff.

It may be immediately apparent when the types involved are obviously dynamic, such as in this example where Python.import is explicitly used. However, my concern is less about the intended use case of dynamic language interop than I am that this feature will be generally available to all types in Swift.

This is big change from AnyObject dispatch. It opens up the dynamism to types and contexts that are not necessarily obviously using dynamic lookup, callable, etc. Maybe this won’t turn out to be a problem in practice but I still think it’s a legitimate concern.

If dynamism if restricted to subclasses of a DynamicObject type, like Xiaodi suggested earlier, then we can protect ourselves from this dynamic dispatch being generally available to all types in Swift.

I like the idea of a class type as the marker.

If we did expose the protocols that DynamicObject conform to, then I would like to see an attribute that would be required at the declaration site.

@dynamic
class MyTypo: SomeDynamicProtocol {...}

I also like the idea of having
@dynamic subscript(...)

I’d like the declaration site to be explicit.

The calling site should just be dot syntax like AnyObject.

···

On Dec 3, 2017, at 8:41 AM, David Hart via swift-evolution <swift-evolution@swift.org> wrote:

On 3 Dec 2017, at 04:11, Matthew Johnson via swift-evolution <swift-evolution@swift.org> wrote:

On Dec 2, 2017, at 7:40 PM, Chris Lattner <clattner@nondot.org> wrote:
On Dec 2, 2017, at 2:13 PM, Matthew Johnson <matthew@anandabits.com> wrote:

When you start mixing in use of native Swift types like dictionaries (something we want to encourage because they are typed!) you end up with an inconsistent mismash where people would just try adding syntax or applying fixits continuously until the code builds.

Beyond that, it is counterproductive to your goals, because it means that people are far less likely to use to use optional returns. Doing so (which produces a safer result) would cause a double tax in syntax, and would be a confusing jumble. I can’t bring myself to do the whole example above, one line - just converting member lookup syntax but not callable syntax - would end up:

  let y = np^.arange?^(24)^.reshape^?(2, 3, 4)

If you made DynamicCallable also return optional it would be:

  let y = np^.arange?^(24)?^.reshape^?(2, 3, 4)!

or something. This is such madness that no one would do that.

Yes, I agree. The interaction with optional chaining makes it unworkable. I hadn’t thought that all the way through. Thank you for indulging in the discussion about this idea.

I’m uncertain what the right answer is. I’m still not really comfortable with opening up dynamic lookup to any user-defined type without some way to indicate to readers that dynamic lookup is happening in a piece of code. Maybe there is a less localized annotation that would indicate dynamic lookup is in effect for a larger chunk of code.

I think that making dynamic calls syntactically different to readers is going too far in the direction of safety. Plus, it makes the language inconsistent as we already have AnyObject dispatch with exactly the same syntax. But I understand why you would want it if *any* type could end up being conformed to a dynamic lookupable/callable protocol. Like said above, I think that a DynamicObject type is enough protection to not bother making the syntax heavier at the point of use.

One idea that hasn’t been explored yet is introducing a dynamic lookup effect. That would provide clean syntax at the expression level while still making it clear that dynamic lookup is happening. This probably isn’t the right approach as it would be viral and wouldn’t even indicate that the code in a specific function body even contains a dynamic lookup. I mention it mostly to broaden the conversation and thought space about what kind of approach might address my concerns without cluttering up expressions.

Another, potentially more viable approach would be to use an approach similar to try and the proposed async of a statement modifier (possibly also allowed at the expression level as with try and async). This could be used in conjunction with an effect as mentioned above but could also be used independently even though there isn’t a current precedent for that. Your example written with this approach might be:

  let np = Python.import("numpy")
  let b = dynamic np.array([6, 7, 8])
  let y = dynamic np.arange(24).reshape(2, 3, 4)
  
  let a = dynamic np.ones(3, dtype: np.int32)
  let b = dynamic np.linspace(0, pi, 3)
  let c = dynamic a + b
  let d = dynamic np.exp(c)

Note: `dynamic` is just a straw man here. This is obviously a lot of boilerplate repetition of the same modifier. I would also be perfectly happy allowing this annotation to apply to a block of code or even a whole function body. Then it might be:

dynamic {
  let np = Python.import("numpy")
  let b = np.array([6, 7, 8])
  let y = np.arange(24).reshape(2, 3, 4)
  
  let a = np.ones(3, dtype: np.int32)
  let b = np.linspace(0, pi, 3)
  let c = a + b
  let d = np.exp(c)
}

The most obvious objection to this is that it introduces nesting and does so in a way that only very indirectly influences control flow (through the potential for dynamic failure).

My objective certainly isn’t to make code ugly and obfuscate logic. It is simply to make the usage of dynamic features that are prone to failure at runtime immediately clear to a reader. This should be as lightweight as possible while still providing valuable information to the reader.

Swift already has a dynamic member lookup feature, "AnyObject dispatch" which does not use additional punctuation, so this would break precedent.

I would prefer if dynamic lookup were visible with AnyObject as well. For that reason I don’t believe it makes a good precedent to follow. In fact, I would prefer to see us go the other direction and perhaps even consider revising dynamic lookup syntax for AnyObject in the future.

This is definitely not going to happen. The change Doug mentioned is to have AnyObject lookup return optional instead of IUO, which forces ? on the clients. Adding other syntax (like you’re suggesting) is certainly not going to happen.

The entire point of AnyObject dispatch is to improve syntactic elegance and clarity of code using it. There is no other reason to exist. Making code that uses it syntactically onerous completely defeats the point of having it in the first place, as I’ve mentioned before.

I agree. The interaction with optional chaining is significantly more onerous than I had considered. I should have worked through an example on my own. I do hope you will consider a less localized approach to usage-site annotation though.

Furthermore, your premise that Swift does not have invisibly failable operations is plainly wrong. Array subscript and even integer addition can fail.

I am well aware of these behaviors. The difference IMO is that programmers tend to be well aware of these preconditions even if they also often choose to ignore them. Dynamic lookup will not be so clear. This is especially true if people use it with types that also have an API available through static lookup.

Even the behavior of AnyObject was carefully designed and considered, and were really really good reasons for it returning IUO.

I am not trying to call into question the choices made in the past. Swift wouldn’t be the great language with a bright future that it is today without an incredibly successful migration of a large user base from Objective-C to Swift. This is a huge accomplishment and couldn’t have happened without making really good decisions about some really hard tradeoffs.

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[jrgoodle (Joe DeCapo)]

One common criticism people keep bringing up is that people will misuse this and end up exposing dynamic api's into general use (and without notation at the call site that draws attention to this, the user may accidentally invoke something dynamic they didn't intend to). I feel like there is enough prior art in current Swift libraries to disabuse us of this notion. Basically every library I've seen goes out of its way to provide a safe api and leverage Swift's strong type system in their public api's. We don't see libraries using force unwraps willy nilly to hide the fact that a non-optional return type is actually coming from an optional value, but this is something that has been possible since day one and is trivially easy to do.

There are essentially 2 "bad" use cases for consumers of these dynamic features: 1) as the consumer of a library that exposes these dynamic features as public api; and 2) as the creator of a library/executable that uses them irresponsibly internally. For (1) it's a very easy decision to simply not use a Swift library that exposes dynamic features like this externally if you object to it, and instead use an alternative. I highly doubt this would become idiomatic in the Swift community, because the enthusiasm for Swift is mostly based on the strong typing it provides. For (2) it is within the creator's complete control to refactor the code to work the preferred way. The moment there's an unintended invocation of a dynamic call, it can simply be wrapped in a more strongly typed Swift wrapper.

As for the autocompletion and jump-to-definition capabilities of the state of the art in Python, I've been working on a Python project for a few years and until recently only used Atom, which provided code completion with a plug-in, but I could have gotten by without it. I recently started using PyCharm, and although it has jump-to-definition capabilities, the dynamic nature of the language means that there's almost always multiple different possible matches, and most often the top choices are the wrong ones. I don't see how Swift could improve on this situation, and for the case of providing quick help, it may actively provide the wrong information. This is not to say that autocompletion and the like aren't worthwhile goals, but I don't think they're really an important barrier to initial adoption of a way to call into dynamic languages.

I get the sense that this is a minor syntactic sugar feature that would probably mostly be used by implementers of library wrappers to provide Swifty api's that call into dynamic language api's. So the concerns about autocompletion and pervasive exposure of these dynamic aspects more generally really feel misplaced. When writing a wrapper around a REST api, I get zero autocompletion when specifying the JSON models, but that in no way hampers my ability to wrap these api's. Sure, there may be more runtime bugs that I have to sort out when initially implementing things, but that's inherent in the nature of dealing with a dynamic programming model. I imagine it will be the same for Swift libraries wrapping dynamic language libraries.

···

On Dec 3, 2017, at 10:41 AM, David Hart via swift-evolution <swift-evolution@swift.org> wrote:

On 3 Dec 2017, at 04:11, Matthew Johnson via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Sent from my iPad

On Dec 2, 2017, at 7:40 PM, Chris Lattner <clattner@nondot.org <mailto:clattner@nondot.org>> wrote:

On Dec 2, 2017, at 2:13 PM, Matthew Johnson <matthew@anandabits.com <mailto:matthew@anandabits.com>> wrote:

For all those reasons, we really do need something like AnyObject dispatch if we care about working with dynamically typed languages. The design I’m suggesting carefully cordons this off into its own struct type, so it doesn’t infect the rest of the type system, and is non-invasive in the compiler.

I am quite familiar with dynamic languages and agree that this is necessary if we are going to fully open up access to these languages from Swift.

Ok, then it appears you agree that something like anyobject dispatch is necessary for effective dynamic language interop.

I strongly urge you to reconsider the decision of that dynamic members must be made available with no indication at usage sites. An indication of dynamic lookup at usage sites aligns very well (IMO) with the rest of Swift (AnyObject lookup aside) by calling attention to code that requires extra care to get right.

I don’t understand this. The proposal is fully type safe, and this approach is completely precedented by AnyObject. Swift’s type system supports many ways to express fallibility, and keeping those decisions orthogonal to this proposal is the right thing to do, because it allows the author of the type to decide what model makes sense for them.

Allowing the author of the type to choose whether the mechanism is hidden or visible is exactly what I don’t want to allow. I think you have the right design regarding types and semantics - the author chooses. But I don’t want these calls to look like ordinary member lookup when I’m reading code.

They inherently have a much greater chance of failure than ordinary member lookup. Further, authors are likely to choose immediate traps or nil IUO as failure modes as forcing users to deal with Optional on every call is likely to be untenable. I believe this behavior should be represented by some kind of syntax at the usage site. I don’t believe it is an undue burden. It would make the dynamic lookup semantic clear to all readers and would help to discourage abuse.

I believe that adding explicit syntax would be counterproductive to your goals, and would not make dynamic lookup syntax more clear. I assume that you would also want the same thing for DynamicCallable too, and operator overloads, subscripts, and every other operation you perform on these values, since they all have the exact same behavior.

If we required some syntax even as minimal as “foo.^bar” and "baz^(42)”, that change would turn this (which uses runtime failing or IUO return values like AnyObject):

  let np = Python.import("numpy")
  let x = np.array([6, 7, 8])
  let y = np.arange(24).reshape(2, 3, 4)
  
  let a = np.ones(3, dtype: np.int32)
  let b = np.linspace(0, pi, 3)
  let c = a+b
  let d = np.exp(c)
  print(d)

into:

  let np = Python.import("numpy")
  let b = np^.array^([6, 7, 8])
  let y = np^.arange^(24)^.reshape^(2, 3, 4)
  
  let a = np^.ones^(3, dtype: np^.int32)
  let b = np^.linspace^(0, pi, 3)
  let c = a+^b
  let d = np^.exp^(c)

This does not improve clarity of code, it merely serves to obfuscate logic. It is immediately apparent from the APIs being used, the API style, and the static types (in Xcode or through static declarations) that this is all Python stuff.

It may be immediately apparent when the types involved are obviously dynamic, such as in this example where Python.import is explicitly used. However, my concern is less about the intended use case of dynamic language interop than I am that this feature will be generally available to all types in Swift.

This is big change from AnyObject dispatch. It opens up the dynamism to types and contexts that are not necessarily obviously using dynamic lookup, callable, etc. Maybe this won’t turn out to be a problem in practice but I still think it’s a legitimate concern.

If dynamism if restricted to subclasses of a DynamicObject type, like Xiaodi suggested earlier, then we can protect ourselves from this dynamic dispatch being generally available to all types in Swift.

When you start mixing in use of native Swift types like dictionaries (something we want to encourage because they are typed!) you end up with an inconsistent mismash where people would just try adding syntax or applying fixits continuously until the code builds.

Beyond that, it is counterproductive to your goals, because it means that people are far less likely to use to use optional returns. Doing so (which produces a safer result) would cause a double tax in syntax, and would be a confusing jumble. I can’t bring myself to do the whole example above, one line - just converting member lookup syntax but not callable syntax - would end up:

  let y = np^.arange?^(24)^.reshape^?(2, 3, 4)

If you made DynamicCallable also return optional it would be:

  let y = np^.arange?^(24)?^.reshape^?(2, 3, 4)!

or something. This is such madness that no one would do that.

Yes, I agree. The interaction with optional chaining makes it unworkable. I hadn’t thought that all the way through. Thank you for indulging in the discussion about this idea.

I’m uncertain what the right answer is. I’m still not really comfortable with opening up dynamic lookup to any user-defined type without some way to indicate to readers that dynamic lookup is happening in a piece of code. Maybe there is a less localized annotation that would indicate dynamic lookup is in effect for a larger chunk of code.

I think that making dynamic calls syntactically different to readers is going too far in the direction of safety. Plus, it makes the language inconsistent as we already have AnyObject dispatch with exactly the same syntax. But I understand why you would want it if *any* type could end up being conformed to a dynamic lookupable/callable protocol. Like said above, I think that a DynamicObject type is enough protection to not bother making the syntax heavier at the point of use.

One idea that hasn’t been explored yet is introducing a dynamic lookup effect. That would provide clean syntax at the expression level while still making it clear that dynamic lookup is happening. This probably isn’t the right approach as it would be viral and wouldn’t even indicate that the code in a specific function body even contains a dynamic lookup. I mention it mostly to broaden the conversation and thought space about what kind of approach might address my concerns without cluttering up expressions.

Another, potentially more viable approach would be to use an approach similar to try and the proposed async of a statement modifier (possibly also allowed at the expression level as with try and async). This could be used in conjunction with an effect as mentioned above but could also be used independently even though there isn’t a current precedent for that. Your example written with this approach might be:

  let np = Python.import("numpy")
  let b = dynamic np.array([6, 7, 8])
  let y = dynamic np.arange(24).reshape(2, 3, 4)
  
  let a = dynamic np.ones(3, dtype: np.int32)
  let b = dynamic np.linspace(0, pi, 3)
  let c = dynamic a + b
  let d = dynamic np.exp(c)

Note: `dynamic` is just a straw man here. This is obviously a lot of boilerplate repetition of the same modifier. I would also be perfectly happy allowing this annotation to apply to a block of code or even a whole function body. Then it might be:

dynamic {
  let np = Python.import("numpy")
  let b = np.array([6, 7, 8])
  let y = np.arange(24).reshape(2, 3, 4)
  
  let a = np.ones(3, dtype: np.int32)
  let b = np.linspace(0, pi, 3)
  let c = a + b
  let d = np.exp(c)
}

The most obvious objection to this is that it introduces nesting and does so in a way that only very indirectly influences control flow (through the potential for dynamic failure).

My objective certainly isn’t to make code ugly and obfuscate logic. It is simply to make the usage of dynamic features that are prone to failure at runtime immediately clear to a reader. This should be as lightweight as possible while still providing valuable information to the reader.

Swift already has a dynamic member lookup feature, "AnyObject dispatch" which does not use additional punctuation, so this would break precedent.

I would prefer if dynamic lookup were visible with AnyObject as well. For that reason I don’t believe it makes a good precedent to follow. In fact, I would prefer to see us go the other direction and perhaps even consider revising dynamic lookup syntax for AnyObject in the future.

This is definitely not going to happen. The change Doug mentioned is to have AnyObject lookup return optional instead of IUO, which forces ? on the clients. Adding other syntax (like you’re suggesting) is certainly not going to happen.

The entire point of AnyObject dispatch is to improve syntactic elegance and clarity of code using it. There is no other reason to exist. Making code that uses it syntactically onerous completely defeats the point of having it in the first place, as I’ve mentioned before.

I agree. The interaction with optional chaining is significantly more onerous than I had considered. I should have worked through an example on my own. I do hope you will consider a less localized approach to usage-site annotation though.

Furthermore, your premise that Swift does not have invisibly failable operations is plainly wrong. Array subscript and even integer addition can fail.

I am well aware of these behaviors. The difference IMO is that programmers tend to be well aware of these preconditions even if they also often choose to ignore them. Dynamic lookup will not be so clear. This is especially true if people use it with types that also have an API available through static lookup.

Even the behavior of AnyObject was carefully designed and considered, and were really really good reasons for it returning IUO.

I am not trying to call into question the choices made in the past. Swift wouldn’t be the great language with a bright future that it is today without an incredibly successful migration of a large user base from Objective-C to Swift. This is a huge accomplishment and couldn’t have happened without making really good decisions about some really hard tradeoffs.

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

[jrgoodle (Joe DeCapo)]

I like Chris's addition to the proposal about only allowing conformance on the original type definition, rather than requiring a class type.

https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438

In the discussion cycle, there was significant concern about abuse of this feature, particularly if someone retroactively conforms a type to DynamicMemberLookupProtocol. For this reason, the compiler only permits conformance of this protocol on the original type definition, not extensions. While the potential for abuse has never been a strong guiding principle for Swift features (and many features can be abused, e.g. operator overloading, emoji identifiers, AnyObject lookup, ImplicitlyUnwrappedOptional, and many more) there is a strong argument that dynamic behavior is a core property of a type that should be part of its declaration. If for some reason there becomes a reason to relax this requirement, we can evaluate that as a separate proposal based on its own merits. See the "Alternatives Considered" section below for further ways to reduce potential for abuse.

It would be a shame if structs were excluded from these capabilities.

···

On Dec 3, 2017, at 11:18 AM, Jose Cheyo Jimenez via swift-evolution <swift-evolution@swift.org> wrote:

On Dec 3, 2017, at 8:41 AM, David Hart via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On 3 Dec 2017, at 04:11, Matthew Johnson via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Sent from my iPad

On Dec 2, 2017, at 7:40 PM, Chris Lattner <clattner@nondot.org <mailto:clattner@nondot.org>> wrote:

On Dec 2, 2017, at 2:13 PM, Matthew Johnson <matthew@anandabits.com <mailto:matthew@anandabits.com>> wrote:

For all those reasons, we really do need something like AnyObject dispatch if we care about working with dynamically typed languages. The design I’m suggesting carefully cordons this off into its own struct type, so it doesn’t infect the rest of the type system, and is non-invasive in the compiler.

I am quite familiar with dynamic languages and agree that this is necessary if we are going to fully open up access to these languages from Swift.

Ok, then it appears you agree that something like anyobject dispatch is necessary for effective dynamic language interop.

I strongly urge you to reconsider the decision of that dynamic members must be made available with no indication at usage sites. An indication of dynamic lookup at usage sites aligns very well (IMO) with the rest of Swift (AnyObject lookup aside) by calling attention to code that requires extra care to get right.

I don’t understand this. The proposal is fully type safe, and this approach is completely precedented by AnyObject. Swift’s type system supports many ways to express fallibility, and keeping those decisions orthogonal to this proposal is the right thing to do, because it allows the author of the type to decide what model makes sense for them.

Allowing the author of the type to choose whether the mechanism is hidden or visible is exactly what I don’t want to allow. I think you have the right design regarding types and semantics - the author chooses. But I don’t want these calls to look like ordinary member lookup when I’m reading code.

They inherently have a much greater chance of failure than ordinary member lookup. Further, authors are likely to choose immediate traps or nil IUO as failure modes as forcing users to deal with Optional on every call is likely to be untenable. I believe this behavior should be represented by some kind of syntax at the usage site. I don’t believe it is an undue burden. It would make the dynamic lookup semantic clear to all readers and would help to discourage abuse.

I believe that adding explicit syntax would be counterproductive to your goals, and would not make dynamic lookup syntax more clear. I assume that you would also want the same thing for DynamicCallable too, and operator overloads, subscripts, and every other operation you perform on these values, since they all have the exact same behavior.

If we required some syntax even as minimal as “foo.^bar” and "baz^(42)”, that change would turn this (which uses runtime failing or IUO return values like AnyObject):

  let np = Python.import("numpy")
  let x = np.array([6, 7, 8])
  let y = np.arange(24).reshape(2, 3, 4)
  
  let a = np.ones(3, dtype: np.int32)
  let b = np.linspace(0, pi, 3)
  let c = a+b
  let d = np.exp(c)
  print(d)

into:

  let np = Python.import("numpy")
  let b = np^.array^([6, 7, 8])
  let y = np^.arange^(24)^.reshape^(2, 3, 4)
  
  let a = np^.ones^(3, dtype: np^.int32)
  let b = np^.linspace^(0, pi, 3)
  let c = a+^b
  let d = np^.exp^(c)

This does not improve clarity of code, it merely serves to obfuscate logic. It is immediately apparent from the APIs being used, the API style, and the static types (in Xcode or through static declarations) that this is all Python stuff.

It may be immediately apparent when the types involved are obviously dynamic, such as in this example where Python.import is explicitly used. However, my concern is less about the intended use case of dynamic language interop than I am that this feature will be generally available to all types in Swift.

This is big change from AnyObject dispatch. It opens up the dynamism to types and contexts that are not necessarily obviously using dynamic lookup, callable, etc. Maybe this won’t turn out to be a problem in practice but I still think it’s a legitimate concern.

If dynamism if restricted to subclasses of a DynamicObject type, like Xiaodi suggested earlier, then we can protect ourselves from this dynamic dispatch being generally available to all types in Swift.

I like the idea of a class type as the marker.

If we did expose the protocols that DynamicObject conform to, then I would like to see an attribute that would be required at the declaration site.

@dynamic
class MyTypo: SomeDynamicProtocol {...}

I also like the idea of having
@dynamic subscript(...)

I’d like the declaration site to be explicit.

The calling site should just be dot syntax like AnyObject.

When you start mixing in use of native Swift types like dictionaries (something we want to encourage because they are typed!) you end up with an inconsistent mismash where people would just try adding syntax or applying fixits continuously until the code builds.

Beyond that, it is counterproductive to your goals, because it means that people are far less likely to use to use optional returns. Doing so (which produces a safer result) would cause a double tax in syntax, and would be a confusing jumble. I can’t bring myself to do the whole example above, one line - just converting member lookup syntax but not callable syntax - would end up:

  let y = np^.arange?^(24)^.reshape^?(2, 3, 4)

If you made DynamicCallable also return optional it would be:

  let y = np^.arange?^(24)?^.reshape^?(2, 3, 4)!

or something. This is such madness that no one would do that.

Yes, I agree. The interaction with optional chaining makes it unworkable. I hadn’t thought that all the way through. Thank you for indulging in the discussion about this idea.

I’m uncertain what the right answer is. I’m still not really comfortable with opening up dynamic lookup to any user-defined type without some way to indicate to readers that dynamic lookup is happening in a piece of code. Maybe there is a less localized annotation that would indicate dynamic lookup is in effect for a larger chunk of code.

I think that making dynamic calls syntactically different to readers is going too far in the direction of safety. Plus, it makes the language inconsistent as we already have AnyObject dispatch with exactly the same syntax. But I understand why you would want it if *any* type could end up being conformed to a dynamic lookupable/callable protocol. Like said above, I think that a DynamicObject type is enough protection to not bother making the syntax heavier at the point of use.

One idea that hasn’t been explored yet is introducing a dynamic lookup effect. That would provide clean syntax at the expression level while still making it clear that dynamic lookup is happening. This probably isn’t the right approach as it would be viral and wouldn’t even indicate that the code in a specific function body even contains a dynamic lookup. I mention it mostly to broaden the conversation and thought space about what kind of approach might address my concerns without cluttering up expressions.

Another, potentially more viable approach would be to use an approach similar to try and the proposed async of a statement modifier (possibly also allowed at the expression level as with try and async). This could be used in conjunction with an effect as mentioned above but could also be used independently even though there isn’t a current precedent for that. Your example written with this approach might be:

  let np = Python.import("numpy")
  let b = dynamic np.array([6, 7, 8])
  let y = dynamic np.arange(24).reshape(2, 3, 4)
  
  let a = dynamic np.ones(3, dtype: np.int32)
  let b = dynamic np.linspace(0, pi, 3)
  let c = dynamic a + b
  let d = dynamic np.exp(c)

Note: `dynamic` is just a straw man here. This is obviously a lot of boilerplate repetition of the same modifier. I would also be perfectly happy allowing this annotation to apply to a block of code or even a whole function body. Then it might be:

dynamic {
  let np = Python.import("numpy")
  let b = np.array([6, 7, 8])
  let y = np.arange(24).reshape(2, 3, 4)
  
  let a = np.ones(3, dtype: np.int32)
  let b = np.linspace(0, pi, 3)
  let c = a + b
  let d = np.exp(c)
}

The most obvious objection to this is that it introduces nesting and does so in a way that only very indirectly influences control flow (through the potential for dynamic failure).

My objective certainly isn’t to make code ugly and obfuscate logic. It is simply to make the usage of dynamic features that are prone to failure at runtime immediately clear to a reader. This should be as lightweight as possible while still providing valuable information to the reader.

Swift already has a dynamic member lookup feature, "AnyObject dispatch" which does not use additional punctuation, so this would break precedent.

I would prefer if dynamic lookup were visible with AnyObject as well. For that reason I don’t believe it makes a good precedent to follow. In fact, I would prefer to see us go the other direction and perhaps even consider revising dynamic lookup syntax for AnyObject in the future.

This is definitely not going to happen. The change Doug mentioned is to have AnyObject lookup return optional instead of IUO, which forces ? on the clients. Adding other syntax (like you’re suggesting) is certainly not going to happen.

The entire point of AnyObject dispatch is to improve syntactic elegance and clarity of code using it. There is no other reason to exist. Making code that uses it syntactically onerous completely defeats the point of having it in the first place, as I’ve mentioned before.

I agree. The interaction with optional chaining is significantly more onerous than I had considered. I should have worked through an example on my own. I do hope you will consider a less localized approach to usage-site annotation though.

Furthermore, your premise that Swift does not have invisibly failable operations is plainly wrong. Array subscript and even integer addition can fail.

I am well aware of these behaviors. The difference IMO is that programmers tend to be well aware of these preconditions even if they also often choose to ignore them. Dynamic lookup will not be so clear. This is especially true if people use it with types that also have an API available through static lookup.

Even the behavior of AnyObject was carefully designed and considered, and were really really good reasons for it returning IUO.

I am not trying to call into question the choices made in the past. Swift wouldn’t be the great language with a bright future that it is today without an incredibly successful migration of a large user base from Objective-C to Swift. This is a huge accomplishment and couldn’t have happened without making really good decisions about some really hard tradeoffs.

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

[anandabits (Matthew Johnson)]

Sent from my iPad

For all those reasons, we really do need something like AnyObject dispatch if we care about working with dynamically typed languages. The design I’m suggesting carefully cordons this off into its own struct type, so it doesn’t infect the rest of the type system, and is non-invasive in the compiler.

I am quite familiar with dynamic languages and agree that this is necessary if we are going to fully open up access to these languages from Swift.

Ok, then it appears you agree that something like anyobject dispatch is necessary for effective dynamic language interop.

I strongly urge you to reconsider the decision of that dynamic members must be made available with no indication at usage sites. An indication of dynamic lookup at usage sites aligns very well (IMO) with the rest of Swift (AnyObject lookup aside) by calling attention to code that requires extra care to get right.

I don’t understand this. The proposal is fully type safe, and this approach is completely precedented by AnyObject. Swift’s type system supports many ways to express fallibility, and keeping those decisions orthogonal to this proposal is the right thing to do, because it allows the author of the type to decide what model makes sense for them.

Allowing the author of the type to choose whether the mechanism is hidden or visible is exactly what I don’t want to allow. I think you have the right design regarding types and semantics - the author chooses. But I don’t want these calls to look like ordinary member lookup when I’m reading code.

They inherently have a much greater chance of failure than ordinary member lookup. Further, authors are likely to choose immediate traps or nil IUO as failure modes as forcing users to deal with Optional on every call is likely to be untenable. I believe this behavior should be represented by some kind of syntax at the usage site. I don’t believe it is an undue burden. It would make the dynamic lookup semantic clear to all readers and would help to discourage abuse.

I believe that adding explicit syntax would be counterproductive to your goals, and would not make dynamic lookup syntax more clear. I assume that you would also want the same thing for DynamicCallable too, and operator overloads, subscripts, and every other operation you perform on these values, since they all have the exact same behavior.

If we required some syntax even as minimal as “foo.^bar” and "baz^(42)”, that change would turn this (which uses runtime failing or IUO return values like AnyObject):

  let np = Python.import("numpy")
  let x = np.array([6, 7, 8])
  let y = np.arange(24).reshape(2, 3, 4)
  
  let a = np.ones(3, dtype: np.int32)
  let b = np.linspace(0, pi, 3)
  let c = a+b
  let d = np.exp(c)
  print(d)

into:

  let np = Python.import("numpy")
  let b = np^.array^([6, 7, 8])
  let y = np^.arange^(24)^.reshape^(2, 3, 4)
  
  let a = np^.ones^(3, dtype: np^.int32)
  let b = np^.linspace^(0, pi, 3)
  let c = a+^b
  let d = np^.exp^(c)

This does not improve clarity of code, it merely serves to obfuscate logic. It is immediately apparent from the APIs being used, the API style, and the static types (in Xcode or through static declarations) that this is all Python stuff.

It may be immediately apparent when the types involved are obviously dynamic, such as in this example where Python.import is explicitly used. However, my concern is less about the intended use case of dynamic language interop than I am that this feature will be generally available to all types in Swift.

This is big change from AnyObject dispatch. It opens up the dynamism to types and contexts that are not necessarily obviously using dynamic lookup, callable, etc. Maybe this won’t turn out to be a problem in practice but I still think it’s a legitimate concern.

If dynamism if restricted to subclasses of a DynamicObject type, like Xiaodi suggested earlier, then we can protect ourselves from this dynamic dispatch being generally available to all types in Swift.

Chris has good reasons for not wanting to use a class. More generally, designs that require subclassing a specific superclass are usually a bad idea. They should only be used with very specific and very good reasons. I’m not sure what this would buy us here. How specifically do you think this helps in ways that requiring conformance to be stated in the original type declaration will not?

···

Sent from my iPad

On Dec 3, 2017, at 10:40 AM, David Hart <david@hartbit.com> wrote:

On 3 Dec 2017, at 04:11, Matthew Johnson via swift-evolution <swift-evolution@swift.org> wrote:

On Dec 2, 2017, at 7:40 PM, Chris Lattner <clattner@nondot.org> wrote:
On Dec 2, 2017, at 2:13 PM, Matthew Johnson <matthew@anandabits.com> wrote:

When you start mixing in use of native Swift types like dictionaries (something we want to encourage because they are typed!) you end up with an inconsistent mismash where people would just try adding syntax or applying fixits continuously until the code builds.

Beyond that, it is counterproductive to your goals, because it means that people are far less likely to use to use optional returns. Doing so (which produces a safer result) would cause a double tax in syntax, and would be a confusing jumble. I can’t bring myself to do the whole example above, one line - just converting member lookup syntax but not callable syntax - would end up:

  let y = np^.arange?^(24)^.reshape^?(2, 3, 4)

If you made DynamicCallable also return optional it would be:

  let y = np^.arange?^(24)?^.reshape^?(2, 3, 4)!

or something. This is such madness that no one would do that.

Yes, I agree. The interaction with optional chaining makes it unworkable. I hadn’t thought that all the way through. Thank you for indulging in the discussion about this idea.

I’m uncertain what the right answer is. I’m still not really comfortable with opening up dynamic lookup to any user-defined type without some way to indicate to readers that dynamic lookup is happening in a piece of code. Maybe there is a less localized annotation that would indicate dynamic lookup is in effect for a larger chunk of code.

I think that making dynamic calls syntactically different to readers is going too far in the direction of safety. Plus, it makes the language inconsistent as we already have AnyObject dispatch with exactly the same syntax. But I understand why you would want it if *any* type could end up being conformed to a dynamic lookupable/callable protocol. Like said above, I think that a DynamicObject type is enough protection to not bother making the syntax heavier at the point of use.

One idea that hasn’t been explored yet is introducing a dynamic lookup effect. That would provide clean syntax at the expression level while still making it clear that dynamic lookup is happening. This probably isn’t the right approach as it would be viral and wouldn’t even indicate that the code in a specific function body even contains a dynamic lookup. I mention it mostly to broaden the conversation and thought space about what kind of approach might address my concerns without cluttering up expressions.

Another, potentially more viable approach would be to use an approach similar to try and the proposed async of a statement modifier (possibly also allowed at the expression level as with try and async). This could be used in conjunction with an effect as mentioned above but could also be used independently even though there isn’t a current precedent for that. Your example written with this approach might be:

  let np = Python.import("numpy")
  let b = dynamic np.array([6, 7, 8])
  let y = dynamic np.arange(24).reshape(2, 3, 4)
  
  let a = dynamic np.ones(3, dtype: np.int32)
  let b = dynamic np.linspace(0, pi, 3)
  let c = dynamic a + b
  let d = dynamic np.exp(c)

Note: `dynamic` is just a straw man here. This is obviously a lot of boilerplate repetition of the same modifier. I would also be perfectly happy allowing this annotation to apply to a block of code or even a whole function body. Then it might be:

dynamic {
  let np = Python.import("numpy")
  let b = np.array([6, 7, 8])
  let y = np.arange(24).reshape(2, 3, 4)
  
  let a = np.ones(3, dtype: np.int32)
  let b = np.linspace(0, pi, 3)
  let c = a + b
  let d = np.exp(c)
}

The most obvious objection to this is that it introduces nesting and does so in a way that only very indirectly influences control flow (through the potential for dynamic failure).

My objective certainly isn’t to make code ugly and obfuscate logic. It is simply to make the usage of dynamic features that are prone to failure at runtime immediately clear to a reader. This should be as lightweight as possible while still providing valuable information to the reader.

Swift already has a dynamic member lookup feature, "AnyObject dispatch" which does not use additional punctuation, so this would break precedent.

I would prefer if dynamic lookup were visible with AnyObject as well. For that reason I don’t believe it makes a good precedent to follow. In fact, I would prefer to see us go the other direction and perhaps even consider revising dynamic lookup syntax for AnyObject in the future.

This is definitely not going to happen. The change Doug mentioned is to have AnyObject lookup return optional instead of IUO, which forces ? on the clients. Adding other syntax (like you’re suggesting) is certainly not going to happen.

The entire point of AnyObject dispatch is to improve syntactic elegance and clarity of code using it. There is no other reason to exist. Making code that uses it syntactically onerous completely defeats the point of having it in the first place, as I’ve mentioned before.

I agree. The interaction with optional chaining is significantly more onerous than I had considered. I should have worked through an example on my own. I do hope you will consider a less localized approach to usage-site annotation though.

Furthermore, your premise that Swift does not have invisibly failable operations is plainly wrong. Array subscript and even integer addition can fail.

I am well aware of these behaviors. The difference IMO is that programmers tend to be well aware of these preconditions even if they also often choose to ignore them. Dynamic lookup will not be so clear. This is especially true if people use it with types that also have an API available through static lookup.

Even the behavior of AnyObject was carefully designed and considered, and were really really good reasons for it returning IUO.

I am not trying to call into question the choices made in the past. Swift wouldn’t be the great language with a bright future that it is today without an incredibly successful migration of a large user base from Objective-C to Swift. This is a huge accomplishment and couldn’t have happened without making really good decisions about some really hard tradeoffs.

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution