The proposal itself has grown a lot of detail as I've worked through all of the issues, and narrowed so that the proposed change neither adds new syntax nor affects source compatibility. Rather, it implicitly opens in cases that would have failed to type-check previously, getting us out of the "existential trap" without increasing the effective surface area of the language or changing the semantics of well-formed code.
And after a lot of thinking about it, I don't want to propose an explicit opening syntax. There's more detail in Alternatives Considered now about what an explicit opening syntax would mean for the language, and my conclusion here is that adding such a feature introduces a significant amount of surface-language complexity without a correspondingly significant improvement in expressiveness. The proposal as written is effectively invisible---and that's a benefit we want to keep.
A short change list for this revision:
Describe contravariant erasure for parameters
Describe the limitation on implicit existential opening to maintain order of evaluation
Avoid opening an existential argument when the existential type already satisfies the conformance requirements of the corresponding generic parameter, to better maintain source compatibility
Introduce as any P and as? any P as syntaxes to suppress the implicit opening of an existential value.
Added discussion on the relationship with some parameters (SE-0341).
Expand discussion of an explicit opening syntax.
Please give it a read-through or try out the toolchain (macOS). I'd love to hear your thoughts.
I get the preference for a proposal that adds (almost) no new syntax. However, in introducing as any P, the lack of support for its dual as some P is only accentuated, so if the latter is not to be supported I'd think there'd need to be some good diagnostics tailored to that.
As you note, there is an expressivity gain within a single function when one has an explicit existential opening syntax, even as it would have to restricted not to 'escape' the dynamic type; to my mind, this would still be a nice-to-have, and I do wonder about the stated goal of not exposing "the notion of opened types" to the end user at all:
I am not sure how one might explain what's going on to the end user without invoking this concept while not muddying the distinction between some P and any P. If naming this operation (and, even, having an explicit syntax for it) can help users understand the behavior they observe and the rationale for its restrictions, then taking the approach of not exposing it might paint us into a corner much as was the case when the language formerly did not expose the distinction between the protocol P and the existential type P, etc.
Thank you for the revised and comprehensive revision of the initial pitch!
One thing I still have trouble understanding is how ‘type(of:)’ would stop relying on special type-checking rules. This function currently takes a parameter of type , which has no constraints. So according to my comprehension of Avoid opening when the existential type satisfies requirements that should result in the existential itself being passed to the function.
That's a good point. You've also made we think that we don't need the as any P syntax at all. I added it because we wanted a way to suppress the implicit opening for cases where we want to pass the existential box. In the course of my implementation work on this, I needed to add as any P in just a handful of places to keep operating on the box. But now that we have the check to only implicitly open when the existential type doesn't fulfill the generic requirements, all of those as any P's I added are unnecessary. I think with the latest rules as any P can only either break code (any P does not conform to P) or push around overload resolution in very rare circumstances. Those don't seem useful enough to specify this capability, especially given the potential for confusion you cite.
[EDIT: I've updated the proposal to remove this special syntax]
The best short explanation I have is "if the box doesn't meet the requirements, we reach into the box to pull out the underlying concrete type that does meet the requirements".
If we wanted to add the explicit opening syntax later, we could: then this feature becomes an inference rule that takes something like:
and interprets a call getA(anyP) as getA(anyP as some P) as any Q.
Even if we don't have the syntax in the language, we could use it for exposition the way we talk about "reverse generics" as being an explanation for opaque result types, despite us not necessarily ever wanting to add such a feature to the language.
Ah, you're right, I think we lost type(of:) along the way. I think you were trying to tell me that on the pull request and I misunderstood, sorry. I'm okay with that trade-off: this new source-compatible formulation is a much better way to address this gap, and type(of:) remaining special is reasonable in my opinion.
[EDIT: I've updated the proposal to eliminate the comment about subsuming the special type(of:) behavior]
I think I agree with this analysis. The "only open when existential doesn't satisfy constraints" rule solves the issues I was thinking of that had me advocate for as any P initially. It would still be nice to have as some P (and it feels to me like it would a fairly natural part of this proposal).
I agree @xwu's intuition here regarding explicit syntax for this operation:
I think the pitch is good. If I have a concern, it's more broadly that I'm not clear how "The Swift Programming Language" will be amended to make all of these things clear. If you already know Swift well, these changes follow along quite naturally. But if you're coming fresh, I struggle with how to teach folks why it's sometimes "any" and sometimes "some" and what the "correct" way to write Costume is. In particular, the pitch feels like it's often in terms of "you did it one way, and don't want all the trouble to refactor it." But does that suggest that "any Costume" was a mistake and this is a hack around that? I don't think that's true; as the Array example shows. But I'm left wondering what to tell students. (I'm on record teaching devs to pass existentials rather than generics when in doubt because I thought it was simpler to understand and had fewer weird edge cases, but I think I was likely wrong about that.)
I'm mostly muttering here. I think this pitch is good and definitely an improvement. Thanks for all the work.
Yeah, that's concerning to me as well. Perfect compatibility with existing source is a worthy goal; however, if compatibility weren't a concern, my sense is that it would usually be preferable to open an existential and pass the underlying value whenever possible, even if the existential could be passed as-is. It will usually be more efficient, because a Swift existential that conforms to its own protocol would do so by trampolining through another layer of witness table, and passing a self-conforming existential opens up the possibility of creating multi-wrapped existentials, which are mostly invisible but create multiple levels of boxing. (Or at least, they're supposed to be mostly invisible, but currently you can only practically create them using Any; it will be exciting to find out how well the runtime deals with other multi-wrapped existential boxes once we expose it to them in practice.) Not opening existentials when possible would also turn adding "self conformance" to a protocol into an evolution hazard, because adding self conformance would either break or add invisible overhead to existing code that passes the existential as a generic argument expecting it to be opened. Overall, I think that favoring opening is the better option. If doing so creates unacceptable practical source breaks in Swift 5, then I think we should still change the behavior in Swift 6.
What kind of performance improvement are we talking about? If these improvements are insubstantial, even in large codebases, I think we should favor source compatibility. Although this source-breaking change would be subtle, and thus rarely relied upon, its subtlety could also incur difficult debugging experiences and nasty bugs. I'm just worried that in our effort to improve Swift by correcting the mistakes of the past, such as changing P to any P, we may risk agitating users without a tangible benefit. There's already been backlash about the warning for marking existentials with any. I bet that if implicit opening changes in Swift 6 and libraries become slightly buggier, as a result, users will not think, "at least Swift is slightly faster"; Swift's already fast. Of course, I'm exaggerating a bit and I understand that in server applications, for example, small gains could add up to big savings, but I think we should maintain a high bar for source-breaking changes.
The primary overhead using the value will be double-indirection—the generic function will interact with the existential box as a generic type, whose implementation of the value witness operations and protocol methods all have to re-dispatch through the corresponding operations on the dynamic type inside the existential. If the existential-as-generic value gets used to create a new existential value somewhere, there will be an additional allocation to copy the existential behind the generic into an existential box. You're right that neither of these overheads are probably a big deal for typical application code that already accepts these costs. Of the concerns I raised, I'm more concerned about the library evolution angle; whether a combination of existential constraints is "self-conforming" or not isn't necessarily a fixed property of the constraints involved, and the subtlety issue you raised cuts both ways—a seemingly innocuous change in one place has unpredictable effects elsewhere in the code.
Although this is a subtle situation for humans to discern, it should be straightforward for the compiler to know where the behavior would change from this proposal. If we were going to gate the behavior change on a language version, then it seems like a migration tool could flag places where the behavior would change, allowing the developer to adapt their code.
Personally, I think that changing the runtime behavior implicitly in Swift 5.x is unacceptable in practice. I agree with you that I would favor opening if source compatibility wasn't a concern. Swift 6 would be an opportunity for such a change, and it feels reasonable to do at that point.
I'll integrate this discussion into the proposal. Thank you!
That sounds like a reasonable compromise to me too, thanks Doug! To maintain Swift 5 compatibility, while also avoiding the problem of API evolution potentially making protocols newly self-conforming in the future, maybe we could limit the non-opening behavior to strictly the cases that are self-conforming today (Any, Error, and @objc protocols) and say that future native-Swift protocol types that gain self conformance in the future still implicitly open as generic arguments in both language modes. It's a weird rule in isolation admittedly, but as a compatibility concession, it strikes a balance that shouldn't break any code today, and will prevent future evolution of the language and standard library from breaking code again in the future.
I went ahead and revised the proposal to incorporate all of this feedback. A brief summary of the changes:
Only apply the source-compatibility rule, which avoids opening an existential argument when the existential box would have sufficed, in Swift 5. In Swift 6, we will open the existential argument whenever we can, providing a consistent and desirable semantics.
Re-introduce as any P and as! any P , now that they will be useful in Swift 6.
Clarify more about the relationship to the explicit opening syntax, which could also be a future direction.
At this point, I think it's a good time to bring this up for a wider review.
A question - in the example Costume.hasBellsMember, what is the type of C in the call to hasBells()? Does that getter function get instantiated with a different type for every type that conforms to C?
It's the Self type of the protocol, which will end up pointing at whatever the type is---be it Int or String or whatever---that conforms to the Costume protocol. Swift doesn't use instantiation as the language model; there's one implementation of hasBells() that dynamically gets provided with type information. That type will always refer to a concrete type that conforms to Costume.
(FYI, the review of this proposal just kicked off, so I'll answer questions over there as well)