SE-0309: Unlock existential types for all protocols

Also, there is a subtle distinction between the two: whereas such a heterogeneous array literal may still be explicitly ascribed to another sensible type, an unconstrained associated type can really only be represented as Any.

The concept of path-dependent types is what we envision interfacing with existentials to land at eventually, and we were careful to make sure the current proposal in all its intricacies does not interfere with a source-compatible future transition. But I doubt things can sort themselves out by making use of opaque types, which are still dependent on and invariant in the generic parameters of the enclosing context (Self in our case).

5 Likes

I am not exactly sure what you meant there. The solution allows any protocol to be used as a type, but a member access on a value of protocol type may still be rejected with an error message showcased in the Diagnostics section. The latter can already happen today with protocol extension members. In other words, we are getting rid of the artificial type-level limitation in favor of the already-existing value-level limitation, which is a type safety barrier we can sometimes reasonably set aside with the help of type erasure.

7 Likes

Thanks for the explanation Filip, I now understand the motivation. Still, I think users will probably just learn that "any P" is the path of least resistance. However, I also don't have a better solution to offer at the moment.

2 Likes

There’s certainly some truth to the saying that when you only have a hammer, everything looks like a nail. However, there’s also some seriously talented teachers in the Swift community, which hopefully counts for something.

Take, for instance, my favorite article on this very subject–A Protocol-Oriented Approach to Associated Types and Self Requirements in Swift by Khawer Khaliq. Though the article will need a few updates if this proposal is accepted, it meticulously demonstrates a variety of techniques to completely avoid compiler errors that are common when misusing associated types and self requirements in protocols.

Not to sound too hyperbolic, but it will kind of fall to all of us in the Swift community to follow exemplars like Khawer. In short, we will need to teach precisely when and when not to reach for the hammer if the any P construct is accepted into the language.

5 Likes

I am a huge +1 on this.

I'm somewhat disappointed that this didn't go the extra mile and include any P as well, but I can see the rationale to break it up into parts. My only concern now is that we won't get to any P quick enough and people are going to start using existentials more and more making the day that we introduce any P more of a headache. I would like to see any P tackled on the same Swift version that this is introduced (I know none of the proposal authors have a say into that) so that when people use this proposal it's coupled with any P.

Overall, I'm extremely happy this is finally going through and huge props to the proposal authors for an amazing writeup!

29 Likes

I happened to be browsing by, just out of nostalgia. I'll add that this is an effect I have always worried the addition of generalized existentials would have. I'm disappointed to see that the proposal doesn't seem to include a careful examination of these implications along with proposing mitigation strategies.

That was one of my earliest proposals for a mitigation strategy. Is it enough? I'm not sure. To me it seems like it helps with the surprising-API-non-availability problem, but maybe less so the language complexity effects.

More broadly, my concern here has always been that in an eagerness to “just get past the restriction already,” we'd brush away the downsides and fail to do an enthusiastic investigation of just what they are, how bad they might be, and how we might counteract them. The sense I get from the proposal is, “we thought about a couple things but decided not to do anything,” which I'm not sure is really adequate, especially after such a thorough exploration of the space from @Joe_Groff.

FWIW-ly y'rs,
Dave

3 Likes

I find myself somewhat confused while reading the discussions about the variances in Swift's types. Is there a general guide/rule somewhere that I can read up on what positions are considered by the type constructor as covariant, invariant, and contra-variant?

1 Like

These four should serve the purpose:

  • Tuple types are covariant in their element types.
  • Generic types are invariant in their generic parameters except for a select few Standard Library types (Optional, Array, Set and Dictionary), which are covariant.
  • inout types are invariant in their underlying type.
  • Function types are contravariant in their parameter types and covariant in their result type.

Unlike with conversions, note that the part about Optional, Array, Set and Dictionary being covariant does not fully apply in our case (the covariant relations are listed at the end of the Proposed Solution).

5 Likes

Was the omission of Set from the corresponding proposal text deliberate?

Yes, covariant type erasure in generic parameter position can cause us to run into the self-conformance problem when there is a conformance requirement.

2 Likes

Thanks!

Is there an explanation or rationale behind these 4 subtyping rules as well? Sorry if I'm asking too much. I want to understand the rules instead of just memorising them.

1 Like

There's a nice article by Mike Ash that talks about the basics - mikeash.com: Friday Q&A 2015-11-20: Covariance and Contravariance and there are some forum posts about it w.r.t generics here and here.

6 Likes

Subtyping rules are generally determined via the Liskov substitution principle, and then variance comes in for compound types to describe how subtyping in a component can relate to subtyping in the compound.

Following the substitution principle, it so happens that the parameter types of f2 must be supertypes of those of f1 and the result type of f2 must be a subtype of that of f1 for f2 to be a subtype of f1. Tuple types are just arrays of element types, and we can prove by induction that subtyping relations in components are propagated out to the tuple. A similar reasoning can be applied to the few Standard Library collections that are covariant by means of hardcoded built-in conversions. Other generic types are invariant in their generic parameters because the compiler does not know how to convert from one specialization to another in the general case. Regarding inout types, I think the reason they are invariant is that deep down, they resemble a generic pointer type like UnsafePointer, but I can't speak to whether UnsafePointer itself is deliberately kept invariant.

4 Likes

+1 to this, and hooray! Long awaited, much needed. Heterogenous [P] collections remain a vexing wall inside the maze of Swift, and this would easy some of the pain. So pleased to see this line of work proceeding at last.

Is there a snapshot build I can use to experiment with this? (Apologies if I missed the link.)

Two questions…er, well, question clusters:


First, is there any value in exposing non-covariant Self as Never rather than making the method disappear altogether? For example, the Equatable existential type would have:

static func == (lhs: Never, rhs: Never) -> Bool

I can imagine one might want to deal generically with method references — reflection, say, or generating diagnostics — without actually needing to call the method. I admit that I don’t have a specific compelling use case for this, and the harm it would do to the already-confusing diagnostic messages would be substantial. Still, worth asking.

I can also imagine that there might be situations where the compiler might be able to determine a better lower bound than Never from generic constraints etc. Does such a situation exist? (For example, the compiler might be able to deduce a less-restrictive bound for a private protocol, where all possible implementing types are known at compile time.)


The second question really just amounts to, “Do I understand the proposal correctly? If so, great!”

Tim Ekl’s nice post on this proposal got me thinking about this limits of this proposal. He considers this example:

protocol Shape {
    func duplicate() -> Self
    func matches(_ other: Self) -> Bool
}

Even with this proposal, the following doesn’t work:

func assertDuplicateMatches(s: Shape) {
    let t = s.duplicate() // t could be any Shape, not nec same as s
    assert(t.matches(s))  // ❌
    assert(s.matches(t))  // ❌
}

…which makes sense, because we need to guarantee to the type checker that s and t actually have the same type:

func assertDuplicateMatches<SpecificShapeType: Shape>(s: SpecificShapeType) {
    let t = s.duplicate() // t also has type SpecificShapeType
    assert(t.matches(s))  // ✅
    assert(s.matches(t))  // ✅
}

(Aside: I like the spelling any Shape better every time I see it in context. It would make the problem with the first version of assertDuplicateMatches much easier to see.)

However, if I understand correctly, the following code still won’t work even if this proposal is accepted:

let testShapes: [Shape] = […]
for shape in testShapes {
    assertDuplicateMatches(shape)
}

…because the compiler can’t determine a non-existential SpecificShapeType for the call to assertDuplicateMatches. Do I have that right?

Given that, I take it this implies the Shape existential does not actually conform to the Shape protocol? The proposal doesn’t say this explicitly, but it must be so. (Given that, what is the error message here? Seems like one that requires extra-special care.)

Given the above, I take it that this future direction in the proposal:

Make existential types "self-conforming" by automatically opening them when passed as generic arguments to functions. Generic instantiations could have them opened as opaque types.

…would make the code above work as written?

1 Like

Not yet (we'll post a link).

Not sure there is practical value in going that far. Besides, exposing non-covariant Self as Never would definitely introduce a source compatibility impediment once we start considering path-dependent types. It is possible to deduce Self internally in the event of a single conformance, but this kind of inference may leave unintended type-erasure unnoticed until the next conformance (also, having a non-public protocol around for the sake of a single conformance is something you should most likely avoid in production code).

Correct. The error message should be the good ol' P does not conform to P (the self-conformance issue).

1 Like

Library authors can already "open" existentials, which can be helpful in the absence of Self-conformances, albeit not being very elegant:

let testShapes: [Shape] = […]
for shape in testShapes {
    let unboxedShape = _openExistential(shape, do: { $0 })
 
    assertDuplicateMatches(unboxedShape)
}

You do not need to use _openExistential to open most existentials. An extension method on the protocol will suffice. _openExistential is only strictly necessary for existentials on whose constraints extensions are not allowed, such as Any and AnyObject.

I wasn’t aware path-dependent types were on the table, even the far-future hypothetical table! It seemed from the responses to the perennial “optionals should be like Kotlin” question that Swift was steering clear of them for the foreseeable future. Interesting.

Yeah, the more I think this through, the more I realize there’s not a concern here. The situations where you can infer a single specific type without actually having type constraints that narrow to that type just…aren’t useful. Thanks for humoring me!

Thanks, good to know. Again, thanks for humoring me.

I do worry about the usability of all these type system features in practice (while still supporting them, to be clear). Messages like P does not conform to P need to make way for more approachable diagnostics, or Swift will become too hostile an environment for people who aren’t PL geeks. Wasn’t there a lovely project a while back that demonstrated type errors with specific examples? That would help immensely here. And as always, I wish for tooling that makes far more robust use of static type info than just type errors and autocompletions, some sort of visualization / contextual annotation / something that makes these kinds of problem apparent before the error message even shows up. That and a flying carpet.

Yes! I almost included similar code in my OP, but decided it muddied my question too much. At least it is possible, if awkward.

Is there an approach as general _openExistential that doesn’t require writing one extension method per method you want to forward to the opened existential? If so, I’d be curious to see it!

Your mention of Kotlin just made me realize that "path-dependent" might sound misleading to some. What we really mean is a notion similar to the type identity in opaque types rather than "smart casts":

3 Likes
Terms of Service

Privacy Policy

Cookie Policy