SE-0309: Unlock existential types for all protocols

[sighoya]

@aerobounce

Excuse me if this sounds silly, however, could I ask why any P is considered to be needed (in the future or as an alternative)?

@aerobounce

To make type declaration clear that "A protocol cannot be conform to itself"?

Two things come to my mind:

• ABI constraints for the old syntax (without any) exist
• any P seems to be a nice dual to some, simply writing the protocol name make it hard to know which kind of existential we mean here (some or any)
• a protocol is theoretically not a type, just a constraint/predicate, but with this proposal we would also refer to the existential type (the protocol as type), instead this should be disambiguated with additional syntax to make the distinction more clear

@aerobounce

(I'm not against the introducing the new keyword at all, but would like to understand them)

Simply because of that:

• We can use any P in parameter position but not some P yet
• some P in argument position would be less powerful regarding non-determinism

[jayton (Jens Ayton)]

@Alejandro_Martinez

One thing I'm still making my mind about is about the introduction of any P .

Note that any P is not part of the proposal being reviewed in this thread.

[Karl]

@aerobounce

Excuse me if this sounds silly, however, could I ask why any P is considered to be needed (in the future or as an alternative)? Read proposal and Improving the UI of generics but still confusing to me.

IMO, the benefit of having new syntax would be if it could clearly distinguish type erasure from generic code. I'm not entirely sure that any P does that.

Basically, today a new developer would read TSPL, learn that "protocols are types", and that they can use a protocol anywhere they would use a type. Then they'd write a function, say...

func frobinate(item: MyType)

And then some time later, when they want to reuse this function, they'll think "hey, this code isn't actually specific to MyType; I only use stuff from the protocol P", and, given what they've learned from TSPL, go away and write:

func frobinate(item: P)

Which will work, and very much looks right - they can call this function with a MyType as before, but also using any other object whose type conforms to P -- but this isn't actually what they wanted to express. What they wanted was to make their function generic, but instead, they've got a non-generic function which accepts a box holding some unknown object conforming to P. That distinction becomes very important once you introduce any sort of complexity to the protocol or the function - associated types or Self won't work, you'll need extra indirection to use inout parameters, and you won't be able to interface with real generic code (the so-called self-conformance problem).

Consider how much clearer this would be if we made things a bit more explicit - rather than just P, or even any P, consider if we wrote BoxContainingSome<P>:

func frobinate(item: BoxContainingSome<P>)

Now it becomes a lot harder to confuse that with a generic function - yes, you can use it with any object whose type conforms to P (if you box it first), but it makes the indirection responsible for all of the above problems more obvious.

People shouldn't be reaching for existentials when they look for generics. There are uses for variables whose type might change at runtime, but they are quite limited, and giving them a more explicit, verbose spelling would probably be for the best.

[aerobounce]

Thanks for taking your time.

@sighoya

• ABI constraints for the old syntax (without any ) exist
• any P seems to be a nice dual to some , simply writing the protocol name make it hard to know which kind of existential we mean here ( some or any )
• a protocol is theoretically not a type, just a constraint/predicate, but with this proposal we would also refer to the existential type (the protocol as type), instead this should be disambiguated with additional syntax to make the distinction more clear

@Karl

IMO, the benefit of having new syntax would be if it could clearly distinguish type erasure from generic code.

Those totally make sense!

@sighoya

• some P in argument position would be less powerful regarding non-determinism

And yes, this.
I was wondering why some P aka Opaque Types isn't sufficient enough as any keyword.

So re-read The Swift Programming Language: Redirect, then noticed that its purpose is different first of all, and understood introducing something distinct is necessary.

So... some is to cover something, any to absorb something.

[grynspan (Jonathan Grynspan)]

@Joe_Groff and I discussed off-forum briefly. My team has a use case in favour of this proposal.

We've got a protocol for a type whose instances may cross process boundaries as XPC messages; for the sake of this discussion, let's call it Action. Action has two associated types, Input: Codable and Output: Codable, and an instance member, func invoke(passing input: Input) throws -> Output. Conforming types implement invoke(passing:) in order to transform an Input into an Output. Nothing too Earth-shattering there.

Now, we want to refer to individual instances of Action in a generic context because:

• we need to enforce type safety and security across the process boundary by defining an array of Action.Types that can be safely decoded after transmission; and
• we want to be able to look up which Action.Type needs to be instantiated and invoked in said array.

But because of Action's associated type requirements, we can't create an Array<Action.Type> directly. We instead need to create a type-erased AnyAction protocol with no associated type requirements. In our case, we imbue this protocol with a single member function static func invoke(usingContentsOf xpcDictionary: xpc_object_t) -> Void, and we make Action conform to it. Action provides a default implementation that does very little: it decodes an instance of Self, decodes an instance of Input, invokes invoke(passing:), and yields either an Output or an Error as an XPC reply. This function has no associated type or Self requirements in its interface—they're all part of its implementation. If we could refer to Action as an existential type, then AnyAction would be unnecessary. We could make invoke(usingContentsOf:) a member of Action and could remove AnyAction from our project.

tl;dr: We have a real-world use case for referring to a protocol with associated types as an existential type. If this proposal is accepted, we'd be able to drop AnyAction and use Action directly in its place, which would mean the elimination of a fair bit of boilerplate and glue.

[Jumhyn (Frederick Kellison-Linn)]

+1 to the proposal overall. Lifting this restriction to individual members is a nice compromise, and I especially like the upcasting trick for Self/associatedtypes in covariant positions. I'm also in support of @xAlien95's suggestion that implicit upcasting to Any require an explicit as coercion, like we require in other places already.

[epam-gor-gyolchanyan (Gor Gyolchanyan)]

Amazing! Huge +1 from me! I've been wanting this for a long time!

I'd like to suggest mentioning in the proposal the unprecedented ability to dynamically check for conformance to a PAT:

func foo(_ value: Any) -> Bool {
    return value is Hashable // not possible before
}

Also, I'd like to suggest mentioning how static members and initializers will behave:

func foo(_ value: RangeReplaceableCollection) -> RangeReplaceableCollection {
    return type(of: value).init() // Okay, even though the type information is missing at compile-time and it might feel like it wouldn't work, accessing the dynamic type recovers it at run-time allowing the initializer to be used.

Also also, I'd like to suggest mentioning in the proposal the inability to refer to static members and initializers when referencing them from the protocol itself:

RangeReplaceableCollection.init() // Error: missing existential type (either static or dynamic)

[Moximillian (Mox)]

@Karl

Consider how much clearer this would be if we made things a bit more explicit - rather than just P , or even any P , consider if we wrote BoxContainingSome<P> :

func frobinate(item: BoxContainingSome<P>)

I know programmer should be nudged to try generics first, and right now existential first wins simply because of too light syntax
func foo<T: MyType>(value: T) // generics
vs
func foo(value: MyType) // existentials

But i think it’s wrong to start punishing existential syntax simply because typically generics are used with named label (such as T above). If generics was used anonymously, we could just have
func foo(value: some MyType) // generics
vs
func foo(value: any MyType) // existentials

I think any P already makes existentials googleable and nudge enough the programmer to understand existentials are not the obvious or even preferred choice.

[filip-sakel]

@aerobounce

Excuse me if this sounds silly, however, could I ask why any P is considered to be needed (in the future or as an alternative)? Read proposal and Improving the UI of generics but still confusing to me.

That’s a good question. I think that others have already pointed out the major advantages; I’m just going to sum them up:

1. Existential types of protocols and those protocols are often conflated. This not only hinders searchability, but it is also unintuitive in certain cases, like Self-conformance — e.g. the existential of Hashable does not conform to Hashable. As a result, language users are met with quite confusing diagnostics: Protocol 'P' as a type cannot conform to the protocol itself

2. The any syntax would help clarify the distinction between generics and existentials, by demoting existentials from being their "default"-abstraction position. Namely, these two abstraction mechanisms differ in terms of performance and usability characteristics. Generics are compile-time type-level abstractions; therefore, they are more performant and they enable type constraints. Existentials, on the other hand, are run-time value-level abstractions and are useful in dynamic environments and in cases where the additional complexity of generics is unnecessary.

[Karl]

@Moximillian

But i think it’s wrong to start punishing existential syntax simply because typically generics are used with named label (such as T above).

...

I think any P already makes existentials googleable and nudge enough the programmer to understand existentials are not the obvious or even preferred choice.

Nobody is suggesting to punish existential syntax - only to make clear exactly what it does. The current spelling is so brief that it fails to communicate important information.

There seems to be some kind of assumption that any P is the heir apparent to our current spelling, given the similarity with some P as used for opaque types. I think there is room to challenge that assumption (hopefully others, who are better at coming up with names for things than I am, will have ideas). That's why I think it's good that this proposal doesn't include alternative spellings - it's an open question.

I also feel that there is some kind of implicit assumption that developers shouldn't even need to care - that they should just write whatever (waves hand) and have the optimiser figure it all out. It may be able to do that in some limited cases today, but my experience profiling Swift code tells me that there is really no good alternative to clearly saying what you mean. I have serious doubts about whether jumping back and forth between type-level and value-level abstractions willy-nilly will really have no impact on the optimiser's ability to do its job.

[filip-sakel]

@xAlien95

I'm a bit concerned about the automatic type erasure for associated types in covariant positions:

Thanks for the feedback!

@xAlien95

In Swift, until now, type erasing has been an explicit action made by the user. An heterogeneous array results in an error instead of a warning

The same way one can currently write let a: Codable = 1, they would be able to write let b: Hashable = 2. Not to mention, that a client using an existential-accepting, library-defined function would lead to the user's implicitly wrapping a value in an existential type.

@xAlien95

func printEnumerated(s: any Sequence) {
  // s is non-mutating, handled as 'some Sequence', same behavior as above
  for (index, element) in s.enumerated() { ✅
    if index < 5 { print(element) } ✅
  }
}

Wouldn't any Sequence also need to be Self-conforming? (The printEnumerated(s:) functions needs to support passing values of existential type any Sequence and some Sequence would be transformed into a generic parameter with a conformance-constraint to Sequence.)

[Moximillian (Mox)]

@Karl

There seems to be some kind of assumption that any P is the heir apparent to our current spelling

any P and very many other alternatives were bikeshedded a lot in the pitch threads in the past. Personally I had hoped we already had a syntax which doesn’t have any issues with the all the possible ways it could be used in the future, at least several other syntaxes had those problems.
For example, this future syntax would work great with any P:

let collection: any Collection<Self.Element == Int> = [1, 2, 3]

[DevAndArtist (Adrian Meister)]

@xAlien95

I would love to see Any replaced by Type .

Unless we would refactor metatypes into meta T and any meta T (there are two kinds of metatypes which are currently merged, which is also why type(of:) cannot be properly implemented in Swift) I don‘t think there is a chance to unlock the T.Type syntax for custom types called Type.

[xAlien95 (Stefano De Carolis)]

My two posts are unfortunately not relevant in some places because, while I was certain to have read the opaque type proposal at the time, what I instead read were the Improving the UI of Generics thread and the Generics Manifesto. I realize that my understanding of opaque types was partially incorrect, shame on me

---

@filip-sakel

The same way one can currently write let a: Codable = 1 , they would be able to write let b: Hashable = 2 .

I was referring to the Covariant Erasure for Associated Types section specifically.

protocol P {
  associatedtype A: Hashable
  var a: A { get }
}

let p: P = pConformingInstance

let member = p.a
// error@-1: 'member' is not concrete; add explicit type annotation if this is intentional
// [fix-it]  Insert ' as Hashable'

It would almost be the same error you receive when using an heterogeneous array literal, in which case ' as [Any]' is suggested.

I suggested it due to my misunderstanding of opaque types, (off-topic)

I thought there could have been room for extending what opaque types can do. Specifically, I thought of them as a way to open existentials in-place:

var collection: any Collection = [1, 2, 3]
// dynamically open an existential on runtime;
// 'opened' has access to the whole protocol interface
let opened = collection as some Collection

Under such assumptions, you would be able to access the whole protocol interface on existentials too, by implicitly referring to the concrete instance in the existential container as an opaque type instance. Every associated type reference would be erased to "some Protocol" with Protocol being the upper conforming bound as in your proposal. You can get any member, but you can't set anything, nor call methods in which the unknown associated type appears in argument position. If you have a method of type signature (some Collection) -> Void (some Collection here refers to the Self type in the protocol interface), then you can call it only if you happen to known what the exact type expected by that function is (which in the context of an existential, in general, is not possible since it depends on the eventually varying concrete underlying type). Under this view, (any Collection) -> Void and (some Collection) -> Void have opposite semantics: the former can accept any Collection-conforming type, while the latter can only accept a single unknown Collection-conforming type (if you're lucky and you know what that type is, then you can call that function, otherwise it will be uncallable).

Users would get diagnostics mentioning opaque types, such as "Method 'foo' expects a parameter of type 'some Collection' (type of 'collection') but got 'Set<Int>' instead" which are more clear in my opinion.

In summary, opaque types are seen as existential containers which cannot change the underlying type on runtime and that can therefore be used where concrete types are expected. Thus, under my distorted view, an immutable existential type instance (so an existential instance that cannot change its type) would be equivalent to an opaque type instance. Therefore in printEnumerated(s:), the parameter s can be treated as some Sequence in the body of the function and optimized accordingly.

If what I wrote makes sense, you can access any member of an existential type without issues by implicitly opening it in-place.

var s: any Sequence = "Swift Evolution"
// s.enumerated() is EnumeratedSequence<some Sequence>
// s.enumerated() is EnumeratedSequence<(type of 's')>
for (index, element) in s.enumerated() { ✅
  // index is Int
  // element is (some Sequence).Element
  // element is (type of 's').Element
  if index < 5 { print(element) } ✅
}

and if you want to type erase, e.g., element, you need to explicitly do so using as Any or

for (index, element): (Int, Any) in s.enumerated() { ... }

So, there would be no need for covariant erasure for associated types. Instead, you would get the underlying type as an unknown type and type erasure only when intentionally and explicitly specified by the user.

But without the wrong assumptions I had (they are in the (off-topic) collapsible section above if you're interested), I fear there would be too much verbosity since any associated type in covariant position would need to be annotated with ' as Protocol' and it may become particularly tedious if generalized existential will come into play...

Insert ' as RandomAccessCollection<.Index == Int, .Element == (key: String, value: Double)>'

[Nobody1707]

I'm so glad this is finally ready for proposal. +1 from me. This has been a rather large pain point in the language, and it's nice to see it being addressed.

I am a little disappointed that any P isn't part of this proposal, but hopefully there'll be a follow on proposal for that. I like Lattner's plan to introduce it, except that I think we should commit to upgrading the warning to an error no later than Swift 7.

Having said that, the thing I'm most happy for in this review is Associated Types with Known Implementations. As far as I'm concerned this section is a bug fix, not a new feature. I'm ecstatic that it's finally being fixed.

@Alejandro_Martinez

One thing I'm still making my mind about is about the introduction of any P . When I read about it on the "improving the UI of generics" it made everything clear, so I like it very much. But sometimes I feel like it could make things much harder to explain, specially since I think other languages don't make this distinction and "just work". (am I right that Java/Kotlin do it?)

Java & Kotlin don't have associated types, and every type that can currently implement an interface is already passed by reference with vtables. Java doesn't make the distinction because it doesn't have anything to distinguish. Even generics in Java are merely syntax sugar for taking an object parameter and casting it to and from the right interface type (Project Valhalla notwithstanding).

[Syre]

@Moximillian

let collection: any Collection<Self.Element == Int> = [1, 2, 3]

It seems like most people in the thread are on board with any P, but I'm just unsure how it is better than simply allowing the use of P (like we already do with homogenous collections).

Maybe I just don't get it, can someone explain to me the benefit in this example compared to:

let collection: Collection<Self.Element == Int> = [1, 2, 3]

[benlings]

My understanding is that the advantage of having a different spelling is that these are two separate (but related) concepts. One is the ‘protocol as a constraint’ (types can conform to it, generic parameters can be restricted to it), the other is ‘protocol as a type’ (a variable/parameter/property has this type).

Without associated types, you can partly ignore this distinction, except when passing a ‘protocol type’ value to a parameter that’s using it as a constraint. However, the compiler error then has to try and explain why they are different in its error message, and it’s difficult when they are both written the same.

By having a different way of writing (‘spelling’) for each concept, it can help clarify the distinction. Protocol as a type is spelled any P, protocol as a constraint is spelled P.

[dinesharjani (Dinesh Harjani)]

Forgive me for not being so well-versed as most of the people here in the Dark Arts of Programming Language Crafting, but I absolutely LOVE this feature. I love almost everything else about Swift, but Protocols are the one aspect of it where I feel severly hindered in comparison with previous languages I've worked with, namely Objective-C and Java. I think this is a huge step in the right direction, and I too agree that making Existentials a source-breaking change with new syntax will be much better in the long-term, because three, four of five years from now, nobody will care about the old syntax anymore (except us who have to maintain it). In the end what will count is that we 'fixed it right', in my opinion.

[anthonylatsis]

@sighoya

Why would A preclude a call to method , is it because Swift doesn't allow for general covariance, e.g. G<T> isn't a subtype of G<Any>

Yes, generic types in Swift are invariant aside from a select few that have built-in support, so A inside G<A> is in invariant position.

@sighoya

If we treat a protocol-inherited associated type different for each type in a type intersection, assuming all types inherit from the same protocol, then this example didn't make sense:

class Class: P {
  typealias A = Int
}

protocol P {
  associatedtype A
}
protocol Q: P {
  func takesA(arg: A)
}

func testComposition(arg: Q & Class) {
  arg.takesA(arg: 0) // OK, 'A' is known to be 'Int'.
}

We must look at each associated type with respect to the base object type to handle the possibility of a known implementation, but we do not treat P.A and Q.A as distinct associated type requirements.

[sighoya]

@anthonylatsis

but we do not treat P.A and Q.A as distinct associated type requirements .

But it was said in the proposal that we might see them different in the future (delivering two vtables I think), but wouldn't that contradict the example included in my post.

I mean if we allow this kind of code and consider Q.A and P.A as distinct in the future, then the compiler infers something it shouldn't infer, the situation is not unique then, but the compiler makes a guess, a step in proliferation as a natural/fuzzy language compiler?