SE-0309: Unlock existential types for all protocols

Evolution Proposal Reviews
82

Why is it not considered the possibility to specify the associated type, like with Generics?
In order to make this possible:

protocol DB {
    associatedtype Element: Identifiable
    func read(id: Element.ID) -> Element
}

struct DependencyLoader {
    let stringDB: DB<String>
    let numbersDB: DB<Int>
    // ...
}
83

We’ve mentioned it in the “Future Directions” section of the proposal. We could introduce new syntax to constrain existential types, so for example in the above scenario you would be able to write something like let stringDB: any DB<.Element == String>.

8 Likes
84

Thanks, I missed that.

1 Like
85

This part of the official Swift docs is actually somewhat misleading.

Currently, Swift actually allows an opaque type that is a protocol type:

import Foundation

@objc protocol P {}
class A: P {}
class B: P {}
enum X {
    /// Opaque type that's always an existential...
    static var p: some P { _p }

    /// ... where the dynamic type is random.
    private static var _p: P {
        let i = Int.random(in: 1...2)
        switch i {
        case 1: return A()
        default: return B()
}}}

assert("\(type(of: X.p))" == "P") // passes
var str = ""
// Cast to Any is a runtime trick to unveil the 
// type wrapped in the existential container
for _ in 0...10 { str += "\(type(of: X.p as Any))" }
print(str) // prints ABAABABBAAA or similar

The above code compiles just fine in Swift 5.3 as long as P is an @objc protocol that does not have any static or initializer requirements.

If we'd add init() or static func foo() to P, now we'd get the following error:

'P' cannot be used as a type conforming to protocol 'P' because 'P' has static requirements

For details on the "trick" mentioned above, see:

How the "any" syntax leads to more confusion about opaques vs. existentials:

I don't have an issue with SE-0309 except the future direction of any and this confusing section about opaques vs. existentials:

When a protocol is used as a type, that type is also known as an existential type. Unlike values of some types, which represent a value of some specific type that conforms to the given constraints, and cannot be reassigned to a value of a different conforming type, an existential value is akin to a box that can hold any value of any conforming type dynamically at any point in time.

In the code example I provided above, static var p: some P is an opaque variable that always gives an existential with a random underlying type. When returned, it can be assigned to a value of A or B if the underlying type is A or B.

Might be nice to see the "Motivation" section of the proposal updated to reflect that the opaque vs. existential dichotomy is not a hard-and-fast rule but it's one with exceptions, since this behavior of @objc protocols is little-known. I only recently learned about it and I think many Swift devs might not know that these actually enjoy self-conformance as long as they don't have static requirements. It's pretty ironic that @objc existentials can be used like concrete types with generics & some whereas, existentials of non-Obj.C protocols can't.

As well I think this tends to rule out introducing any as a future direction, since it would lead to more confusion, especially given the other future direction to possibly increase the situations where self-conformance can apply beyond just @objc protocols without static requirements.

86

Please do tell us where exactly this syntactic change is being suggested or implied in the Proposed Solution section, and we will gladly remove it..

87

Regarding the syntax change I must have misread where it was in the proposal, apologies. Will update my comment.

The "Motivation" section that I quoted seems to imply that existentials are mutually exclusive from opaque types. Perhaps you could update it to mention there are exceptions to this.

I do note you have mentioned in Future Directions about the proposal possibly increase the number of cases where an existential self-conforms. I wonder if you could mention whether you think an implementation of this proposal should avoid changing which existentials can self-conform, in case it might be a side effect of the other changes? (I personally think it would be a good side-effect to have, if it can be done without ABI/lib. evolution concerns, but if it's reserved to the future for some specific reasons it might be nice to have that clarified). Just some thoughts.

88

I’m really +1 on this proposal!

I think, that it will greatly improve working with protocols and existentials and that it fits well into the direction of Swift.

I’d like to add that I’d also really like to see at least the possibility to write an existential as any P as well.

89

@1oo7 I’m still trying to understand, what’s exactly your problem with any P vs some P. AFAICS, both syntaxes make total sense: An opaque type can hold a value of some specific type that conforms to P, while an existential can hold a value of any type that conforms to P.

Maybe I’m misunderstanding you or I have overlooked something from one of your previous replies, but I want to comment on that.
I don’t think that there are exceptions to opaque types. They can really only hold one type that conforms to P ever. Of course that means that they can hold the existential of P as well, if (and only if) P conforms to itself. That is not an exception from the rules of opaque types but rather a consequence thereof.

7 Likes
90

The second part of what you said is true: it's a consequence of the rules that an existential of P that conforms to P can be returned as an opaque type.

However my point is that therefore it doesn't make any sense to use any to refer to existentials, as opposed to some, because there are cases where some can be an existential. So if the rule is that if it's an existential then it has to be marked as any , that conflicts with the scenario where it's a concrete type.

Please see below for further clarification.

Perhaps you could clarify what "orthogonal to the difference" means in this context?

If we can go back to what you said earlier:

The point I was trying to make is, they're actually not different at all:

  • in func x<T: Zish>(_ z: T) — protocol Zish constrains type T
  • in let z: Zish— protocol Zish constrains the invisible, auto-synthesized, existential container type

This is explained in detail within the SIL docs' definition of the various existential functions:

protocol and protocol composition types consist of an existential container, which is a generic container for a value of unknown runtime type, referred to as an “existential type”

The existential container's type is constrained by a generic type, the "existential type". Therefore a generic constraint and a protocol conformance are not "different yet equally spelled things", they are the same exact thing, just with a vastly different level of sugar (all of Swift is sugar for SIL to varying degrees).

I fully respect your feeling that Swift would be more approachable if this could be better clarified. I'm of the view that we should address this with an IDE-level feature similar to "show-invisibles" mode, where you could see more of what the compiler is actually doing. This would transform any declaration like let x: Protocol into let x: ExistentialContainer<Protocol>.

Here's what it might look like if the compiler generated something similar in native Swift, like what the compiler generates in SIL for some protocol P when it's used as an existential:

protocol P {
    var id: Int { get }
}
class A: P {
    var id: Int
    init(id: Int) { self.id = id }
}

@dynamicMemberLookup
struct ExistentialContainer<X: P> {
    var _contents: X
    var contents: some P {
        _contents
    }
    init(_ contents: X) { self._contents = contents }
    
    subscript<T>(dynamicMember dynamicMember: KeyPath<X, T>) -> T {
        (contents as! X)[keyPath: dynamicMember]
    }
}
extension ExistentialContainer: P {
    var id: Int {
        self[dynamicMember: \.id]
    } 
}

Usage:

func doSomething(with p: P) {
    print(p.id)
}

let existential = ExistentialContainer(A(id: 1))
doSomething(with: existential) 
// prints 1
1 Like
91

The statements in the "Motivation" do not imply that either kind of type cannot act as an abstraction over the other. Your example with an @objc protocol is a mere consequence of our type system. The underlying type of an opaque type can indeed be an existential type that conforms to its protocol, and an existential value can as well be handed a value of opaque type with the appropriate constraints (the underlying value will be handed). Just as with a generic parameter type:

@objc protocol P {}
class Class: P {}

func foo<T: P>(arg: T) {
  let p: P = arg
}

foo(arg: Class() as P)

It may be silly to do so in practice, but these cases do not make opaque types and existential types potentially equivalent by definition. They are always different abstractions and always provide different guarantees, and are not identically equal to what is being abstracted. The existence of "self-conforming" existentials does not affect this.

10 Likes
92

This proposal has no "self-conformance" side-effects (we would have had to mention them otherwise). Did I get you right?

FYI: The only protocols that "self-conform" today are Swift.Error, @objc protocols that do not have any static/init requirements (as you mentioned), and @_marker protocols. Here's the code for that.

6 Likes
93

Pay attention to my bold highlight in the following quotation from your "Motivation" section:

When a protocol is used as a type, that type is also known as an existential type. Unlike values of some types which represent a value of some specific type that conforms to the given constraints, and cannot be reassigned to a value of a different conforming type, an existential value is akin to a box that can hold any value of any conforming type dynamically at any point in time.

To me this strongly implies a sort of dichotomy where a value of an existential type cannot also be a value of a some type, because of the phrasing "unlike values of some types".

I might suggest instead:

When a protocol is used as a type, that type is also known as an existential type. An existential value is akin to a box that can hold any value of any conforming type dynamically at any point in time.

There's no need here to reference some types; they are unrelated to existentials and the two concepts are not mutually exclusive, so I do not see much of a point here to bringing it up.

2 Likes
94

When the opaque type holds an existential then it still just holds some specific type, namely any P. It’s completely irrelevant for the opaque type that it’s underlying type can dynamically hold differing types, because it just holds the container, which is a specific type.

11 Likes
95

The limitation of some still holds, you can't change the type behind some from any P to Int over assignment, though the any P may allow to change the inner type for certain operations.

Citing the motivation of the proposal:

I think this sentence bites me a bit.
Why an existential must be a box? I think we refer to mutability here inducing to change the type at runtime, but the concept of existentials exists outside of mutability.
Even if you can't change the underlying type of some P after initialization, it is still an existential as we can initially bound any type to some P.

96

Both opaque types (some T) and existentials (any P) can be thought of as boxes (even if technically it’s not necessarily accurate)

In case of opaque type (some), the box is locked to single type inside that cannot be changed. This locked type can even be an existential in some cases (as shown upthread), but that doesn’t change the fact that box itself is still opaque, not existential. An array of some P can only contain same single type in all of the array’s elements.

In case of existential (any), the box is ambiquous in what it contains, and essentially erases all specifics about the type inside, except what is exposed by the P itself, i.e the protocol’s methods. The existential box can contain many different types, e,g when used in an array, and all those types inside promise to provide the methods from the protocol P.

2 Likes
97

For me, a box is an implementation detail, but the term is generally not well-defined

Type some P is the unification of all conforming types to P, hence some P is an existential type, the same holds for any P.

In difference to any P, a variable of type some P is restricted not to change the underlying type once a value was assigned.

98

I will take thought of how the wording here can be improved to address yours and everyone else's feedback, thanks! The contrast with some types is deliberate and is meant to help readers grasp a distinction in comparison. In the wild, these two abstraction models are very much related due to their deceptive resemblance, and it is not uncommon for people to wonder why we need one when we have the other.

1 Like
99

In my understanding, that is not accurate. some P refers to the return type of a property or method, and the compiler enforces the actual identity of P to be the same from all return paths in the property or method definition. That means each usage of some P only ever refers to one concrete P. This cannot change at build time, never mind at runtime.

2 Likes
100

Referring to some P as ‘reverse generics’ initially made the concept of some much clearer to me.
Whereas the generic types of a function are decided by the caller, the specific type of P in a some P return type is decided by the function implementation.

1 Like
101

I do agree we need to disambiguous the meaning of existential type and protocol constraints, but I doubt the 'any' is a good choice. As far as I read the discussion above, there are no clear contrasting relationship between existential types and opaque types that any/some suggests.

I really agree this concern. For me it feels exactly any concrete type that conforms to Usable. It also seems similar to generic result types.

Also, if I understand correctly, currently, it is not the clearly-agreed direction to use some for generics discussed here. If we don't take the direction, there is possibility to use any for other purpose, for example, some for 'reverse generics' and any for 'generics'.

I'm +1 on unlocking existential types for all protocols, but -1 on making it stable direction to use any P for existential types. At least, it should be decided after the whole direction of generics UI is decided. For what the any should be used, should be more carefully considered.