SE-0309: Unlock existential types for all protocols

So, I believe that this side discussion is fundamentally about a disagreement over the definition of "existential".

IIUC, your definition of "existential" is similar to:

a value of an existential of protocol P is a value v for which there exists a type T, such that v is of type T and T conforms to P.

Under this definition the value boxed in any P and both possible extensions of some P are existentials.


However, IIUC the definition of "existential" in general use in the Swift community is similar to:

a value of an existential of protocol P is a value b which contains a value v for which there exists a type T, such that v is of type T and T conforms to P, and b contains proofs of those statements about v.

Under this definition, any P, but not the value boxed in it, is an existential, but some P is not, because the proofs for some P are not stored in the value, but in the context.

4 Likes

Thanks everyone for participating in the review! The proposal has been accepted.

20 Likes

First of all congratulations on the proposal acceptance! I think this is a great step forwards for Swift as a language.

Now let me please try to clear up some of the above confusion about opaque types vs. existentials.

Using "some" with "any" would be confusing because "some Foo" is a placeholder for static type information known at compile-time, whereas "any Foo" would be a placeholder for dynamic type information only known at runtime.

Since Swift is a statically-typed language, and its type abstractions always refer to statically-known type information. We should keep it that way.

For details see below. (Warning, it's long.)

Definitions

  • existential container type (noun): the statically-known, but invisible to the programmer, type of the value/object that contains an instance of a value/object of a non-statically-known type that conforms to the protocol associated with the existential container. AKA "protocol-type". An instance of this would be a "protocol-type object".
  • existential type (noun): the dynamically-known type of the value/object contained within an existential container. This is NOT the type of the protocol-type object, it's the type of what's inside it. This type can only be determined via type-casting, is, or calling type(of:) at runtime. (Protip: In some cases, the protocol-type object must first be cast to Any to "open the container" before the dynamic existential type can be accessed.)
  • existential (noun): this is used in confusing ways. When referring to something at runtime, "an existential" could be used to refer to a specific value or object whose static type is a protocol-type. When referring to something at compile-time, it usually refers to the existential container type itself.
  • opaque type (noun): the statically-known, but invisible to the programmer or consuming code, return type of a function or property type of a variable. This type is not a container of some type—it is that type. In a static sense, the opaque type is merely a placeholder to be replaced by a specific type at compile-time. An opaque type is a protocol name prefixed by "some".
  • generic type (noun): the statically-known, visible to the programmer, parameter type or return type of a function, or property type of a variable. This acts very similarly to an opaque type except that consuming code will know exactly what the type is, at compile time.
  • associated type (noun): a placeholder in a protocol for a specific type that must be declared (or be able to be inferred) on any type that conforms to the protocol; very often, this requirement can be satisfied by a generic type parameter via a typealias. In current Swift, an associated type can never be opaque but it can be a self-conforming existential.
  • self-conforming (adjective): describes an existential container type that is statically known to conform to the same protocol that's also conformed to by the dynamic, existential type of its contents. A value/object of a self-conforming protocol-type will forward all member access to its contents.
  • static (adjective): when referring to type information, it means type information that's known at compile time. When referring to a protocol requirement, it means a requirement that specifies a static var, static func, or init method.
  • dynamic (adjective): when referring to type information, it means type information that's known only at runtime. This can be determined with type-casting (as/is), reflection, or functions like type(of:). When used as a declaration modifier, e.g. dynamic var foo it grants some Obj. C runtime features to that property.
  • protocol requirements (noun): in Swift, a protocol is just a set of requirements. Every func, var, and associatedtype in a protocol, is a requirement. For static protocol requirement, see the definition of static above.
  • protocol-type (noun): see existential container type

Swift is a Statically-Typed Language

When reading a novel, it's confusing if the character whose point-of-view the story is being told from, keeps randomly changing every other sentence, or if we keep jumping around from a past-tense to a future-tense, etc.

To avoid this problem, Swift tells the broad strokes of its story entirely from the static point-of-view. As a statically-typed language, Swift keeps its explicit, declarative syntax abstractions squarely in the static perspective. Its declarative abstractions are all placeholders for static type information that will be determined at compile-time: generics, opaque types, associated types, protocol-types, etc.

Meanwhile, Swift handles dynamic type information in a totally imperative manner, always using dynamic type-casting like as, is, or functions like type(of:) to make it clear which type information is dynamically determined—that which requires some lines of code to actually run before it will be accessible in code.

This is why the future direction of "any" just does not work for Swift, because it would make a declaration func foo() -> any Bar sound like "any Bar" is a placeholder for static type information, which it simply isn't.

Opaque Types can be Existential Protocol-Types

Incorrect. As explained above in the thread, an opaque type can be an existential container type as long as the protocol is one of the three kinds of protocols in Swift whose protocol-type objects can self-conform:

  • @objc protocols that don't have any static or initializer requirements
  • the new @_marker protocols
  • compiler-known protocols (such as Swift.Error—there's a short list of protocols built into Swift that are allowed to self-conform despite not fitting any of the above categories, because they have special witness tables created in advance to facilitate self-conformance)

See this part of the Swift compiler code, thanks to @anthonylatsis for linking it above.

Existential container types are known at compile time so they can indeed qualify as "some"/opaque type that conforms to their own protocol—as long as the compiler agrees they can conform to their own protocol (self-conformance).

However the existential type of the value/object contained inside an instance of a given existential container type is not known at compile time. Whether or not a given type is an existential container type is irrelevant to the question of whether it can conform to its own protocol and therefore be used with generics/opaque types.

Swift's Static vs. Dynamic Type Information Is Confusing

The fact that opaque types and existential container types seem related in the minds of people who don't understand them is not a bad reason to clarify the language, but we should be very careful that whatever changes we make don't further confuse the issue about what type information is dynamically known vs. what type info is statically known.

Ideally we can add syntax that stays true to declarative = static, imperative = dynamic distinction that Swift has.

However the proposed future direction for syntax "any" would make an existential seem like a placeholder for static type info, as if the compiler has been upgraded to where now I can do:

protocol Bar {
    associatedtype X
    static var x: X
}

func bar() -> any Bar { ... }

struct Foo<T: Bar> {
    var bar: T
}

let foo = Foo(bar: bar()) 
// this can never compile because `bar`'s return type
// is not statically known to conform to `protocol Bar`
// (meanwhile it would be, if this was `some Bar`)

That's why the opaque type feature is extremely different from the feature of existentials. The fact that people could get these concepts mixed up we can do a better job of educating Swift developers and that "some" might not have been the best word for opaque types.

"Opaque type" is not a very self-explanatory term, and "some" is extremely vague. It makes sense if you already know what it means, but if you don't, it gives you no clues about what is meant.

Opaque types are types that are statically known by the compiler at build-time, but are purposefully hidden from the consumers of the API. It probably should have been called something more descriptive, like a "concealed type" with "concealed" used instead of "some", like:

var body: concealed View // compiler knows what type this is, not you

Meanwhile, the concept of existential containers is totally different. The container type is statically known, but what goes inside is a purely dynamic behavior. The underlying type of a given protocol-type object is not known at compile time.

Future Directions to Clarify Swift

I won't pretend to know what will make the dynamic vs. static aspects of Swift less confusing for most developers, because everyone's brain works differently and this is a very abstract and arcane aspect of the language to understand.

Personally I'm a visual thinker and I like the principle of least surprise. For someone like me, I think it would hep if there was a visual aid combined with an explicit, unambiguous, plain-English keyword, which makes it clear that an existential container is a box whose type is statically known but whose contents' type is only known at runtime. We already have a word, "variable", which means "something that's known at runtime." So we could combine that with a visual aid in the form of | characters to make it clear what's meant:


struct Thing {
    typevariable FooType: Foo
    var foo: |FooType|
    // foo's type is a container type whose contents are of a variable type
    // that's only known at runtime
}

... but for someone whose brain works differently than mine, I'm not sure this would be less confusing than the existing behavior. YMMV

6 Likes

Those are two different things. Just because the type referred to by some P can be an existential does not mean that every some P is an existential.

That claim would be the same as saying that since some P can refer to Int, all some P refer to Int.

4 Likes

I don't understand why this would be the case. Would the compiler not verify that bar() is actually returning a type that conforms to Bar?

:+1:t2: I never said they were the same. I said "can be" ... that means they are different, but not mutually exclusive, concepts.

That claim would be the same as saying that since some P can refer to Int , all some P refer to Int

... what claim?

My point was just that this earlier statement of yours is inaccurate:

It's not an existential because the concrete type must be known to the compiler at build time

This is an inaccuracy because certain existentials can indeed count as concrete types at compile-time, whereas your statement makes it sound like if the compiler knows a concrete type therefore it must not be an existential.

Maybe that's what you meant though? I could be misreading.

1 Like

Do you mean concrete type? I think of associated types as generic types, but not as a placeholder for a generic type.

This is what I would think of as a placeholder for a generic type:

protocol P {
    associatedtype T: Hashable
    func f(a: T)
}

struct S: P {
    // this doesn't fulfil the protocol requirement and
    // there's no way to alias T = U
    func f<U: Equatable>(a: U) {}
}

Or this: https://github.com/apple/swift/blob/main/docs/GenericsManifesto.md#generic-associatedtypes :point_left: Generic associatedtypes

irrelevant point

In my mind the only question here is: does Bar conform to itself? That can be statically known, right? Edit: just realized that Bar (or any protocol for that matter) will never conform to itself - oops!

As someone who liked the any P syntax, I have to admit this a compelling argument against it.

This seems too verbose to me and the |FooType| looks too similar to [FooType]. I would rather have a syntax like protocol<Foo> over that. Maybe capitalize it so it looks more like a type: Protocol<Foo>

3 Likes

The point is that Bar in this example has associated type and static requirements, which means, an existential of type Bar cannot conform to Bar because the compiler will have no way to know what the associated type actually is, or where the implementations of the static methods are to be found.

struct Foo<T: Bar> requires some type T that conforms to Bar, but existentials of Bar can never conform to Bar. Therefore let foo = Foo(bar: bar()) can never compile.

That's why the "any" syntax doesn't seem viable to me, because any Bar makes it sound like the compiler statically knows that this function returns an existential container whose type is statically known to conform to Bar, which it doesn't and can't because Bar has associated type and static requirements.

With the expanded rules about existentials, perhaps we could have a protocol-type object of type Bar (the existential container type), however that object can't enjoy self-conformance because the compiler can't infer the existential type within the existential container at compile-time. It's a logical impossibility sadly.

The usual kind of proof that's given for this is a reduction to absurdity. (Forgive me if I butcher this)

Given:

protocol Bar {
    associatedtype X
    static var x: X { get }
    init()
}

class A: Bar {
    typealias X = Int
    static var x: Int = 42
    init() {}
}

class B: Bar {
    typealias X = String
    static var x: String = "blah"
    init() {}
}

... and a function that returns "any Bar" ...

func bar() -> any Bar { .random() ? A() : B() }

If the statically known type "any Bar" conforms to Bar protocol, then we reach absurdity:

struct Foo<T: Bar> {
    var bar: T
}

let bar = Foo(bar: bar()).bar // what's the static type of bar?

func impossible<T: Bar>(_:T) { print(T.x) }

impossible(bar)
// what gets printed, according to the compiler? 

But I probably butchered it. I think there was a better example given by @xwu on this thread: Very confused about some compiler warnings related to protocols—bugs or intended?

1 Like

Good point. It would have been more accurate for me to say that it's a placeholder that must resolve to a specific type at compile time. This is often satisfied via a typealias to a generic type parameter but it doesn't have to be. I'll update the post. Thanks.

Hmm, if what you're asking is, "does (the protocol-type object) Bar conform to Bar"? You're correct that it can be statically known whether or not a given protocol is able to have a protocol-type object that enjoys self-conformance, because there are rules about this in Swift's compiler. The example I gave would be totally fine if Bar was an @objc protocol with no static or initializer requirements. However, in my example, Bar has both an associated type and a static requirement, which means a protocol-type object for Bar cannot conform to Bar.

1 Like

I think we know statically that any Bar can't exist, yet?
But maybe with more features integrated, it could?

A compiler that can auto-synthesize all possible implementations at compile time, including extensions in other modules that don't exist yet, is certainly appealing. I suppose I can't argue against this without incurring Roku's Basilisk, and I definitely want a quantum Mac Cryo cube one day. One more thing?

1 Like

Err... I confused the ability of making existentials with self-conformance. Sorry. I'm reading the thread you linked now. Good stuff! :smile:

1 Like

Relooking onto your example, I'm relative sure it would be allowed as the associated type stays in covariant position and the variable bounded by it isn't directly mutable.

So typeof bar would be any Bar and T would be any Bar too s.t. printing T.x would be allowed?

But maybe I'm wrong, don't know exactly.

I don't see any problem here; the some P returned can actually be an any P because any P is a concrete type. It is a type-erasing wrapper type that can hold any instance conforming to P. (Incidentally, clarifying this concept is why I previously pitched spelling it Any<P> instead.) It's also an existential container type, which is just a fancy way of saying "wrapper type".

If func makeP() -> some P actually wants to return an any P, that's fine. It doesn't matter what the any P itself contains; the concrete type that it is returning is any P. That type is statically known to the compiler.

8 Likes

FWIW, instead of only contrasting the opaque type / existential (some/any), there’s the matter of syntactical weight that ”any” was thought to be a solution of.

Currently we have generic code which people should be using most of the time:

func foo<T: SomeProtocol>(value: T)

But what people gravitate to is:

 func foo(value: SomeProtocol)

Now, if compiler, behind the scenes, refactors existential code to generics, then this problem is moot for those cases. However, not all existentials can be converted.

Also, using protocol as constraint is not the same as using protocol as a type. But currently they look the same. Irrespective of whether things are static or dynamic, having ”any SomeProtocol” in places where protocol is used as a type, makes things clearer to me.

And specifically using ”any” is to highlight that there can be many different types inside, vs just single locked down type.

6 Likes

I agree that any P is a static type known to the compiler. However it cannot necessarily be returned for some P because existentials don't necessarily enjoy self-conformance.

The mistake you are making is that of confusing any P, the "type-erasing wrapper type" that has its own allocation of memory, with some P, which is merely a reference used by the compiler to get the type of a variable that has already been initialized with a statically-known type into existing memory. In the latter case, the type hidden by opaqueness must conform to P, whereas in the former case, any P is the type of a wrapper that doesn't necessarily conform to P—it just wraps something that conforms to P.

In current Swift, there's a very limited number scenarios where an existential container conforms to its own protocol, but it's so rare to see in actual production code that many Swift developers aren't even aware this is possible. The vast majority of the time, the existential container doesn't self-conform, i.e. any P can not satisfy some P.

7 Likes

First of all... what problem?

Second of all, "if compiler, behind the scenes, refactors existential code to generics" this would very likely break ABI stability and defeat the whole point of having existentials, which is to abstract away the type information of an object by using a protocol-type object (existential container) so that you can have heterogeneous collections.

Constraints are always inside angled braces <T: Fooable> or after a where clause: where T: Fooable. How much further disambiguation do we need?

func f<T: Fooable>(_ t: T)
// versus
func f(_ t: Fooable)

Is the problem to you, that they both use a colon?

Seems like we could solve that by switching out the colon in generic type constraints for another character, such as:

func f<T 👤Fooable>(_ t: T) // T whose shadow is Fooable
//or
func f<T; Fooable>(_ t: T) // the semicolon returns to Swift!

Or as I suggested earlier use | to wrap the existential form so it's clearly a box that contains something conforming to Fooable:

func f(_ t: |Fooable|)

PS—I think some of the confusion stems from the fact that, when you have assigned the result of a function that returns an opaque type into a new variable, the compiler will often implicitly cast this into an existential, which could easily give the false impression that it was an existential all along. However, this is just a strange behavior of the compiler ex post facto. For a demo of what I mean, see See example here: Demonstrates the bizarre, blurry world of static type information in Swift · GitHub

You'll see that for a non-@objc protocol P, the existential is not considered to conform to P, unlike the opaque some P. And if you modify the playground to add an associatedtype to P, you'll see it goes downhill from there. But given the chance the compiler will call an overloaded function with the existential instead of with the opaque type itself, making it falsely seem like the opaque type is an existential when in reality it's just a really idiosyncratic behavior of the way Swift resolves overloaded functions.

1 Like

Sure. And if any P (the wrapper) doesn't conform to P (the protocol), then you cannot return any P from a function that returns some P. The compiler will produce an error when you try to do so. I never said that any P can always be returned for some P, and I don't think anyone else has either.

It's not clear to me what this has to do with your objections. You have said it is incorrect to say that some P always refers to a concrete type because it might actually refer to a self-conforming existential. But since an existential is itself a concrete type, I don't see the contradiction.

10 Likes

Exactly. :beers:

OK, I feel you. I read your post initially in the context of the foregoing general discussion about the nature of how opaques and existentials broadly work, but I can certainly accept that you were referring to the exceptional 1% case of self-conforming existentials.

Hmm, no, I said the following assertions are incorrect (trimmed down so you can see which parts I meant):

Both of these statements make the false assertion that some P isn't an existential.

That's a faulty assumption for the reasons we just discussed: an instance of some P can very well be an instance of an existential container type, AKA "an existential".

What should we write for these? var x: some any P ... ? Or should it be, var x: any some P...? Which abstraction should trump the other when they intersect?

The problem with "any" syntax

At the risk of sounding repetitive, for the benefit of @bjhomer here's a summary of why I think we can do better than using "any" to make protocol-type objects explicit.

The problem is that "any" syntax is being discussed as though it refers to a different abstraction than opaque types, when in fact the two concepts are not mutually exclusive because an opaque type can also be an existential, and when they're not, the compiler often prefers to implicitly cast them to existentials even when a generic overload is available (see example here). We should avoid baking a fake "distinction" into the syntax, because it would make it even more confusing than what we have now by selling "any" as being the yin to some's yang, a very seductive yet misleading dichotomy. To me, "any" syntax seems like a high-risk idea because of its potential to be misunderstood:

var x: some P // type "some P" always conforms to P
              // the type that conforms to P is statically known

var y: any P // type "any P" almost never conforms to P
             // the type that conforms to P is not statically known
             // unless it's "any P" in which case it can also be "some P"

Does that make any actual sense to you or seem really better than what we have now?

Basing any future decisions about Swift syntax on such a fundamentally tortured methaphor would be disastrous IMHO for the cohesiveness of Swift:

  • it would blur the lines between static and dynamic type information
  • it would cement a tortured metaphor into Swift
  • it would further overload the most vague word in the English language, "any", which is already used extensively as both the protocol Any and a prefix indicating type-erasure
// can we please not make this possible:
let something: [any AnyCancellable] as Any

Lets Look For a Better Alternative to "Any"

Any word or symbol would be better than further spreading the word "any" to fit yet another purpose.

We should look at how other programming languages represent existentials and consider what the Swift version of that should look like. Considering how similar Swift is to Scala, I might suggest we take a page from them regarding the use of underscore as a wildcard type parameter:

var array: [_ & Hashable]
var myView: _ & View

This kind of syntax will be familiar to anyone from how underscore operates as a wildcard within Swift's pattern-matching syntax, such as inside a Switch statement, except here it's a wildcard for a type parameter instead of a value.

The nice thing with this is that it could be used with some:

var myView: some _ & View // an existential type that must conform to itself
                          // and be statically known at compile-time

Considering that ampersand literally means "etc.", maybe we could just drop the underscore and represent existentials via an appended "etc.":

var myView: View etc.
var array: [Hashable etc.]

Whatever we go with ultimately, I just ask that if we must add special syntax for existential container types, can we please do it in a very distinct and clear way that does not appear to be a "dual syntax" with "some"? That way we can clarify Swift instead of adding confusion to it.

Ideally the new syntax should make it clear that the existential container itself doesn't conform to the protocol, and hopefully if you google it, you get the exact result that explains what you want to know rather than returning every use of the word "any" throughout Swift.

1 Like
protocol P {}

struct S1: P {}
struct S2: P {}

func f() -> some P {
    .random()
        ? S1()
        : S2()
}

func g() -> P {
    .random()
        ? S1()
        : S2()
}

f() is illegal, because some P isn't an existential. One cannot return any conforming type, but must choose a single type.

g() returns an existential, and thus can return an instance of any type which conforms to P.

Ergo, some P is not an existential. It's a stand-in for a concrete type, chosen by the developer, and checked by the compiler. If the compiler cannot verify that the underlying concrete type is the same, it refuses to compile the function.

To me, an existential is a sort of proto-type (in Swit, a protocol) whose concrete type is not known at compile time. some P is not that.

3 Likes