SE-0309: Unlock existential types for all protocols

[jayton (Jens Ayton)]

Thank you. I’ve written several drafts on the theme of “some T isn’t a type, it’s a type variable”, but your presentation is much clearer, and correct.

[sighoya]

@QuinceyMorris

OTOH it makes no sense to ask whether type name “ SecretSauce ” is existential. A type SecretSauce is existential or not, regardless of how it’s named or referred to.

Well, yes, but when we talk about types as existentials then we usually mean existential type.

@QuinceyMorris

“Opaque” is a qualifier of type names, not of types. “Existential” is a qualifier of types, not type names. Asking “Is a given type name existential?” is of course nonsensical.

In case of Swift, I would say no. Both any and opaque would be type qualifiers.
I don't know what you mean with name qualifiers, there are type qualifiers which can be seen as type constructors, source here.

And again, it is unclear if the existential in your mind is the same for the rest. The problem is that you talk about the meaning of existential in the term/grammar system in Swift. Yes, it is a qualifier, but we talk about the semantic difference between existential types and opaque types which your conclusion doesn't contribute to.

[1oo7 (J. Shad.)]

@Zollerboy1

@1oo7

> // can we please not make this possible:
let something: [any AnyCancellable] as Any
> ```

This would not really happe

Need I remind you that Any is a protocol?

And so under the regime of "any" we would have to declare like this:

let x: any Any

[xwu (Xiaodi Wu)]

@1oo7

And so under the regime of "any" we would have to declare like this:

let x: any Any

I think we can safely say that under no circumstance would Swift adopt the spelling any Any. Any is not an ordinary protocol (for example, you cannot extend Any), and there is no reason why it would have to adopt such a silly spelling.

[1oo7 (J. Shad.)]

@xwu

struct E: Error { }

func f() -> /* any */ Error { E() }
func g() -> some Error { f() }

var x = f()
x = g() // Allowed because `x` can be bound to any error (it is of existential type).

var y = g()
y = g() // Allowed because this is *always* the same underlying type.
y = f() // Compiler error. Not allowed because one cannot use knowledge of the actual underlying type.

This is an essential point to understand. The underlying type is immaterial: some Error is not an existential type.

I require further clarification on this.. not convinced yet.

If we change the last line of your above code to the following, then we will actually get a better picture of what is really happening:

func o<T: Error>(_ t: T) -> some Error {t}

y = o(f()) 

This gives the more detailed error:

Cannot assign value of type 'some Error' (result of 'o') to type 'some Error' (result of 'g()')

Your argument is that 'some Error' (result of 'o') is the actual type here, not Error—however I'm not convinced yet, because I believe this is just the compiler lying to us on purpose.

In the SIL, I noticed that indeed, each opaque type is a just a function that returns the actual type it conceals. For an opaque-typed variableName that's a member of TypeName in ModuleName, this opaque type function will be of the form:

 $@_opaqueReturnTypeOf("$s4ModuleName3TypeNameV7variableNameQrvp", 0) 🦸

Yes, that disguised super-heroine emoji is actually in the SIL. Her face is aptly concealed by a mask. She's... some Superheroine.

Anyway, the reason I'm not persuaded that this function should be considered as the actual type of the opaque value returned by g(), is because clearly this is just function for getting that actual type.

And as we shall see, we can easily force the compiler to call this function and thereby prove that this is, in fact, the existential.

To prove that indeed some Error actually is the existential type, given the above declarations of yours, I submit to you:

func a<T>(_ x: T, _ y: T) -> T.Type { T.self }

struct R<W> {
    var t: W.Type { W.self }
    init(_: W.Type) {}
}

let t: Any.Type = R(a(g(), f())).t
print("\(t)") // prints "Error" (the existential!)

Unless you can convince me that, in fact, T != T, or that T somehow does not capture the static type of whatever gets returned from g(), or that print somehow coerces variable t (a constant!) to change its type to the existential, then I am prepared to rest my case that the actual type of of some Error returned by g() is, in fact the existential container type of Error.

For further evidence, I present:

struct B<T: Error> {
    var a: T
    var b: T
}

var b = B(a: y, b: f()) // works
b.a = f() // works
b.b = y // works
b.a = g() // works
b.b = f() // works

This works just like the other case, because if a's type is concealed but b's isn't, since they have to be the same type in order for the code to compile, this forces the compiler to check whether "getOpaqueReturnType(g())" returns the same type as the already-known type of b (Error).

The complier figures that since you already know the type Error, then there's no harm in tipping it's hand, because we're not increasing the level of implementation details that we share with the client so it's fine to spill the beans.

Meanwhile:

var c = B(a: y, b: y)
c.a = f() // doesn't compile

This doesn't compiler because a this stage, the type information of both a and b is still concealed behind a function that hasn't been called yet. Revealing this type information could expose hidden implementation details to the client since at this point the compiler has no evidence that the caller already knows what the return type would be.

However the fact remains that concealing something doesn't change what it fundamentally is; it's just slight-of-hand. A shell game.

Or perhaps you'll convince me why I'm wrong.

[1oo7 (J. Shad.)]

@Joe_Groff

There is an "existential specializer" pass that turns some functions with existential arguments into their generic forms, which allows generic specialization passes to further optimize them. You could maybe use @inline(never) to suppress specialization, but there still isn't a fully designed and developed suite of attributes for controlling optimizer behavior.

This is extremely interesting.

I really hope this "suite of attributes for controlling optimizer behavior" can soon be added to Swift, because the most legitimate complaint I've heard about Swift recently from our principal engineer (who is primarily a Java/Kotlin dev) relates to the fact that Swift has unpredictable/inconsistent treatment of how it resolves which overloaded generic method should get called.

Their expectation is that the most specialized method should always get called, since allows you to write code that is "closed for modification but open for exetnsion".

You see, if the consumer of a module can add more specialized methods in extensions, which overload less specialized methods in the base implementation, then they can extend and modify its behavior as if it were an open class and they were overriding its methods with a subclass. Except now you don't have the limitations of class inheritance to worry about—it's all protocols.

However currently we can't really do this in Swift, and clearly we'd need something like an open protocol to accomplish it. And of course if we're doing an open protocol then people will likely want a sealed protocol too.

[Jumhyn (Frederick Kellison-Linn)]

@1oo7

This works just like the other case, because if a 's type is concealed but b 's isn't, since they have to be the same type in order for the code to compile, this forces the compiler to check whether "getOpaqueReturnType(g())" returns the same type as the already-known type of b ( Error ).

I haven’t followed this discussion closely enough to comment on the broader aspects, but to this point specifically—I suspect that this works not because the compiler is allowing the user to discover the type behind the opaque type, but rather because Error is an acceptable supertype of some Error, and so once Error is discovered as the type for b the compiler is free to upcast y from some Error to Error (which is also what happens in the subsequent assignments).

[1oo7 (J. Shad.)]

@Avi

protocol P {}

struct S1: P {}
struct S2: P {}

func f() -> some P {
    .random()
        ? S1()
        : S2()
}

func g() -> P {
    .random()
        ? S1()
        : S2()
}

f() is illegal, because some P isn't an existential. One cannot return any conforming type, but must choose a single type.

g() returns an existential, and thus can return an instance of any type which conforms to P .

Ergo, some P is not an existential. It's a stand-in for a concrete type, chosen by the developer, and checked by the compiler. If the compiler cannot verify that the underlying concrete type is the same, it refuses to compile the function.

Let me explain why this is incorrect.

Your first mistake was that you started with a protocol P that could not self-conform, so even if .random() ? S1() : S2() had the implicit type P (it doesn't), it still wouldn't be able to be returned from f() because the return type some P requires that whatever type is returned by this function must conform to P.

Your second mistake was that if you want to return an existential from f() then you need to wrap your heterogenous .random statement in parentheses and then dynamically cast it to the existential (because the compiler won't do that for you).

Here's the fixed version, which compiles and runs fine :D

import Foundation

@objc protocol P {}
class S1: P {}
class S2: P {}

func f() -> some P { (.random() ? S1() : S2()) as P }
func g() -> P { .random() ? S1() : S2() }

[1oo7 (J. Shad.)]

@Jumhyn

I suspect that this works not because the compiler is allowing the user to discover the type behind the opaque type, but rather because Error is an acceptable supertype of some Error , and so once Error is discovered as the type for b the compiler is free to upcast y from some Error to Error (which is also what happens in the subsequent assignments).

Yeah on further inspection I think you might be right.

[xwu (Xiaodi Wu)]

@1oo7

This gives the more detailed error:

Cannot assign value of type 'some Error' (result of 'o') to type 'some Error' (result of 'g()')

Your argument is that 'some Error' (result of 'o') is the actual type here, not Error —however I'm not convinced yet, because I believe this is just the compiler lying to us on purpose.

In the SIL, I noticed that indeed, each opaque type is a just a function that returns the actual type it conceals.

It’s irrelevant to this discussion how something is implemented in SIL. The point is that, in Swift (not SIL), the type is 'some Error' (result of 'o'). That is the user-facing model of the language; the compiler is telling you as much. That is why this feature is called an opaque type.

@1oo7

Or perhaps you'll convince me why I'm wrong.

This analysis is incorrect for the reason explained by @Jumhyn:

• In the case of var b = B(a: y, b: f()), B is inferred as B<Error>, because y (like any other value of type some Error) can be upcast to Error, which conforms to Error.
• In the case of var c = B(a: y, b: y), B is inferred as B<some Error (result of 'o')>.

@1oo7

But I think that doesn't mean I'm necessarily wrong, because due to the circular nature of the argument, both can be true.

Problem is, that whole example was to show that when a self-conforming existential is returned as some P , it's still an existential type once we trick the compiler into tipping it's hand.

An opaque type is always opaque: that is the entire point of an opaque type. In no case here does the compiler “accidentally” allow you to make use of your knowledge of the underlying type; it would be a bug if it did. To emphasize, in no way is the underlying type opaque to the compiler, it is opaque only to the user.

[1oo7 (J. Shad.)]

@xwu

It’s irrelevant to this discussion how something is implemented in SIL.

So on the one hand, how something is implemented in SIL is irrelevant to this discussion—but on the other hand, we must consider an invisible, inferred casting behavior as the explanation?

BTW I think you're right, I just think that it's also valid to bear in mind that Swift is a front-end to write SIL. It has really helped me to understand how Swift works to look at that, even if I usually understand it wrong until you explain what's really happening :D

[Jumhyn (Frederick Kellison-Linn)]

@1oo7

I'll be persuaded this might not be the case if you can show me an example where the same kind of "automatic upcast" happens for anything other than a self-conforming existential of the same protocol as the opaque type.

An opaque type some P is known to be a type that conforms to P, so the compiler will always be able to perform this upcast (now that we can form an existential of any protocol):

protocol P {}
struct S: P {}

func f() -> some P { S() }

let p: P = f() // automatic upcast from 'some P' to 'P'

[Avi]

You're the only one going on about self-conformance, and some P has nothing to do with self-conformance. You keep introducing it where it's not wanted to prove a point that is not relevant to the concept of some P. You didn't "fix" my example. You turned it into something no one was talking about.

[xwu (Xiaodi Wu)]

@1oo7

So on the one hand, how something is implemented in SIL is irrelevant to this discussion—but on the other hand, we must consider an invisible, inferred casting behavior as the explanation?

Perhaps the language isn’t terribly clear, but I’m just describing the same type inference that applies everywhere. There is nothing magical going on.

[Avi]

That example is legal today, because it's not functionally different than

protocol P {}
struct S: P {}

func f() -> S { S() }

let p: P = f()

However, the reverse is not true. You cannot do this:

protocol P {}
struct S: P {}

func f() -> some P { S() }

let p: S = f()

Because the entire point of opaque types is that the type is hidden from the developer. Allowing the type to be explicitly declared by the user (let p: S ...) would undermine that.

[Jumhyn (Frederick Kellison-Linn)]

Sure, that example was very specifically targeted at @1oo7’s request to show an opaque type getting upcast to its existential when the underlying type of the opaque type is not the existential itself.

[1oo7 (J. Shad.)]

@xwu

Perhaps the language isn’t terribly clear, but I’m just describing the same type inference that applies everywhere. There is nothing magical going on.

It's the impossibility of the other direction of the cast that seemed magical to me... at least, until I understood that an opaque type (under the hood) is basically a typealias to a function type for a function that returns the underlying type that's been made opaque, e.g. typealias opaqueReturnType<T>() -> T: Foo.Type ... where the return value can very well be the existential type Foo.

Because If that's the case, then it makes it much more sense why we cannot cast to an opaque type, since in Swift you can't cast one closure type to another.

I find that explains why an opaque type is a different type than the existential type it might conceal when its value is of that existential type.

So thanks for that clarification

[Avi]

I understand what you're saying, but I think a point is being missed. Opaque types require the caller to always treat the return value as the existential. It's not something that happens only in specific examples. That doesn't ever mean the compiler doesn't know the actual type, only that the caller must act as if it's unknown. Dynamic casts work as usual (as! or as?), but not static casts (as).

[1oo7 (J. Shad.)]

@Avi

Opaque types require the caller to always treat the return value as the existential

What about opaque types where the protocol is not allowed to have an existential? (As in Swift 5.4 if it has an associatedtype requirement.)

Or are you referring to what will happen after this proposal goes in?

[Avi]

@1oo7

until I understood that an opaque type (under the hood) is basically a typealias to a function type for a function that returns the underlying type that's been made opaque, e.g. typealias opaqueReturnType<T>() -> T: Foo.Type ... where the return value can very well be the existential type Foo .

I think you're mental model of opaque types is needlessly complicated. Opaque types simply aren't types, in the computer science sense of the term. They aren't like Int or String and they aren't like Hashable or Equatable.

Opaque types describe a mechanism for hiding the statically-known type of the return value from the caller. The caller is not allowed to act as if it knows the static type, even if the developer writing the code knows the type. Whether that static type is a concrete type or an existential has no bearing on the model.