SE-0309: Unlock existential types for all protocols

All “self-conformances” are special cases: there can be no implicit self-conformance in the general case. We are talking about them because @1oo7 is concerned about how these special cases work. There is nothing special about Error here (for the purposes of this discussion) other than it’s a self-conformance that you can actually try for yourself on all platforms.

4 Likes

I hope that the compiler did the right thing in most cases when considering generics.
However, I wish some kind of annotation to overwrite the decision of the compiler.

Using generics or not shouldn't fix boxing or code generation for the backend.

The compiler can specialize either form—they are isomorphic—so this is not a reliable way of controlling code size.

7 Likes

In first case instance of SomeProtocol can have different underlying types. In second case Type is known at compile time. What do you mean, saying "The compiler can specialize either form"?

"this is not a reliable way of controlling code size" – totally agree here, I only handle that such idea exists. Does Swift 5.4 compiler generates binaries of the same size for this two functions, or may be it will do in future?

In cases where SomeProtocol existentials are statically resolvable, i.e. you know the underlying type, and the underlying type doesn't change in the function body you can specialize functions accepting runtime existentials, but not always.

Neither of these statements is always true. If you pass a value of known static type into an existential, then the compiler can see that and still specialize the function based on the type going in. Similarly, if you call the generic function with a dynamic type, the compiler does not necessarily know what that type is, and may not be able to specialize. Both functions accept a generic type and a value of that type; the existential form just packs them into a single value instead of two separate arguments.

There is an "existential specializer" pass that turns some functions with existential arguments into their generic forms, which allows generic specialization passes to further optimize them. You could maybe use @inline(never) to suppress specialization, but there still isn't a fully designed and developed suite of attributes for controlling optimizer behavior.

8 Likes

Thanks for sharing these interesting details.

4 Likes

This would not really happen…A protocol should never have the prefix Any, normally it’s given to a type-erasing wrapper type (which normally is a struct and could never have another any in front of it).
So the Array type would instead be written either as [any Cancellable] or as [AnyCancellable] (depending on whether you want an array of existentials or an array of wrapper types), but never [any AnyCancellable] (unless you aren’t following standard naming rules and give your protocol the name AnyCancellable).

1 Like

The discussion of some P vs. any P has gone off the rails. It’s conflating type names with types. Look at this example:

func getSauce() -> some Sauce {
    return SecretSauce()
}

Here, the returned value has type SecretSauce, whether the caller knows it or not. There are two different names, “SecretSauce” and “some Sauce”, but the names really do refer to the same type (well, they do in this context — “some Sauce” can refer to a different type when it’s the return type of a different function).

It makes no sense to ask whether SecretSauce is an opaque type, whatever that might mean. The opaqueness here belongs to the type namesome Sauce”, and indicates that the caller typically doesn’t know what it names.

OTOH it makes no sense to ask whether type nameSecretSauce” is existential. A type SecretSauce is existential or not, regardless of how it’s named or referred to.

“Opaque” is a qualifier of type names, not of types. “Existential” is a qualifier of types, not type names. Asking “Is a given type name existential?” is of course nonsensical.

7 Likes

Thank you. I’ve written several drafts on the theme of “some T isn’t a type, it’s a type variable”, but your presentation is much clearer, and correct.

1 Like

Well, yes, but when we talk about types as existentials then we usually mean existential type.

In case of Swift, I would say no. Both any and opaque would be type qualifiers.
I don't know what you mean with name qualifiers, there are type qualifiers which can be seen as type constructors, source here.

And again, it is unclear if the existential in your mind is the same for the rest. The problem is that you talk about the meaning of existential in the term/grammar system in Swift. Yes, it is a qualifier, but we talk about the semantic difference between existential types and opaque types which your conclusion doesn't contribute to.

Need I remind you that Any is a protocol?

And so under the regime of "any" we would have to declare like this:

let x: any Any

I think we can safely say that under no circumstance would Swift adopt the spelling any Any. Any is not an ordinary protocol (for example, you cannot extend Any), and there is no reason why it would have to adopt such a silly spelling.

3 Likes

I require further clarification on this.. not convinced yet.

If we change the last line of your above code to the following, then we will actually get a better picture of what is really happening:

func o<T: Error>(_ t: T) -> some Error {t}

y = o(f()) 

This gives the more detailed error:

Cannot assign value of type 'some Error' (result of 'o') to type 'some Error' (result of 'g()')

Your argument is that 'some Error' (result of 'o') is the actual type here, not Error—however I'm not convinced yet, because I believe this is just the compiler lying to us on purpose.

In the SIL, I noticed that indeed, each opaque type is a just a function that returns the actual type it conceals. For an opaque-typed variableName that's a member of TypeName in ModuleName, this opaque type function will be of the form:

 $@_opaqueReturnTypeOf("$s4ModuleName3TypeNameV7variableNameQrvp", 0) 🦸

Yes, that disguised super-heroine emoji is actually in the SIL. Her face is aptly concealed by a mask. She's... some Superheroine.

Anyway, the reason I'm not persuaded that this function should be considered as the actual type of the opaque value returned by g(), is because clearly this is just function for getting that actual type.

And as we shall see, we can easily force the compiler to call this function and thereby prove that this is, in fact, the existential.

To prove that indeed some Error actually is the existential type, given the above declarations of yours, I submit to you:

func a<T>(_ x: T, _ y: T) -> T.Type { T.self }

struct R<W> {
    var t: W.Type { W.self }
    init(_: W.Type) {}
}

let t: Any.Type = R(a(g(), f())).t
print("\(t)") // prints "Error" (the existential!)

Unless you can convince me that, in fact, T != T, or that T somehow does not capture the static type of whatever gets returned from g(), or that print somehow coerces variable t (a constant!) to change its type to the existential, then I am prepared to rest my case that the actual type of of some Error returned by g() is, in fact the existential container type of Error.

For further evidence, I present:

struct B<T: Error> {
    var a: T
    var b: T
}

var b = B(a: y, b: f()) // works
b.a = f() // works
b.b = y // works
b.a = g() // works
b.b = f() // works

This works just like the other case, because if a's type is concealed but b's isn't, since they have to be the same type in order for the code to compile, this forces the compiler to check whether "getOpaqueReturnType(g())" returns the same type as the already-known type of b (Error).

The complier figures that since you already know the type Error, then there's no harm in tipping it's hand, because we're not increasing the level of implementation details that we share with the client so it's fine to spill the beans.

Meanwhile:

var c = B(a: y, b: y)
c.a = f() // doesn't compile

This doesn't compiler because a this stage, the type information of both a and b is still concealed behind a function that hasn't been called yet. Revealing this type information could expose hidden implementation details to the client since at this point the compiler has no evidence that the caller already knows what the return type would be.

However the fact remains that concealing something doesn't change what it fundamentally is; it's just slight-of-hand. A shell game.

Or perhaps you'll convince me why I'm wrong.

1 Like

This is extremely interesting.

I really hope this "suite of attributes for controlling optimizer behavior" can soon be added to Swift, because the most legitimate complaint I've heard about Swift recently from our principal engineer (who is primarily a Java/Kotlin dev) relates to the fact that Swift has unpredictable/inconsistent treatment of how it resolves which overloaded generic method should get called.

Their expectation is that the most specialized method should always get called, since allows you to write code that is "closed for modification but open for exetnsion".

You see, if the consumer of a module can add more specialized methods in extensions, which overload less specialized methods in the base implementation, then they can extend and modify its behavior as if it were an open class and they were overriding its methods with a subclass. Except now you don't have the limitations of class inheritance to worry about—it's all protocols.

However currently we can't really do this in Swift, and clearly we'd need something like an open protocol to accomplish it. And of course if we're doing an open protocol then people will likely want a sealed protocol too.

I haven’t followed this discussion closely enough to comment on the broader aspects, but to this point specifically—I suspect that this works not because the compiler is allowing the user to discover the type behind the opaque type, but rather because Error is an acceptable supertype of some Error, and so once Error is discovered as the type for b the compiler is free to upcast y from some Error to Error (which is also what happens in the subsequent assignments).

4 Likes

Let me explain why this is incorrect.

Your first mistake was that you started with a protocol P that could not self-conform, so even if .random() ? S1() : S2() had the implicit type P (it doesn't), it still wouldn't be able to be returned from f() because the return type some P requires that whatever type is returned by this function must conform to P.

Your second mistake was that if you want to return an existential from f() then you need to wrap your heterogenous .random statement in parentheses and then dynamically cast it to the existential (because the compiler won't do that for you).

Here's the fixed version, which compiles and runs fine :D

import Foundation

@objc protocol P {}
class S1: P {}
class S2: P {}

func f() -> some P { (.random() ? S1() : S2()) as P }
func g() -> P { .random() ? S1() : S2() }
1 Like

Yeah on further inspection I think you might be right.

It’s irrelevant to this discussion how something is implemented in SIL. The point is that, in Swift (not SIL), the type is 'some Error' (result of 'o'). That is the user-facing model of the language; the compiler is telling you as much. That is why this feature is called an opaque type.

This analysis is incorrect for the reason explained by @Jumhyn:

  • In the case of var b = B(a: y, b: f()), B is inferred as B<Error>, because y (like any other value of type some Error) can be upcast to Error, which conforms to Error.
  • In the case of var c = B(a: y, b: y), B is inferred as B<some Error (result of 'o')>.

An opaque type is always opaque: that is the entire point of an opaque type. In no case here does the compiler “accidentally” allow you to make use of your knowledge of the underlying type; it would be a bug if it did. To emphasize, in no way is the underlying type opaque to the compiler, it is opaque only to the user.

6 Likes

So on the one hand, how something is implemented in SIL is irrelevant to this discussion—but on the other hand, we must consider an invisible, inferred casting behavior as the explanation?

BTW I think you're right, I just think that it's also valid to bear in mind that Swift is a front-end to write SIL. It has really helped me to understand how Swift works to look at that, even if I usually understand it wrong until you explain what's really happening :D