SE-0309: Unlock existential types for all protocols

Both opaque types (some T) and existentials (any P) can be thought of as boxes (even if technically it’s not necessarily accurate)

In case of opaque type (some), the box is locked to single type inside that cannot be changed. This locked type can even be an existential in some cases (as shown upthread), but that doesn’t change the fact that box itself is still opaque, not existential. An array of some P can only contain same single type in all of the array’s elements.

In case of existential (any), the box is ambiquous in what it contains, and essentially erases all specifics about the type inside, except what is exposed by the P itself, i.e the protocol’s methods. The existential box can contain many different types, e,g when used in an array, and all those types inside promise to provide the methods from the protocol P.

2 Likes

For me, a box is an implementation detail, but the term is generally not well-defined

Type some P is the unification of all conforming types to P, hence some P is an existential type, the same holds for any P.

In difference to any P, a variable of type some P is restricted not to change the underlying type once a value was assigned.

I will take thought of how the wording here can be improved to address yours and everyone else's feedback, thanks! The contrast with some types is deliberate and is meant to help readers grasp a distinction in comparison. In the wild, these two abstraction models are very much related due to their deceptive resemblance, and it is not uncommon for people to wonder why we need one when we have the other.

1 Like

In my understanding, that is not accurate. some P refers to the return type of a property or method, and the compiler enforces the actual identity of P to be the same from all return paths in the property or method definition. That means each usage of some P only ever refers to one concrete P. This cannot change at build time, never mind at runtime.

2 Likes

Referring to some P as ‘reverse generics’ initially made the concept of some much clearer to me.
Whereas the generic types of a function are decided by the caller, the specific type of P in a some P return type is decided by the function implementation.

1 Like

I do agree we need to disambiguous the meaning of existential type and protocol constraints, but I doubt the 'any' is a good choice. As far as I read the discussion above, there are no clear contrasting relationship between existential types and opaque types that any/some suggests.

I really agree this concern. For me it feels exactly any concrete type that conforms to Usable. It also seems similar to generic result types.

Also, if I understand correctly, currently, it is not the clearly-agreed direction to use some for generics discussed here. If we don't take the direction, there is possibility to use any for other purpose, for example, some for 'reverse generics' and any for 'generics'.

I'm +1 on unlocking existential types for all protocols, but -1 on making it stable direction to use any P for existential types. At least, it should be decided after the whole direction of generics UI is decided. For what the any should be used, should be more carefully considered.

Of course, if you put a box inside a box, then the combination of those boxes results in a different behaviour than using just a single box. This is true for any similar situation in programming.

For example:

class C {
  var a: String = ""
}

struct S {
  var c = C()
}

The fact that you can put a class C inside a struct S means that struct no longer is a full value type, but instead now partially has reference semantics. But that doesn't mean S should now be called a "class" or should be created with some other syntax, it just means you have put a box inside another box and the resulting behaviour is the combination of those two boxes.

I don't know if putting any P inside some T has any real life use cases, but if there are, then those behaviours are then wanted and expected similar to using classed inside structs. Language design is not done by exceptional situations or corner cases, and in case of any P and some T in most situations I think they would be used separately where the behavior is then actually different.

1 Like

Where to use these keywords and when, needs to be carefully considered, indeed, since people don't understand the difference of behaviour that there is between existentials and generics, and that is made worse by the Swift's syntax not making enough difference between the two, causing people to make assumptions that they are the same or similar.

NOTE: below is my best understanding of the concepts, feel free to point out anything that isn't correct in regards to the way that Swift implements these concepts.

Both generics and existentials can be used in many places, as return type, as function parameter, or as property declaration, for example. Each of those uses have slightly different peculiarities, but for now let's just acknowledge that we are not only talking about return types with any P / some T.

Now, since "any" isn't in use in Swift yet, we don't have a formal description of what it's used for, but the idea so far has been to syntactically highlight existentials as any P, which means the protocol should be treated as a type, and to distinguish them from using a protocol as a constraint for some other type (i.e. a another type conforms to a certain set of rules). So the primary reason for that syntax is to separate protocols from protocols (existentials from constraints).

Generics and some T
Opaque types (where syntax is some T) are used in Swift e.g. as a return type. Their implementation is based on generics, i.e. locking down to a single type at compile time, which cannot be changed, and obviously doesn't change at run time either. You can think of compiler replacing the generics syntax with the actual concrete type at compile time and thus at runtime there isn't even any generic-related abstractions to be seen, which makes the code have very good performance.

Currently we have reverse generics with opaque type both in return type and as property

protocol P {}
extension Int: P {}
extension String: P {}

let value: some P = 1
func (value: Int) -> some P {
    return value
}

You could in the future have also some in function parameter position (possibly normal generics or reverse generics?)

func foo(value: some P) 

For example, if you pass 1 to the function foo, then only Int can be stored, and no changes can be done to that inside the function or at runtime.

so the word "some" here is meant to signify "a single concrete type from a set of possible types"

Existentials and any P
Existentials have type erasing behaviour, i.e. by nature those types only know about what is provided by the constraint (the interface defined by the protocol) and allows for using any of the types that conform to that interface to be used in that type. In case of collections with existential we typically talk about heterogenous collections i.e. collection does not need to contain single concrete type, instead it can contain many different types that just appear on the surface as the same existential. This means compiler keeps all the type abstractions around; both when compiling and at runtime, and this means the concrete type may change in the code, as long as it conforms to the constraints (i.e. the protocol). While this is less performant code, it allows you to have collections which can can contain many different types.

protocol P {}
extension Int: P {}
extension String: P {}

func foo(collection: any [P]) 

For example you could pass [1, "a", 3] to the function foo, and it allows using it, whereas in generics this would be an error.

so the word "any" here is meant to signify "any of the concrete types from a set of possible types"

EDIT: in a separate thread it was noted that having some as sugar for normal generics in function parameter might not make sense, but instead some would make more sense when used as reverse generics in that situation. Reverse generics for argument type? - #14 by xAlien95
I have updated this post to reflect those thoughts.

4 Likes

I totally agree with your explanation.

But I’m worried about this point. Is it agreed direction? As far as I know using some for 'anonymous generics' is still not agreed direction now. If this syntax were to reject, the contrast of any and some collapsed. Then it would become more confusing than ever. If this direction is already formally agreed and there is no possibility to be rejected, I also think using any for existential type is great.

I'm now considering about reverse generics for argument. I think this would potentially threaten the direction to use some for normal generics. Because it would be off-topic, I opened a new topic. I'd happy if you check it.

I hope my concerns are wrong. In that case, I also agree with the use of any P for existential types.

I've reread the motivation section and I see that the term existential == existential value is tied to the term box and dynamic and indirectly to the term mutability because mutability requires to use a box.

I think we are confusing the abstraction with the implementation, here.
An instance of an existential type is generally an existential independent if we use a box or any other kind of implementation in the background.
So an instance of some P isn't an existential because we can optimize it out at compile time?

Given the following example ( I know it isn't currently possible, but I'm sure it will in the future):

func method(variable:some P){...}

Imagine, that we don't generate specialized methods expecting an instance of some P for each different concrete type, instead the compiler chooses the way to simply make a box for the variable of some P and generates only one method in the backend.
Does the variable of type some P then becomes an existential value?

Note, that some P isn't even transparent to the type system, rather it is transparent to the optimizer. Otherwise, some P would influence dispatching and allow concretizing the type behind some P at compile time.
So I wouldn't even consider some P as a type alias.

Edit: change opaque to transparent, sorry.

It's not an existential because the concrete type must be known to the compiler at build time. Otherwise, the compiler can't ensure that all return paths are returning the same type.

4 Likes

The latter doesn't imply the former.

If you had read the explanation I wrote upthread, you'd know that some refers to generics, where type is locked down to a single concrete type already at compile time. If you want to have existential, then you would need to use any P. With existentials the type remains abstracted and undecided at compile time.

With some the type is opaque to user of the function (e.g. app code using a function from a library) but it's transparent (i.e. the concrete type) to the compiler and optimizer, meaning you get the best performance from it.

Agree.

If you had read the explanation I wrote upthread, you'd know that some refers to generics,

It has a relation to them, but it isn't the same as I said upthread, at least not then anymore when they are allowed in argument type position, then the behavior differs between generics and some P.

I disagree. But maybe the term existential is redefined in Swift context, but this would lead to more confusion for users coming from other languages already knowing existentials.

I don't know how it differs from other programming languages, but in swift the existential means protocol as a type, which is not the same as protocol itself. Protocol on its own is just a definition of constraints (set of rules).

For swift, the description of existentials is here: The Swift Programming Language: Redirect and in context of swift forums, i would expect existentials to be used in that definition unless explicitly mentioned what meaning is meant for it.

Some Info: I meant opaque types are not-transparent (opaque) to the type system and transparent to the optimizer, sorry.

They are mentioning existential types, only. Not existential values. And why should some P not be a protocol type, i.e. an existential type.
Both some P and any P allow the same set of values from a type theoretic perspective.

The fundamental difference between some and any is what happens when that code is compiled. Just as with e.g. structs and classes, a very similarly looking code has very different behaviour. In case of structs and classed you end up in a very different place behaviour wise, when you make a copy of it.

In case of generics (i.e. some) you end up with only the concrete type in the resulting compiled code. There's no "generics" to be found anymore, it's been replaced by that concrete type everywhere. And this means when code is compiled, the P is locked to single type, e.g. Int. So all values in this case are "Int", not a "generic value" or "protocol value".

In case of existentials (i.e. any), you have abstracted code even after it has been compiled, and if you want to know the real concrete type, you have to explicitly query for it. This means when you use existential as a function parameter, you can "open" it to reveal the instance of the concrete type that actually is being stored in there. Existential itself isn't really a value, it's merely a box that contains a concrete type value which can be one of many. So after compile, any P still can be any of Int, String, ..., whatever type that complies with the rules of P.

Holds only for return type position and is as I said an implementation detail.

I've heard that this is also true for some P types in some cases.

Existentials relate to the variable or more concretely to the instance behind the variable which isn't the same as the value.

I already said it upthread, I don't consider existentials tied to the term box or dynamic/runtime, existentials exists outside these concepts.
For me, an existential is an instance of an existential type, which some P is.

I don't think the definitions in Swift programming language agree with your view of the term "existential" and also Opaque types, as currently implemented in Swift 5.4, which use the "some P" as their syntax, do not have anything to do with existentials (as defined by Swift), but are instead implemented using generics, as far as I know. Generics uses protocols (as constraints) as much as existentials do, but generics does not use protocols as a type, they instead replace the constrained type with the real concrete type. The keywords protocol and existential cannot be used interchangeably.

This all feels like it has very little to do with reviewing the proposal. If the definition of "existential" is in question, perhaps that discussion can be had in a separate thread?

16 Likes