SE-0309: Unlock existential types for all protocols

Of course, if you put a box inside a box, then the combination of those boxes results in a different behaviour than using just a single box. This is true for any similar situation in programming.

For example:

class C {
  var a: String = ""
}

struct S {
  var c = C()
}

The fact that you can put a class C inside a struct S means that struct no longer is a full value type, but instead now partially has reference semantics. But that doesn't mean S should now be called a "class" or should be created with some other syntax, it just means you have put a box inside another box and the resulting behaviour is the combination of those two boxes.

I don't know if putting any P inside some T has any real life use cases, but if there are, then those behaviours are then wanted and expected similar to using classed inside structs. Language design is not done by exceptional situations or corner cases, and in case of any P and some T in most situations I think they would be used separately where the behavior is then actually different.

1 Like

Where to use these keywords and when, needs to be carefully considered, indeed, since people don't understand the difference of behaviour that there is between existentials and generics, and that is made worse by the Swift's syntax not making enough difference between the two, causing people to make assumptions that they are the same or similar.

NOTE: below is my best understanding of the concepts, feel free to point out anything that isn't correct in regards to the way that Swift implements these concepts.

Both generics and existentials can be used in many places, as return type, as function parameter, or as property declaration, for example. Each of those uses have slightly different peculiarities, but for now let's just acknowledge that we are not only talking about return types with any P / some T.

Now, since "any" isn't in use in Swift yet, we don't have a formal description of what it's used for, but the idea so far has been to syntactically highlight existentials as any P, which means the protocol should be treated as a type, and to distinguish them from using a protocol as a constraint for some other type (i.e. a another type conforms to a certain set of rules). So the primary reason for that syntax is to separate protocols from protocols (existentials from constraints).

Generics and some T
Opaque types (where syntax is some T) are used in Swift e.g. as a return type. Their implementation is based on generics, i.e. locking down to a single type at compile time, which cannot be changed, and obviously doesn't change at run time either. You can think of compiler replacing the generics syntax with the actual concrete type at compile time and thus at runtime there isn't even any generic-related abstractions to be seen, which makes the code have very good performance.

Currently we have reverse generics with opaque type both in return type and as property

protocol P {}
extension Int: P {}
extension String: P {}

let value: some P = 1
func (value: Int) -> some P {
    return value
}

You could in the future have also some in function parameter position (possibly normal generics or reverse generics?)

func foo(value: some P) 

For example, if you pass 1 to the function foo, then only Int can be stored, and no changes can be done to that inside the function or at runtime.

so the word "some" here is meant to signify "a single concrete type from a set of possible types"

Existentials and any P
Existentials have type erasing behaviour, i.e. by nature those types only know about what is provided by the constraint (the interface defined by the protocol) and allows for using any of the types that conform to that interface to be used in that type. In case of collections with existential we typically talk about heterogenous collections i.e. collection does not need to contain single concrete type, instead it can contain many different types that just appear on the surface as the same existential. This means compiler keeps all the type abstractions around; both when compiling and at runtime, and this means the concrete type may change in the code, as long as it conforms to the constraints (i.e. the protocol). While this is less performant code, it allows you to have collections which can can contain many different types.

protocol P {}
extension Int: P {}
extension String: P {}

func foo(collection: any [P]) 

For example you could pass [1, "a", 3] to the function foo, and it allows using it, whereas in generics this would be an error.

so the word "any" here is meant to signify "any of the concrete types from a set of possible types"

EDIT: in a separate thread it was noted that having some as sugar for normal generics in function parameter might not make sense, but instead some would make more sense when used as reverse generics in that situation. Reverse generics for argument type? - #14 by xAlien95
I have updated this post to reflect those thoughts.

4 Likes

I totally agree with your explanation.

But I’m worried about this point. Is it agreed direction? As far as I know using some for 'anonymous generics' is still not agreed direction now. If this syntax were to reject, the contrast of any and some collapsed. Then it would become more confusing than ever. If this direction is already formally agreed and there is no possibility to be rejected, I also think using any for existential type is great.

I'm now considering about reverse generics for argument. I think this would potentially threaten the direction to use some for normal generics. Because it would be off-topic, I opened a new topic. I'd happy if you check it.

I hope my concerns are wrong. In that case, I also agree with the use of any P for existential types.

I've reread the motivation section and I see that the term existential == existential value is tied to the term box and dynamic and indirectly to the term mutability because mutability requires to use a box.

I think we are confusing the abstraction with the implementation, here.
An instance of an existential type is generally an existential independent if we use a box or any other kind of implementation in the background.
So an instance of some P isn't an existential because we can optimize it out at compile time?

Given the following example ( I know it isn't currently possible, but I'm sure it will in the future):

func method(variable:some P){...}

Imagine, that we don't generate specialized methods expecting an instance of some P for each different concrete type, instead the compiler chooses the way to simply make a box for the variable of some P and generates only one method in the backend.
Does the variable of type some P then becomes an existential value?

Note, that some P isn't even transparent to the type system, rather it is transparent to the optimizer. Otherwise, some P would influence dispatching and allow concretizing the type behind some P at compile time.
So I wouldn't even consider some P as a type alias.

Edit: change opaque to transparent, sorry.

It's not an existential because the concrete type must be known to the compiler at build time. Otherwise, the compiler can't ensure that all return paths are returning the same type.

4 Likes

The latter doesn't imply the former.

If you had read the explanation I wrote upthread, you'd know that some refers to generics, where type is locked down to a single concrete type already at compile time. If you want to have existential, then you would need to use any P. With existentials the type remains abstracted and undecided at compile time.

With some the type is opaque to user of the function (e.g. app code using a function from a library) but it's transparent (i.e. the concrete type) to the compiler and optimizer, meaning you get the best performance from it.

Agree.

If you had read the explanation I wrote upthread, you'd know that some refers to generics,

It has a relation to them, but it isn't the same as I said upthread, at least not then anymore when they are allowed in argument type position, then the behavior differs between generics and some P.

I disagree. But maybe the term existential is redefined in Swift context, but this would lead to more confusion for users coming from other languages already knowing existentials.

I don't know how it differs from other programming languages, but in swift the existential means protocol as a type, which is not the same as protocol itself. Protocol on its own is just a definition of constraints (set of rules).

For swift, the description of existentials is here: The Swift Programming Language: Redirect and in context of swift forums, i would expect existentials to be used in that definition unless explicitly mentioned what meaning is meant for it.

Some Info: I meant opaque types are not-transparent (opaque) to the type system and transparent to the optimizer, sorry.

They are mentioning existential types, only. Not existential values. And why should some P not be a protocol type, i.e. an existential type.
Both some P and any P allow the same set of values from a type theoretic perspective.

The fundamental difference between some and any is what happens when that code is compiled. Just as with e.g. structs and classes, a very similarly looking code has very different behaviour. In case of structs and classed you end up in a very different place behaviour wise, when you make a copy of it.

In case of generics (i.e. some) you end up with only the concrete type in the resulting compiled code. There's no "generics" to be found anymore, it's been replaced by that concrete type everywhere. And this means when code is compiled, the P is locked to single type, e.g. Int. So all values in this case are "Int", not a "generic value" or "protocol value".

In case of existentials (i.e. any), you have abstracted code even after it has been compiled, and if you want to know the real concrete type, you have to explicitly query for it. This means when you use existential as a function parameter, you can "open" it to reveal the instance of the concrete type that actually is being stored in there. Existential itself isn't really a value, it's merely a box that contains a concrete type value which can be one of many. So after compile, any P still can be any of Int, String, ..., whatever type that complies with the rules of P.

Holds only for return type position and is as I said an implementation detail.

I've heard that this is also true for some P types in some cases.

Existentials relate to the variable or more concretely to the instance behind the variable which isn't the same as the value.

I already said it upthread, I don't consider existentials tied to the term box or dynamic/runtime, existentials exists outside these concepts.
For me, an existential is an instance of an existential type, which some P is.

I don't think the definitions in Swift programming language agree with your view of the term "existential" and also Opaque types, as currently implemented in Swift 5.4, which use the "some P" as their syntax, do not have anything to do with existentials (as defined by Swift), but are instead implemented using generics, as far as I know. Generics uses protocols (as constraints) as much as existentials do, but generics does not use protocols as a type, they instead replace the constrained type with the real concrete type. The keywords protocol and existential cannot be used interchangeably.

This all feels like it has very little to do with reviewing the proposal. If the definition of "existential" is in question, perhaps that discussion can be had in a separate thread?

16 Likes

It's not merely an implementation detail. It's a compiler-enforced rule. That is, the rule that all returned values must be of the same type logically allows the compiler to treat the method as-if it had been declared to return the concrete type.

Usage of some P in argument position would seem to make no sense. At that point, it's not hiding anything from the caller, and one can hide the concrete type from the callee today, by using generics.

This is a circular definition. What is an existential type?

In Swift a protocol type, whatever this means. Generally, a type which steal literals/values from other types, usually base types like integer, structs ... .

The discussion about that has moved to reverse-generics-for-argument-type

By that definition, some P isn't existential. some P is shorthand, with the added benefit of hiding the real type from the caller. But that's all a compiler trick. some P is a stand-in from the perspective of the developer, never the compiler.

some P behaves like the following:

// magic modifier to hide the real type from everyone else.
private(type) typealias HiddenP = Int

public func f() -> HiddenP {}

One would not consider HiddenP to be an existential, and some P is nothing more than Swift's actual implementation of my made-up typealias modifier.

4 Likes

It’s also very confusing for an average Swift user (like myself). I’m completely lost now. The waters have been very muddled and my understanding of this topic has regressed. I’m silently hoping for one of the more recognisable names to “lay down the law” here and declares what is what to get back on solid ground.

Now it might not matter whether I understand all the little details here since I can’t offer any sensible contribution anyway. I can only hope that in the end this feature and its nuances will be explainable and understandable to the average user.

2 Likes

I'm sorry for this. A new topic was opened for further discussion on the topic opaque types.
Our discussion about existentials vs opaque types hasn't changed the proposal yet, so you can ignore it.

1 Like

I'm in the same boat as you. I can't follow all the nuance and detail because I don't have a solid computer science background. But don't consider your contributions to be worthless or insensible. Swift has to be approachable by ordinary app developers, and not just language theorists and compiler devs. Any time you ask a question, even out of ignorance, you're signaling that something is not quite right.

Of course, there's a time and place. In the middle of a very technical discussion might not be the best place. But I think a Review post is -- it's the last time you and I get a voice unless and until an amendment is proposed in the future.

3 Likes