[Pitch] Implicitly opening existentials

Great proposal! I can’t wait to get all the pieces from the generics manifesto to play with, and this proposal is yet another step forward.

I think there’s a typo in the document which confused me for a bit.

func hasBells<C: Costume>(_ costume: Costume) -> Bool {
  return costume.hasSameAdornments(as: costume.withBells())
}

should probably be

func hasBells<C: Costume>(_ costume: C) -> Bool {
  return costume.hasSameAdornments(as: costume.withBells())
}

because a function signature needs to refer to all generic type parameters. If I then understand it correctly, the existential type opening is done when hasBells(_:) is invoked, and hasBells(_:) only deals with a “concrete” type C.

I presume this ends up doing the same thing as the function _openExistential?

There have definitely been times that tool seemed like the right thing, however it runs into he problem that the closure is not async or more generally is just a bit un-ergonomic; so this pitch definitely makes that considerably more approachable and feels like the right way of approaching the problem.

For someone who has used similar functionality via the use of underscored functions that do similar things a big +1 from me.

Agreed, having to explicitly open the existential here does feel like something we could do without, it would be great if it were implicit. I know there is a mountain of complexity with generics and existential types, but ultimately for developers the desire is to reason about capabilities over types and remove wasteful code duplication as well as increase type safety.

1 Like

I'm excited to see this, opening existentials is a major gap in Swift expressiveness.


I'm a bit concerned about the need to repeat the protocol name in some P. It does not look that bad in proposal with single-letter names, but I'm afraid that in production code it may be pretty verbose:

func authenticate(alternativeAuthenticationProviders: [any AlternativeAuthenticationProvider]) {
    for provider: some AlternativeAuthenticationProvider in alternativeAuthenticationProvider {
        ...
    }
}

Would it be possible to combine with proposal with SE-0315 to be able to use some _ when we want to open the existential, but don't want to change the type otherwise?

func authenticate(alternativeAuthenticationProviders: [any AlternativeAuthenticationProvider]) {
    for provider: some _ in alternativeAuthenticationProvider {
        ...
    }
}

I'm a bit paranoid about lack of name for the opened type when opening using some P. I would feel safer having something like let <T: P> openedX = x in my toolbox in addition to the proposed mechanism. But as a workaround, one can extract code into a generic function. So I guess it's not really an issue.


func cannotOpen7<T: P>(_ value: T) -> X<T> { /*...*/ }

For now, we could allow opening in this case, but erase result to Any. Or better to an existential for all protocols which X conforms to:

protocol P {}
protocol Q {}
protocol R {}
struct X<T: P>: Q, R {}
func cannotOpen7<T: P>(_ value: T) -> X<T> { /*...*/ }
let x: any P = ...
let y = cannotOpen7(x) // type of y is any Q & R

I've had cases like this in production, I think it has value. But if it is not supported, there is a workaround:

func cannotOpen7Boxed<T: P>(_ value: T) -> any Q & R {
    return cannotOpen7(value)
}
let y = cannotOpen7Boxed(x)

To fully support all cases of erasure after opening we would need generalised existentials which can box arbitrary complex generic type. Then this case would be type-erased to any<T: P> X<T>. And similarly result of

func decomposeQ<T: Q>(_ value: T) -> (T, T.B, T.B.A) {}

would erase to any<T: Q> (T, T.B, T.B.A).

If we get generalised existentials in the future, would changing erased type from (any Q, any P, Any) to any<T: Q> (T, T.B, T.B.A) be a source-breaking change?


I think there is a typo here - overloaded1 has two arguments, but is called with one:

protocol P { }

func overloaded1<T: P, U>(_: T, _: U) { } // A
func overloaded1<U>(_: Any, _: U) { }     // B

func changeInResolution(p: any P) {
  overloaded1(p) // used to choose B, will choose A with this proposal
}

I suspect some people will try to open existential in return statement. I don't think anything can be done to improve this, other than provide an informative error message:

func noOpeningInReturn(_ x: any P) -> some P {
    let y: some P = x // works
    return x // Does not work
    return y // Does not work either
}
1 Like

Never is convertible to every existential type, but it does not conform to every protocol:

protocol P {
    static func make() -> Self {}
}

func f<T: P>(x: T?) {
   print(x ?? T.make())
}

let x: (any P)? = nil
f(x) // It is not valid to substitute [T = Never] here
1 Like

Ahh thank you.
I realized that instance of Never never exists, but metatype of Never exists.
So static func of protocol cause problem about conforming by Never .

Yeah, _openExistential(x, f) becomes equivalent to f(x) with Doug's pitch. We could probably remove _openExistential() entirely, unfortunately a search of GitHub the other day revealed a handful of open source projects had introduced usages of it.

6 Likes

I can only speak for myself but I don't mind receiving the breakage of an underscore method that becomes full fledged language feature (provided I can still get the same effects for its usage).

So doing it the way outlined seems to avoid the closure (which might allocate), avoid captures (which might cause ref counting, allow for async/await. This all looks like win/win/win to me at the cost of a handful of folks being happy to be supported (e.g. they have to fix a small breakage of something they signed up for to have broken).

13 Likes

In layman's terms does this mean that type(of: existential) would return Concrete.Type? Isn't this source breaking and would break code which relies on it returning any (Protocol.Type)?

Small bit of confusion, is func hasBells<C: Costume>(_ costume: Costume) -> Bool meant to be func hasBells<C: Costume>(_ costume: C) -> Bool ?

Under this proposal, type(of:) would behave exactly as it does today, but without any special treatment. type(of: existential) implicitly opens existential, calls type<T>(of: T) with T bound to the dynamic type of existential, and then type-erases the returned T.Type back to the existential any Protocol.Type.

8 Likes

I think giving library developers some time to update their code isn’t that hard. Yes, this is an underscored feature and it can mostly be avoided by performing an operation in an extension of the target protocol. But there are cases where this feature is currently required for extending non-nominal protocols, which are found in large codebases. For example, SwiftUI’s AnyView.init(_fromValue:) probably uses _openExistential. The difference between large frameworks and independent libraries being that maintenance on the former is much easier. Of course, removing _openExistential won’t be the end of the world for smaller projects and neither will it cause significant breakage, but keeping it around for a little longer doesn’t seem too difficult either.

One thing I'm a little unsure of is how the compiler knows what's in the box. Afaiu the compiler knows the concrete type of some P, but not any P. In which case, I started pondering over how it could know the concrete type of any P to "unbox". A simple thought which comes to mind is for passing through function call arguments it's easy to trace from source, through a set of calls, to target where the unbox occurs and pass that concrete type information downstream. But what about properties, I don't see properties mentioned in the proposal. If some arbitrary class with arbitrary property let storage: any P were to be implicitly unwrapped, how would the compiler know what the concrete type is? Maybe it's similar to the procedure I already mentioned or maybe I'm way off?

The compiler doesn't know what the concrete type is, but it knows there's some concrete type (since the underlying value must be of some concrete type), and it knows how to pull that type info out of the existential's box in order to pass it to the generic function. The information "what is the concrete type of this existential" must be available to the runtime—after all, you could already today use an as? cast to recover the concrete type information from the existential if you wanted.

2 Likes

I see, so this is closer to a dynamic cast, but then how is overload resolution performed from deeper within functions with generic arguments, when an unwrapped existential is being passed through these sets of generic functions? I had thought that for an opaque type as well as a generic type that overload resolution was decided upon during compile time in order to reduce dynamic dispatch. If it only knows there is some concrete type wouldn't that imply it would have to perform dynamic dispatch?

The overload will be resolved at compile time based on the constraints known from the existential. For instance:

protocol P {}
protocol Q: P {}

func takesT<T>(_: T) {}
func takesT<T: P>(_: T) {}
func takesT<T: Q>(_: T) {}

func takesQ(_ q: Q) {
  takesT(q)
}

At the call site takesT(q), the compiler knows that basically, when it 'opens' Q, it will have a value of type some Q (with the underlying type known only at runtime). However, the bound to Q means that the compiler can call takesT<T: Q> since that is the "most specific" overload in a sense.

1 Like

Ah, this is clear, thanks. I had assumed that a some P which was truly an Int would enter func takesT(_: Int) if defined, but the answer is clearly no.

1 Like

Right, that would allow users to observe the underlying type of the opaque type, which isn't really allowed. Of course, you can do this with a dynamic cast to the proper type:

protocol P {}
struct S: P {}

func p() -> some P {
    S()
}

func takesP<T: P>(_: T) {
    print("P")
}

func takesP(_: S) {
    print("S")
}

takesP(p()) // P
print(p() is S) // true

but with dynamic casts it's hopefully at least a bit more clear that you are doing something which could break at any time (and notably, it wouldn't be possible to do this if S was a type that was private to p:

func p() -> some P {
    struct S: P {}
    return S()
}

print(p() is S) // error: cannot find type 'S' in scope

)

3 Likes

Thanks for the proposal!

Just out of curiosity, will the hidden API _openExistential be eliminated when this feature lands?
Some of our code is relying on this API today, though we know it is not a recommended way to do such things.

First off, I’m very glad to see this proposal, along with all the other generics ergonomics proposals of late! But…I think I’m ultimately with @xwu and @Jumhyn on this. The core feature here—which not much time is spent on in the proposal!—is formalizing _openExistential in some form, whether it’s let thisP: some P = anyP or let thisP = anyP as some P. I think that’s great and we should absolutely have it even though there’s a syntactic bikeshed to be had.

Then there are rules for when that’s implicitly applied, like inout-to-pointer conversions, that only work in certain cases. That’s a bit stickier, because it doesn’t eliminate the friction in all cases. It can’t. But it absolutely does in the most common cases, just like inout-to-pointer and array-to-pointer and string-to-pointer. So I’m torn…but ultimately I think that it’s not necessary, and not the important part of the proposal, as long as there’s a fix-it to invoke the explicit syntax when it’s possible. Because otherwise the cases where it still doesn’t work, and can’t, are going to be even more inscrutable than they are today.

It’s possible that with very good diagnostics we could pull this off, akin to Rust’s lifetime diagnostics. “You can’t do this because arg1 [highlight] could have type Foo: P…and arg2 [highlight] could have type Bar: P, but the function [highlight] requires that they be the same type.” But it’d be an uphill battle. I sympathize with someone who says “if there’s a fix-it to write the code you need, and it doesn’t make anything clearer at the call site, why do I have to write it explicitly?”, but I think there’s a value in saying the rules of a feature are simple and easy to explain even if the compiler will conspire with you on how to bend them correctly.

The note about self-conforming protocols also worries me. I think we’re going to end up with more self-conforming protocols, not sweep them into a corner, and honestly features like this are part of that story. Which means there has to be a way to stay heterogeneous sometimes and open up other times.

We have a lot of one-way conversions in Swift (subclass-to-superclass, value-to-optional) that are implicit in one direction but explicit and failable in the other. We also have some rarer bidirectional ones (bridging only?) that are explicit in both directions, partly because there may be a performance cost, and partly because neither type is “obviously” a base of the other. With some P and any P we’re in a situation where one direction is implicit—as it should be!—and the other direction doesn’t currently exist [with a supported syntax]. We should have some way to write that other direction. It can’t fail at run time. It should probably be explicit in general. But…it wouldn’t be the first implicit-only-for-arguments conversion we have.

P.S. On a more mundane note, I agree with @Nickolas_Pohilets that there should be a way to spell the operation without repeating the protocol name, but I don’t have a great suggestion for one. some _ makes sense by analogy but also starts to look like line noise.

13 Likes