SE-0352: Implicitly Opened Existentials

Asking more “Is it supposed to work this way?” questions as I mull over the proposal…

I tried reimplementing type(of:) as the proposal suggests, and as the proposal seems to imply, it doesn’t work as expected in this Swift 5 build:

func 𝒕𝒚𝒑𝒆<T>(of value: T) -> T.Type {
    T.self
}

let a = [3, 4] as any Collection
type(of: a)  // Array<Int>
𝒕𝒚𝒑𝒆(of: a)  // ❌ Collection

Would that work in Swift 6 mode?

1 Like

Yes, that's a good idea. I'd be inclined to require the very explicit as any Collection on the expression itself, rather than try to make use of contextual type information like the type of the variable being initialized.

That explanation is definitely not good enough, and not even really accurate. The issue here is that you cannot, in general, take an X<T> (where T conforms to P) and treat it as an X<any P>, because X might contain instances of T in it that affect its memory layout. For example, let's say we had this:

class X<T: P> {
  var first: T
  var second: T
}

Now, let's assume that Int conforms to P. X<Int> will be laid out as an object header + two Ints. X<any P> will be a bit larger, because an instance of any P has to be able to store a value of any type that conforms to P, so it has some buffer space + some metadata. That's our fundamental problem: we cannot treat an X<Int> instance as X<any P>, because the layouts are different, and we cannot break apart the X<Int> instance and reconstitute it as an X<any P> because that has semantic implications.

There's a little bit of discussion about this in alternatives considered, but most of it comes down to this alternative approach making opened existential types part of the user-visible type system, which is a significant jump in complexity.

It's motivated both by preserving source compatibility and also by my comment above, where this proposal is trying to avoid introducing the notion of "opened types" into the user-visible type system.

Yes, I believe it is. Consistent order-of-evaluation matches up with a developer's mental model. If we break that mental model for one narrow case, they have to throw out the model. @xwu had a good answer to this as well.

This looks like a bug in the implementation, thank you!

Doug

5 Likes

Does SE-0309 have similar issue? If so, I think we should align the solution for this sort of problem.

Yes, this issue was introduced by SE-0309. Here's a self-contained example that doesn't use anything from this proposal:

protocol P {
  associatedtype A
}

protocol Q {
  associatedtype B: P where B.A == Int

  func f() -> B
}

func g(q: any Q) {
  let a = g.f()  // okay: a has type `any P`
}

Whatever solution we arrive at should apply equally to this proposal and SE-0309, but it's going to end up changing somewhat if the constrained existentials pitch becomes an accepted proposal, since now we'll be able to express the existential result with the bounds. There's a procedural question for the Core Team here as to whether this proposal should try to address this problem or whether they'd prefer a separate amendment to SE-0309, which is at the "root" of this issue and which this proposal builds upon.

I'd also love to hear the opinions of the authors of SE-0309: @anthonylatsis, @filip-sakel, and @suyashsrijan.

Doug

2 Likes

Just a quick reminder to all that the origins of the covariant upper bound erasure story predate SE-0309 in the face of covariant Self erasure, which likewise depends on the fate of constrained existentials:

protocol P {
  associatedtype A
}
extension P where A == Int {
  func sillyCopy() -> Self
}

func foo(p: any P<Int>) { 
  let copy = p.sillyCopy() // would be expected to have type 'any P<Int>'.
}

I am having trouble devising an example, do you have anything in mind? A hypothetical any Pany P<T> transition seems benign as per the substitution principle.

1 Like

Overloading tends to ruin the substitution principle. You could in theory have something like:

func foo(_: any P)
func foo(_: any P<Int>)

and if we change the implicit type-erased bound of a call with an opened existential in an expression like foo(bar(someExistential)), then we'll start calling the other foo overload.

I’m still not sure if, according to this proposal, type(of: { MyError() as any Error }() ) returns Error.self, (any Error).self, or MyError.self. Does the answer differ between Swift 5.7 and Swift 6? The section “Type-erasing resulting values” implies the answer is (any Error).self, but the current behavior is returning MyError.self. How does this relate to the (dis)continuance of _openExistential()?

I think that example would return MyError.self as any (Error.Type) in both language modes, since in Swift 5, type(of:) has the unconditional opening behavior, with no direct way to suppress, and in Swift 6, we'd also have the unconditional opening behavior, and this doesn't match the "explicit coercion of the argument" rule. If you'd written type(of: { MyError.self }() as any Error), then I'd expect you to get (any Error).self in Swift 6 mode.

Thanks. I think what I was missing here is that type(of:)’s signature is not <T>(_: T.Type) -> T. It’s <T, Metatype>(_: T.Type) -> Metatype. That’s why the return value isn’t subject to re-boxing.

Its current type signature is really an illusion, since it has special case treatment by the type checker today, that is effectively equivalent to always opening an existential argument and type-erasing the result back to the existential metatype. In Swift 6 mode, I think we'd want to change it to the more appropriate <T> (T) -> T.Type signature and remove the special case behavior. With regards to _openExistential, it might still be needed to open Any, AnyObject, Error, and other self-conforming protocols in Swift 5 mode, but we should be able to deprecate it with the proposed Swift 6 rule.

The rules regarding contravariant erasure seem quite complex. When thinking about how I’d explain this feature, this proposal would provide a good mental model in its current form, but only for simple cases. Hiding existential opening from users will only lead to more confusion as evident by the current proliferation of workarounds and special cases.

Perhaps it’d be better if existentials were transparently opened, such that the type returned after opening is an opaque type. Admittedly, this adds significant complexity to the type system, but I think it’s well worth given its intuitive mental model. Instead of superficially hiding the dynamic generic type, which can be uncovered by a simple ‘print’, this approach would be explained as the compiler optimizing the code, since the underlying type of the existential can’t be changed in a generic-accepting function (AKA: the current motivation). The difference being that users would actually see this upfront, and not try to figure out why a printed type is different at runtime. If users want to cast back to the existential type, they can do so explicitly, reflecting the actual implementation detail that is hidden under the current model. This approach also has the benefit of clearly positioning opaque types as the default abstraction over types conforming times a given protocol. As for compatibility, behavior is already going to change in Swift 6, so source breaks are more acceptable IMO, not to mention that the migrator could automatically add an ‘as any P’ where required.

I’m interested to hear your thoughts and if/what I missed in terms of semantics and compatibility.

I infer from this that T.Type behaves analogously to T.AssociatedType in covariant return position according to this pitch.

Can _openExistential realistically be deprecated without implementing “Explicitly opened existentials” (aka as some P)? If current users of _openExistential refactor their call sites into one-off generic functions, might that cause code bloat?

Unfortunately my understanding is that this proposal does not enable _openExistential to be written natively even in Swift 6, because it takes a closure as an argument and closures cannot be generic.

It's not quite the same thing as an associated type because metatypes still have special recognition in the language. We know that existentials of the form any Constraints.Type exist, so we can do the covariant erasure from T.Type to any Constraints.Type.

_openExistential already requires you to refactor your call site into a one-off generic function. With this proposal, you still can't write _openExistential yourself, but you shouldn't need to, because you don't need to write anything at all to get its behavior.

1 Like

My belated review:

I’m a +1, and delighted to see this longstanding language gap filled at last. Still, it seems to me this proposal is necessary but not sufficient, so I hope the work does not end here.

Several things about the proposal originally gave me pause:

  1. It’s a bit…magic-y. Is it too magic? Will people be surprised when they accidentally encounter the feature? Does the magic introduce undesired behavior?
  2. Forcing a new function to open an existential limits refactoring options / expressivity, and doesn’t compose nicely with other language features. Couldn’t we do some lexically scoped version of this?
  3. Is the feature discoverable? When people need to unwrap an existential, how do they realize they need to pull the code that works with the concrete type into its own function? That is a curious leap.

Reading the proposal and the discussion, and doing my own fumbling experiments, I’m convinced on point 1: yes, it is a bit magical to have a type invisibly transform like this from caller to callee, but the behavior of that magic (1) is almost always going to be what people actually wanted, to the extent they won’t notice it’s magic, and (2) the magic almost never shadows other behavior people would want instead. To the extent it’s magical, it’s also desirable and intuitive.


Points 2 and 3 still stick in my craw. It is a basic building block of Alogol’s many descendants that it is generally possible to inline a function in source, as it were:

a
f(…)
d                             a
                —becomes→     b
func f(…) {                   c
    b                         d
    c
}

Under this proposal, a function that opens an existential is no longer thus inlinable. By forcing the creation of new functions that may interrupt the logic flow of code and may not have a meaningful name, this proposal will thus create a scattering of fragmentary functions. The situation is akin to how Objective-C’s target/selector approach to callbacks could leave classes littered with tiny methods that represented the tail ends of thoughts started elsewhere, much like reading a Choose Your Own Adventure book straight through out of order.

Swift’s nested functions mitigate this problem, though the resulting dance is hardly ideal:

    let foo: any P
    ...
    openFoo(foo)
    func openFoo(_ foo: some P) {
        ...
    }

Worried about all that, this sentence from the proposal put me at peace with its approach as a good starting point:

Opening an existential means looking into the existential box to find the dynamic type stored within the box, then giving a "name" to that dynamic type. That dynamic type name needs to be captured in a generic parameter somewhere, so it can be reasoned about statically, and the value with that type can be passed along to the generic function being called.

Yes, right, we need a way to name that type that resulted from the opening, or the opening isn’t worth much — and right now, Swift can only introduce generic types at function or type boundaries. Any sort of solution that offers the inlining I’m looking for necessitates a large increase in language surface area, whereas the proposal’s answer makes an urgently needed feature available with minimal impact. I’m convinced. +1, good proposal, yes, do it.


I do hope we don’t stop there, however. This proposal leaves a gap in the shape of the undocumented _openExistential. (Aside: I think it is not possible to fully recreate the behavior of _openExistential for a generic protocol P using this proposal, even in Swift 6 mode, because there is no way to specify type parameters <P, T:P> in Swift. Is that correct?)

Edit: I misremembered _openExistential as accepting a closure. I was picturing this nonexistent beast:

let foo: any Widget
...
openExistential(foo) { fooOpened in
  ...
}

…Or better yet, I like the alternative/future direction the proposal mentions of allowing opening via coercion from any P to some P. The introduction of opaque result types introduced the problem of types that are unnameable, which IIRC greatly vexed both @Chris_Lattner3 and me during the review of that proposal. Being able to refer to a specific some type would open up this future direction:

let foo: any Widget  // where P has associated type Doodad
...
let fooOpened: some Widget = foo
let bar: TypeOf(fooOpened).Doodad = fooOpened.doodad  // bad hypothetical syntax

…and this would grant my “descendants of Algol” wish from above. I do hope we head in that direction eventually.

6 Likes

That’s what @Joe_Groff said:

Thanks for the clarifications @Joe_Groff. I’m really excited for how this sets up Swift 6 to have less magic.

There does appear to be a version of _openExistential that accepts function types. It is used in swift-corelibs-foundation to implement AttributedString.

func openLHS<LHS>(_ lhs: LHS) -> Bool {
    if let rhs = rhs as? LHS {
        return CheckEqualityIfEquatable(lhs, rhs).attemptAction() ?? false
    } else {
        return false
    }
}
return _openExistential(lhs, do: openLHS)

Right, but the second argument has to actually be a function; it crashes if you pass it a closure.

Works:

func f(foo: P) {
    _openExistential(foo, do: g)
}

func g<T: P>(openedFoo: T) {
    print(openedFoo)
}

Crashes the compiler:

func f(foo: P) {
    _openExistential(foo) { openedFoo in
        print(foo)
    }
}

Without having investigated, I assume this is because closures in Swift can’t be generic, only named functions can.

(And the closure is necessary to grant my wish of having openedFoo inside the lexical scope of foo, the capability around which I’m arguing future work is necessary. Nested functions work, as per my post, but are awfully awkward.)

3 Likes

The key difference between accepting functions and accepting a closure is that a function can have type parameters which the implementation of _openExistential can bind the underlying type to.

Effectively, the Swift 6 behavior described in this pitch is like if all calls to generic functions* were made via _openExistential.

* all calls to generic functions which accept only one type parameter and exactly one formal argument, where the type of the formal argument is equal to the type parameter

2 Likes

Ah, got it.