Unlock Existential Types for All Protocols

I would greatly appreciate existentials' Self and associated types erasure (or hiding) to opaque types. Opaque types are self-explanatory for unexperienced users (I guess, I hope) and have good diagnostics too. You immediately understand why p2.takesB(b1) isn't valid since the following error gets prompted:

Cannot convert value of type '(some P).B' (associated type of protocol 'P')
to expected argument type '(some P).B' (associated type of protocol 'P')

giving the user to assumption that "some P" and "some P" may be different types.

Your points about the situation being inexplicable are well-taken, if slightly overstated. But I would agree that it's certainly not something that can be explained in any diagnostic of reasonable length.

That's not very fair; I've already indicated in this thread that I don't consider it to be an explanation.

I do think my proposed syntax would make it almost impossible to teach people about existentials without also giving them an explanation for an “inexplicable[-in-diagnostics]” result, as the Swift book does. The any P syntax would not discourage anyone from presenting existentials without calling attention to the link between constraints and missing API. I also think the syntax I proposed will make it much harder for people to forget about that link when actually using existentials.

By mentioning “protocol cannot conform to protocol” I think you may be proving one of my points: lots of people who want generalized existentials are confused about what the feature actually provides and imagine it will solve problems that it will not. Protocol self-conformance is not fixed by generalized existentials.

I want self-conforming existential/protocols too, but to make that work we need a completely different feature: the user needs to be able to declare that the protocol self-conforms and the compiler needs to be able to prevent you from adding requirements that are incompatible with self-conformance. And I'd be perfectly happy if that category of existential types was spelled by the unadorned protocol name.

We're all in agreement about that. We disagree about whether “simply lifting the restriction” results in a better, and less confusing, world for Swift programmers. I'm simply not content to open the floodgates to all of the issues I've raised about generalized existentials and hope that someday, somehow, improving diagnostics will lead to a good result. That's not a plan. Once you make a fundamental mistake in language design it is usually extremely difficult to walk it back.

If someone can lay out a design that actually deals with the problems (or give reasons why the problems are just in my imagination), even if not all of it can be implemented right away, then we have something to talk about.

1 Like

A missing feature caused by a broken model, IMO. We can apparently access members of existentials, even though the existential doesn’t actually have any members and just forwards to the contained object.

To actually fix this, we need to stop accessing members via this forwarding mechanism, unbox the object and access its members directly. Overall this would lead to a much simpler and more easily understood model.

Yes, we can bend the existing model a bit further, but I don’t think we should. I suspect the remaining limitations will still leave us with something that is frustratingly complex and incomplete.

1 Like

Can you explain why you want this?

AFAIK the only reason for self-conformance is to make existentials play nicely with generics, and there are much better options for that which don’t require new kinds of protocols or impose restrictions on what they may include.

That's why. When you have a protocol and a bunch of models, and need to add another model that works via type erasure, it's very inconvenient that the existential doesn't work directly.

and there are much better options for that which don’t require new kinds of protocols or impose restrictions on what they may include.

I'm listening…

Unboxing. So if you have a function:

protocol MyProto {}

func foo<T: MyProto>(_ value: T)

Currently, when you try to call that with an existential of type MyProto, the compiler will attempt to bind T == MyProto (the existential), and will fail:

let existential: MyProto = ...
foo(existential) // T == MyProto. Error: MyProto does not conform to MyProto.

Instead, the contained value should be unboxed (either manually or by the compiler) to some local generic type, and that is what should be used instead:

let existential: MyProto = ...
unbox(existential) { <X: MyProto>(unboxedValue: X) in
  foo(unboxedValue) // T == X. X: MyProto is true, all is good.
}

Inside the scope, X is a real generic type. I don't see any reason why we couldn't enable the full expressiveness of generic code for X.

For X to be meaningful outside this local scope, we may need to introduce something that represents "an unknown type satisfying some constraints" (note that this is different to existentials, which represent "any type which satisfies some constraints").

To illustrate the difference, consider an Array<MyProto> - each element is an independent existential, and may have a different type to its neighboring elements. There is currently no way in the language to express that all elements have the same type -- and that even though we don't know which specific type they are, we know that it conforms to MyProto.

I haven't been following along in detail to this thread, but just saw this and wanted to comment, any particular reason you want the compiler to do this vs. forcing ourselves to manually open the existential? I.e.

let existential: MyProto = ...
let <X: MyProto> openedMyProto = existential
foo(openedMyProto)

It would certainly be less magical, but I view that as a good thing because now it's clear after opening the existential that the function takes a generic parameter conforming to the type vs. an existential.

1 Like

No - you’re right, I probably shouldn’t call the unboxing „automatic“. I don’t see any reason it couldn’t be manual, and less magic could very well lead to a clearer model.

I updated the post to be less opinionated about who does the unboxing :)

1 Like

Well, that's interesting for some purposes, but it doesn't solve my problem. I need an actual model of the protocol, e.g. that I can use to fulfill an associated type requirement.

protocol P0 {
  associatedtype A: P1
}

The existential type P1 can't be used as the A of a P0 model unless it actually conforms to P1. Just goes to show: the number of wrinkles in the things people hope to get from existentials is pretty impressive. To me it looks like a Pandora's box, and I'd really like someone to sort out all the monsters, give them names, and describe a workable programming model for them before we open it further.

2 Likes

What feature do you imagine can make that work without violating type soundness?

Without any type system enhancements, we could say returnAssoc can be invoked and the result treated as Any.

If path-dependent types are added then the result would have type p1.Q. This would allow the result to be used as input to methods on p1 that accept Q.

1 Like

Opaque types.


Swift already provides path-dependent types when working with opaque types. This works today:

protocol P {
  associatedtype B
  var b: B { get set }
  func takesB(_: B)
}

struct S1: P {
    typealias B = Double
    var b = 5.3
    func takesB(_: B) {}
}
struct S2: P {
    typealias B = String
    var b = "swift"
    func takesB(_: B) {}
}

var p1: some P = S1()
var p2: some P = S2()

let b1 = p1.b
let b2 = p2.b
p1.takesB(b1) // ok
p2.takesB(b1) // error
p2.takesB(b2) // ok
p1.takesB(b2) // error

var p1too = p1 // 'p1too' has the same type of 'p1'
p1.takesB(p1too.b) // ok
p1too.takesB(p1.b) // ok

What we are missing are:

  • a way to refer to p1's concrete type explicitly;
  • some type level expressions to be used in typealias declarations (instead of value level expressions) such as concreteType(of: instance) to get the concrete type of an instance or concreteType(commonTo: instance1, instance2) to get the least common concrete super type of a list of instances.

The underlying structure is partially already there because of the behavior of opaque types. Some syntaxes have already been proposed for opening existentials (Generics Manifesto: Opening existentials):

// from Generics Manifesto
if let p1asT = p1 openas T {
  // you can use T as a typealias in this scope
}

// from Alejandro in this thread
let <T: P> p1: T = S1()

// with type level functions
let p1: some P = S1()
typealias T = concreteType(of: p1)
3 Likes

Yeah, that’s what I meant by “work.” Okay, that could potentially be designed. Another monster in the box named…

I'm using any _ to refer to the existential protocol type for simplicity. If both any P0 and any P1 do not conform to P0 and P1 respectively, using opaque types in methods and properties having Self or associated types involved in them (SoAT) will let you use those protocols as existential types and will let you access all the non-SoAT properties and methods like you are already do with non-PATs. With opaque types you can even access SoAT properties and methods, but with their types being hidden.

protocol P1 {
  associatedtype One
  var one: One { get set }
}
protocol P0 {
  associatedtype A: P1
  var a: A { get set }
}

var pzeroes: [any P0] = [pzero1, pzero2, pzero3]
var x = pzeroes[0]  // x: any P0
var xa = x.a        // xa: (any P0).A,        i.e. (some P0).A
var xaone = xa.one  // xaone: (any P0).A.One, i.e. (some P0).A.One

var xas = pzeroes.map(\.a)  // xas: [any P1]
pzeroes[0].a = xas[0]  // error: instance of type 'any P1' cannot be
                       // assigned to an instance of type '(some P0).A'
xas[0] = pzeroes[0].a  // valid: instance of type '(some P0).A' is 'any P1'

We have:

(any P0).A.self == (some P0).A.self  // because .A is path-dependent
(some P0).A.self == some P1.self     // because .A conforms to P1

pzeroes[0].a = xas[0]  // invalid, 'any P1' unassignable to 'some P1'
xas[0] = pzeroes[0].a  // valid, 'some P1' assignable to 'any P1'

However, if you explicitly conform the existential type any P0 to P0:

// for simplicity, in existential extensions, `self` refers to
// the concrete instance, `Self` refers to its concrete type
// thus here we have: self: some P1, Self = some P1
extension any P1: P1 {
  typealias One = Any
  var one: One {
    get { self.one }
    set { self = newValue }
  }
}
extension any P0: P0 {
  typealias A = any P1
  var a: A {
    get { self.a }
    set { self = newValue }
  }
}

// now we can use '(any P0).A' which is 'any P1'
var pzeroes: [any P0] = [pzero1, pzero2, pzero3]
var x = pzeroes[0]  // x: any P0
var xa = x.a        // xa: (any P0).A,        i.e. any P1
var xaone = xa.one  // xaone: (any P0).A.One, i.e. Any

var xas = pzeroes.map(\.a)  // xas: [any P1]
pzeroes[0].a = xas[0]       // valid: same type assignment
xas[0] = pzeroes[0].a       // valid: same type assignment

We have:

(any P0).A.self == (any P1).self  // .A is not path-dependent now

pzeroes[0].a = xas[0]  // valid, 'any P1' assignable to 'any P1'
xas[0] = pzeroes[0].a  // valid, 'any P1' assignable to 'any P1'
A concrete example: any Hashable
extension any Hashable: Hashable {
  var hashValue: Int { self.hashValue }

  func hash(into hasher: inout Hasher) {
    self.hash(into: &hasher)
  }
}

extension any Hashable: Equatable {
  static func == (lhs: any Hashable, rhs: any Hashable) -> Bool {
    // least common concrete super type
    typealias T = concreteType(commonTo: lhs, rhs)

    guard T.self is Equatable.Type else { return false }
    return lhs as! T == rhs as! T
  }
}

I don't understand what you mean by this. The underlying type of an opaque type is ultimately known statically to the compiler because it works like a generic parameter. This is not the case for an existential type, and that is indeed much of the point. How could the compiler statically know the type of something that's not known, by construction?

I really only meant separating the notions of the protocol and the type would make that possible to explain, but yes, you still got me in a way, so I would like to take this opportunity to ask: why can an existential not simply conform abstractly to its protocol like an archetype does? Is the following a reason for that?

func firstIndex<C: Collection>(c: C) -> C.Index { c.startIndex }

var c: Any<Collection where .Element == Int> = ...

let idx = firstIndex(c: c) // What's the type of idx?

Explained here in the thread referenced previously. [Edit:] and I'm not trying to “get you.”

The only interesting part of Hashable when it comes to existentials is the Equatable conformance, and as I pointed out in the Protocol Oriented Programming talk, that's a special case for which we already have an answer. If you want to prove something about how a feature like path dependent types will allow us to do new things, I suggest picking a case that we don't know how to solve today.

I was only referring to the behavior of (any P0).A. In my opinion it should behave like an opaque type: you can get x.a of type concreteType(of: x).A and, since A conforms to P1, you don't know anything more than the fact that it will be "some P1" at runtime, i.e. a concrete unknown type conforming to P1.

Terms of Service

Privacy Policy

Cookie Policy