Lifting the "Self or associated type" constraint on existentials

[dabrahams (Dave Abrahams)]

@anandabits

the existential type

As I mentioned previously, in the world of generalized existential there is not just one existential type for a protocol. There will often be an unbounded number of existential types with different constraints. I can't imagine how you intend to declare clearly which members are available on all of these potential existential types.

This is the one argument I think really holds water against what I've suggested, and I've been waiting for someone to make it; sorry I missed your previous mention. My best answer is that I think the unconstrained existential is special. I think it's a serious problem that you can have a type called, simply, SomeProtocol and go look at the declaration of that type, and not be able to access some part of the API declared there. If it were spelled “SomeProtocol where _”, then I'd probably be more comfortable with some of the protocol's API being implicitly unavailable, because there's this attached “where I don't know anything other than that the type conforms” right there in the code.

Generalized existentials are an extremely powerful feature that includes a nontrivial amount of essential complexity. Learning how to work with them will necessarily involve some effort. I don't see any way around this.

I don't think this means we should not support the feature or be afraid that users won't be able to use it effectively (one thing I love about Swift is that it has mostly avoided that kind of paternalism). I think it means we need put more effort than usual into educating people about this feature, how to use it well, and what pitfalls or traps they to be aware of. This is especially necessary as generalized existentials is a relatively sophisticated feature for which I don't think there is a clear point of reference in the previous experience of most programmers.

Agreed 100%. Precisely because the feature “includes a nontrivial amount of essential complexity,” we need to be careful about how we introduce it, how it will interact in users' minds with the rest of the language, whether we're putting that complexity in too close at hand when features with fewer pitfalls are farther away, and what new problems arise when we introduce part, but not all, of the feature.

[Joe_Groff]

@xwu

• Allow all existentials to be spelled P that have no Self or associated type requirements (status quo)

This is not a fixed property of public protocols, since they can resiliently add new requirements. We wouldn't want it to be source-breaking to do so.

[David_Sweeris]

@anandabits

@dabrahams

the existential type

As I mentioned previously, in the world of generalized existential there is not just one existential type for a protocol. There will often be an unbounded number of existential types with different constraints.

Have we closed the door on having different behavior for compiling executables vs libraries? I can't recall if anyone has already asked... If not, this seems like a good candidate for it: the compiler can assume all types are present & accounted for when compiling "My Great App" (but not when compiling "libMyGreatLibrary" or however we're libraryifying names these days), and if there's only one type that satisfies the requirements, great, use that. Otherwise throw an "ambiguous type" error or tell the developer they'll need to make the function/type generic (depending on which is more applicable).

[xwu (Xiaodi Wu)]

@Joe_Groff

• Allow all existentials to be spelled P that have no Self or associated type requirements (status quo)

This is not a fixed property of public protocols, since they can resiliently add new requirements. We wouldn't want it to be source-breaking to do so.

...but it is the status quo. And if we evolve in such a way that a new requirement with Self or associated type requirements merely requires a protocol opt in to the existing spelling via annotation, then even that would not be source-breaking.

[Joe_Groff]

If we were going to introduce a new syntax for existentials, I think it'd be much less confusing if we did so uniformly, deprecating the existing syntax after a late-enough language version, than to have two syntaxes where one only works some of the time. That seems to me like it'd only make things more confusing, personally.

[David_Sweeris]

@dabrahams

@Moximillian

Why is this? Why would we want to hard-code protocols into two distinct categories?
I have always thought that the ability to use PAT's or Self-constrained protocols

FWIW, I don't know the term “PAT.”

As to why, I thought I explained it, but let me try again, in fewer words this time.

• An existential type is fundamentally different from a type conforming to the corresponding protocol. When existentials are allowed to be created from protocols with arbitrary requirements, we create the “some of P's requirements can't be used on a P” and “P does not conform to P” paradoxes, which are very bad for usability and comprehensibility.
• We intend to support protocols evolving to add new requirements, as long as they are defaulted. If we allow arbitrary new requirements to be added to a protocol that has been used as an existential, we either have to live with “P does not conform to P,” or adding new requirements with defaults may break existing code.
• Declaring these protocols differently would allow the compiler to prevent us from creating these difficulties.

Is there a reason (besides inertia) to not "declare these protocols differently"?

[Joe_Groff]

It isn't clear that they are fundamentally different, and one "kind" can morph into the other over time.

[David_Sweeris]

@Joe_Groff

It isn't clear that they are fundamentally different, and one "kind" can morph into the other over time.

Does "over time" mean "during execution", "as code evolves", or something else?

[Joe_Groff]

As code evolves, sorry.

[jrose (Jordan Rose)]

@David_Sweeris

Have we closed the door on having different behavior for compiling executables vs libraries? I can't recall if anyone has already asked... If not, this seems like a good candidate for it: the compiler can assume all types are present & accounted for when compiling "My Great App" (but not when compiling "libMyGreatLibrary" or however we're libraryifying names these days), and if there's only one type that satisfies the requirements, great, use that. Otherwise throw an "ambiguous type" error or tell the developer they'll need to make the function/type generic (depending on which is more applicable).

Apps aren't the end of the line either. Unit tests often add dummy types; the Swift runtime supports dynamically-loaded in-process plug-ins; and you can even declare new types or add new conformances in the debugger if you want.

[David_Sweeris]

@jrose

@David_Sweeris

Have we closed the door on having different behavior for compiling executables vs libraries? I can't recall if anyone has already asked... If not, this seems like a good candidate for it: the compiler can assume all types are present & accounted for when compiling "My Great App" (but not when compiling "libMyGreatLibrary" or however we're libraryifying names these days), and if there's only one type that satisfies the requirements, great, use that. Otherwise throw an "ambiguous type" error or tell the developer they'll need to make the function/type generic (depending on which is more applicable).

Apps aren't the end of the line either. Unit tests often add dummy types; the Swift runtime supports dynamically-loaded in-process plug-ins; and you can even declare new types or add new conformances in the debugger if you want.

Cool! I did not know that.

[xwu (Xiaodi Wu)]

Here is a good example of a genuine use case where our current restriction would have saved some confusion if the protocol in question had Self or associated type requirements:

Pattern matching Errors

I ran into some strange behavior while testing pattern matching in catch blocks: protocol FooError : Error { func whatWentWrong() -> String } extension Array : FooError, Error where Element: FooError { func whatWentWrong() -> String { return "idk i'm an array" } } struct MyError : FooError { func whatWentWrong() -> String { return "something happened" } } func test() throws { throw [MyError()] } do { try test() } catch let err where err is [FooError] { // Why is this n…

[Joe_Groff]

That's a bug (I misunderstood, disregard) The non-self-conformance issue is also a good indication of why the existing restriction already doesn't really hide the complexity.

[xwu (Xiaodi Wu)]

(See other thread.)

[dabrahams (Dave Abrahams)]

@Joe_Groff

If we were going to introduce a new syntax for existentials, I think it'd be much less confusing if we did so uniformly, deprecating the existing syntax after a late-enough language version, than to have two syntaxes where one only works some of the time. That seems to me like it'd only make things more confusing, personally.

I "loved" this comment when I saw it originally, but now I'm having second thoughts, because an idea I buried in an aside earlier is growing on me. Suppose:

1. Self-conforming-protocol existentials can be spelled with just the protocol name
2. Other existentials always need a where clause, which may be “empty:” Collection where _.

I think that might be enough to solve my problem with implicitly-unavailable APIs, and it wouldn't punish many uses of generalized existentials, which will have a where clause anyway. Finally, if we decide this was all a mistake, we can always lift restriction #2.

I realize this doesn't give us an immediate way to fix the technical problems @Joe_Groff is trying to address with this pitch, but it is an eventual future I think I could support. If we can agree that it's a good idea, maybe we can find a reasonable evolution path to get there that includes an immediate fix for the technical problems.

Thoughts?

[Joe_Groff]

If We Could Do It All Over Again, it might've been nice to make the type system a Lisp-2 and put protocols and types in separate namespaces. That would give you the ability to express the "simple existential" idea as a typealias, e.g.:

protocol Runcible {
  mutating func runce()
}

typealias Runcible = Any<Runcible>

or spell the canonical type eraser for a less trivial protocol:

protocol Collection { ... }

struct Collection<T>: Collection {
  var _value: Any<Collection where .Element == T>

  /* Collection conformance here */
}

As for where we are today, since "self-conforming-protocol existentials" don't exist now except in limited circumstances, and declaring that self-conformance when desired will necessarily be explicit, making a rule that non-conforming existentials must be spelled differently doesn't really help the immediate source compatibility problems, since non-self-conforming existentials already exist and are spelled without decoration.

[jrose (Jordan Rose)]

I very much hate separate namespaces but you can have the required Any without that.

[dabrahams (Dave Abrahams)]

• I agree with @jrose so don't see the point of what you're saying. Is it about the namespaces or the required Any?
• I recognize my proposal doesn't solve any immediate source-compatibility problems; in fact it creates an eventual source-compatibility problem. The questions I'm trying to ask are:

  1. How does my proposal sound as a place to end up?

     Note that I think Foo where _ does more to call out the unknown types and therefore the possible unavailability of API than Any<Foo> does, and it doesn't burden constrained existentials with the extra Any<…> syntax which AFAICT only benefits the unconstrained existential case.

     Note also that I include solving the technical problems your proposal addresses as part of the goal.

  2. If “good,” then what would the source evolution story be?

  3. What does that story imply about immediate next steps in the language?

[rbishop (Russ Bishop)]

Thinking through this over the holiday and reading through the arguments, ultimately I think I come down in favor of @Joe_Groff's pitch. tl;dr: I don't think we should let the perfect be the enemy of the good.

@dabrahams raises a lot of good points but IMHO relaxing the artificial restriction on existentials doesn't change the status quo (we already run into these issues and relaxing the rules solves some real problems). From what I can tell, source compatibility means future versions of the language must solve this problem in a way that isn't seriously constrained by this pitch and the restriction imposes its own set of artificial design constraints that are no worse than the theoretical ones Dave raises (though reasonable people might disagree on that).

This discussion has been enlightening; it certainly raises the desire to tackle the existential problem in Swift 6.

(Life's existential problems are left as an exercise for the reader)

[L_Mihalkovic]

A type system that needs these kind of hack is at best unfinished.. ... ABI stability serves Apple’s needs, but does not make the language usable for people coming from more complete type systems (no .. that does not define python).
Btw... this is JUST an opinion