[Pitch] Introduce existential `any`

Hello Swift Evolution!

I recently started a discussion about easing the learning curve of introducing generic parameters. In that discussion, it was brought up multiple times that the existing syntax of existential types is actively harmful, and we should consider making the implications explicit in the language. I wrote up a proposal for introducing an any keyword for this purpose.

I've pasted the first iteration of the proposal below, and you can view the latest version here: https://github.com/hborla/swift-evolution/blob/existential-any/proposals/NNNN-existential-any.md

Please let me know your questions, thoughts, and other constructive feedback!

-Holly


Introduction

Existential types in Swift have an extremely lightweight spelling: a plain protocol name in type context means an existential type. Over the years, this has risen to the level of active harm by causing confusion, leading programmers down the wrong path that often requires them to re-write code once they hit a fundamental limitation of value-level abstraction. This proposal makes the impact of existential types explicit in the language by annotating such types with any.

Motivation

Existential types in Swift have significant limitations and performance implications. Some of their limitations are missing language features, but many are fundamental to their type-erasing semantics. For example, given a protocol with associated type requirements, the existential type cannot conform to the protocol itself without a manual conformance implementation, because there is not an obvious concrete associated type that works for any value conforming to the protocol, as shown by the following example:

protocol P {
  associatedtype A
  func test(a: A)
}

func generic<ConcreteP: P>(p: ConcreteP, value: ConcreteP.A) {
  p.test(a: value)
}

func useExistential(p: P) {
  generic(p: p, value: ???) // what type of value would P.A be??
}

Existential types are also significantly more expensive than using concrete types. Because they can store any value whose type conforms to the protocol, and the type of value stored can change dynamically, existential types require dynamic memory unless the value is small enough to fit within an inline 3-word buffer. In addition to heap allocation and reference counting, code using existential types incurs pointer indirection and dynamic method dispatch that cannot be optimized away.

Despite these significant and often undesirable implications, existential types have a minimal spelling. Syntactically, the cost of using one is hidden, and the similar spelling to generic constraints has caused many programmers to confuse existential types with generics. In reality, the need for the dynamism they provided is relatively rare compared to the need for generics, but the language makes existential types too easy to reach for, especially by mistake. The cost of using existential types should not be hidden, and programmers should explicitly opt into these semantics.

Proposed solution

I propose to make existential types syntactically explicit in the language using the any keyword. This proposal introduces the new syntax, and this syntax should be required under the Swift 6 language mode.

Detailed design

Grammar of explicit existential types

This proposal adds the following production rules to the grammar of types:

type -> existential-type

existential-type -> 'any' type

Semantics of explicit existential types

The semantics of any types are the same as existential types today. Explicit any can only be applied to protocols and protocol compositions; any cannot be applied to nominal types, structural types, and type parameters:

struct S {}

let s: any S = S() // error: 'any' has no effect on concrete type 'S'

func generic<T>(t: T) {
  let x: any T = t // error: 'any' has no effect on type parameter 'T'
}

any also cannot be applied to Any and AnyObject (unless part of a protocol composition).

Rationale: any Any and any AnyObject are redundant. Any and AnyObject are already special types in the language, and their existence isn’t nearly as harmful as existential types for regular protocols because the type-erasing semantics is already explicit in the name.

The existential metatype becomes (any P).Type, and the protocol metatype remains P.Protocol.

Examples

protocol P {}
class C {}

any P
any Any           // error
any AnyObject     // error
any P & AnyObject
any C             // error
any P & C
any () -> Void    // error
(any P).Type

func test<T>(_: T) where T == any P

Source compatibility

Enforcing that existential types use the any keyword will require a source change. To ease the migration, I propose to start allowing existential types to be spelled with any with the Swift 5.6 compiler, and require existential types to be spelled with any under the Swift 6 language mode. The old existential type syntax will continue to be supported under the Swift 5 language mode, and the transition to the new syntax is mechanical, so it can be performed automatically by a migrator.

SE-0306 Unlock existentials for all protocols enables more code to be written using existential types. To minimize the amount of new code written that will become invalid in Swift 6, I propose requiring any immediately for protocols with Self and associated type requirements. This introduces an inconsistency for protocols under the Swift 5 language mode, but this inconsistency already exists today (because you cannot use certain protocols as existential types at all), and the syntax difference serves two purposes:

  1. It saves programmers time in the long run by preventing them from writing new code that will become invalid later.
  2. It communicates the existence of any and encourages programmers to start using it for other existential types before adopting Swift 6.

Effect on ABI stability

None.

Effect on API resilience

None.

Alternatives considered

Instead of leaving Any and AnyObject in their existing spelling, an alternative is to spell these types as any Value and any Object, respectively. Though this is more consistent with the rest of the proposal, this change would have an even bigger source compatibility impact. Given that Any and AnyObject aren’t as harmful as other existential types, changing the spelling isn’t worth the churn.

Future Directions

Extending existential types

This proposal provides an obvious syntax for extending existential types in order to manually implement protocol conformances:

extension any Equatable: Equatable { ... }

Re-purposing the plain protocol name

In other places in the language, a plain protocol name is already sugar for a type parameter conforming to the protocol. Consider a normal protocol extension:

extension Collection { ... }

This extension is a form of universal quantification; it extends all types that conform to Collection. This extension introduces a generic context with a type parameter <Self: Collection>, which means the above syntax is effectively sugar for a parameterized extension:

extension <Self> Self where Self: Collection { ... }

Changing the syntax of existential types creates an opportunity to expand upon this sugar. If existential types are spelled explicitly with any, a plain protocol name could always mean sugar for a type parameter on the enclosing context with a conformance requirement to the protocol. For example, consider the declaration of append(contentsOf:) from the standard library:

extension Array {
  mutating func append<S: Sequence>(contentsOf newElements: S) where S.Element == Element
}

Combined with a syntax for constraining associated types in angle brackets, such as in [Pitch] Light-weight same-type constraint syntax, the above declaration could be simplified to:

extension Array {
  mutating func append(contentsOf newElements: Sequence<Element>)
}

This sugar eliminates a lot of noise in cases where a type parameter is only referred to once in a generic signature, and it enforces a natural model of abstraction, where programmers only need to name an entity when they need to refer to it multiple times.

Acknowledgments

Thank you to Joe Groff, who originally suggested this direction and syntax in Improving the UI of generics, and to those who advocated for this change in the recent discussion about easing the learning curve for generics.

104 Likes

I haven't yet thought through all of the implications, but I think the proposal could use a section which compares the Swift 5 and 6 language modes with this feature. If I understand this correctly, it makes naked use of an existential type a compiler error, requiring any Existential instead. Perhaps the "Proposed solution" section could summarize the change?

4 Likes

Thrilled to see any get a proper pitch, it will be interesting to see how the community feels about the tradeoffs of a source break here. This will require a lot of code churn, but it will hopefully be an easy migration and worth it in the long run.

I’m sure I’ll have more to say once I’ve had a chance to think through everything a bit more, but just wanted to say:

Perhaps it’s come up before in discussions of any, but this clarification of the confusing P.Type vs P.Protocol situation is a nice little bonus win for deprecating bare existential syntax that I hadn’t considered previously. :slight_smile:

14 Likes

I’m also really excited and will be glad if the proposal is accepted and implemented before SE309 is rolled out.

One question: how would one write ‘[Any]’ with the new syntax since ‘any Any’ is prohibited?

I assume it'll be the same as with some types, but might be good to call out precedence rules (e.g., requiring parens to spell (any P)?, which admittedly is much wordier than P?).

I see the rationale in not requiring any for these, but I don't see the harm that necessitates a special rule to ban them. It's visually redundant but certainly not confusing, and it'd be nice not to have errors pop up midway through writing a protocol composition that begins any AnyObject & ....

Wouldn't that just remain [Any]?

6 Likes

That must mean that the as Protocols turn into as any Protocols, yes?

If so, we can't reasonably have support for as any without as some as well. If the latter comes along with the former, this is a doubly great pitch.

I’m extremely in favor of this, thank you for making it happen! I want to offer one suggestion: allow any P.Type. That’s easier to type, obviously, but I’ll argue there’s a semantic reason as well. This type serves two purposes: it’s both the result of type(of: someExistentialValue), and it’s the type that has the operations marked static in the protocol. In that sense, today’s P.Type is an existential type as well, not merely the metatype for an existential type. An API that has a parameter of type P.Type probably has arguments that look like NSObject.self, not the type of an existing instance. In another language, P.Type could be a protocol that NSObject.Type conforms to. That means a hypothetical any (P.Type) is a reasonable type, which also happens to be the same type, in Swift, as (any P).Type. Which works out nicely!

Now, if only I could convince myself the same applies to any P?

10 Likes

Unrelatedly: the Rust discussion about their own change in this vein: `dyn Trait` Syntax for Trait Objects: Take 2 by Ixrec · Pull Request #2113 · rust-lang/rfcs · GitHub

5 Likes

Hi, very excited about this changes, thanks for summarizing all the points.
Just one question: how self-conforming protocols will be expressed?

let error: Error = NetworkError() // or `let error: any Error = ...` ?

generic(error)

func generic<SomeError: Error>(error: SomeError) {
  let errorCopy: SomeError // or `let errorCopy: any SomeError` if argument of Error existential type is passed?
}

Huge +1 on this, I'm so glad it's getting a pitch.

+1 on the observation that the current syntax causes active harm. I wager a majority of working Swift users never properly learned the concept of existential due to confusion caused by it.

8 Likes

Isn’t Any considered an existential here? If so, then any Any seems more consistent to me. Yes, it’s not great, but we can introduce Value and deprecate Any in Swift 6 for a less disruptive source break.

This is a great proposal and I welcome the introduction of the any keyword. There is one thing that I'd like to push for reconsideration. Why is it a requirement to disallow any S where S a value type or a class, etc.?

Can't any not only represent existentials but also sub-types? While any S won't have any effect as we don't have sub typing for value types now, Never as a bottom type will/could fall under this umbrella in the future. any C for a class could already make sense and it would be equivalent to the today's C.

class C {} 
class D: C {}

// today
let c: C = D() // okay
// after the proposal
let anyC: any C = D() // theoretically the same as above

I personally think that some C would equally make sense where one could express that there is an opaque class involved, especially when we discuss the possibility of the generic footprint reduction via the some keyword in a context of generic type parameter T constrained by a class type.

<T: C>(_: T) ---- becomes ----> (_: some C)

If there is some C, then there should be any C as well.


TLDR; can't we make those errors as warnings instead?

Of course we could relax this rule in the future, but I'd like to know if it would be okay to do it with this proposal already.

Thank you in advance.

1 Like

I'm also going to bump the ideas regarding the issues with the current metatype spelling. There are meta types which are static and the result of Type.self and there are meta type existentials. The current state of the proposal pitch does not cover this whole area, which can be improved with the introduction of the any keyword.


Without derailing the current topic from the any keyword, in the UX related thread we also briefly discussed the idea of another keyword named meta to improve metatypes. Here's a short explanation of the types it would cover compared against the any syntax:

typealias Meta<T> = meta T

P.Protocol == (any P).Type == Meta<any P> == Meta<P> == meta P
    P.Type ==   any P.Type == any Meta<P> == any meta P
              ~~~~~~~~~~~~

I'm not asking for an additional introduction of that keyword, I just mention it to further clarify what any keyword can already cover and why.

This also further emphasis why the any keyword should support not only existentials (e.g. protocol conformances) but sub-typing (e.g. inheritance) as well!

+1 for this very nice proposal.

I like Any->Value and AnyObject->Object transition, maybe it's time to make this change through 5.6 release.

Any protocol means any value by literal in Swift, now we have any P syntax to support this subtle conception, formalize Value and any Value would be much better.

IIRC, AnyObject naming problem been raised and discussed several times in forum, it should be Object by initial design. Now finally we have a chance to fix this problem in this proposal. any Object is the right design by nature, and furthermore I really hate protocol SomeP: AnyObject class bound protocol declaration. So weird to look and read~

any Value/Object change is the right thing to do, and it's the right time to do this right thing. Please fixing it!

7 Likes

Heh, perhaps the (any P).Type win was not as clear as I initially thought. Between this proposal, @jrose's suggestion that any P.Type and (any P).Type be the same, and @Joe_Groff's earlier suggestion that P.Protocol become (any P).Type, we have three different (and incompatible) ideas of how any syntax should apply to protocol metatypes. IMO, having P.Type become (any P).Type makes sense, since to me that syntax naturally reads as "the metatype of all types which conform to P", and I'm ambivalent-to-negative on having any P.Type mean the same thing. I think I big plus of (any P).Type is that it puts some visual distance and a bit of syntactic clarification between the former P.Type and P.Protocol. If we continued to allow P.Type as a syntactic form (even if it were required to prefix with any, we'd lose that benefit.

In the same vein, I'd be strongly against allowing both any P.Type and (any P).Type and having those mean different things. I suspect that would lead to even more confusion than the Type-Protocol issue we have today.


On another note, I'd like to discuss a bit more about the source break here. As I mentioned previously, this is going to be a big break if adopted as is. Hopefully it will be as simple as prepending any to all existing existential type names, but I do wonder whether there are ways that we could mitigate the amount of churn that will be required.

As came up in a previous thread, some uses of existentials are isomorphic to generics thanks to Swift not requiring monomorphization for generic functions. That is, if we remove the associated type from the proposal's example, there's not a fundamental issue with:

protocol P {
  func test()
}

func generic<ConcreteP: P>(p: ConcreteP) {
  p.test()
}

func useExistential(p: P) {
  generic(p: p)
}

In fact, we can already call generic from within useExistential today, with _openExistential:

func useExistential(p: P) {
  _openExistential(p) { concrete in
    generic(p: concrete)
  }
}

I suspect that we could come up with a reasonable list of non-problematic uses of existentials that maybe don't need to require any syntax. In the above situation, we could auto-open the existential so that the generic(p: p) call succeeds without any dance.

The proposal also calls out concerns that "the type of value stored can change dynamically" but this is only true insofar as the value can change at all—if the value of existential type is a let (as with non-inout function parameters), then we needn't worry about the concrete type changing dynamically.

I think we could possibly also get away with allowing bare existentials for @objc protocols.

Together with auto-opening of existentials, I do wonder if a rule like the following would get us most (all?) of the benefit while reducing the code churn required:

An existential type must be annotated with any if the protocol is not @objc and either of the following is true:

  • The protocol has Self or associatedtype requirements
  • The value of existential type is mutable

Since SE-0306 hasn't shipped with an official version of Swift yet (right?), that would allow us to only require updating the uses of mutable values of existential type (which are not @objc). This would likely include any Swift protocol delegates, since most of those will be weak vars, but it might carve out enough exceptions to make the source break more palatable.

Of course, I have a generally higher tolerance for source breaks, so if this problem is deemed big enough for the big, hard break in Swift 6, then I'm all for it. I'd rather end up in a world with fewer complex rules for type syntax, so a rule of "existential types are prefixed with any" is preferable to me in that regard. Just wanted to raise a couple of alternative paths forward in the event that the widest-reaching version of this proposal is unpalatable to some. :slight_smile:

I agree with you that those things are extremely close to each other, yet they mean two totally different things. Metatatypes have two kinds merged into them today, this is rather awkward because not only does it limit our way into expressing a specific type constraint, but it's also the fact on why type(of:) remains compiler magic instead of having a pure swift function signature.

  • any P.Type is the existential that accepts all meta types, which inherit from P.Type (e.g. T.Type from T.self where T: P which also means that T.Type: P.Type)

  • (any P).Type is a static metatype (not an existential itself) of an existential for P (e.g. the result of type(of: existentialP) or P.self)

A small side note: The "true" P.Type is not expressible in Swift. Let this sink in for a bit. ;)

2 Likes

Hm—I'm not sure I understand this distinction fully. In Swift today, I thought that the result of type(of: existentialP) would be a P.Type (i.e., the metatype of some type conforming to P) while P.self is of type P.Protocol.

Even if I can bring myself to grasp this distinction fully, I don't think we should expose it to users as (any P).Type vs. any P.Type. We should come up with a better way of spelling those two (?) types (perhaps with the suggested meta keyword). Regardless, I don't think it's a problem that this proposal really needs to solve, as long as we're not boxing ourselves out of a potential future.

1 Like

One more thing, about 5 years ago we already tried to move this topic forward, but we (the original proposal authors) failed and withdrew our proposal. Later on @beccadax helped us to clear things a bit, but the proposal never seen any daylight. I still have it dusting in my outdated swift-evolution repository.

If anyone is interested in reading it and get a feeling on the issues around meta types, here's the link to it: Refactor Metatypes

The introduction of any keyword is probably our (second to) last chance to fix this in the language.

1 Like

What exactly do you mean by this? Is this proposal consuming design space that would be required for a (separate) metatype-focused proposal, or is your concern that metatype issues won't on their own justify a source break and so would have to 'ride along' with the any proposal if the fixes require their own source breaks?