Unlock Existential Types for All Protocols

Evolution Pitches
, ,
1

Unlock Existentials for All Protocols

Introduction

Swift currently offers the ability for protocols that meet certain criteria to be used as types. Trying to use an unsupported protocol as a type yields the error [the protocol] can only be used as a generic constraint because it has 'Self' or associated type requirements. This proposal aims to relax this artificial constraint imposed on such protocols.

Motivation

Currently, any protocol that has non-covariant Self references or associated type requirements is not allowed to be used as a type. Initially, this restriction reflected technical limitations (as Joe states here); however, such limitations have now been alleviated. As a result, users are left unable to utilize a powerful feature for certain protocols. That’s evident in a plethora of projects. For instance, the Standard Library has existential types such as AnyHashable and AnyCollection and SwiftUI has AnyView.

Generics are most often the best mechanism for type-level abstraction, which relies on the compiler knowing type information during compilation. However, type information is not always a gurantee, which is why the value-level abstraction of existential types is extremely useful in some cases.

One such case is heterogenous collections, which require value-level abstraction to store their elements of various types:

protocol Identifiable {
  associatedtype ID: Hashable 
    
  var id: ID { get }
}

let invalidIdentifiables: [Identifiable] ❌
// The compiler doesn't currently allow that.
// So, we work around that by creating a
// custom existential type: 'AnyIdentifiable'.


struct AnyIdentifiable { 
  typealias ID = AnyHashable
    
  var id: ID { ... }
}

let myIdentifiables: [AnyIdentifiable] ✅

Furthermore, dynamic environments are also known to lack type information. Therefore value-level abstraction can be exploited in cases such as previewing an application, where the application's components are dynamically replaced, in the file system where a file representing an unknown type might be stored, and in server environments, where various types could be exchanged between different computers.

Morover, the availability of an existential type for a given protocol is sometimes quite unintuitive. That is, today, a protocol qualifies for an existential type provided that it lacks any associated type or non-covariant Self requirements; however, the associated types of a protocol can be fixed via the same-type constraint. As a result, post after post has been created asking for this restriction's removal:

protocol User: Identifiable
  where ID == UUID {
  
  var username: Strng { get }
}
  
let myUsers: [User] ❌
// This is forbidden today
// for no apparent reason.

All in all, supporting existential types for all protocols is useful for many situations that involve dynamicity. Not to mention that there are many questions by language users asking about this behavior. Taking everything into consideration, we are confident that addressing this abnormality in the language will build a stronger foundation for future additions.

Proposed Solution

The constraint prohibiting the use of some protocols as types will be lifted. Consequently, boilerplate code in many projects — especially libraries and frameworks — will be significantly reduced.

Detailed Design

The compiler will no longer differentiate between protocols that don’t have Self or associated type requirements and those that do. However, some restrictions will apply to the use of requirements referencing associated types as seen in the below examples.

Examples:

Protocol with Self and Associated Type Requirements
protocol A {
  associatedtype A
    
  var a: A { get }
}

struct S: A { 
  let a: Int = 5
}

let myA: A = S() ✅ 


let a: Any = myA.a ❌
// We don’t know what associated type 
// 'A' is on the existential type of 'A'.


extension A {
  var opaqueA: some Any {
    a 
  }
  // Note that it references 
  // the associated type 'A'.
}

let opaqueA: some Any = myA.opaqueA ❌
// Opaque result type don't 
// type-erase; they just conceal
// the underlying value from the
// user. As a result, the above 
// is not allowed.
Protocol with Known Associated Types
protocol A {
  associatedtype A

  var a: A { get }
}

protocol RefinedA: A
  where A == Int {}

struct S: RefinedA { 
  let a: Int = 5
}

let myRefinedA: RefinedA = S() ✅

let intA: Int = myRefinedA.a ✅
// Here we know that the associated
// type 'A' of 'RefinedA' is 'Int'.
Protocol Composition
protocol A {
  associatedtype A

  var a: A { get }
}

protocol RefinedA: A
  where A == Int {}

protocol B {
  associatedtype B

  var b: B { get }
}

struct S: RefinedA & B { 
  let a: Int = 5
  let b: Int = 5
}


let myRefinedAB: RefinedA & B = S() ✅


let a: Int = myRefinedAB.a ✅

let b: some Any = myRefinedAB.b ❌
// We don’t know what type 'B' is
// on the type-erased value 'myRefinedAB'.

Note on conflicting associated types: Consider the following sample:

...

protocol AFixedToInt: A 
  where A == Int {}

protocol AFixedToBool: A 
  where A == Bool {}

let composition: AFixedToInt & AFixedToBool ⚠️
// The associated type 'A' has conflicting 
// implementations as both 'Int' and 'Bool'
// in this composition type.

The above composition type emits a warning, as providing a value bound to the associated type 'A' is not possible. It was also disussed to outright emit an error in this case; however, due to source compatibility constraints, the fact that the two protocols' associated types 'A' are formally distinct and potential for protocol member disambiguation in the future, warnings were chosen.

Source compatibility

This is an additive change with no impact on source compatibility.

Effect on ABI stability

This is an additive change with no impact on ABI stability.

Effect on API resilience

This is an additive change with no impact on API resilience.

Alternatives Considered

We could leave Swift as is. That, however — as discussed in the Motivation section — produces boilerplate code and a lot of confusion for language users.

Future Directions

Separate Existential Types from Protocols

To alleviate confusion between existential types and protocols it has been proposed that when referring to the former some different way be used. Some advocate for the modifier 'any' to serve that purpose: any Foo, while others propose parameterizing 'Any': Any<Foo>. Whatever the way of achieving this is, differentiation between the two would be useful as it would — among other reasons — prevent beginners from unknowingly using existential types, which can adversely affect performance.

Introduce Constraints for Existential Types

After introducing existential types for all protocols, constraining them seems like the next logical step. Constraining refers to constraining a protocol’s associated types which will, therefore, only be available to protocols that have unspecified associated types. These constraints would probably be the same-type constraint: where A == B and the conformance constraint: where A: B:

typealias A = Any<
  Identifiable where .ID == String
>

typealias B = Any<
  Identifiable where .ID: Comparable 
>

Allow Accessing Associated Types

Currently, accessing associated types through a protocol's existential type is invalid. However, we could ease that constraint by replacing every associated type of the 'base' protocol with its existential type:

protocol Identifiable {
    assciatedtype ID: Hashable

    var id: ID { get }
}

struct S: Identifiable {
  let id: Int = 5
}

let myIdentifiable: Identifiable = S()

let id: Any<Hashable> = myIdentifiable.id ✅

Make Existential Types Extensible

Today, no protocol’s existential type can conform to the protocol itself (except for @objc protocols). This is quite unintuitive — as is evident by countless questions asking about it. Such a feature could automatically apply to protocols that lack initializers, static requirements and functions with parameters bound to Self (as discussed in related post). To handle the cases that do not meet the aforementioned criteria for implicit conformance, the following syntax has been proposed:

extension Any<Hashable>: Hashable {
  …
}

Other protocols that do meet these criteria would have existential types that automatically gain conformance to their corresponding protocol. In other words, a type such as Error would automatically gain support for conformance to the Error protocol.

33 Likes
2

I'm not sure I understand what the future potential described here is. It is explicit that a type cannot (now or ever) conform to a protocol in two different ways, so the existential illustrated here is uninhabitable. What am I missing?

Note on the text here and in the subsections below: since the preceding future direction is orthogonal to the subsequent directions, I would urge using the existing spelling for existential syntax instead of one of the strawman syntaxes that don't currently work. This leads to confusion since Any<Hashable> isn't a thing but AnyHashable is (and its distinct from the existential Hashable).

On that note, please discuss what will happen to manually created types such as AnyHashable. It will be confusing when Hashable can be used. Should library authors be allowed to disable the existential type and perhaps even annotate that users should use the manually created type instead? (I think so.)

3

I think the intent here is to show that accessing a member whose type involves an unknown associatedtype is not supported.

However, the specific example shown—storing a value to an Any—seems on its face like it ought to work. Sure, the type of myA.a is statically unknown, but at runtime it must dynamically have some type, and whatever that type happens to be, an instance of it can of course be type-erased to Any.

7 Likes
4

While I support this change, we should talk about the subtle differences this is gonna introduce in code, and the potential performance penalties of such code. For example, the proposal cites AnyView as an example of a type eraser for an existential value, which is both true and also not exactly true. Existential values don’t conform to their underlying protocols (which is something we should also lift) and that implies code like this:

struct MyView: View {
  var body: View {
    ...
  }
}

Will compile, but not satisfy the protocol requirements. Plus, we may see some folks write APIs mistakenly taking View as an existential and then running up against cliffs when trying to pass it as a generic parameter:

func makeList(_ v: View) -> View {
  List { v } // won’t compile, View doesn’t conform to View
}

We should at least sample the diagnostics that users are likely to see, and make them much clearer after allowing this.

Also:

func f() -> View { ... }
func g() -> some View { ... }

These two have very different performance characteristics, one of them returning a concrete type, another returning an existential that may need to spill over into a heap allocation. We gotta make sure users understand the differences.

10 Likes
5

Given the Swift goal to promote clarity over brevity, I would prefer having to explicitly mark existentials (at least for protocols having 'Self' or associated type requirements, PATs) with any Protocol instead of simply using Protocol.
Whenever a PAT is placed as an existential, the compiler would let the user choose which fix-it to apply: either using a generic or explicitly marking the protocol as existential with any.

The benefits of having explicit existentials are available in the Clarifiying existential types section of Improving the UI of generics.

13 Likes
6

To reinforce what @Nevin said, I find it somewhat surprising that properties of unconstrained type aren’t available as Any. This should be clearly motivated and/or discussed as a future direction. Perhaps this is implied by the Allow Accessing Associated Types section (since this is just the minimally constrained case), but I don’t think that’s clear to the typical Swift developer.

Relatedly, I would like to see code like this work (for principle-of-least surprise reasons rather than a specific use case):

protocol P {
    associatedtype A: Equatable

    func f() -> A
    func g() -> A
}

func fIsG(_ p: P) -> Bool {
    p.f() == p.g() // This is statically correct even though we don’t know what A is
}

Again, this might fall out of Allow Accessing Associated Types, but it’s pretty subtle.

3 Likes
7

In private discussion, @Joe_Groff brought up an example of how retaining such requirements formally distinct could prove useful in the future. If I understand correctly (correct me if I’m wrong) he says that we could assign separate witnesses in the future in order to enable something like this:

any P<.A == Int> & Q<.A == Bool>

I tried to omit what could constitute future directions from the note to keep it simple. Do you think it should be clarified nonetheless?

The future directions sections tries to retain the features discussed in the order that they’ll probably be introduced into the language. This is one of many proposals to come that will refine existential types. Therefore, the features discussed in the section build on each other, hence the use of Any syntax. We could clarify that — if you think it’s necessary or consider discussing the features separately without one referencing others.

Good point, I will add the way Standard Library existential types will be treated in the proposal.

8

You’re right, we would prefer it if we were able to write this:

let identifiable: Identifiable = ...
let hashable: Hashable = identifiable.id

However, this proposal is incremental. We don’t aim to completely overhaul existential types in Swift. Instead we take a small step towards improving this feature. This is also discussed as a future direction, exactly for this reason.

Feel free, though, to suggest a more appropriate explanation for why that example fails.

1 Like
9

I'm not sure I quite understand your reply here.

It appears that in your reply you're giving a different example of two protocols P and Q that both coincidentally have associated types named A; that is to say, two distinct associated types P.A and Q.A. It is a possible future direction to allow disambiguation so that a type can conform to both P and Q and explicitly specify different associated types.

That is a different situation from the example in your draft proposal text, with two protocols both refining the same protocol A with an associated type A.A that is constrained in two distinct ways. A.A cannot simultaneously be Int and Bool. There aren't two distinct associated types in that example, and no type can (now or ever) both be Int and Bool.

Perhaps it's just a typo and you mean to write the following?

protocol P { associatedtype A }
protocol Q { associatedtype A }
protocol AFixedToInt: P where A == Int { }
protocol AFixedToBool: Q where A == Bool { }
let composition: AFixedToInt & AFixedToBool

Just because a feature will be proposed doesn't mean that it'll be accepted! If another feature cannot be introduced or explained without a preceding feature being accepted, then it makes sense to write a section that builds on a preceding section. That is perhaps a hint that they're not two separate features after all, though.

On the other hand, if a feature can stand alone even if another feature is rejected, then it should also be possible to describe it without reference to that other feature (otherwise, it's a red flag that they can't stand as separate features). So, it's a useful exercise to ensure that the features are properly scoped. Moreover, it adds clarity for the reader of the text, since it should not be necessary to require the reader to learn a strawman syntax from a preceding section in order to understand a separate feature in another section that doesn't even rely on the strawman syntax.

:+1:

10

Does a protocol only get an existential under this proposal if it fully conforms to itself? If so, I’m in support of this. If not, I think it needs to be required to be marked in such a way that it’s clear at the site of use that it is not “complete” when it doesn’t self-conform. I am partial to the spelling partial P (pun intended) from the earlier thread for anything that is an existential that doesn’t have all the promised functionality of the protocol it represents.

11

No, that would be greatly source-breaking and a non-starter.

That would also be source-breaking. Even if it were optional (and therefore not strictly source-breaking), it would constrain library authors because otherwise innocuous requirements could not be added without being source-breaking (where currently one can add requirements as long as they have default implementations). Sprinkling in new footguns like that doesn't seem acceptable to me.

1 Like
12

Then I can’t support this proposal. Allowing existentials that don’t conform to their protocols without a specifier to that effect leads to a bad (and confusing) user experience. Better to wait until we can do it in conjunction with a new spelling.

13

There is no impediment to a new spelling for all existential types (as discussed in the proposal) but I do not see how it can be workable to have a design where the conformance of a type to a protocol is indicated at the point of use, for the reasons outlined above. I, at least, would not support such a design at any time in Swift's evolution; it's not an issue of waiting. Others may differ on this, but ultimately I do agree that distinguishing existential types from their corresponding protocols is useful and, arguably, could sufficiently address the concern you bring up.

2 Likes
14

Without a more concrete plan of where existentials are headed, I’d be hesitant to lift this restriction.

For storage (in instance/global variables), existentials are great. As function arguments, they are usually a poor substitute for actual generic functions. I think we need a spelling that clearly communicates existentials as type-erasing boxes, so that users only use them when that particular feature is really required.

Until we do that, or at least have some kind of agreed-upon roadmap to answer the questions raised by the alternative designs (e.g. around partial protocols or “self-conformance”), existentials should be on pause. I think it is broadly accepted that this area of the language needs work beyond simple lifting of restrictions.

3 Likes
15

OTOH, I don't see the eventual answers to those questions changing what this fundamentally looks like when it's supported. I like the idea of an explicit any modifier for existential types, but we must maintain compatibility with the existing syntax at least for protocol types we currently allow, and I don't think we're going to change the syntax of compositions again. As it stands, the restriction creates a real roadblock for many developers, and lifting the restriction makes working within the other restrictions easier. Remember that the "self or associated type" restriction is not because of any fundamental property of those protocols, but merely because Swift 1.x had an incomplete implementation of protocol witness tables that didn't allow associated type metadata to be recovered dynamically.

20 Likes
16

You’re right View’s existential won’t conform to View. However, what this proposal will help with is simplifying AnyView’s implementation. View currently looks similar to this:

protocol View {
    static func _makeView(_ instance: Self, inputs: _BlockInputs) -> _BlockOutputs

    // ... More hidden requirements


    associatedtype Body: View

    var body: Body { get }
}

Currently AnyView can’t just create a View existential from the View type to be erased and then compile successfully. Instead, it has to undergo more elaborate boxing, similar to AnyHashable. This proposal make type-erasing easier for library authors. For example AnyView could with this proposal look something like this:

struct AnyView: View where Body == Never {
    var _viewExistential: View
    // No need for AnyHashable-style boxing

    init<Content: View>(erasing content: Content) {
        self._viewExistential = content
    }

    static func _makeView(_ instance: Self, inputs: _BlockInputs) -> _BlockOutputs {
       ...
    }

    var body: Never {
        ...
    }
}

Do you think we should make explicitly note that View won’t outright replace AnyView in the proposal text?

I agree that beginners may opt in for the simpler existential syntax not understanding the important differences between generics and existentials. However, this is also a problem today and to address it has been proposed to distinguish existentials from protocols with special ‘Any‘, ‘any Identifiable’ and ‘boxed Identifiable’ syntax (which is also discussed as a future direction).

Something we can do in the proposal is clarify that existential types are not a substitute for generics. They are different solutions for different kinds of problems. I think this is evident in the proposal as we don’t discuss using such types in cases where generics could be used, but in cases which require dynamic solutions. Is there something specific you think we could specify in the proposal text?

2 Likes
17

I'd appreciate this, but also I think the proposal ought to include some details about the kinds of diagnostics users will start seeing now that "protocol cannot be used since it has Self or associatedtype requirements" won't be emitted. Just in an effort to survey the land and see if the existing diagnostics are adequate or if there's an opportunity to educate.

I still very much support this change, and am super glad y'all are pushing it.

3 Likes
18

I think there's a more fundamental issue here: the proposal suggests that it provides a better way of implementing type-erased wrappers:

This opening strongly implies that this proposal will allow these types to be replaced, but this is far from true. In all 3 cases, there is a lot more needed to achieve this. As it stands, creating a stand-alone [Collection] would add little value above an array of Any, even if it would now be possible. The proposal makes the claim of reduction of boilerplate code, but doesn't give examples of such boilerplate being eliminated.

The proposal would probably benefit from a more motivating example than Identifiable. While a [User] array might be useful, an [Identifiable] collection is not, particularly; like with Collection, there's not much else you could do with these identifiable things than cast them back to concrete types.

This suggests there may be a more coherent minimum viable proposal that only allows protocols with fully constrained associated types to be used as existentials, but not ones that still have unconstrained ones. This is probably not true – the feature as proposed is useful – but the proposal doesn't give any real world examples of this.

6 Likes
19

To motivate removing the restriction today, I think it helps to look at associated types that don't want to be constrained in type erasure. With Collection, for instance, you probably want to specify the Element, but you almost certainly don't want to constrain the Index except in special circumstances, because the index implementation is generally tightly coupled to a specific collection implementation. The subset of Collection's API that doesn't traffic in Index is still useful—you can ask for individual elements, and perform operations like map and filter whose interfaces traffic only in element types because they abstract over their specific index traversal patterns. There are other protocols with associated types for which constraining any of the associated types is usually undesirable, such as SwiftUI's View.

11 Likes
20

I started getting confused as I read this proposal because so many examples had the associatedtype with the same name as the surrounding protocol (A). Is that intentional? I'm not sure what it's supposed to mean.

3 Likes