Unlock Existential Types for All Protocols

[Ben_Cohen]

This is another case where a good realistic (but simple) protocol would improve the readability as well as motivation of the proposal.

[filip-sakel]

@xAlien95

Given the Swift goal to promote clarity over brevity, I would prefer having to explicitly mark existentials (at least for protocols having 'Self' or associated type requirements, PATs) with any Protocol instead of simply using Protocol .
Whenever a PAT is placed as an existential, the compiler would let the user choose which fix-it to apply: either using a generic or explicitly marking the protocol as existential with any .

The benefits of having explicit existentials are available in the Clarifiying existential types section of Improving the UI of generics.

I’d love to have a version of something similar to what your propose; however as I’ve said before this isn’t in the scope of this proposal. That’s why it’s discussed in the future directions section.

I’ve also read that post and it has been of much inspiration to this proposal.

[filip-sakel]

@Ben_Cohen

This opening strongly implies that this proposal will allow these types to be replaced, but this is far from true. In all 3 cases, there is a lot more needed to achieve this.

Okey, we’ll clarify that.

@Ben_Cohen

As it stands, creating a stand-alone [Collection] would add little value above an array of Any , even if it would now be possible. The proposal makes the claim of reduction of boilerplate code, but doesn't give examples of such boilerplate being eliminated.

The proposal would probably benefit from a more motivating example than Identifiable . While a [User] array might be useful, an [Identifiable] collection is not, particularly; like with Collection , there's not much else you could do with these identifiable things than cast them back to concrete types.

Suggestion taken! If I understand correctly you would appreciate concrete, useful, real-world examples to really showcase the motivation and prove how boilerplate would be reduced.

Also, if you have any particular cases with protocols that would significantly benefit from this proposal in mind feel free to share them.

[vns]

@Ben_Cohen

The proposal would probably benefit from a more motivating example than Identifiable .

I will try to give more realistic example I have written for real project.

Example 1. It was quite hard for me on the first time to create type-erasure wrappers, so I followed Swift's library examples and get something like that just to have an ability to work with the protocol without not creating a lot of generics:

Code

public protocol UpdateHandler: AnyObject {

    associatedtype Value

    func handle(_ result: Result<Value>)
    func canHandle<T>(_ value: T.Type) -> Bool

}

public final class AnyUpdateHandler: UpdateHandler {
    public typealias Value = Any

    private let _box: _AnyUpdateHandlerBox

    public init<Handler>(_ handler: Handler) where Handler: UpdateHandler {
        _box = _ConcreteUpdateHandlerBox(handler)
    }

    public func handle(_ result: Result<Value>) {
        _box.handle(result)
    }

    public func canHandle<T>(_ value: T.Type) -> Bool { _box.canHandle(value) }
}

private protocol _AnyUpdateHandlerBox {
    func handle(_ result: Result<Any>)
    func canHandle<T>(_ value: T.Type) -> Bool
}

private class _ConcreteUpdateHandlerBox<Base: UpdateHandler>: _AnyUpdateHandlerBox {
    private let base: Base

    init(_ base: Base) {
        self.base = base
    }

    func handle(_ result: Result<Any>) {
        base.handle(result.flatMap {
            if let value = $0 as? Base.Value {
                return .success(value)
            }
            return .failure(result.error ?? Cancelled())
        })
    }

    func canHandle<T>(_ value: T.Type) -> Bool { base.canHandle(value) }
}

And it is hard to get work with that for people - I've noticed that so many times there was a necessity to describe this to someone in details and, to be honest, I still confused with some things.

Example 2. This is actually the first one I've implemented in the project and it is actually has a quite big interface, which leads to significant time needed to maintain it in case of some changes - around 10 methods and 5 properties right now, and almost zero compilers checks on that. In short, it was originally designed to hide dependency from Apple's NSFetchedResultsController type and be able to replace it with any other implementation. In code sample below I've added just a little of that protocol and type-erasure wrapper.

Code

public protocol ResultsController: AnyObject {
    associatedtype Entity

    var isEmpty: Bool { get }

    subscript (indexPath: IndexPath) -> Entity { get }

    func numberOfItems(in section: Int) -> Int
    func contains(where predicate: (Entity) -> Bool) -> Bool
}

// this wrapper is everywhere where I need to use `ResultsController` type
public final class AnyResultsController<EntityType>: ResultsController {
    public typealias Entity = EntityType
    
    public var isEmpty: Bool { box.isEmpty }
    private let box: _AnyResultsControllerBase<Entity>
    
    public subscript(indexPath: IndexPath) -> Entity { box[indexPath] }

    public func numberOfItems(in section: Int) -> Int {
        box.numberOfItems(in: section)
    }

    public func contains(where predicate: (EntityType) -> Bool) -> Bool {
        box.contains(where: predicate)
    }
}

// then goes class with a lot of `fatalError` methods intended to be overwritten
private class _AnyResultsControllerBase<EntityType>: ResultsController {
    var isEmpty: Bool { fatalError() }
    subscript(indexPath: IndexPath) -> Entity { fatalError() }
    func numberOfItems(in section: Int) -> Int { fatalError() }
    func contains(where predicate: (EntityType) -> Bool) -> Bool { fatalError() }
}

// and finally we got to concrete object passing
private final class _AnyResultsControllerBox<Concrete>: _AnyResultsControllerBase<Concrete.Entity> where Concrete: ResultsController {
    // there are pretty the same code as two times before, so I omit it here.
}

This two examples live in the project for around 6-7 month for now, they do help us to implement a lot in a good way, especially second example - it is our base object to get results. But it is a lot of code, a lot of time to understand and support and to be understood by others.

But I like to call these two examples to be unique in the code I prefer to write and still trying to keep away those wrappers as much as I can. In other words, I am looking for other options, rather getting into type-erasure wrappers world again. Several years ago I was against using them, and when we first time faced with need of type-erasure, we haven't chose that way, but only wrote base "abstract" class with a lot of fatalError calls in each method.

[filip-sakel]

Thanks for sharing these examples!

I think many aspects of what you shared will make for a much better motivation. Namely, in your examples we can see that maintenance of type-erasuring code is quite demanding with protocol requirements scattered around different protocol and concrete types. Furthermore, such code also needs to be thoroughly explained adding to the maintenance cost. All in all, we could argue that such code is hard to update, explain and overall maintain becoming a burden to code based.

[drewster99 (Andrew Benson)]

@toph42

I am partial to the spelling partial P (pun intended) from the earlier thread for anything that is an existential that doesn’t have all the promised functionality of the protocol it represents.

Why partial? Won’t that break many things? What is the use case for something that partially conforms to the protocol?

[toph42 (Topher Hickman)]

I don’t want to hijack this thread but you can read more about it on the previous thread. The idea is for the compiler to require the partial identifier at the call site so the user has to acknowledge that they understand that the existential they are using is only as much of its protocol’s API as possible to be “the most specific possible common supertype of all types conforming to the protocol” when that’s not everything. If it is constrained enough to fully conform to the protocol (or requires no constraint to do so) the partial qualifier would not be used at the call site and would have the current spelling.

[drewster99 (Andrew Benson)]

@toph42

The idea is for the compiler to require the partial identifier at the call site so the user has to acknowledge that they understand that the existential they are using is only as much of its protocol’s API as possible to be “the most specific possible common supertype of all types conforming to the protocol” when that’s not everything.

But, if the rules all confirm to the protocol, then.... they all confirm to the protocol.. so every part of the protocol’s API would be available... right? How can anything that conforms to the protocol not conform to the entire protocol?

Or are we talking about types conforming to different protocols which happen to share some common interface pieces being held in the same existential?

I’ll check out the other thread.

Thanks!

[toph42 (Topher Hickman)]

That was an assumption I made before those threads, too, and the reason why I think the compiler should warn that it doesn’t work they way you’d assume.

[nicklockwood]

I'm a bit torn on this one.

On the one hand the inability to use PATs as types is a major point of confusion and inconsistency in the Swift language, so I'm in favor of removing that restriction.

However I'm not sure it's actually very useful to be able to do this in practice. For example, a natural thing that Obj-C users might try when moving to Swift is:

let objects: [Equatable] = someHeterogeneousObjects
if objects.contains(someObject) { ... }

These users will initially stumble on the first line here due to the PAT restriction (which this proposal would remove), but then they'll find the second line doesn't work either for an even more confusing reason.

Similarly, it's pretty common to want to do this:

let objects: [Codable] = someHeterogeneousObjects
let data = try JSONEncoder().encode(objects)

Here the first line already works fine, but the second fails because protocols don't conform to themselves in the general case. If (as mentioned in the future directions IIUC) this restriction was lifted then the code would technically now work (in the sense it would compile) but would probably just be setting the user up for a bigger and more confusing limitation when they go to try and decode the array again.

In a sense, while not immediately intuitive, the inability to use PATs as types is a sort of "fail fast" feature, forcing users to learn about a fundamental issue early on rather than travelling further down the wrong path before hitting it.

It would be good to see some examples of common, valid use-cases for this capability.

[Joe_Groff]

The way I see it, making Equatable and Hashable existentials ought to be something we handle as a special case even without a general make-protocols-conform-to-themselves features, as two values of different dynamic types can be compared by wrapping both in AnyHashable and using its implementation of equality and hashing. Absent a general language feature, we should do what we've done for tuple equality/hashability and special case Equatable and Hashable existentials to be Equatable and Hashable as well.

[Moximillian (Mox)]

Yes please! Have been waiting for this restriction to be lifted for such a long time!

As many have mentioned this restriction is a big stumbling block for anyone learning to use protocols, and the biggest reason why it is like that is because it’s an arbitratry firewall; it actively prevents you from understanding what is wrong, by bluntly stating ”PAT”s are nono in here”.

I understand the worry that someone might make inefficient code with existentials, but learning requires making mistakes, and getting exact error messages about specific problems. Once a person experiences the real limitations of existentials, not the arbitrary ones, then it’s possible to understand when to use what approach (e.g existentials vs generics).

This combined with ”any Collection” would make Swift protocols so much easier to learn.

[anthonylatsis]

Hey there,

As you may recall, a protocol member with direct Self references is allowed to be used with an existential base if said references are in covariant positions, since we can safely erase these references to their upper bounds – which coincide with the base type – following the Liskov substitution principle:

protocol P {
  func f1() -> Self
  func f2(_: (Self) -> ())
}

func takesP(p: P) {
  let x = p.f1() // Okay, x: P

  p.f2 { y in } // Okay, y: P
}

With the current behavior limited to just Self, we feel like allowing associated type references to be covariantly erased as well would be a well-fit inclusion and hopefully alleviate some concerns around the usefulness of these protocols when used as types. Any thoughts?

protocol P {
  associatedtype B: Class
  var b: B { get }
}

func takesP(p: P) {
 let x = p.b // Okay, x: Class
}

[anandabits (Matthew Johnson)]

I think we want path-dependent types here.

protocol P {
  associatedtype B: Class
  var b: B { get }
  func takesB(_: B)
}

func takesP(p1: P, p2: P) {
 let b1 = p1.b 
 let b2 = p2.b
 p1.takesB(b1) // ok
 p2.takesB(b1) // error: b1 may be of the wrong type
 p2.takesB(b2) // ok
 p1.takesB(b2) // error: b2 may be of the wrong type

}

[Joe_Groff]

We probably want path-dependent types for Self types too, but that would already require a source break to make happen. It seems reasonable to me to have unconstrained associated types behave isomorphically to Self types on existentials in the meantime.

[anthonylatsis]

As an aside, I imagine we could get this done by covariantly erasing to an opaque type.

[dabrahams (Dave Abrahams)]

As I have written many times elsewhere, I have several concerns with this direction, but I'll just repeat the most serious one here: unless something is done about the spelling of existential types, we will have introduced many type declarations into Swift that seem to say instances of the type have those APIs when they in fact do not. IMO that creates an unacceptable understandability/explainability barrier, particularly for new users.

I don't believe the syntaxes mentioned-but-not-proposed for “separating existential types from protocols” do enough to help clarify the situation. Since every P is declared to have a foo, it's not evident why any P or Any<P> doesn't also have a foo—that's not the way the English language or logic work—and the explanation is anything but simple.

The mentioned-but-not-proposed syntax for existential type constraints is inconsistent with what we've done for other constraints: where clauses now always appear outside of the <…>.

Given that we'll need room for constraints in the future, and the lack of constraints is the reason for the missing APIs on a bare existential of type P, I see no advantage in introducing the word “any”, and would like to use a syntax that points explicitly to the cure for the missing API: P where _.

As noted, I have raised other concerns (and suggested ways to address most), which I'm not going to repeat here, about “unlocking existentials” in past discussions. If the community wants to say, “yeah, we thought about those, and here's why we don't think they're a problem” I can accept that, but it's discouraging to see yet another pitch to “just lift the constraint already” without any of those concerns or suggestions having been explicitly addressed.

[sveinhal (Svein Halvor Halvorsen)]

I’m not sure I’m qualified to comment on your specific concerns, at this time. But I wholeheartedly agree that whatever serious concerns are raised, should explicitly be mentioned and discussed in a proposal.

[anthonylatsis]

@dabrahams

Given that we'll need room for constraints in the future, and the lack of constraints is the reason for the missing APIs on a bare existential of type P

It is true that the current implementation allows us to safely use an API when all the type parameters in its signature have been constrained to something non-dependent, but constraints are not an explanation as to why we cannot use them otherwise, nor an answer as to whether we could do something about it in the future. Similarly, P where A == Int is simply a refinement, not necessarily the reason you could call f() -> A where A: SignedInteger or a similar method featuring a different associated type on that type. It seems like a more accurate diagnostic could make a decent enough explanation compared to any of the syntaxes hitherto mentioned.

I am a bit concerned about leaving the syntactic part on the shelf though. This proposal would obviously aggravate a future source break, if the accepted syntax for generalized existentials ever comes to be one.

[dabrahams (Dave Abrahams)]

It is true that the current implementation allows us to safely use an API when all the type parameters in its signature have been constrained to something non-dependent,

Presumably you mean the proposed implementation? Otherwise, I cannot imagine what you're saying.

but constraints are not an explanation as to why we cannot use them otherwise,

Never claimed they were an explanation of anything. I said my proposed syntax points explicitly to the cure for the missing API.

nor an answer as to whether we could do something about it in the future.

Sorry, I don't know what you mean by that. What is “it?”

Similarly, P where A == Int is simply a refinement,

Not in the usual sense we use the word “refinement,” which refers to what's sometimes called ”protocol inheritance.“ I'm not sure what you mean by calling it a refinement.

not necessarily the reason you could call f() -> A where A: SignedInteger or a similar method featuring a different associated type on that type.

I'm sorry, I'm afraid I don't understand what you're trying to say here either. How could the name of a type (P where A == Int) be the reason for anything?

It seems like a more accurate diagnostic could make a decent enough explanation compared to any of the syntaxes hitherto mentioned.

sigh; diagnostics are not a substitute for understandability.