Allowing self-conformance for protocols

[Ben_Cohen]

@Karl

Personally I find the whole idea of existential conformances to be absurd. An existential that is bound to some protocol is "a box which contains a thing that conforms to that protocol". The box itself does not conform.

I don't know why you think this is "absurd" and "nonsense" and I'd actually ask you tone the rhetoric down a little bit. Perhaps you are both talking about different things and need to get on the same page, but throwing extreme terms around won't help.

If this were a custom box type for a specific protocol, instead of one the compiler creates automatically for you, then it would be very reasonable for that box to conform to the protocol too. The conformance would be implemented by forwarding calls on to the boxed value. You wouldn't need to then unbox to use the thing with generic functions. And this is just what AnyCollection and friends do.

Now, because protocols are special and the compiler does a bunch of work for them behind the scenes, conforming their "box" functionality might not work exactly like that. But it's not absurd or nonsense to think of it similarly.

[Ben_Cohen]

@bjhomer

I'd be interested in seeing examples of where it would be useful, though.

The most common use for existentials is to be able to store heterogenous types in a collection that requires a single homogenous element type. It's also common to want to write extension Collection where Element: SomeProtocol in order to do generic processing on any collection where an element conforms to a protocol. Lack of existential conformance confounds this.

For example:

// our docs admonish you not to use CustomStringConvertible
// this way, but let's ignore that for now...
extension Collection where Element: CustomStringConvertible {
  func prettyPrint() {
    let strings = self.lazy.map { $0.description }
    print( "[\(strings.joined(separator: ", "))]" )
  }
}

let someInts: [Int] = [1,2,3]
someInts.prettyPrint()
// [1, 2, 3]

let somePrintableThings: [CustomStringConvertible] = [1,"two",3]
somePrintableThings.prettyPrint()
// error: value of protocol type 'CustomStringConvertible' cannot conform 
// to 'CustomStringConvertible'; only struct/enum/class types can conform
// to protocols

// And yet you can just take the body of prettyPrint and use it...
let strings = somePrintableThings.lazy.map { $0.description }
print( "[\(strings.joined(separator: ", "))]" )
// [1, two, 3]

This to me makes the clear case that this is a hole in the language that should be fixed.

I do worry that existentials are already significantly overused, and I wish people would treat them as a last rather than first resort more often. But we shouldn't not fix their legitimate use just because of that, even if I fear that will lead to even more misuse.

[suyashsrijan]

Hmm, good point. I think you'll also have to omit the fix-it if the function returns T. Perhaps a note without fix-it would be enough?

Is such a fix-it always source-compatible with the call sites of f if Proto can be used as an existential?

I think so (as long as T is not used in ways as described above and I am not forgetting anything else!):

passing an existential as an argument is isomorphic to passing a generic argument with its own type variable—anything that's possible with one representation is possible on the other. We've even discussed giving them the same underlying ABI.

[harlanhaskins]

Just a small clarification: through a little bit of compiler magic, the capability to open an existential does existing in the shipping compiler through the _openExistential builtin.

import Foundation

protocol Attachment {
    var fileURL: URL { get }
}

struct Photo: Attachment {
    var fileURL: URL {
        URL(fileURLWithPath: "/tmp/photo.jpg")
    }
}
struct Video: Attachment {
    var fileURL: URL {
        URL(fileURLWithPath: "/tmp/video.mov")
    }
}

struct AnyAttachment: Attachment {
    var attachment: Attachment

    var fileURL: URL {
        func project<T: Attachment>(_ attachment: T) -> URL {
            attachment.fileURL
        }
        return _openExistential(attachment, do: project)
    }
}

let photo = Photo()
let anyPhoto = AnyAttachment(attachment: photo)
anyPhoto.fileURL // "file:///tmp/photo.jpg"

let video = Video()
let anyVideo = AnyAttachment(attachment: video)
anyVideo.fileURL // "file:///tmp/video.mov"

[dfsweeney (Daniel Sweeney)]

Thanks @harlanhaskins. I have sort of struggled with _openExistential while trying to figure this out--I read the source code, looked at stuff, tried to make it work. I want to state my understanding out loud so somebody can tell me I'm right or wrong:

1. in the AnyAttachment struct, the attachment var is an existential because it's defined as an Attachment. So when. you make an AnyAttachment you give it something that conforms to the Attachment protocol.

2. In the normal world, at that point you would be able to call .fileURL directly--if I replace the body of the AnyAttachment.fileURL property in the code above with just a direct call to attachment.fileURL it works the same as if I use the _openExistential code.

3. So I'm trying to work through what's really happening there--in the direct call there is dynamic dispatch to .fileURL property. In the _openExistential code, something with an existential type goes in to _openExistential along with a closure that's generic over the existential type, and _openExistential calls the closure. The mechanics are different but I am having trouble seeing a difference in effect.

I guess I am asking, what is the difference in the typing environment inside the closure? Is there a more complicated example that would show a difference in behavior?

Thanks. I know this is off-topic from self-conformance but it's not completely off-topic, and it might be good to get the knowledge out there.

[harlanhaskins]

Y'know, my example should have actually included some code that demonstrates the utility here. The utility is being able to pass the value from the existential over to a generic function that expects <T: Attachment>, something like this:

func printFileURL<T: Attachment>(of attachment: T) {
    print(attachment.fileURL)
}

func printFileURLFromExistential(of attachment: Attachment) {
    printFileURL(of: attachment) // does not compile

    _openExistential(attachment, do: printFileURL) // does compile
}

[dfsweeney (Daniel Sweeney)]

Thank you! That is good.

The generic function (requiring a conformance to the protocol) printFileURL would in real life exist somewhere else and just be written to do its thing, like people do. You can't pass the existential directly as an argument to the generic function printFileURL because it's trying to make the argument (an existential) into the same type as the generic argument T, which must be a concrete type conforming to the Attachment protocol. I put the full error message into the line below. In the error message , the 'Value of protocol type' is the existential argument, it does not self-conform, and it can't conform to the protocol so it can't fulfill T.

printFileURL(of: attachment) // does not compile-- Value of protocol type 'Attachment' cannot conform to 'Attachment'; only struct/enum/class types can conform to protocols

printFileFromExistential is not generic, it just takes an existential as an argument. You could do some things with the existential in the body, but you can't call generic functions that want an argument of the type of the protocol. Which is why this is confusing to talk about.

---

Going back to another example higher in the thread, this almost works:

  func prettyPrint<T>(x: T) where T : Collection,
                                  T.Element : CustomStringConvertible {
    let s = x.map {$0.description}
    print( "[\(s.joined(separator: ", "))]" )
  }

  let somePrintableThings: [CustomStringConvertible] = [1, "two", 3]

/// ALMOST! 
  _openExistential(somePrintableThings,
                   do: prettyPrint) // Type of expression is ambiguous without more context

  prettyPrint(x: somePrintableThings) // Value of protocol type 'CustomStringConvertible' cannot conform to 'CustomStringConvertible'; only struct/enum/class types can conform to protocols

I think the problem is that somePrintableThings is an Array of CustomStringConvertible, not an object conforming to the CustomStringConvertible protocol. But I'm not completely sure about that. It's possible that there is some type annotation I could add somewhere that would make that work.

Thank you @harlanhaskins!

[Alejandro (Alejandro Alonso)]

I believe the issue is that [CustomStringConvertible] is itself not an existential, but rather a generic type containing one. The following works just fine:

func prettyPrint<T: CustomStringConvertible>(_ x: T) {
  print(x.description)
}

let somePrintableThings: CustomStringConvertible = [1, "two", 3]

_openExistential(somePrintableThings, do: prettyPrint(_:))

[harlanhaskins]

One of the issues in this example is that [CustomStringConvertible] is a potentially heterogenous collection, whereas prettyPrint<T>(x: T) where T: Collection, T.Element: CustomStringConvertible requires all the elements in the collection to be the same.

[brentdax (Brent Royal-Gordon)]

The magic bit of _openExistential is that, if you tried to write out its declaration, you’d end up with something like:

func _openExistential<Existential, Result>(
   _ existential: Existential,
   do fn: <Opened: Existential>(Opened) -> Result
) -> Result

The problem is that, although this signature might (might!) make sense to a human Swift programmer, to the compiler it is utter gibberish that doesn’t even parse correctly. Since Existential is a generic parameter, Opened: Existential is not valid to write, and the type of a parameter (or of any variable/value) can’t be generic anyway. And yet special cases have been hacked into the compiler to make _openExistential behave like it has a signature like this.

(The fact that _openExistential’s type is so weird is also why the type checker tends to say vague things like “type of expression is ambiguous without more context” instead of telling you what’s actually wrong. _openExistential’s type checking failures look different from failures of types you can actually express in the language, and little effort has been put into polishing them.)

[dfsweeney (Daniel Sweeney)]

_openExistential is a good puzzle and helped me understand some things. I don't want to derail the self-conformance thread though. Thank you all for pushing my understanding (and hopefully everybody else's understanding) forward one step!

[John_McCall]

It's important to distinguish two ideas when talking about self-conformance:

1. The ability of a protocol or protocol composition type to conform to a protocol, most likely (but not necessarily) itself.
2. The ability of such a conformance to automatically satisfy its requirements by forwarding to the underlying value.

The first is always logically possible. The second is fundamentally restricted because the operation on the underlying value may not be able to satisfy the requirements (or even the type signature) of the protocol as applied to the protocol type. For example:

• A static method requirement cannot simply forward to the conformance for the underlying value because there is no underlying value. When a static method of some type T is called, it's passed a value of the type T.Type. For a protocol self-conformance P: P, this corresponds to the protocol metatype P.Protocol, which doesn't carry a specific conforming type, so we don't know which conformance to forward to. It obviously isn't reasonable behavior to just pick one at random.

• An initializer requirement has the same restriction for essentially the same reason.

• Requirements whose signature uses the Self type in an input position (e.g. as a parameter) cannot simply forward because there's no always-valid way to turn an input value of type P into an input value of the same type as the underlying value. This problem is defined away in the current language because we don't allow protocols with these requirements to be used in protocol types, but if we lift that restriction ("generalized existentials"), it'll surface here immediately.

So, if we allow self-conformance but inherently tie it to the ability to forward implementations, we're actually only allowing self-conformance for protocols where all the members satisfy these restrictions. And maybe that's okay, but it becomes a permanent constraint on those protocols: once they declare self-conformance, they can never add a requirement that violates these restrictions without irrevocably breaking clients that were relying on self-conformance.

So I think the right design direction here is to pursue fully-general conformances of protocol types to protocols, and then say that conformances on protocols have extra defaulting powers. But that would start with just allowing people to add members to protocols that are actually members of the protocol type rather than being implicitly added to all conforming types.

[Jumhyn (Frederick Kellison-Linn)]

Thanks for teasing apart these two concepts, John—I agree it's good to think about them independently.

@John_McCall

once they declare self-conformance, they can never add a requirement that violates these restrictions without irrevocably breaking clients that were relying on self-conformance.

This doesn't seem more problematic than the situation we have today with the fact that adding Self and associatedtype requirements breaks any clients that were formerly using the protocol as an existential. (Unless, perhaps, it's more common for authors to add static/init requirements to an existing protocol than it is Self/associatedtype requirements, so making their introduction a source-breaking change is more problematic.)

In some ways, self-conformance even seems less problematic than the status quo—if we require the self-conformance to be marked explicitly, then the author cannot break clients silently by adding members. E.g., if in FooKit v1.0 I have:

// Straw syntax
@selfconforming
protocol Foo {
  func frobnicate()
}

and I try to update Foo in v2.0 to

@selfconforming
protocol Foo {
  func frobnicate()
  static func frobnicateStatically()
}

I would get a compile error.

OTOH, in Swift today, adding a Self/associatedtype requirement to a protocol P won't necessarily break my own module (since I might only be using P as a generic constraint anyway), but once I ship, clients who were using P as an existential are suddenly broken!

If the fully-generalized "custom conformances for existentials" is considered the natural next step of self-conformance, it doesn't strike me as obvious that just allowing self-conformance for protocols is a bad resting place, aside from the fact that we would want to choose a syntax for declaring the conformance that could be extended to cover the general case as well.

(I have minor concerns about the general protocol-existentails-conform-to-protocols direction as well, but if the discussion is going to take that direction it's probably worth starting another thread for the general feature.)

[John_McCall]

@Jumhyn

In some ways, self-conformance even seems less problematic than the status quo—if we require the self-conformance to be marked explicitly, then the author cannot break clients silently by adding members.

It's a fair point that protocols have this evolution problem today because of the restrictions on protocol types. On the other hand, that's something we specifically want to solve by generalizing existentials, not something we want to double down on.

We do also have this evolution restriction on @objc protocols, although there it's also tied to the ObjC class model, which doesn't support adding new methods through defaulting.

@Jumhyn

If the fully-generalized "custom conformances for existentials" is considered the natural next step of self-conformance, it doesn't strike me as obvious that just allowing self-conformance for protocols is a bad resting place, aside from the fact that we would want to choose a syntax for declaring the conformance that could be extended to cover the general case as well.

I agree that the syntax for declaring the conformance seems to be the main thing informed by the general picture.

[lassejansen]

@John_McCall

So, if we allow self-conformance but inherently tie it to the ability to forward implementations (...) it becomes a permanent constraint on those protocols: once they declare self-conformance, they can never add a requirement that violates these restrictions without irrevocably breaking clients that were relying on self-conformance.

This may be a phantasy because it's quite source-breaking, but would it make sense if clients would need to opt in to let's call them "meta-type requirements" (static, init, Self)?

The some keyword could be used to require that the type must be known at compile time:

func takeASpecificAnimal<T: some Animal>(_ animal: T) {
    // access to T.someStaticMethod()
}

func takeASpecificAnimal<T: Animal>(_ animal: T) {
    // no access to T.someStaticMethod() 
}

Essentially, without the some keyword the client would only get access to a subset of the protocol requirements (i.e. the non-meta-type requirements) and adding requirements later on wouldn't break clients that were relying on (partial) self-conformance.

[GreatApe (Gustaf Kugelberg)]

@Jumhyn

In some ways, self-conformance even seems less problematic than the status quo—if we require the self-conformance to be marked explicitly, then the author cannot break clients silently by adding members.

Couldn't the solution be even more flexible, by simply requiring, and allowing, the user to implement any static methods/inits that may be required, when he declares the conformance?

protocol Foo: Self { // I would prefer this syntax
    func frobnicate()
    // Error: "Static methods on self-conforming protocols need to have an implementation"
    static func frobnicateStatically() 
}

[Nickolas_Pohilets (Nickolas Pokhylets)]

I think it should be another way around to preserve source compatibility:

func takeASpecificAnimal<T: Animal>(_ animal: T) {
    // access to T.someStaticMethod()
    // T cannot be existential container
}

func takeASpecificAnimal<T: any Animal>(_ animal: T) {
    // no access to T.someStaticMethod()
    // T can be existential container
}

Where T: any Animal means T is a type which is a subtype of (or should it be implictly castable to?) the existential container for Animal (aka any Animal or Any<Animal>).

[lassejansen]

Yes, but reading Improving the UI of generics and the linked discusstion thread, it looks like the any syntax might also be source breaking eventually (with a deprecation period):

• Making existential types require an explicit keyword. This one is particularly interesting because of the source compatibility concerns. Some previous discussion of this point came up in a thread about lifting the "self or associated type" constraint on existential types.

[Nickolas_Pohilets (Nickolas Pokhylets)]

I'm not aware of any strong reasons to make this change source-breaking. We could still promote P to any P when there is no ambiguity in the role of a protocol. And require any only when there is need to disambiguate.

But regardless of the source compatibility, my main point was to give kudos to the idea. I think it solves the original problem of using existential containers with generics better than self-conformance.

protocol Q {}

protocol P {
    associatedtype AT: Q
    func a()
    static func b()
    func c() -> AT
    func d(_ x: AT)
}

func f<T: any P>(...) { ... }

Is equivalent to following code:

protocol Q {}

protocol _P {
    func _a()
    func _c() -> any Q
}

protocol P: _P {
    associatedtype AT: Q
    func a()
    static func b()
    func c() -> AT
    func d(_ x: AT)
}

extension P {
    func _a() { self.a() }
    func _c() -> Q { return self.c() }
}

extension (any P): _P {
    func _a() {
        let <T: P> zelf = self
        zelf.a()
    }
   func _c() -> any Q {
        let <T: P> zelf = self
        return zelf.c()
   }
}

func f<T: _P>(...) { ... }

It is more powerful, as it allows protocol in question to have static requirements and associated types. And it will not break when new members are added to the protocol.

On the other hand, such equivalence shows that we don't strictly need new kind of generic constraint, if we have fully-generalized "custom conformances for existentials". But that's a lot of boiler-plate code to write, and understanding what code needs to be written actually requires deep understanding of protocols, existential containers and generics.

[John_McCall]

If the caller knows the type statically, that's just generics.

The idea of using something like some P as a shorthand for declaring a generic function, and making it easier to call generics with values of existential and existential-metatype type, is something that I know @Joe_Groff has thought about a lot.