Allowing self-conformance for protocols

Well, we have Error which was special-cased to allow for this, plus countless users confused by "protocol type P does not conform to P" error messages, but I agree that more concrete motivation would be useful. The thread containing Slava's response above provides a good example, IMO, and it's even more problematic when applied to types (since you can't overload a type). E.g., if I define a FooConsumer<T> with a member func consume(_ foo: T), there is no way for me to express that FooConsumer can be parameterized by any concrete Foo-conforming type, or by the Any<Foo> existential.

Ah, yeah I should have been clearer—I'm specifically talking about implicit conformance for protocols which satisfy the constraints that Slava listed: i.e., those without contravariant Self, associatedtype, static, or init requirements.

I don't think this is the case. If it were possible today to write an extension for Any<Attachment> (and conform it to protocols), it seems like it would be perfectly valid to write:

extension Any<Attachment> : Attachment {
  var fileURL: URL {
    return self.fileURL
  }
}

Why wouldn't this be valid?

I don't even think the ability to open existentials helps us here. In the body of a static member on an existential, we might not even have a type that we can open! After all, it would be perfectly valid to write Any<Attachment>.contentType.

I think I lean the other way, in I think the conformance Any<P>: P should be implicitly supplied for eligible protocols (if it can be done in a source-compatible way), but that's mainly because I can't think of a good spelling for specifying that conformance that doesn't invite declaring arbitrary members of Any<P>. Maybe that's not a problem in practice...

1 Like

It’s infinitely recursive, unless ‘self’ means something different inside an existential extension.

1 Like

Ah, yep, thanks. :man_facepalming:

ETA:

Though really, I would expect to be able to just write

extension Any<Attachment> : Attachment {}

and have the witnesses determined automatically.

1 Like

In the meantime, do you think it’s worth improving the diagnostics by emitting a note (and possibly a fix-it if method lies in the same module) to let the user know that they could change func f<T: Proto>(arg: T) to func f(arg: Proto) as an alternative (as long as protocol doesn’t have associated types and etc) in order to pass a protocol typed value?

4 Likes

At the very least, such a diagnostic would have to check against the entire body of f to make sure that arg was not used in ways the require a concrete type, and that T was not referenced directly. Is such a fix-it always source-compatible with the call sites of f if Proto can be used as an existential?

I find that many cases where self-conformance is needed, can be solved by using subtyping relation instead of protocol conformance.

Original thread - Existential subtyping as generic constraint
MR - https://github.com/apple/swift/pull/33552

I’m suggesting syntax f<T: P> for protocol conformance requirement and f<T: any P> for existential subtyping.

First one allows T to be any concrete type confirming to P, and allows usage of static requirements and associated types.

Second one allows T to be any concrete type conforming to P, but also existential type for P or any derived protocol. But static requirements and associated types are not available inside f.

Witness table for the existential subtyping can have single entry - a function that returns an existential container for P given a value of T.

2 Likes

Yes.

I don't think so. The additional diagnostic would be more actionable: you could e.g. split your protocol in static and non-static protocols.

I'd say yes.

Fwiw this is the example written in Rust:

trait Animal {
    fn is_mammal(&self) -> bool;
    //fn static_is_mammal() -> bool;
}

struct Dog {}

impl Animal for Dog {
    fn is_mammal(&self) -> bool { true }
    //fn static_is_mammal() -> bool { true }
}

fn take_a_specific_animal<T: Animal + ?Sized>( _t: &T) {}

fn main() {
    let dog: Box<dyn Animal> = Box::new(Dog{});
    take_a_specific_animal( &*dog);
}

Without static_is_mammal() the code compiles.

If you add the static_is_mammal() function you get this error:

the trait Animal cannot be made into an object
the trait bound dyn Animal: Animal is not satisfied

I don't know why you think this is "absurd" and "nonsense" and I'd actually ask you tone the rhetoric down a little bit. Perhaps you are both talking about different things and need to get on the same page, but throwing extreme terms around won't help.

If this were a custom box type for a specific protocol, instead of one the compiler creates automatically for you, then it would be very reasonable for that box to conform to the protocol too. The conformance would be implemented by forwarding calls on to the boxed value. You wouldn't need to then unbox to use the thing with generic functions. And this is just what AnyCollection and friends do.

Now, because protocols are special and the compiler does a bunch of work for them behind the scenes, conforming their "box" functionality might not work exactly like that. But it's not absurd or nonsense to think of it similarly.

13 Likes

The most common use for existentials is to be able to store heterogenous types in a collection that requires a single homogenous element type. It's also common to want to write extension Collection where Element: SomeProtocol in order to do generic processing on any collection where an element conforms to a protocol. Lack of existential conformance confounds this.

For example:

// our docs admonish you not to use CustomStringConvertible
// this way, but let's ignore that for now...
extension Collection where Element: CustomStringConvertible {
  func prettyPrint() {
    let strings = self.lazy.map { $0.description }
    print( "[\(strings.joined(separator: ", "))]" )
  }
}

let someInts: [Int] = [1,2,3]
someInts.prettyPrint()
// [1, 2, 3]

let somePrintableThings: [CustomStringConvertible] = [1,"two",3]
somePrintableThings.prettyPrint()
// error: value of protocol type 'CustomStringConvertible' cannot conform 
// to 'CustomStringConvertible'; only struct/enum/class types can conform
// to protocols

// And yet you can just take the body of prettyPrint and use it...
let strings = somePrintableThings.lazy.map { $0.description }
print( "[\(strings.joined(separator: ", "))]" )
// [1, two, 3]

This to me makes the clear case that this is a hole in the language that should be fixed.

I do worry that existentials are already significantly overused, and I wish people would treat them as a last rather than first resort more often. But we shouldn't not fix their legitimate use just because of that, even if I fear that will lead to even more misuse.

19 Likes

Hmm, good point. I think you'll also have to omit the fix-it if the function returns T. Perhaps a note without fix-it would be enough?

Is such a fix-it always source-compatible with the call sites of f if Proto can be used as an existential?

I think so (as long as T is not used in ways as described above and I am not forgetting anything else!):

Just a small clarification: through a little bit of compiler magic, the capability to open an existential does existing in the shipping compiler through the _openExistential builtin.

import Foundation

protocol Attachment {
    var fileURL: URL { get }
}

struct Photo: Attachment {
    var fileURL: URL {
        URL(fileURLWithPath: "/tmp/photo.jpg")
    }
}
struct Video: Attachment {
    var fileURL: URL {
        URL(fileURLWithPath: "/tmp/video.mov")
    }
}

struct AnyAttachment: Attachment {
    var attachment: Attachment

    var fileURL: URL {
        func project<T: Attachment>(_ attachment: T) -> URL {
            attachment.fileURL
        }
        return _openExistential(attachment, do: project)
    }
}

let photo = Photo()
let anyPhoto = AnyAttachment(attachment: photo)
anyPhoto.fileURL // "file:///tmp/photo.jpg"

let video = Video()
let anyVideo = AnyAttachment(attachment: video)
anyVideo.fileURL // "file:///tmp/video.mov"
6 Likes

Thanks @harlanhaskins. I have sort of struggled with _openExistential while trying to figure this out--I read the source code, looked at stuff, tried to make it work. I want to state my understanding out loud so somebody can tell me I'm right or wrong:

  1. in the AnyAttachment struct, the attachment var is an existential because it's defined as an Attachment. So when. you make an AnyAttachment you give it something that conforms to the Attachment protocol.

  2. In the normal world, at that point you would be able to call .fileURL directly--if I replace the body of the AnyAttachment.fileURL property in the code above with just a direct call to attachment.fileURL it works the same as if I use the _openExistential code.

  3. So I'm trying to work through what's really happening there--in the direct call there is dynamic dispatch to .fileURL property. In the _openExistential code, something with an existential type goes in to _openExistential along with a closure that's generic over the existential type, and _openExistential calls the closure. The mechanics are different but I am having trouble seeing a difference in effect.

I guess I am asking, what is the difference in the typing environment inside the closure? Is there a more complicated example that would show a difference in behavior?

Thanks. I know this is off-topic from self-conformance but it's not completely off-topic, and it might be good to get the knowledge out there.

2 Likes

Y'know, my example should have actually included some code that demonstrates the utility here. The utility is being able to pass the value from the existential over to a generic function that expects <T: Attachment>, something like this:

func printFileURL<T: Attachment>(of attachment: T) {
    print(attachment.fileURL)
}

func printFileURLFromExistential(of attachment: Attachment) {
    printFileURL(of: attachment) // does not compile

    _openExistential(attachment, do: printFileURL) // does compile
}
4 Likes

Thank you! That is good.

The generic function (requiring a conformance to the protocol) printFileURL would in real life exist somewhere else and just be written to do its thing, like people do. You can't pass the existential directly as an argument to the generic function printFileURL because it's trying to make the argument (an existential) into the same type as the generic argument T, which must be a concrete type conforming to the Attachment protocol. I put the full error message into the line below. In the error message , the 'Value of protocol type' is the existential argument, it does not self-conform, and it can't conform to the protocol so it can't fulfill T.

printFileURL(of: attachment) // does not compile-- Value of protocol type 'Attachment' cannot conform to 'Attachment'; only struct/enum/class types can conform to protocols

printFileFromExistential is not generic, it just takes an existential as an argument. You could do some things with the existential in the body, but you can't call generic functions that want an argument of the type of the protocol. Which is why this is confusing to talk about.


Going back to another example higher in the thread, this almost works:

  func prettyPrint<T>(x: T) where T : Collection,
                                  T.Element : CustomStringConvertible {
    let s = x.map {$0.description}
    print( "[\(s.joined(separator: ", "))]" )
  }

  let somePrintableThings: [CustomStringConvertible] = [1, "two", 3]

/// ALMOST! 
  _openExistential(somePrintableThings,
                   do: prettyPrint) // Type of expression is ambiguous without more context

  prettyPrint(x: somePrintableThings) // Value of protocol type 'CustomStringConvertible' cannot conform to 'CustomStringConvertible'; only struct/enum/class types can conform to protocols

I think the problem is that somePrintableThings is an Array of CustomStringConvertible, not an object conforming to the CustomStringConvertible protocol. But I'm not completely sure about that. It's possible that there is some type annotation I could add somewhere that would make that work.

Thank you @harlanhaskins!

I believe the issue is that [CustomStringConvertible] is itself not an existential, but rather a generic type containing one. The following works just fine:

func prettyPrint<T: CustomStringConvertible>(_ x: T) {
  print(x.description)
}

let somePrintableThings: CustomStringConvertible = [1, "two", 3]

_openExistential(somePrintableThings, do: prettyPrint(_:))
2 Likes

One of the issues in this example is that [CustomStringConvertible] is a potentially heterogenous collection, whereas prettyPrint<T>(x: T) where T: Collection, T.Element: CustomStringConvertible requires all the elements in the collection to be the same.

3 Likes

The magic bit of _openExistential is that, if you tried to write out its declaration, you’d end up with something like:

func _openExistential<Existential, Result>(
   _ existential: Existential,
   do fn: <Opened: Existential>(Opened) -> Result
) -> Result

The problem is that, although this signature might (might!) make sense to a human Swift programmer, to the compiler it is utter gibberish that doesn’t even parse correctly. Since Existential is a generic parameter, Opened: Existential is not valid to write, and the type of a parameter (or of any variable/value) can’t be generic anyway. And yet special cases have been hacked into the compiler to make _openExistential behave like it has a signature like this.

(The fact that _openExistential’s type is so weird is also why the type checker tends to say vague things like “type of expression is ambiguous without more context” instead of telling you what’s actually wrong. _openExistential’s type checking failures look different from failures of types you can actually express in the language, and little effort has been put into polishing them.)

8 Likes

_openExistential is a good puzzle and helped me understand some things. I don't want to derail the self-conformance thread though. Thank you all for pushing my understanding (and hopefully everybody else's understanding) forward one step!

It's important to distinguish two ideas when talking about self-conformance:

  1. The ability of a protocol or protocol composition type to conform to a protocol, most likely (but not necessarily) itself.
  2. The ability of such a conformance to automatically satisfy its requirements by forwarding to the underlying value.

The first is always logically possible. The second is fundamentally restricted because the operation on the underlying value may not be able to satisfy the requirements (or even the type signature) of the protocol as applied to the protocol type. For example:

  • A static method requirement cannot simply forward to the conformance for the underlying value because there is no underlying value. When a static method of some type T is called, it's passed a value of the type T.Type. For a protocol self-conformance P: P, this corresponds to the protocol metatype P.Protocol, which doesn't carry a specific conforming type, so we don't know which conformance to forward to. It obviously isn't reasonable behavior to just pick one at random.

  • An initializer requirement has the same restriction for essentially the same reason.

  • Requirements whose signature uses the Self type in an input position (e.g. as a parameter) cannot simply forward because there's no always-valid way to turn an input value of type P into an input value of the same type as the underlying value. This problem is defined away in the current language because we don't allow protocols with these requirements to be used in protocol types, but if we lift that restriction ("generalized existentials"), it'll surface here immediately.

So, if we allow self-conformance but inherently tie it to the ability to forward implementations, we're actually only allowing self-conformance for protocols where all the members satisfy these restrictions. And maybe that's okay, but it becomes a permanent constraint on those protocols: once they declare self-conformance, they can never add a requirement that violates these restrictions without irrevocably breaking clients that were relying on self-conformance.

So I think the right design direction here is to pursue fully-general conformances of protocol types to protocols, and then say that conformances on protocols have extra defaulting powers. But that would start with just allowing people to add members to protocols that are actually members of the protocol type rather than being implicitly added to all conforming types.

17 Likes

Thanks for teasing apart these two concepts, John—I agree it's good to think about them independently.

This doesn't seem more problematic than the situation we have today with the fact that adding Self and associatedtype requirements breaks any clients that were formerly using the protocol as an existential. (Unless, perhaps, it's more common for authors to add static/init requirements to an existing protocol than it is Self/associatedtype requirements, so making their introduction a source-breaking change is more problematic.)

In some ways, self-conformance even seems less problematic than the status quo—if we require the self-conformance to be marked explicitly, then the author cannot break clients silently by adding members. E.g., if in FooKit v1.0 I have:

// Straw syntax
@selfconforming
protocol Foo {
  func frobnicate()
}

and I try to update Foo in v2.0 to

@selfconforming
protocol Foo {
  func frobnicate()
  static func frobnicateStatically()
}

I would get a compile error.

OTOH, in Swift today, adding a Self/associatedtype requirement to a protocol P won't necessarily break my own module (since I might only be using P as a generic constraint anyway), but once I ship, clients who were using P as an existential are suddenly broken!

If the fully-generalized "custom conformances for existentials" is considered the natural next step of self-conformance, it doesn't strike me as obvious that just allowing self-conformance for protocols is a bad resting place, aside from the fact that we would want to choose a syntax for declaring the conformance that could be extended to cover the general case as well.

(I have minor concerns about the general protocol-existentails-conform-to-protocols direction as well, but if the discussion is going to take that direction it's probably worth starting another thread for the general feature.)