Allowing self-conformance for protocols

Since the topic of self-conforming protocols has been brought up a couple times this week, I thought it would be good to start a separate discussion to determine whether this problem is worth addressing directly. The main motivation, to borrow some examples from @bjhomer, is to allow the following to compile:

protocol Animal {}
struct Cat: Animal {}
struct Dog: Animal {}

// A variable of type 'Animal' is an existential; it can hold *any* Animal
var anyAnimal: Animal
anyAnimal = Cat()
anyAnimal = Dog()

func takeASpecificAnimal<T: Animal>(_ animal: T) {}

// error: value of protocol type 'Animal' cannot conform to 'Animal'; only struct/enum/class types can conform to protocols
takeASpecificAnimal(anyAnimal)

This fails today with the noted error.

This has been mentioned many times over the years, but I couldn't find any threads dedicated towards this feature specifically. The most comprehensive account of the technical limitations I could find was from @Slava_Pestov, several years back:

If anyone can track down other thorough treatments of this issue, please call them out and I can link them here! The primary question to answer initially is whether the technical landscape has changed in any relevant ways that makes this feature infeasible (e.g., from an ABI stability perspective). If it's still technically possible, here are some questions I have to guide initial discussion:

  1. Do people feel that it is worthwhile at all to lift this limitation?
  2. If so, does the additional diagnostic "protocols with init or static func requirements cannot self-conform" leave us in a worse place than the existing "protocols cannot conform to other protocols" diagnostic?
  3. Is such a self-conformance implicitly supplied simply by declaring the protocol?
  4. If not, how should the self-conformance be spelled in source?
  5. If we offer a way to spell self-conformance explicitly, should we extend this syntax to allow protocol types to conform to other 'simple' protocols?

Would love to hear everyone's thoughts on these topics as well as anything else that comes to mind regarding self-conforming protocols!

8 Likes
1 Like

I regret that we talk about this as "self-conformance"—I think that's a misleading way to phrase it. A protocol cannot conform to itself, that would be like declaring protocol P : P {}. The question here is whether an existential of protocol P should be allowed to conform to P itself.

1/2:

I'm not convinced that it's useful to allow it right now when existentials can only sometimes be formed, and when we have no way to extract the original type from an existential. I'd be interested in seeing examples of where it would be useful, though.

3:

Implicit conformance wouldn't always be possible. Consider the following example (which uses my proposed Any<P> syntax for clarity to represent the existential itself.)

protocol Attachment {
	var fileURL: URL { get }
}

struct Photo: Attachment { ... }
struct Video: Attachment { ... }

/* implicitly generated by compiler:
extension Any<Attachment> : Attachment {
	var fileURL: URL {
		let unwrappedSelf: <T: Attachment> = self // "Opening" the existential to get the underlying type
		return unwrappedSelf.fileURL
	}
}
*/

The implementation of this conformance requires features that we cannot even write in Swift right now -- specifically, the ability to open an existential. Now imagine that we add another requirement:

protocol Attachment {
	static var contentType: String { get }
	var fileURL: URL { get }
}

The compiler cannot automatically generate the conformance for us, and we have no way to write the conformance ourselves because we cannot write the implementation for Any<Attachmen>.contentType without the ability to open existentials. So by adding a static var, we've now lost the ability for the existential to conform to the protocol at all. We can't automatically generate it, and we cannot write it ourselves.

So again, I think it's a pitfal to allow conformance without the ability to open existentials.

4.

Because the compiler cannot always generate a conformance, I think there must be a way to spell it in source.

5.

If we have a way to spell existential conformances in source, I see no reason they should be limited to only the originating protocol.

3 Likes

Personally I find the whole idea of existential conformances to be absurd. An existential that is bound to some protocol is "a box which contains a thing that conforms to that protocol". The box itself does not conform.

In a world with existential conformances, you would be able to have a box which contains any Attachment instance, then ask it for the content type of the box. It's just nonsense.

It gets worse when associated types enter the fray. Suddenly you have to define "default" types which are applicable to the box.

Other solutions, like path-dependent types or unboxing to local generic type parameters, don't have this conceptual weirdness - which shows that it is not fundamentally required to solve the problem.

1 Like

Well, we have Error which was special-cased to allow for this, plus countless users confused by "protocol type P does not conform to P" error messages, but I agree that more concrete motivation would be useful. The thread containing Slava's response above provides a good example, IMO, and it's even more problematic when applied to types (since you can't overload a type). E.g., if I define a FooConsumer<T> with a member func consume(_ foo: T), there is no way for me to express that FooConsumer can be parameterized by any concrete Foo-conforming type, or by the Any<Foo> existential.

Ah, yeah I should have been clearer—I'm specifically talking about implicit conformance for protocols which satisfy the constraints that Slava listed: i.e., those without contravariant Self, associatedtype, static, or init requirements.

I don't think this is the case. If it were possible today to write an extension for Any<Attachment> (and conform it to protocols), it seems like it would be perfectly valid to write:

extension Any<Attachment> : Attachment {
  var fileURL: URL {
    return self.fileURL
  }
}

Why wouldn't this be valid?

I don't even think the ability to open existentials helps us here. In the body of a static member on an existential, we might not even have a type that we can open! After all, it would be perfectly valid to write Any<Attachment>.contentType.

I think I lean the other way, in I think the conformance Any<P>: P should be implicitly supplied for eligible protocols (if it can be done in a source-compatible way), but that's mainly because I can't think of a good spelling for specifying that conformance that doesn't invite declaring arbitrary members of Any<P>. Maybe that's not a problem in practice...

1 Like

It’s infinitely recursive, unless ‘self’ means something different inside an existential extension.

1 Like

Ah, yep, thanks. :man_facepalming:

ETA:

Though really, I would expect to be able to just write

extension Any<Attachment> : Attachment {}

and have the witnesses determined automatically.

1 Like

In the meantime, do you think it’s worth improving the diagnostics by emitting a note (and possibly a fix-it if method lies in the same module) to let the user know that they could change func f<T: Proto>(arg: T) to func f(arg: Proto) as an alternative (as long as protocol doesn’t have associated types and etc) in order to pass a protocol typed value?

4 Likes

At the very least, such a diagnostic would have to check against the entire body of f to make sure that arg was not used in ways the require a concrete type, and that T was not referenced directly. Is such a fix-it always source-compatible with the call sites of f if Proto can be used as an existential?

I find that many cases where self-conformance is needed, can be solved by using subtyping relation instead of protocol conformance.

Original thread - Existential subtyping as generic constraint
MR - [Generics Manifesto] Existential subtyping as generic constraint by nickolas-pohilets · Pull Request #33552 · apple/swift · GitHub

I’m suggesting syntax f<T: P> for protocol conformance requirement and f<T: any P> for existential subtyping.

First one allows T to be any concrete type confirming to P, and allows usage of static requirements and associated types.

Second one allows T to be any concrete type conforming to P, but also existential type for P or any derived protocol. But static requirements and associated types are not available inside f.

Witness table for the existential subtyping can have single entry - a function that returns an existential container for P given a value of T.

2 Likes

Yes.

I don't think so. The additional diagnostic would be more actionable: you could e.g. split your protocol in static and non-static protocols.

I'd say yes.

Fwiw this is the example written in Rust:

trait Animal {
    fn is_mammal(&self) -> bool;
    //fn static_is_mammal() -> bool;
}

struct Dog {}

impl Animal for Dog {
    fn is_mammal(&self) -> bool { true }
    //fn static_is_mammal() -> bool { true }
}

fn take_a_specific_animal<T: Animal + ?Sized>( _t: &T) {}

fn main() {
    let dog: Box<dyn Animal> = Box::new(Dog{});
    take_a_specific_animal( &*dog);
}

Without static_is_mammal() the code compiles.

If you add the static_is_mammal() function you get this error:

the trait Animal cannot be made into an object
the trait bound dyn Animal: Animal is not satisfied

I don't know why you think this is "absurd" and "nonsense" and I'd actually ask you tone the rhetoric down a little bit. Perhaps you are both talking about different things and need to get on the same page, but throwing extreme terms around won't help.

If this were a custom box type for a specific protocol, instead of one the compiler creates automatically for you, then it would be very reasonable for that box to conform to the protocol too. The conformance would be implemented by forwarding calls on to the boxed value. You wouldn't need to then unbox to use the thing with generic functions. And this is just what AnyCollection and friends do.

Now, because protocols are special and the compiler does a bunch of work for them behind the scenes, conforming their "box" functionality might not work exactly like that. But it's not absurd or nonsense to think of it similarly.

13 Likes

The most common use for existentials is to be able to store heterogenous types in a collection that requires a single homogenous element type. It's also common to want to write extension Collection where Element: SomeProtocol in order to do generic processing on any collection where an element conforms to a protocol. Lack of existential conformance confounds this.

For example:

// our docs admonish you not to use CustomStringConvertible
// this way, but let's ignore that for now...
extension Collection where Element: CustomStringConvertible {
  func prettyPrint() {
    let strings = self.lazy.map { $0.description }
    print( "[\(strings.joined(separator: ", "))]" )
  }
}

let someInts: [Int] = [1,2,3]
someInts.prettyPrint()
// [1, 2, 3]

let somePrintableThings: [CustomStringConvertible] = [1,"two",3]
somePrintableThings.prettyPrint()
// error: value of protocol type 'CustomStringConvertible' cannot conform 
// to 'CustomStringConvertible'; only struct/enum/class types can conform
// to protocols

// And yet you can just take the body of prettyPrint and use it...
let strings = somePrintableThings.lazy.map { $0.description }
print( "[\(strings.joined(separator: ", "))]" )
// [1, two, 3]

This to me makes the clear case that this is a hole in the language that should be fixed.

I do worry that existentials are already significantly overused, and I wish people would treat them as a last rather than first resort more often. But we shouldn't not fix their legitimate use just because of that, even if I fear that will lead to even more misuse.

19 Likes

Hmm, good point. I think you'll also have to omit the fix-it if the function returns T. Perhaps a note without fix-it would be enough?

Is such a fix-it always source-compatible with the call sites of f if Proto can be used as an existential?

I think so (as long as T is not used in ways as described above and I am not forgetting anything else!):

Just a small clarification: through a little bit of compiler magic, the capability to open an existential does existing in the shipping compiler through the _openExistential builtin.

import Foundation

protocol Attachment {
    var fileURL: URL { get }
}

struct Photo: Attachment {
    var fileURL: URL {
        URL(fileURLWithPath: "/tmp/photo.jpg")
    }
}
struct Video: Attachment {
    var fileURL: URL {
        URL(fileURLWithPath: "/tmp/video.mov")
    }
}

struct AnyAttachment: Attachment {
    var attachment: Attachment

    var fileURL: URL {
        func project<T: Attachment>(_ attachment: T) -> URL {
            attachment.fileURL
        }
        return _openExistential(attachment, do: project)
    }
}

let photo = Photo()
let anyPhoto = AnyAttachment(attachment: photo)
anyPhoto.fileURL // "file:///tmp/photo.jpg"

let video = Video()
let anyVideo = AnyAttachment(attachment: video)
anyVideo.fileURL // "file:///tmp/video.mov"
6 Likes

Thanks @harlanhaskins. I have sort of struggled with _openExistential while trying to figure this out--I read the source code, looked at stuff, tried to make it work. I want to state my understanding out loud so somebody can tell me I'm right or wrong:

  1. in the AnyAttachment struct, the attachment var is an existential because it's defined as an Attachment. So when. you make an AnyAttachment you give it something that conforms to the Attachment protocol.

  2. In the normal world, at that point you would be able to call .fileURL directly--if I replace the body of the AnyAttachment.fileURL property in the code above with just a direct call to attachment.fileURL it works the same as if I use the _openExistential code.

  3. So I'm trying to work through what's really happening there--in the direct call there is dynamic dispatch to .fileURL property. In the _openExistential code, something with an existential type goes in to _openExistential along with a closure that's generic over the existential type, and _openExistential calls the closure. The mechanics are different but I am having trouble seeing a difference in effect.

I guess I am asking, what is the difference in the typing environment inside the closure? Is there a more complicated example that would show a difference in behavior?

Thanks. I know this is off-topic from self-conformance but it's not completely off-topic, and it might be good to get the knowledge out there.

2 Likes

Y'know, my example should have actually included some code that demonstrates the utility here. The utility is being able to pass the value from the existential over to a generic function that expects <T: Attachment>, something like this:

func printFileURL<T: Attachment>(of attachment: T) {
    print(attachment.fileURL)
}

func printFileURLFromExistential(of attachment: Attachment) {
    printFileURL(of: attachment) // does not compile

    _openExistential(attachment, do: printFileURL) // does compile
}
4 Likes

Thank you! That is good.

The generic function (requiring a conformance to the protocol) printFileURL would in real life exist somewhere else and just be written to do its thing, like people do. You can't pass the existential directly as an argument to the generic function printFileURL because it's trying to make the argument (an existential) into the same type as the generic argument T, which must be a concrete type conforming to the Attachment protocol. I put the full error message into the line below. In the error message , the 'Value of protocol type' is the existential argument, it does not self-conform, and it can't conform to the protocol so it can't fulfill T.

printFileURL(of: attachment) // does not compile-- Value of protocol type 'Attachment' cannot conform to 'Attachment'; only struct/enum/class types can conform to protocols

printFileFromExistential is not generic, it just takes an existential as an argument. You could do some things with the existential in the body, but you can't call generic functions that want an argument of the type of the protocol. Which is why this is confusing to talk about.


Going back to another example higher in the thread, this almost works:

  func prettyPrint<T>(x: T) where T : Collection,
                                  T.Element : CustomStringConvertible {
    let s = x.map {$0.description}
    print( "[\(s.joined(separator: ", "))]" )
  }

  let somePrintableThings: [CustomStringConvertible] = [1, "two", 3]

/// ALMOST! 
  _openExistential(somePrintableThings,
                   do: prettyPrint) // Type of expression is ambiguous without more context

  prettyPrint(x: somePrintableThings) // Value of protocol type 'CustomStringConvertible' cannot conform to 'CustomStringConvertible'; only struct/enum/class types can conform to protocols

I think the problem is that somePrintableThings is an Array of CustomStringConvertible, not an object conforming to the CustomStringConvertible protocol. But I'm not completely sure about that. It's possible that there is some type annotation I could add somewhere that would make that work.

Thank you @harlanhaskins!

I believe the issue is that [CustomStringConvertible] is itself not an existential, but rather a generic type containing one. The following works just fine:

func prettyPrint<T: CustomStringConvertible>(_ x: T) {
  print(x.description)
}

let somePrintableThings: CustomStringConvertible = [1, "two", 3]

_openExistential(somePrintableThings, do: prettyPrint(_:))
2 Likes

One of the issues in this example is that [CustomStringConvertible] is a potentially heterogenous collection, whereas prettyPrint<T>(x: T) where T: Collection, T.Element: CustomStringConvertible requires all the elements in the collection to be the same.

3 Likes