Lifting the "Self or associated type" constraint on existentials

Given what Slava said above, I'm guessing you wouldn't be able to call param.a or param.b.

You would be able to create arrays of P or Q though, and pass those elements in to functions (must be non-generic due to protocol existentials not self-conforming) or use them directly when their superclass constraints allow it.

A more complete solution would also lift that self-conformance limitation. I wonder if we could...

2 Likes

+1 for
var array: [P]

This will allow [Equatable] and create some more confusion, but hey I‘m still +1 for this :slight_smile:

1 Like

It's awkward, but you can in fact "open" the existential by invoking a protocol extension method, which will give you Self as the dynamic type of the value inside the method, which you can then pass to generic functions:

func foo<T: P>(_: T)

extension P {
  func _forwardToFoo() { foo(self) }
}

func callFooOnEach(_ ps: [P]) {
  for p in ps {
    p._forwardToFoo()
  }
}

Another nice stepping stone on the way to generalized existentials (which I'm not proposing immediately) would be to make it so that existentials automatically open when passed as generic arguments to functions. Oftentimes this is closer to what you want than protocol self-conformance, since you really want to operate on the value inside the existential, not the existential itself.

30 Likes

Wait :scream: you can pass Self to generic context today? :exploding_head: That‘s interesting to know.

This is a pretty awesome trick I have used a few times. Unfortunately, because Swift does not have higher-ranked types we cannot pass a generic function to call. The good news is that we don't have to hard code the function that gets called (foo in the above example). There are many variations on this trick and a very useful one to know is how to use a "receiver" protocol to do double dispatch:

protocol _ReceiveP {
    func _receiveP<T: P>(_: T)
}

extension P {
    func _openP<Receiver: _ReceiveP>(for receiver: Receiver) {
        receiver._receiveP(self) 
    }
}

extension Bar {
    func doSomethingWithPs(_ ps: [P]) {
        for p in ps {
            p._openP(for: self)
        }
    }
  
    func _receiveP<T: P>(_ value: T) {
        // do somehing with value
    }
}

I hope you can imagine several variations on this theme. For example, you can pass dynamic context through the dispatch, you can have a receive method that returns a value which is directly returned by the open method, etc.

15 Likes

I love this. Thank you and Joe for sharing this, I coudln‘t come up with this

Just for my understanding, do you mean by ‚higher ranked types‘ ‚higher kinded typed‘?

Does this mean:

protocol Foo: Hashable { 
   ...
}

func something(with foo: Foo) { // <- This is OK ?
   return Set<Foo>([foo]) // but this will fail because the Hashable/Equatable members won't be accessible on the Foo existential?
}
1 Like

That's correct. By itself, this wouldn't address the issue of existentials not being Equatable/Hashable.

2 Likes

Hmm, lifting this constraint will be fantastic, but I think it would need to be done in conjunction with the implementation of a solid slate of diagnostics to educate users.

For example, users who really mean to write func f<T : P>(_: T) are often steered away from func f(_: P) by the impossibility of using an existential P, but without that limitation I can foresee users writing themselves into various tricky situations much further down the line.

6 Likes

So the idea here is that you could use any protocol as a type, and when working with an instance of a protocol you could access any member that doesn't include Self or another associated type in its declaration? That is, with the Collection protocol we all know and love, you could write something like:

func hasAtLeastTwoElements(_ c: Collection) -> Bool {
    if c.isEmpty { return false }
    return c.count >= 2
}

But using methods like contains(where:), or subscripts, or for-in loops would still be disallowed, since they use the Element, Index, and Iterator associated types. Likewise, I could create a custom Equatable-inheriting protocol for a group of custom data types so that I could store them all in a single array, but I wouldn't be able to use == with the array's members or use Equatable-constrained array methods.

It feels a bit like this will make it harder to diagnose and help people fix problems in the right place, since they'll get an error message inside a function they're writing that they'll need to fix in the function's signature. Are there benefits of relaxing this constraint that would apply before we get all the way to generalized existentials?

4 Likes

The benefits I see are:

  • It defines away some otherwise straightforward library evolution questions with protocols. Removing this constraint means that adding defaulted method or associated requirements to a protocol is always OK, since it can't interfere with existing uses of the protocol.
  • Even if it doesn't define away the need for manual type-erased containers, it allows for them to be written in a way that's easier for the compiler to optimize, and ABI-compatible with future generalized existentials, by wrapping the unconstrained existential type instead of boxing a value in a subclass or closure.
  • Some protocols may have associated types or contravariant requirements, but still have a useful subset of functionality that doesn't rely on those requirements.
  • Although you can't access the requirements on the existential directly, it is possible through the convolution of writing protocol extension methods to open the value inside and have full access to the protocol interface inside the protocol extension, and thereby wrap needed functionality in an existential-friend way.
15 Likes

You will be able to call methods that return Self, because those are erased to the existential type. This works today:

protocol Cloneable {
  func clone() -> Self
}

func foo(_ c: Cloneable) -> Cloneable {
  return c.clone()
}

Similarly, we could allow you to call methods that return associated types and erase them to their upper bounds.

9 Likes

I'm definitely +1000 for taking steps in this direction...

Could you give an example of what this would look like? I have a bunch of type-erased types, and it would be nice to know the correct method for doing it once this change is made.

Is this predictable enough that the compiler could create some sort of thunk for doing it? What are the design/implementation obstacles that keep us from having a clean syntax to open things?

4 Likes

+ all the 1s

I agree that we'll need good diagnostics, but I think that lifting this constraint will actually be a good thing to help more people to understand what's going on. You hit the existing "self or associated type" error earlier, but I see a lot of questions about what it means and why this limitation exists on various forums.

Having a decent error message at point of use of a protocol method that involves a contravariant Self or an associated type can explain much better what the actual problem is.

So big +1 to this proposal.

3 Likes

Will this change be possible within the Swift 5 timeframe?

Would it be difficult to support this without fully implementing Generalized Existentials? Having to write type erased boxes is a bit of a hassle.

1 Like

Well, this is all predictable enough that the language and compiler ought to do it for you, in the fullness of time. You can wrap a more specific type-erasing container around the unconstrained existential by doing something like this:

protocol Foo {
  associatedtype Bar

  func foo(_: Bar) -> Bar
}

private extension Foo {
  // Forward to the foo method in an existential-accessible
  // way, asserting that the _Bar generic argument matches the
  // actual Bar associated type of the dynamic value
  func _fooThunk<_Bar>(_ bar: _Bar) -> _Bar {
    assert(_Bar.self == Bar.self)
    let result = foo(unsafeBitCast(bar, to: Bar.self))
    return unsafeBitCast(result, to: _Bar.self)
  }
}

struct AnyFoo<Bar>: Foo {
  private var _value: Foo

  init<F: Foo>(_ value: F) where F.Bar == Bar {
    self._value = value
  }
  
  func foo(_ bar: Bar) -> Bar {
    return self._value._fooThunk(bar)
  }
}

The Bar == _Bar assertion and unsafeBitCasts should optimize away, and in the future when we support Foo<.Bar == T> as a first-class existential type, it should have a compatible ABI with the unconstrained Foo existential, as well as this struct wrapper, since one-field structs have the same ABI as their one field. Because of the need for unsafe type casts, though, this is probably code you'd want a code generator to produce for you instead of writing out by hand.

The "open existential" operation exists in SIL as a general-purpose operation; however, our optimizers and code generation are not necessarily set up to handle the operation in full generality outside of the fixed patterns we emit as part of method calls. There's also no surface-level language design for expressing a generalized open existential operations; @Douglas_Gregor had a strawman syntax sketched out in his generics manifesto, but I hope we can come up with something a little bit more polished in the end (though perfect is the eternal enemy of good.)

For the brave, there is in fact a general _openExistential primitive hidden in the standard library you can play with, though I don't recommend using it in your code, since it may break or miscompile, and like all underscored things in the standard library, is subject to change without notice.

9 Likes

I support lifting this restriction, but note that there are implementation reasons why lifting it might not be trivial. I've noticed a fair amount of code at the SIL level that seems to assume that an opened archetype is always the primary archetype, and IIRC opened archetypes still don't carry a generic environment, which will prevent IRGen from being able to follow requirement paths from the root metadata+conformance for an opened type.

2 Likes