SE-0309: Unlock existential types for all protocols

Hi everyone. The review of SE-0309, "Unlock existentials for all protocols", begins now and runs through May 1, 2021.

Reviews are an important part of the Swift evolution process. All review feedback should be either on this forum thread or, if you would like to keep your feedback private, directly to the review manager. When emailing the review manager directly, please keep the proposal link at the top of the message.

What goes into a review?

The goal of the review process is to improve the proposal under review through constructive criticism and, eventually, determine the direction of Swift. When writing your review, here are some questions you might want to answer in your review:

  • What is your evaluation of the proposal?
  • Is the problem being addressed significant enough to warrant a change to Swift?
  • Does this proposal fit well with the feel and direction of Swift?
  • If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
  • How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

More information about the Swift evolution process is available at

https://github.com/apple/swift-evolution/blob/master/process.md

Thank you,

Joe Groff
Review Manager

57 Likes

No sleep 'till dub dub. :sleepy:

8 Likes

This proposal looks really great, it is exceptionally well written and clear.

I think the proposal is sensible, will remove much of the friction of PATs and will make the unsupported cases more explainable, this is a clear win.

I think that we should carefully consider the point in the alternatives considered section about introducing “any P” syntax. There are good reasons to do this, and coupling this with an expansion of the model makes sense. For source compatibility, we could take a policy of:

  • new cases require the use of ‘any P’ in all language modes, and ‘any P’ is supported for all existentials in all language modes
  • Swift 6 warns about existentials that lack the any marker with a fixit hint that adds it.
  • we may or may not ever make it an error for the existing cases to elide the any.

Such an approach would dovetail and build on the upcoming migration to Swift 6: the migrator should add the ‘any’s automatically for anyone moving to Swift 6 (presumably to get concurrency features).

In any case, again, I love this proposal. Beyond looking technically great, it is very well written. Thank you for driving this forward!

-Chris

45 Likes

+1

I think this is definitely the right direction.

Some points:

Why would A preclude a call to method, is it because Swift doesn't allow for general covariance, e.g. G<T> isn't a subtype of G<Any>, maybe some kind of wildcard would help us here but is out of scope for now.

If we treat a protocol-inherited associated type different for each type in a type intersection, assuming all types inherit from the same protocol, then this example didn't make sense:

class Class: P {
  typealias A = Int
}

protocol P {
  associatedtype A
}
protocol Q: P {
  func takesA(arg: A)
}

func testComposition(arg: Q & Class) {
  arg.takesA(arg: 0) // OK, 'A' is known to be 'Int'.
}

Because only Class.A is known but not Q.A, I think the compiler ought to be too clever here, don't know if this is causing trouble when incorporating the described semantic here:

Likewise, Swift may provide a way give formally distinct requirements, like P1.A and Q1.A, distinct implementations in the future

I'm also favoring the any P over P syntax as a nice dual of some P, but a hard breakage may be suboptimal, maybe slowly deprecating the old is the way to go.

I was indeed avoiding associated types in protocols because of the limitations around existentials. This will be very welcome.

Personally, I'd rather not have this policy. I don't mind migrating to any P when a fix-it appears. But I'd go for consistency (all existentials should be using the same syntax in a given version of Swift) rather than making it a temporary stylistic choice with weird restrictions (having no associated types or Self requirements). We're trying to remove this distinction about existentials, so let's not reintroduce it at the syntactic level (even if it's temporary).

7 Likes

I think the proposal addresses this well:

Though, if we were going to introduce a new syntax for existentials, we think it'd be much less confusing if we took the potentially source-breaking path and did so uniformly, deprecating the existing syntax after a late-enough language version, than to have [...] two syntaxes where one only works some of the time. We also believe that drawing a tangible line between protocols that "do" and "do not" have limited access to their API is ill-advised due to the relative nature of this phenomenon.

1 Like

+1. This will fix a big wart in Swift, I'm glad it's being tackled. I really like the notion of any P.

1 Like

Huge +1, fantastic to see this happen finally :slightly_smiling_face:

As someone with a long background with Scala and the JVM the limitations around existential have been mind boggling and annoying ever since I hopped over to Swift, an otherwise fantastic language :wink:

I read through the proposal in depth and have no real concerns about it, looks great :+1:

6 Likes

FINALLY! I have been waiting for this for a long time.

Also, I don’t know if ”any P” is part of this review or not, but also support for changing to it, and rather break source than do it halfway.

8 Likes

It‘s a great evolution of the language and I for my part would align my opinion somewhere in what @Chris_Lattner3 outlined above. I do prefer the idea of the any P convention for existential types. This is also great for the scenario where the protocal-as-type conforms to the protocol itself: extension any P: P { ... }.

2 Likes

Long waiting and too late to arrive, see you in Swift@5.5.

Very positive +1.
Definitely Swifty direction.

With this proposal, sounds like Swift would gain something like (a part of) Objective-C's flexibility while keeping Swift's typed safeness.
This would eliminate lots of boilerplates and workarounds straightforward.

Very nice!

2 Likes

Thank you for pushing this proposal forward! Aside from the any Protocol explicit syntax, I'm a bit concerned about the automatic type erasure for associated types in covariant positions:

Covariant Erasure for Associated Types

When invoking a member, Self-rooted associated types that

  • do not have a known implementation and
  • appear in covariant position within the type of the member

will be type-erased to their upper bounds as per the generic signature of the existential that is used to access the member.

In Swift, until now, type erasing has been an explicit action made by the user. An heterogeneous array results in an error instead of a warning

let array = [1, "two"]
// error@-1: Heterogeneous collection literal could only be inferred to '[Any]';
//           add explicit type annotation if this is intentional
// [fix-it]  Insert ' as [Any]'

and you need to explicitly add as [Any] in order to be able to compile your code. I would like the same treatment for associated types.

As a general rule (and under the assumption of expanding opaque types to be available also in non-return positions and as a tool for expressing unnamed generic parameters), I wouldn't distinguish between covariant and contravariant positions and would indiscriminately replace Self and associated types with opaque types of their upper bounds. Opaque types are a powerful and easier to understand concept than variance IMHO, since they can be explained as "unknown" types.

protocol P {}
protocol Q {
  associatedtype A
  associatedtype B: P

  var a: A { get set }
  var b: B { get set }
  func takesA(a: A)
  func takesB(b: B)
}

var anyQ: any Q = instanceConformingToQ
anyQ.a is some Any
anyQ.b is some P
anyQ.takesA is (some Any) -> Void // practically uncallable
anyQ.takesB is (some P) -> Void   // practically uncallable

Then, if the user's desire is to work with existential types, they must explicitly use as any Q, in line with what we do for heterogenous collections:

var anyQ: any Q = instanceConformingToQ
var b = anyQ.b as any P  // or 'var b: any P = anyQ.b'

Opaque types also egregiously handle mutability and path-dependent types and may allow some of the rejected code to compile in the future. As a future direction, if anyQ were non-mutating and since protocol existential types are value types, then it could be implicitly handled as an instance of some Q upon initialization:

let anyQ: any Q = instanceConfomingToQ  // equivalent to 'anyQ: some Q'
anyQ.a is (some Q).A    // i.e. (type of 'anyQ').A
anyQ.takesA is ((some Q).A) -> Void
anyQ.takesA(a: anyQ.a)  // valid today if anyQ is some Q

The example of RandomAccessCollection.dropLast(_:) would then become:

Proposed solution in this proposal
let collection: any RandomAccessCollection = [1, 2, 3]
// head is any RandomAccessCollection
let head = collection.dropLast()

// error since +(_: any RandomAccessCollection, _: any RandomAccessCollection) is not available
let y = collection + head ❌
Current behavior of opaque types
let collection: some RandomAccessCollection = [1, 2, 3]
// head is (some RandomAccessCollection).SubSequence
// head is (type of 'collection').SubSequence
let head = collection.dropLast()

let y = collection + head ✅
print(y) // prints [1, 2, 1, 2, 3]
With opaque type inferred on non-mutating instances
let collection: any RandomAccessCollection = [1, 2, 3]
// head is (some RandomAccessCollection).SubSequence
// head is (type of 'collection').SubSequence
let head = collection.dropLast()

// no error, collection is immutable, same behavior as above
let y = collection + head ✅

while the example of Sequence.enumerated() would become:

Proposed solution in this proposal
func printEnumerated(s: any Sequence) {
  // error: member 'enumerated' cannot be used on value of type protocol type 'any Sequence'
  // because it references 'Self' in invariant position; use a conformance constraint
  // instead. [fix-it: printEnumerated(s: any Sequence) -> printEnumerated<S: Sequence>(s: S)]
  for (index, element) in s.enumerated() { ❌
    if index < 5 { print(element) } ❌
  }
}
Current behavior of opaque types
let s: some Sequence = "Swift Evolution"
// s.enumerated() is EnumeratedSequence<some Sequence>
// s.enumerated() is EnumeratedSequence<(type of 's')>
for (index, element) in s.enumerated() { ✅
  // index is Int
  // element is (some Sequence).Element
  // element is (type of 's').Element
  if index < 5 { print(element) } ✅
}
With opaque type inferred on non-mutating instances
func printEnumerated(s: any Sequence) {
  // s is non-mutating, handled as 'some Sequence', same behavior as above
  for (index, element) in s.enumerated() { ✅
    if index < 5 { print(element) } ✅
  }
}

The any syntax

Regarding the any syntax, I agree with the proposal, in the sense that it should be implemented for all existential protocol types regardless of them being PATs or not. I think that it should be introduced as a warning along with this "unlocking" though, and not in a subsequent proposal. Users would be able to type Collection where a type is expected, their code will compile, but a warning+fixit should appear to automatically add any in front of it. As suggested by @Chris_Lattner3, it may be enforced and become an error later on.

I also think that having any Any and some Any should be avoided and in that regard I would love to see Any replaced by Type. Type as an identifier is already in a gray area, since MyType.Type cannot refer to a custom type (even though diagnostics may currently suggest otherwise):

struct MyType {
  // error@+4: Type member must not be named 'Type', since
  //           it would conflict with the 'foo.Type' expression
  // [fix-it]  If this name is unavoidable, use backticks to escape it
  //           Replace 'struct Type {}' with 'struct `Type` {}'
  struct Type {}
}
struct MyType {
  struct `Type` { static let foo = 2 }
}

let foo = MyType.Type.foo    // error: Type 'MyType.Type' has no member 'foo'
let foo = MyType.`Type`.foo  // error: Type 'MyType.Type' has no member 'foo'

I would like Type to be the empty protocol, i.e. the protocol which all types implicitly conform to, and Any become a typealias:

typealias Any = any Type

We could even introduce Unknown as a typealias for some Type. With this rename in place, Any would be what it already is, i.e. the existential type of all types and it's usage in code wouldn't change since nobody is really using it as a protocol (unless someone wrote protocol MyAny: Any {}, which is of dubious use). Therefore, the Swift 6 migrator wouldn't have to replace all the Anys.

2 Likes

Why practically uncallable?
Afaik, (some Any) -> Void is isomorph to T->Void \forall T unless the callee is able to change the type behind the opaque type by changing the parameter in a certain way.
Anyway, this doesn't constraint what gets in.

Further, as far as I'm concerned, doesn't some implies a compile time existential a la impl Trait in Rust?
This would unnecessarily constrain non-determinism, i.e. disallow changing the type behind the opaque type non-deterministically at runtime.

IMHO, I think any Value would be a better replacement for any Any since any Type would suggest passing any type as argument to function.

1 Like

Good proposal. +1

I've only read through and haven't thought about the issues very deeply but my instinct is go with the proposal as is rather than introduce new syntax (any P) as I don't really see any ambiguity or confusion that would avoid so it would seem like semantic clutter to me (although I may be missing something).

The only part I'd wish some improvement to is the error messages could potentially become more beginner friendly (although I don't have specific better wording in mind) and I certainly wouldn't want the search for that to block or delay such a major improvement.

Because some P would be an unknown concrete type conforming to P. You can pass only an instance of that concrete type in order to call the function, but you don't know what that concrete type actually is, therefore you can't pass anything. Under the hypothesis of non mutability of anyQ, its type cannot change, thus we can refer to it as of type some Q. The type signature of anyQ.takesB would then be

((type of 'anyQ').B) -> Void

If you happen to have a variable of type (type of 'anyQ').B (e.g. anyQ.b), then you can pass it as a parameter.

No, some Type implies an unknown concrete type, not an existential. Existentials would be spelled with the bare protocol name alone or eventually with any in front of it. There are already rules in place for opaque types to prevent type changes and assignments with types different than the one the variable already has.

Diagnostics also refer explicitly to the name of the variable in some simple cases

The reason why I used (type of 's') is because diagnostics already refer to path-dependent types that way:

var a: some RangeReplaceableCollection = [1]
var b = a
var c: some RangeReplaceableCollection = [1]
let ab = a + b  // valid
let ac = a + c  // error

The type of a and b is considered to be the same, while the type of c is considered as an unknown concrete type conforming to RangeReplaceableCollection unrelated to a's one.


I proposed Type since it's somewhat forbidden in some places, so we can officially "demote" it to a reserved keyword as much as Any is. The less reserved keywords there are, the better it is and Swift has a ton of reserved keywords already. I also think it reads better: p: any Type means that the type of p can be any type, but of course suggestions would be very welcome (anything reads better than any Any). Value, on the other hand, may recall the dichotomy between value- and reference-types.

1 Like

But which side is aware of the underlying type?
In case of a parameter, this is usually the (nth) caller, but not the callee opposed to opaque types in return type position.

The former is true unless inferring the type from error messages.

Unfortunately, "some" Types aren't really unknown

I don't even like some types being compared to each other, changing the underlying type in one may break linking against this library just because opaque types can be compared to their identity, but that being said, it was already decided.

The latter seems not to be true, some P and any P are both existentials, the former is a compile time existential, the latter a runtime existential.
The former is less powerful than the latter but enables for better inference leading to better performance.

Another reason against to use opaque types here is that they can't be used in argument position yet.
They have to be introduced and semantically defined in another proposal.

+1, been waiting for this for a long time

5 Likes

I'm very happy to see this moving forward finally. Improving this is definitely pushing Swift in the right direction.

One thing I'm still making my mind about is about the introduction of any P. When I read about it on the "improving the UI of generics" it made everything clear, so I like it very much. But sometimes I feel like it could make things much harder to explain, specially since I think other languages don't make this distinction and "just work". (am I right that Java/Kotlin do it?)

1 Like

Excuse me if this sounds silly, however, could I ask why any P is considered to be needed (in the future or as an alternative)? Read proposal and Improving the UI of generics but still confusing to me.

To make type declaration clear that "A protocol cannot be conform to itself"?
In that case, why some P is not sufficient?

(I'm not against the introducing the new keyword at all, but would like to understand them)

2 Likes