Opaque result types

@Douglas_Gregor This is an exciting proposal! I'm happy to see more work being done to address leaking of implementation details.

You allude to it in the Source compatibility section, but I hope that the Swift team will seriously consider allowing source-breaking changes that might fall out of this proposal in order to stabilize APIs in the standard library or synthesized code.

One example that immediately comes to mind is CaseIterable:

protocol CaseIterable {
  associatedtype AllCases: Collection where Self.AllCases.Element == Self
  static var allCases: AllCases { get }
}

When the compiler synthesizes the implementation of allCases in a conforming type, today it must synthesize a signature with a concrete type (at the current time, [Self]). The major drawback here is that if someone ever opts-in to the compiler-synthesized implementation of allCases, then they can never replace it with a different type without it being a potentially source-breaking change if a caller refers to that array type specifically.

If we update the compiler to synthesize this instead:

enum SomeEnum: CaseIterable {
  static var allCases: opaque Collection where _.Element == Self { ... }
}

Then that will be a source-breaking change in the short-term, but it eliminates a major restriction in the ability of the type author to improve it later on.

Since there are probably many places in the standard library that would benefit from this, I hope that we won't prevent those changes from being made for the long-term health of those APIs.

14 Likes

On the surface, opaque types are quite similar to existential types: in each case, the specific concrete type is unknown to the static type system, and can be manipulated only through the stated capabilities (e.g., protocol and superclass constraints). The primary difference is that the concrete type behind an opaque type is constant at run-time, while an existential's type can change.

My initial feedback is that the keyword opaque does not really convey the "constant" nature of an opaque type. As that is a key difference between opaque types and existentials it would be better if syntax could be found that communicates that more clearly.

2 Likes

The storage can be writable. The underlying concrete type will still be determined by the getter's return statements, and will be used as the type of the value provided to the setter. I added some examples to the proposal here.

Yes, see my reply to Matthew Johnson.

Fixed in the document, thanks!

No, this is called out in restrictions on opaque result types.

Doug

They won't use existential boxes because the type metadata and associated conformances are fixed (not dynamic), and reachable via accessor functions. The opaque result type will be represented by an ArchetypeType in the type system. Type metadata for the ArchetypeType can be retrieved by calling an accessor; similarly for any protocol conformance requirement listed in the opaque result type (e.g., there will be an accessor to call to get the conformance of the ArchetypeType to the protocol P for opaque P).

When the opaque result types is non-resilient or we are in the same resilience domain, we don't need to call the accessors to get the type metadata or protocol conformances, because we know the underlying concrete type.

Doug

We could generalize this rule to be the common super type of all of the return expressions, but doing so could very easily put us in a place where we need to type-check all of the expressions together. For example, you mentioned literal types:

func foo() -> opaque Numeric {
  if Bool.random() {
    return UInt(5)
  }

  return 1 + 2
}

As written, the proposal would reject this code because UInt != Int. If we took a "common type" rule (here it's not even a super type), we would type-check the second return statement as producing an Int because that's the only consistent solution.

I'm inclined to keep the simpler-to-implement rule in place, and then evaluate the "common super type" rule once the other pieces are in place and we have some usage experience.

It falls out of the model and I see no reason to restrict it.

Doug

1 Like

Yes, that's a reasonable way to think about this feature.

Huh, interesting! My main complaint with this approach is that it always forces you to write out the full type name, which can get ugly. Here's a silly little example:

func foo<C: Collection>(_ c: C) -> opaque Collection where _.Element == String {
  return c.lazy.map { String(describing: $0) }.filter { Bool.random() }
}

The opaque result type lets me avoid having to write out that ugly type. I should add this motivation to the proposal!

I also don't like the idea that I have to come up with a name for a opaque type, which is often going to be an UpperCamelCased version of the oneFunctionThatReturnsThatValue. I think it causes a different kind of API surface area expansion, and makes me wonder whether it's really any better than defining a public, resilient struct that wraps the returned type.

From the design perspective, I think opaque result types is a simpler feature. We don't have to deal with the type-identity issues that plague the design of generalized existentials, e.g., the problem of how one can refer to the associated types of a generalized existential, "open" a generalized existential to give a name to the dynamic type it holds, etc.

I also expect that the demand for generalized existentials will be reduced by opaque result types, because opaque result types subsume the use cases for generalized existentials that involve hiding the result type of the operation. And they do so in a manner that works far better with the generics system, because generalized existentials still don't address the issue that an existential doesn't conform to its own protocol.

opaque constants are fine; there's an example now.

I think that should be an error. Opaque types are a stand-in for a concrete-but-unknown type; composing them should mean that you're composing the concrete types, which doesn't make sense.

Doug

Well in that particular example they're not finalized since there is no default concrete type for any of the opaque type nor is there any return nearby (assuming P and Q are not value types, nor a final class). Isn't that reasonable enough to allow their composition like with existentials in general?


Let's assume we have a type that looks like this:

struct MyStruct {
  ...
  var collection: opaque Collection where Element == String { ... }
  ...
}

How well will that play with key-paths?

// Is this valid?
let keyPath: KeyPath<MyStruct, opaque Collection where Element == String> = \MyStruct.collection

In the last situation I'd rather prefer a typealias solution to make life easier:

struct MyStruct {
  ...
  typealias OpaqueStringCollection = opaque Collection where Element == String
  var collection: OpaqueStringCollection { ... }
  ...
}

let keyPath: KeyPath<MyStruct, OpaqueStringCollection> = \MyStruct.collection

For the "opaque typealias" case, maybe opaque belongs in the decl modifiers? That avoids the "two equal signs with different meaning" issue. The grammar could be:

opaque typealias <Name> [generic constraints on opaque type] = [concrete type]

e.g.

opaque typealias IntCollection: Collection
   where IntCollection.Element == Int = [Int]

Maybe this is a crazy idea of mine, but can this potentially remove the need of opaque keyword in some places?

The grammar you pitched does not make any sense to be reused for 'existentials', at least from my point of view. That makes it unambiguous for opaque types.

// The following 3 version cannot exist in the same scope because of the name collision

// Version 1: Implicitly `opaque` with a default concrete type
typealias IntCollection : Collection where IntCollection.Element == Int = [Int]

// Version 2: Implicitly `opaque` meant for re-usage (has no concrete type)
typealias IntCollection : Collection where IntCollection.Element == Int

// Version 3: True existential
typealias IntCollection = Collection where IntCollection.Element == Int

What does it mean for a typealias to have a default concrete type in this case?

That is needed for the example above asked by @anandabits. We need a way to say that a specific set of opaque types is the same.

// Slightly modified
protocol Proto {
  associatedtype SomeCollection : Collection
  func someValue() -> SomeCollection
  func someOtherValue() -> SomeCollection
}

struct T : Proto {
  typealias SomeCollection : Collection where SomeCollection.Element == Int = [Int]
  func someValue() -> SomeCollection { ... }
  func someOtherValue() -> SomeCollection { ... }
}

// This would be also valid
let t = T()
var value = t.someValue()
value = t.someOtherValue()

I don't think it's a significant amount of implementation work on top of the rest of this feature. Since writing the above I'm a bit less excited about this direction... an opaque typealias seems like it doesn't provide all that much benefit over defining a wrapper struct, and you still don't get to avoid writing the type.

Doug

I think the core team would be willing to introduce some source incompatibilities in the short term for, e.g., allCases, or specific standard library APIs. My main concern about the standard library is the ABI: I don't know if this feature can be implemented and adopted quickly enough to make it in for the ABI stability guideline.

I haven't done the work of auditing the standard library to find all of the places where opaque result types would make sense.

Doug

The wrapper struct forces you to box and unbox values every time you cross an API boundary. That has a syntactic cost and could also have a runtime cost if the opaque type is used as part of a collection or function type.

opaque typealias Foo = Bar

func makeFoos() -> [Foo] { 
   return functionThatReturnsLargeArrayOfBars()
}

// vs 

struct Foo {
   private let bar: Bar
}

func makeFoos() -> [Foo] { 
   let bars = functionThatReturnsLargeArrayOfBars()
   // now we have to map
   return bars.map(Foo.init)
}

The runtime cost could possibly be avoided if we're careful to wrap Bar right away internally but then the cost of the wrapper struct approach increases quite a bit and the concern of type abstraction permeates the implementation of our library.

I guess I don't understand what an opaque typealias that is not "finalized" would be. Your earlier examples don't state the concrete type underlying the opaque typealias. Where is that type provided?

Doug

There's some syntactic weight from the wrapper struct, sure, but I wouldn't expect a resilient struct to be all that different from a resilient opaque result type from the performance standpoint. The former introduces a distinct type at runtime, but it still has the same kind of metadata accessors.

Doug

I wouldn't expect the overhead from the type itself to be that much different, but the mapping between concrete types that have more structure than just T could. That's what I tried to demonstrate with the array example more than the syntactic overhead.

Which is very nice indeed. But later you write an overload of the same function:

func foo<C: Collection>(_ c: C, of unlikeliness: Int) -> opaque Collection where _.Element == String {
  return c.lazy.map { String(describing: $0) }.filter { Int.random(in: 0 ..< unlikeliness) == 0 }
}

And the result isn't interchangeable for the silly reason that two functions create two opaque types that are distinct even if the underlying concrete type is the same. The client cannot write this:

let result = condition ? foo([1,2,3]) : foo([4,5,6], unlikeliness: 5)

A typealias for the return type would allow this. But then can you migrate the return type from the anonymous opaque type in the old version of a library to a typealias in the new version without breaking the ABI?

1 Like

I share your concern that opaque typealiases will end up being NameOfMethodCollection<T,U,V> and will force the type to be written out.

On the other hand:

  • These opaque types are pretty wordy as it is, at least for collections, which seem like they're the dominant use case. Now, if we had generalized existentials, you'd be able to take advantage of standard typealiases and just write opaque AnyCollection<Element>; but we don't, so you need this where clause instead, which is really bloated-feeling.
  • If there's a separate typealias declaration, there's also an obvious syntax for declaring the opaque type's conditional conformances. In practice I think our opaque return types are going to want, like, a million conditional conformances.
8 Likes

fixed? hidden? Not clear there is anything better.