Opaque result types

An opaque type alias would certainly make sense. We would need to somehow get both the opaque type and its binding into the syntax, e.g.,

private struct MyOpaqueCollectionImpl<T> { ... }
public typealias MyOpaqueCollection<T> = opaque Collection where _.Element == T
   = MyOpaqueCollectionImpl<T>

Having two different meanings for = is a little odd, so maybe we could use a different symbol for one of them. This would let us use MyOpaqueCollection<Something> as a type basically anywhere in our APIs. It's known by its capabilities but its underlying type is unknown.

Doug

I have a few questions about the implementation that I would like to understand: specifically how would this be SILGened/represented in SIL and how would the optimizer chew through this abstraction if the caller is in the same resilience domain? And in the caller how would this be represented in SIL... as a special form of existential box where the type is not dynamic?

One small suggestion: rather than using _ as the name of this opaque type, allow a name for the type, usable only in the where clause, if provided:

public func f<T>(...) 
  -> opaque FungibleCollection : Collection
     where FungibleCollection.Element == T {...}

or perhaps using =:

public func f<T>(...) 
  -> opaque FungibleCollection = Collection
     where FungibleCollection.Element == T {...}

which is a little more like a short-lived typealias.

8 Likes

I imagine off the top of my head that there are two codegen strategies that lead to either a specialization based approach or an approach that is more similar to how we devirtualize today (i.e. thunk + concrete impl). Specifically:

  1. Specialization: In this case we would codegen a generic function and then the optimizer would specialize/clone the code for optimization purposes. This could have interesting code-size implications.
  2. Since the type is static, we could instead take an approach similar to how we implement devirtualization and create a completely concrete implementation and a resilient thunk. In the same resilience domain, the optimizer would be able to see the body of the thunk and via inlining eliminate it. This IMO would give the smallest code-size and potentially the best performance since we would not have a generic implementation.

That being said I haven't completely thought about this and am not sure how optimizing this fits into broader ideas like generalized existentials (if it does at all).

Do these really need to be the same type, or can they be a set of types with a common supertype?

For example can I return a value with a type and another value with a type that inherits from the other type, or perhaps return an Int? in one place and an Int in the other? What if a literal is returned in one place and a type that is ExpressibleBy... that literal type is returned in another?

What's the motivation behind allowing this?

2 Likes

Hi Douglas, first of all I'd like to thank you for this really interesting pitch. I read your proposal and the comments before my post and here is my feedback. I really like the vision of opaque types presented in the proposal, especially the part which adds more flexibility to work with generics. However I have also a few concerns - most syntax-wise though.

To begin with I noticed that this feature overlaps with some parts of the existential world quite heavily, especially when there is a where clause applied. From my perspective this whole idea can be treated as a constraint on existentials in general. The only thing that I could observe in your proposal is that an opaque type is constrained to be of the same type and the dynamic type is not exposed anymore, compared to an existential. Since I'm no compiler developer, I'm in no position to judge the type system, but this is my vision as a daily Swift developer.

// Assuming that `P` and `R` are existentials with possible 
// associated types iff any of them is a protocol or a class

// The following function can return any type that conforms
// to the required protocol or is a subclass of a required
// superclass constraint
func foo() -> P
func bar() -> P & Q

// In the future we should be able to further constrain
// existentials with a `where` clause, which also does 
// not prevent us from combining them even further.
typealias MyP = P where AssociatedType == Int
typealias MyPQ = MyP & Q

// Now I see the `opaque` keyword as a possible constraint
// on existentials.
typealias OpaqueMyP = opaque MyP

// These functions can only return the same type that satisfies
// the requirement.
func baz() -> opaque P
func buz() -> OpaqueMyP
func bum() -> opaque MyPQ

Furthermore I'm not a fan of the proposed _.AssociatedType syntax nor do I like the dozens of where clauses as shown in the original post. Instead I would like you to consider a slightly different approach. David (cc @hartbit) and I were talking about the where clause on existentials, considering the previous work from Austin Zheng, and came to the conclusion that when introducing this feature it should be restricted to be declared on a typealias only. In short: If you want to create an existential that is constrained with a where clause you have use a typealias for that matter. This removes a little of the flexibility but we think that the tradeoff is worth it because:

  • it makes you reason more about your type names
  • it reduces possible ambiguity of multiple where clauses in a single delcaration
  • it forces the developer to a re-usable approach
  • it removes the need of prefix dot .AssociatedType or even the _.AssociatedType of this proposal (which is consistent to protocol MySequence : Sequence where Element == Int { ... } syntax)
  • var something: GoodTypeName is preferred over var something: A & B & C where AssociatedType == SomethingElse

In that sense I think we should force the declaration of a constrained existential and an opaque type into the typealias.

// Instead of this:
var strings: opaque MutableCollection where _.Element == String = ["hello", "world"]

// We would have this
typealias OpaqueMutableStringCollection = opaque MutableCollection where Element == String
var strings: OpaqueMutableStringCollection = ["hello", "world"]

I mentioned before that this pitch feels to me like a constraint over existentials, which raises another question: Why can we introduce opaque types with a where clause before we even have existentials with a where clause?

I have re-written a few of your examples by using a few other missing features to see how I'd prefer the syntax to look like visually and declaratively compared to the original pitch.

extension BidirectionalCollection {
  // In this example the returned type cannot be a `RandomAccessCollection`
  public func reversed() -> OpaqueBidirectionalCollection<Element> { ... }
}

// This existential would potentially eliminate the current struct in the sdlib
typealias AnyBidirectionalCollection<T> = BidirectionalCollection where Element == T

// Creating a new `Opaque*` family of types similar to `Any*` family
typealias OpaqueBidirectionalCollection<T> = opaque AnyBidirectionalCollection<T>

// Missing feature: extending existentials
extension AnyBidirectionalCollection where Element == String {
  public func joined(separator: String) -> String
}

In case of ambiguity of the where clause I had a discussion a while ago where I pitched a different keyword for constraining conditionally a type directly from the declaration.

extension BidirectionalCollection {
  public func reversed() -> OpaqueBidirectionalCollection<Element>
    constraints 
    OpaqueBidirectionalCollection : RandomAccessCollection where Self : RandomAccessCollection,
    OpaqueBidirectionalCollection : MutableCollection where Self : MutableCollection {
    return ReversedCollection<Self>(...)
  }
}

// If there are no other generic parameters to constrain,
//  we could omit the reference to the type and just write
extension BidirectionalCollection {
  public func reversed() -> OpaqueBidirectionalCollection<Element>
    constraints 
    RandomAccessCollection where Self : RandomAccessCollection,
    MutableCollection where Self : MutableCollection {
    return ReversedCollection<Self>(...)
  }
}

One thing that we have to consider is that we lose the ability to conditionally conform the currently exposed types or the returned opaque type to custom protocols. There is no way to inject any further constraints into the above reverse function. It would be a major breaking change and removing some of the current flexibilities until we can extend opaque types ourselves.

extension ReversedCollection : MyProtocol where C : MyProtocol { ... }

// This is a required alternative that must exist!
extension opaque BidirectionalCollection : MyProtocol where C : MyProtocol { ... }

How about something like this?

extension Int : P {}
struct T : R {
  // Uses the same extra keyword `constraints` to avoid ambiguity 
  // with possible `where` clause
  typealias OpaqueP = opaque P constraints OpaqueP == Int
  func someValue() -> OpaqueP { ... }
  func someOtherValue() -> OpaqueP { ... }
}

To sum up I would like to see the evolution in the following order:

  1. where clause for existentials (and opaque types later) allowed only on a typealias
  2. adding opaque types as a constraint over existentials
  3. adding a constraints keyword (or similar) to not create confusion or ambiguity with the where clause
  4. allow extending existentials and opaque types

There are still some questions left:

  • Why is is it not possible to have opaque constants? A result of an query function that returns an opaque type does not need to be always mutable, what do I miss here?

  • Can we combine multiple opaque types?

typealias OpaqueP = opaque P
typealias OpaqueQ = opaque Q

// Is this possible?
typealias OpaquePQ = OpaqueP & OpaqueQ // means `opaque P & Q`
4 Likes

This is tremendous!!!

The Opaque result types vs. existentials paragraph looks like it is the most important point of the pitch.

If only it were possible to drop the opaque keyword, and with it the distinction between opaque types and existentials, the language would make a major step forward. But I expect that you have all thought pretty hard about it. Maybe opaque types are the best compromise so far, in that they make it possible to implement a much desired feature, while avoiding existential dragons. (Edit: I wrote this paragraph because my second gut reaction (after enthusiasm) was that opaque types look like an implementation detail that leaks into userland).

All my trust and congratulations to @Douglas_Gregor :heart:

I hope we'll have time to ship it in Swift 5, considering the effect on ABI stability :crossed_fingers:!!!

2 Likes

I hope I didn't totally missed the point below.

If it is true that changing from () -> P to () -> opaque P has no impact (API-wise) on clients, then can we say that opaque types are both:

  • an optimization opportunity for demanding libraries such as a stdlib who care about ABI and performance,
  • a partial implementation strategy for generalized existentials?

If so, opaque types would become a concern for libraries, much less for their clients. We could de-emphasize them, as below:

In Swift 5:

protocol P {}

// implemented as today, with an existential
func f() -> P

// NEW: implemented with opaque types, with all their limitations
// such as all return statements must use the same concrete type.
func f() -> Collection where ...

// NEW: long-term commitment to opaque types ABI and optimization opportunities
func f() -> @opaque P
func f() -> @opaque Collection where ...

And when eventually support for existentials improves, we'll get, in Swift 6:

// same: implemented with an existential
func f() -> P 

// NEW (ABI-breaking, API-compatible)
// implemented with existentials, with limitations of opaque types lifted
func f() -> Collection where ...

// same: long-term commitment to opaque types ABI and optimization opportunities
func f() -> @opaque P 
func f() -> @opaque Collection where ...

?

Do I understand this correct? Basically you want a theoretical existential type with a where clause but without the opaque keyword to represent an opaque type? Then when existential support is improved in the compiler the same type (with a where clause and no opaque keyword) will become a true constrained existential that can return any type instead of always the same concrete type.

typealias StringCollection = Collection where Element == String 
typealias OpaqueStringCollection = opaque Collection where Element == String // or just `opaque StringCollection`

In Swift 5 both would be the same, but in Swift 6 the limitations of an existential being an opaque type would be removed.

I'm not sure if this will work out well. Do existentials supposed to work with generics like the proposed opaque types? If not then if we allow the above in Swift 5 then there is a potential risk that people will start using existentials in generics as opaque types which would make it impossible to lift that restriction in Swift 6. Please correct me if I'm wrong, I'm no expert on this topic.


Edit: I can answer my question myself. No existentials won't work with generics like opaque types.

Here is the snippet from above that proves it.

var c = makeMeACollection(Int.self)
c.append(17)         // okay: it's a RangeReplaceableCollection with Element == Int
c[c.startIndex] = 42 // okay: it's a MutableCollection with Element == Int
print(c.reversed())  // okay: all Collection/Sequence operations are available

func foo<C: Collection>(_ : C) { }
foo(c)               // okay: unlike existentials, opaque types work with generics

This is exactly that. Based on my limited understanding of the subject, of course, and the limited time I spent on a topic I have just discovered like one hour ago. Deep apologies to compiler experts if I'm totally out of bounds.

I'm not sure if this will work out well.

Me neither. I just feel that opaque types are a fantastic tool, but I'm wondering if it is a good idea to throw them at all users if opaque T is a subtype of T, and we can use covariance (not at the ABI level, ok, but at least at the API level).

EDIT: again some apologies for using the "subtype" and "covariance" words, maybe in a wrong way, because I'm in the learning phase of those powerful concepts.

Well then I get your point. ;) As I said above, this topic is really interesting to explore no matter how much expertise each one of us have on that area.

All right. Thanks for spotting where the subtyping relation breaks in the current state of the pitch :-)

Cool, this is exactly what I had in mind. How much additional implementation work would be involved in adding something like this? It would lift what could possibly end up being a frustrating limitation.

@Douglas_Gregor This is an exciting proposal! I'm happy to see more work being done to address leaking of implementation details.

You allude to it in the Source compatibility section, but I hope that the Swift team will seriously consider allowing source-breaking changes that might fall out of this proposal in order to stabilize APIs in the standard library or synthesized code.

One example that immediately comes to mind is CaseIterable:

protocol CaseIterable {
  associatedtype AllCases: Collection where Self.AllCases.Element == Self
  static var allCases: AllCases { get }
}

When the compiler synthesizes the implementation of allCases in a conforming type, today it must synthesize a signature with a concrete type (at the current time, [Self]). The major drawback here is that if someone ever opts-in to the compiler-synthesized implementation of allCases, then they can never replace it with a different type without it being a potentially source-breaking change if a caller refers to that array type specifically.

If we update the compiler to synthesize this instead:

enum SomeEnum: CaseIterable {
  static var allCases: opaque Collection where _.Element == Self { ... }
}

Then that will be a source-breaking change in the short-term, but it eliminates a major restriction in the ability of the type author to improve it later on.

Since there are probably many places in the standard library that would benefit from this, I hope that we won't prevent those changes from being made for the long-term health of those APIs.

14 Likes

On the surface, opaque types are quite similar to existential types: in each case, the specific concrete type is unknown to the static type system, and can be manipulated only through the stated capabilities (e.g., protocol and superclass constraints). The primary difference is that the concrete type behind an opaque type is constant at run-time, while an existential's type can change.

My initial feedback is that the keyword opaque does not really convey the "constant" nature of an opaque type. As that is a key difference between opaque types and existentials it would be better if syntax could be found that communicates that more clearly.

2 Likes

The storage can be writable. The underlying concrete type will still be determined by the getter's return statements, and will be used as the type of the value provided to the setter. I added some examples to the proposal here.

Yes, see my reply to Matthew Johnson.

Fixed in the document, thanks!

No, this is called out in restrictions on opaque result types.

Doug

They won't use existential boxes because the type metadata and associated conformances are fixed (not dynamic), and reachable via accessor functions. The opaque result type will be represented by an ArchetypeType in the type system. Type metadata for the ArchetypeType can be retrieved by calling an accessor; similarly for any protocol conformance requirement listed in the opaque result type (e.g., there will be an accessor to call to get the conformance of the ArchetypeType to the protocol P for opaque P).

When the opaque result types is non-resilient or we are in the same resilience domain, we don't need to call the accessors to get the type metadata or protocol conformances, because we know the underlying concrete type.

Doug

We could generalize this rule to be the common super type of all of the return expressions, but doing so could very easily put us in a place where we need to type-check all of the expressions together. For example, you mentioned literal types:

func foo() -> opaque Numeric {
  if Bool.random() {
    return UInt(5)
  }

  return 1 + 2
}

As written, the proposal would reject this code because UInt != Int. If we took a "common type" rule (here it's not even a super type), we would type-check the second return statement as producing an Int because that's the only consistent solution.

I'm inclined to keep the simpler-to-implement rule in place, and then evaluate the "common super type" rule once the other pieces are in place and we have some usage experience.

It falls out of the model and I see no reason to restrict it.

Doug

1 Like

Yes, that's a reasonable way to think about this feature.

Huh, interesting! My main complaint with this approach is that it always forces you to write out the full type name, which can get ugly. Here's a silly little example:

func foo<C: Collection>(_ c: C) -> opaque Collection where _.Element == String {
  return c.lazy.map { String(describing: $0) }.filter { Bool.random() }
}

The opaque result type lets me avoid having to write out that ugly type. I should add this motivation to the proposal!

I also don't like the idea that I have to come up with a name for a opaque type, which is often going to be an UpperCamelCased version of the oneFunctionThatReturnsThatValue. I think it causes a different kind of API surface area expansion, and makes me wonder whether it's really any better than defining a public, resilient struct that wraps the returned type.

From the design perspective, I think opaque result types is a simpler feature. We don't have to deal with the type-identity issues that plague the design of generalized existentials, e.g., the problem of how one can refer to the associated types of a generalized existential, "open" a generalized existential to give a name to the dynamic type it holds, etc.

I also expect that the demand for generalized existentials will be reduced by opaque result types, because opaque result types subsume the use cases for generalized existentials that involve hiding the result type of the operation. And they do so in a manner that works far better with the generics system, because generalized existentials still don't address the issue that an existential doesn't conform to its own protocol.

opaque constants are fine; there's an example now.

I think that should be an error. Opaque types are a stand-in for a concrete-but-unknown type; composing them should mean that you're composing the concrete types, which doesn't make sense.

Doug

Well in that particular example they're not finalized since there is no default concrete type for any of the opaque type nor is there any return nearby (assuming P and Q are not value types, nor a final class). Isn't that reasonable enough to allow their composition like with existentials in general?


Let's assume we have a type that looks like this:

struct MyStruct {
  ...
  var collection: opaque Collection where Element == String { ... }
  ...
}

How well will that play with key-paths?

// Is this valid?
let keyPath: KeyPath<MyStruct, opaque Collection where Element == String> = \MyStruct.collection

In the last situation I'd rather prefer a typealias solution to make life easier:

struct MyStruct {
  ...
  typealias OpaqueStringCollection = opaque Collection where Element == String
  var collection: OpaqueStringCollection { ... }
  ...
}

let keyPath: KeyPath<MyStruct, OpaqueStringCollection> = \MyStruct.collection