Opaque result types

Yeah, this is the direction in which I was thinking. Do you have any syntax in mind for doing this? Does the idea of an opaque typealias make sense as a mechanism for doing this? Perhaps that would also support "naming" an opaque type without having to involve a protocol.

I don't know much about this area, so forgive the naïve question. The proposed syntax for where clauses:

opaque Collection where _.Element == U
opaque MutableCollection & RangeReplaceableCollection where _.Element == T

suggests to me an alternative syntax here

_ where _: Collection, _.Element == U
_ where _: MutableCollection & RangeReplaceableCollection, _.Element == T

which, at first glance, aligns with the other meanings of the underscore and makes things seem more uniform to me. It might also be easier to extend to the multiple opaque type case. Did you consider this spelling and decide against it? Perhaps it's too cryptic, or ambiguous, or hard to search for when you encounter it.

Yes, types can be @usableFromInline. One could make the function @inline and the type @usableFromInline (or @_fixedContents even) to retain optimizability. If we need it, we could attribute the opaque further to let the type identity be known to the optimizer as well, e.g., public func f() -> @_fixedContents opaque P.


1 Like

That's correct, thank you! I've updated the linked document.


1 Like

It's "the opaque result type of A.foo()". It's going to be weird when it shows up in diagnostics, and I hope we can come up with a good way to describe it.

Yes, that would be fine.

Presumably Q is another protocol, in which case it would be a compile-time error because the result of foo() is not known to conform to Q.


1 Like

An opaque type alias would certainly make sense. We would need to somehow get both the opaque type and its binding into the syntax, e.g.,

private struct MyOpaqueCollectionImpl<T> { ... }
public typealias MyOpaqueCollection<T> = opaque Collection where _.Element == T
   = MyOpaqueCollectionImpl<T>

Having two different meanings for = is a little odd, so maybe we could use a different symbol for one of them. This would let us use MyOpaqueCollection<Something> as a type basically anywhere in our APIs. It's known by its capabilities but its underlying type is unknown.


I have a few questions about the implementation that I would like to understand: specifically how would this be SILGened/represented in SIL and how would the optimizer chew through this abstraction if the caller is in the same resilience domain? And in the caller how would this be represented in SIL... as a special form of existential box where the type is not dynamic?

One small suggestion: rather than using _ as the name of this opaque type, allow a name for the type, usable only in the where clause, if provided:

public func f<T>(...) 
  -> opaque FungibleCollection : Collection
     where FungibleCollection.Element == T {...}

or perhaps using =:

public func f<T>(...) 
  -> opaque FungibleCollection = Collection
     where FungibleCollection.Element == T {...}

which is a little more like a short-lived typealias.


I imagine off the top of my head that there are two codegen strategies that lead to either a specialization based approach or an approach that is more similar to how we devirtualize today (i.e. thunk + concrete impl). Specifically:

  1. Specialization: In this case we would codegen a generic function and then the optimizer would specialize/clone the code for optimization purposes. This could have interesting code-size implications.
  2. Since the type is static, we could instead take an approach similar to how we implement devirtualization and create a completely concrete implementation and a resilient thunk. In the same resilience domain, the optimizer would be able to see the body of the thunk and via inlining eliminate it. This IMO would give the smallest code-size and potentially the best performance since we would not have a generic implementation.

That being said I haven't completely thought about this and am not sure how optimizing this fits into broader ideas like generalized existentials (if it does at all).

Do these really need to be the same type, or can they be a set of types with a common supertype?

For example can I return a value with a type and another value with a type that inherits from the other type, or perhaps return an Int? in one place and an Int in the other? What if a literal is returned in one place and a type that is ExpressibleBy... that literal type is returned in another?

What's the motivation behind allowing this?


Hi Douglas, first of all I'd like to thank you for this really interesting pitch. I read your proposal and the comments before my post and here is my feedback. I really like the vision of opaque types presented in the proposal, especially the part which adds more flexibility to work with generics. However I have also a few concerns - most syntax-wise though.

To begin with I noticed that this feature overlaps with some parts of the existential world quite heavily, especially when there is a where clause applied. From my perspective this whole idea can be treated as a constraint on existentials in general. The only thing that I could observe in your proposal is that an opaque type is constrained to be of the same type and the dynamic type is not exposed anymore, compared to an existential. Since I'm no compiler developer, I'm in no position to judge the type system, but this is my vision as a daily Swift developer.

// Assuming that `P` and `R` are existentials with possible 
// associated types iff any of them is a protocol or a class

// The following function can return any type that conforms
// to the required protocol or is a subclass of a required
// superclass constraint
func foo() -> P
func bar() -> P & Q

// In the future we should be able to further constrain
// existentials with a `where` clause, which also does 
// not prevent us from combining them even further.
typealias MyP = P where AssociatedType == Int
typealias MyPQ = MyP & Q

// Now I see the `opaque` keyword as a possible constraint
// on existentials.
typealias OpaqueMyP = opaque MyP

// These functions can only return the same type that satisfies
// the requirement.
func baz() -> opaque P
func buz() -> OpaqueMyP
func bum() -> opaque MyPQ

Furthermore I'm not a fan of the proposed _.AssociatedType syntax nor do I like the dozens of where clauses as shown in the original post. Instead I would like you to consider a slightly different approach. David (cc @hartbit) and I were talking about the where clause on existentials, considering the previous work from Austin Zheng, and came to the conclusion that when introducing this feature it should be restricted to be declared on a typealias only. In short: If you want to create an existential that is constrained with a where clause you have use a typealias for that matter. This removes a little of the flexibility but we think that the tradeoff is worth it because:

  • it makes you reason more about your type names
  • it reduces possible ambiguity of multiple where clauses in a single delcaration
  • it forces the developer to a re-usable approach
  • it removes the need of prefix dot .AssociatedType or even the _.AssociatedType of this proposal (which is consistent to protocol MySequence : Sequence where Element == Int { ... } syntax)
  • var something: GoodTypeName is preferred over var something: A & B & C where AssociatedType == SomethingElse

In that sense I think we should force the declaration of a constrained existential and an opaque type into the typealias.

// Instead of this:
var strings: opaque MutableCollection where _.Element == String = ["hello", "world"]

// We would have this
typealias OpaqueMutableStringCollection = opaque MutableCollection where Element == String
var strings: OpaqueMutableStringCollection = ["hello", "world"]

I mentioned before that this pitch feels to me like a constraint over existentials, which raises another question: Why can we introduce opaque types with a where clause before we even have existentials with a where clause?

I have re-written a few of your examples by using a few other missing features to see how I'd prefer the syntax to look like visually and declaratively compared to the original pitch.

extension BidirectionalCollection {
  // In this example the returned type cannot be a `RandomAccessCollection`
  public func reversed() -> OpaqueBidirectionalCollection<Element> { ... }

// This existential would potentially eliminate the current struct in the sdlib
typealias AnyBidirectionalCollection<T> = BidirectionalCollection where Element == T

// Creating a new `Opaque*` family of types similar to `Any*` family
typealias OpaqueBidirectionalCollection<T> = opaque AnyBidirectionalCollection<T>

// Missing feature: extending existentials
extension AnyBidirectionalCollection where Element == String {
  public func joined(separator: String) -> String

In case of ambiguity of the where clause I had a discussion a while ago where I pitched a different keyword for constraining conditionally a type directly from the declaration.

extension BidirectionalCollection {
  public func reversed() -> OpaqueBidirectionalCollection<Element>
    OpaqueBidirectionalCollection : RandomAccessCollection where Self : RandomAccessCollection,
    OpaqueBidirectionalCollection : MutableCollection where Self : MutableCollection {
    return ReversedCollection<Self>(...)

// If there are no other generic parameters to constrain,
//  we could omit the reference to the type and just write
extension BidirectionalCollection {
  public func reversed() -> OpaqueBidirectionalCollection<Element>
    RandomAccessCollection where Self : RandomAccessCollection,
    MutableCollection where Self : MutableCollection {
    return ReversedCollection<Self>(...)

One thing that we have to consider is that we lose the ability to conditionally conform the currently exposed types or the returned opaque type to custom protocols. There is no way to inject any further constraints into the above reverse function. It would be a major breaking change and removing some of the current flexibilities until we can extend opaque types ourselves.

extension ReversedCollection : MyProtocol where C : MyProtocol { ... }

// This is a required alternative that must exist!
extension opaque BidirectionalCollection : MyProtocol where C : MyProtocol { ... }

How about something like this?

extension Int : P {}
struct T : R {
  // Uses the same extra keyword `constraints` to avoid ambiguity 
  // with possible `where` clause
  typealias OpaqueP = opaque P constraints OpaqueP == Int
  func someValue() -> OpaqueP { ... }
  func someOtherValue() -> OpaqueP { ... }

To sum up I would like to see the evolution in the following order:

  1. where clause for existentials (and opaque types later) allowed only on a typealias
  2. adding opaque types as a constraint over existentials
  3. adding a constraints keyword (or similar) to not create confusion or ambiguity with the where clause
  4. allow extending existentials and opaque types

There are still some questions left:

  • Why is is it not possible to have opaque constants? A result of an query function that returns an opaque type does not need to be always mutable, what do I miss here?

  • Can we combine multiple opaque types?

typealias OpaqueP = opaque P
typealias OpaqueQ = opaque Q

// Is this possible?
typealias OpaquePQ = OpaqueP & OpaqueQ // means `opaque P & Q`

This is tremendous!!!

The Opaque result types vs. existentials paragraph looks like it is the most important point of the pitch.

If only it were possible to drop the opaque keyword, and with it the distinction between opaque types and existentials, the language would make a major step forward. But I expect that you have all thought pretty hard about it. Maybe opaque types are the best compromise so far, in that they make it possible to implement a much desired feature, while avoiding existential dragons. (Edit: I wrote this paragraph because my second gut reaction (after enthusiasm) was that opaque types look like an implementation detail that leaks into userland).

All my trust and congratulations to @Douglas_Gregor :heart:

I hope we'll have time to ship it in Swift 5, considering the effect on ABI stability :crossed_fingers:!!!


I hope I didn't totally missed the point below.

If it is true that changing from () -> P to () -> opaque P has no impact (API-wise) on clients, then can we say that opaque types are both:

  • an optimization opportunity for demanding libraries such as a stdlib who care about ABI and performance,
  • a partial implementation strategy for generalized existentials?

If so, opaque types would become a concern for libraries, much less for their clients. We could de-emphasize them, as below:

In Swift 5:

protocol P {}

// implemented as today, with an existential
func f() -> P

// NEW: implemented with opaque types, with all their limitations
// such as all return statements must use the same concrete type.
func f() -> Collection where ...

// NEW: long-term commitment to opaque types ABI and optimization opportunities
func f() -> @opaque P
func f() -> @opaque Collection where ...

And when eventually support for existentials improves, we'll get, in Swift 6:

// same: implemented with an existential
func f() -> P 

// NEW (ABI-breaking, API-compatible)
// implemented with existentials, with limitations of opaque types lifted
func f() -> Collection where ...

// same: long-term commitment to opaque types ABI and optimization opportunities
func f() -> @opaque P 
func f() -> @opaque Collection where ...


Do I understand this correct? Basically you want a theoretical existential type with a where clause but without the opaque keyword to represent an opaque type? Then when existential support is improved in the compiler the same type (with a where clause and no opaque keyword) will become a true constrained existential that can return any type instead of always the same concrete type.

typealias StringCollection = Collection where Element == String 
typealias OpaqueStringCollection = opaque Collection where Element == String // or just `opaque StringCollection`

In Swift 5 both would be the same, but in Swift 6 the limitations of an existential being an opaque type would be removed.

I'm not sure if this will work out well. Do existentials supposed to work with generics like the proposed opaque types? If not then if we allow the above in Swift 5 then there is a potential risk that people will start using existentials in generics as opaque types which would make it impossible to lift that restriction in Swift 6. Please correct me if I'm wrong, I'm no expert on this topic.

Edit: I can answer my question myself. No existentials won't work with generics like opaque types.

Here is the snippet from above that proves it.

var c = makeMeACollection(Int.self)
c.append(17)         // okay: it's a RangeReplaceableCollection with Element == Int
c[c.startIndex] = 42 // okay: it's a MutableCollection with Element == Int
print(c.reversed())  // okay: all Collection/Sequence operations are available

func foo<C: Collection>(_ : C) { }
foo(c)               // okay: unlike existentials, opaque types work with generics

This is exactly that. Based on my limited understanding of the subject, of course, and the limited time I spent on a topic I have just discovered like one hour ago. Deep apologies to compiler experts if I'm totally out of bounds.

I'm not sure if this will work out well.

Me neither. I just feel that opaque types are a fantastic tool, but I'm wondering if it is a good idea to throw them at all users if opaque T is a subtype of T, and we can use covariance (not at the ABI level, ok, but at least at the API level).

EDIT: again some apologies for using the "subtype" and "covariance" words, maybe in a wrong way, because I'm in the learning phase of those powerful concepts.

Well then I get your point. ;) As I said above, this topic is really interesting to explore no matter how much expertise each one of us have on that area.

All right. Thanks for spotting where the subtyping relation breaks in the current state of the pitch :-)

Cool, this is exactly what I had in mind. How much additional implementation work would be involved in adding something like this? It would lift what could possibly end up being a frustrating limitation.

@Douglas_Gregor This is an exciting proposal! I'm happy to see more work being done to address leaking of implementation details.

You allude to it in the Source compatibility section, but I hope that the Swift team will seriously consider allowing source-breaking changes that might fall out of this proposal in order to stabilize APIs in the standard library or synthesized code.

One example that immediately comes to mind is CaseIterable:

protocol CaseIterable {
  associatedtype AllCases: Collection where Self.AllCases.Element == Self
  static var allCases: AllCases { get }

When the compiler synthesizes the implementation of allCases in a conforming type, today it must synthesize a signature with a concrete type (at the current time, [Self]). The major drawback here is that if someone ever opts-in to the compiler-synthesized implementation of allCases, then they can never replace it with a different type without it being a potentially source-breaking change if a caller refers to that array type specifically.

If we update the compiler to synthesize this instead:

enum SomeEnum: CaseIterable {
  static var allCases: opaque Collection where _.Element == Self { ... }

Then that will be a source-breaking change in the short-term, but it eliminates a major restriction in the ability of the type author to improve it later on.

Since there are probably many places in the standard library that would benefit from this, I hope that we won't prevent those changes from being made for the long-term health of those APIs.


On the surface, opaque types are quite similar to existential types: in each case, the specific concrete type is unknown to the static type system, and can be manipulated only through the stated capabilities (e.g., protocol and superclass constraints). The primary difference is that the concrete type behind an opaque type is constant at run-time, while an existential's type can change.

My initial feedback is that the keyword opaque does not really convey the "constant" nature of an opaque type. As that is a key difference between opaque types and existentials it would be better if syntax could be found that communicates that more clearly.