Opaque result types

[Douglas_Gregor]

@anandabits

The wrapper struct forces you to box and unbox values every time you cross an API boundary. That has a syntactic cost and could also have a runtime cost if the opaque type is used as part of a collection or function type.

There's some syntactic weight from the wrapper struct, sure, but I wouldn't expect a resilient struct to be all that different from a resilient opaque result type from the performance standpoint. The former introduces a distinct type at runtime, but it still has the same kind of metadata accessors.

Doug

[anandabits (Matthew Johnson)]

I wouldn't expect the overhead from the type itself to be that much different, but the mapping between concrete types that have more structure than just T could. That's what I tried to demonstrate with the array example more than the syntactic overhead.

[michelf (Michel Fortin)]

@Douglas_Gregor

func foo<C: Collection>(_ c: C) -> opaque Collection where _.Element == String {
  return c.lazy.map { String(describing: $0) }.filter { Bool.random() }
}

The opaque result type lets me avoid having to write out that ugly type. I should add this motivation to the proposal!

Which is very nice indeed. But later you write an overload of the same function:

func foo<C: Collection>(_ c: C, of unlikeliness: Int) -> opaque Collection where _.Element == String {
  return c.lazy.map { String(describing: $0) }.filter { Int.random(in: 0 ..< unlikeliness) == 0 }
}

And the result isn't interchangeable for the silly reason that two functions create two opaque types that are distinct even if the underlying concrete type is the same. The client cannot write this:

let result = condition ? foo([1,2,3]) : foo([4,5,6], unlikeliness: 5)

A typealias for the return type would allow this. But then can you migrate the return type from the anonymous opaque type in the old version of a library to a typealias in the new version without breaking the ABI?

[John_McCall]

I share your concern that opaque typealiases will end up being NameOfMethodCollection<T,U,V> and will force the type to be written out.

On the other hand:

• These opaque types are pretty wordy as it is, at least for collections, which seem like they're the dominant use case. Now, if we had generalized existentials, you'd be able to take advantage of standard typealiases and just write opaque AnyCollection<Element>; but we don't, so you need this where clause instead, which is really bloated-feeling.
• If there's a separate typealias declaration, there's also an obvious syntax for declaring the opaque type's conditional conformances. In practice I think our opaque return types are going to want, like, a million conditional conformances.

[Step (Step C)]

fixed? hidden? Not clear there is anything better.

[hartbit (David Hart)]

@John_McCall

These opaque types are pretty wordy as it is, at least for collections, which seem like they're the dominant use case. Now, if we had generalized existentials, you'd be able to take advantage of standard typealias es and just write opaque AnyCollection<Element> ; but we don't, so you need this where clause instead, which is really bloated-feeling.

Add an extra reason: it could greatly improve error messages which would use the typealias name instead of "the result type of makeMeACollection".

[Michael_Ilseman]

This looks really neat!

@Douglas_Gregor

The primary difference is that the concrete type behind an opaque type is constant at run-time, while an existential's type can change.

What are the library evolution implications here?

E.g., the actual type can change in a new library version without causing an ABI break, just like un-annotated structs.

• public func foo() -> opaque Collection can change the return type version-to-version without breaking binary compatibility.

However, does adding @inlinable force the concrete return type to be ABI? I.e. is it possible to have the choice of keeping the return type opaque at compile time for inlinable functions, leaving the following options:

1. @inlinable public func foo() -> opaque Collection
2. @inlinable public func bar() -> @fixedLayout opaque Collection

But, I don't see how it's possible to support #1 and enforce same-type at runtime, unless inits/calls are automagically virtualized requiring any new version's return type to also supports those inits/calls.

This seems like quite a bit of complexity, and I suppose a library author can always declare a resilient struct type to return for this purpose. However, the same-type runtime constraint would mean that @inlinable requires both semantic equivalence in new versions and the same return type, which may not be obvious.

Have you considered ways to forward an opaque return type from a helper function call, or otherwise constrain multiple function's opaque return types to be the same at runtime? Fine-tuning a library's ABI often involves hand-outlining just the right parts, and without this the library author reverts to concrete return types for all outlined helper functions. This isn't a big deal, just wondering if this was considered.

[Paul_Cantrell]

@Douglas_Gregor

From the design perspective, I think opaque result types is a simpler feature. We don't have to deal with the type-identity issues that plague the design of generalized existentials, e.g., the problem of how one can refer to the associated types of a generalized existential, "open" a generalized existential to give a name to the dynamic type it holds, etc.

This answers some of my immediate questions. My first reaction was that it’s putting the cart before the horse to create a specially constrained version of generalized existentials for optimization purposes before generalized existentials exist in their more general form. The rationale makes a bit more sense now.

I do wonder about the opaque keyword, and the mental model it implies viewed from the far (hopefully not too far) future perspective where we have generalized existentials, and this might look less like a separate feature and more like a special constraint.

@Douglas_Gregor

I also expect that the demand for generalized existentials will be reduced by opaque result types, because opaque result types subsume the use cases for generalized existentials that involve hiding the result type of the operation.

My own use cases for generalized existentials have all been either:

1. heterogenous collections (i.e. some Foo<T> grouped as [Foo] where the T varies between elements), or
2. less commonly, methods that need to accept an input whose type params are unknown to the caller (i.e. process(foo: Foo); usually comes up because of #1).

This proposal doesn’t address either of those concerns, correct?

[Joe_Groff]

@Paul_Cantrell

I do wonder about the opaque keyword, and the mental model it implies viewed from the far (hopefully not too far) future perspective where we have generalized existentials, and this might look less like a separate feature and more like a special constraint.

If it weren't too late to break source compatibility, I would wager that opaque concrete types are the more commonly useful feature than existentials, and in hindsight, maybe we should have saved the protocol-in-type-position syntax for an opaque type rather than using that syntax for existentials, and decorated existentials in some way instead. Rust is making that pivot even despite having source compatibility.

[IanPartridge]

This occurred to me too. Then maybe an existential could be any Foo which reads better to me.

[dabrahams (Dave Abrahams)]

As @Douglas_Gregor already knows, I'm a big fan of this feature: LGTM!

A few minor points:

I'm a little concerned about overloading “_” in the way proposed; everywhere else it means a kind of nothingness (no label, no pattern constraint, no name) but here it's being used to refer to a specific thing. It's hard to see that on a continuum with its other uses.

I'm also concerned about the arguable overloading of “->” and lack of symmetry in this example:

@Douglas_Gregor

extension BidirectionalCollection {
  public func reversed() -> opaque BidirectionalCollection
      where _.Element == Element
      where Self: RandomAccessCollection -> _: RandomAccessCollection
      where Self: MutableCollection -> _: MutableCollection {
    return ReversedCollection<Self>(...)
  }
}

To me something like this would be more logical:

extension BidirectionalCollection {
  public func reversed()
      -> opaque BidirectionalCollection where _.Element == Element,
      -> opaque RandomAccessCollection where Self : RandomAccessCollection,
      -> opaque MutableCollection where Self : MutableCollection 
  {
    return ReversedCollection<Self>(...)
  }
}

Restrictions on opaque result types

Opaque result types can only be used within the result type of a function, the type of a variable, or the element type of a subscript. However, they can occur within some part of the structure of that type.

I know what you mean, but I don't think subscript has an “element type” does it? Can we just say “return type” since a subscript always has a getter? The next sentence doesn't sound like a “however” to me; it's just a restatement, so “however” is confusing.

Contextual named types (e.g., .Element ) can only be used within the where clause of an opaque result type

It's not clear what you mean here. If I read it literally it seems to be saying that this proposal introduces contextual named types to the language by allowing them in opaque result type constraints. If so, IMO you should say so more directly. If not, maybe you could clarify.

For example, say we want to return two opaque types that conform to Collection but whose element types are the same:

func g() -> (opaque Collection where _.Element: Equatable, opaque Collection) 
    where /* Element types of both opaque types are equivalent? */ {
  // ...
}

The _.Element syntax that works nicely for saying that the Element type of the first Collection is Equatable doesn't allow us to relate that Element type to the second Collection element type. We would need to invent more syntax, e.g., where _.0.Element == _.1.Element .

Is it possible to work around that with something like this as long as you were willing to return a nominal type instead of a tuple?

protocol SameElement {
    associatedtype C0 : Collection
    associatedtype C1 : Collection where C1.Element == C0.Element
    var c0: C0 { get }
    var c1: C1 { get }
}
func g() -> opaque SameElement { }

[beccadax (Becca Royal-Gordon)]

I wonder if we might be better off thinking of this feature as an annotation on a type declaration, not on a return type. That is, maybe we want LazyCompactMapCollection to be opaque in all public uses, not to have its use in compactMap(_:) be opaque.

[torquato]

Excuse me for thinking aloud, if I may…

The proposal introduces a new keyword. It has often been mentioned that scattering the language up with too many keywords is not a good idea. What if we introduce something like a placeholder type, which from the outside would more look like a 'concrete' type.

func foo() -> opaque Collection where _.Element: Hashable {}

could become something like

func foo() -> Opaque<Collection where _.Element: Hashable> {}

I think this would also immediately solve the question raised upthread whether opaque P & opaque Q could be allowed. No.

Thinking further. Would Opaque<> or Qpaque<_> then be allowed? Basically it would mean nothing more than Any, any type without any constraints. If so, why not call the opaque placeholder type Any in the first place?

The above could become

func foo() -> Any<Collection where _.Element: Hashable> {}

There was concern about using the underscore to refering to the type itself, instead of the omission of something. This probably could be solved with what we're doing elswhere in a generic context:

func foo() -> Any<T: Collection where T.Element: Hashable> {}

or

func foo() -> Any<This: Collection where This.Element: Hashable> {}

Lots of room for bikeshedding left… ;-)

I could imagine that this approach could also simplify the question what type(of:) should give back.

Excuse me for an amateur thinking aloud. I hope I'm not completely off the track…

[hartbit (David Hart)]

@Douglas_Gregor

extension BidirectionalCollection {
  public func reversed() -> opaque BidirectionalCollection
      where _.Element == Element
      where Self: RandomAccessCollection -> _: RandomAccessCollection
      where Self: MutableCollection -> _: MutableCollection {
    return ReversedCollection<Self>(...)
  }
}

This syntax worries me because the where can be very confusing: the first one prefixes the list of truths about the opaque type, whereas the following ones prefix conditions.

[torquato]

@dabrahams

I'm a little concerned about overloading “_” in the way proposed; everywhere else it means a kind of nothingness (no label, no pattern constraint, no name) but here it's being used to refer to a specific thing. It's hard to see that on a continuum with its other uses.

Perhaps "$" would be a better fit. We use $0, $1 and so forth as a placeholder/ substitution for values passed in into closures. Why not reusing it as a placeholder sign in this case?

func foo() -> opaque BidirectionalCollection where _.Element == Element

would be

func foo() -> opaque BidirectionalCollection where $.Element == Element

[DevAndArtist (Adrian Meister)]

I think you forgot the zero there, should be $0.Element no?

I'd like to extend that idea a little because I kind of like it. I still think that personally I'd prefer normal where clauses, that do not require any knowledge about the parent type where the returned opaque type is declared, that it should be moved into a typealias (especially when used on constants and variables).

Accumulated some syntax forms from this thread:

// Generalized `$0` shorthand syntax
public opaque typealias ReversedCollection<T> : BidirectionalCollection where T : BidirectionalCollection

extension BidirectionalCollection {
  public func reversed() -> ReversedCollection
    where 
    $0.Element == Element,
    // The following constraints are crystal clear conditional conformances
    $0 : RandomAccessCollection where Self : RandomAccessCollection,
    $0 : MutableCollection where Self : MutableCollection {
    return ReversedCollection<Self>(...)
  }
}

// Or allowing extension on opaque types which would extend the concrete
// type hidden by the opaque type
extension opaque Collection ... {
  ...
}
extension ReversedCollection ... {
  ...
}

The generalized $ shorthand syntax could potentially solve some issues where we want to return multiple opaque types at once.

// Only the elements of the second opaque collection required to conform to `Equatable`.
// Without further constraints the elements of both collections are considered different.
func g() -> (opaque Collection, opaque Collection) where $1.Element : Equatable {
  ...
}

[hlovatt (Howard Lovatt)]

At WWDC Tim Cook said "We created Swift to make it easy — to make it so easy to learn to code that is easy as our products are to use." I think this message is totally lost with a rash of proposals that require to say the least 'extreme' annotations.

This opaque protocols proposal is unfortunately another example of this trend.

There is a much simpler alternative make protocols generic, then we wouldn't need associated types, existentials, generalised existentials, nor opaque protocols. A lot simpler.

There is also an 'elephant in the room' (for none english natives this phrase means that there is something no one is saying); competition from other languages like Kotlin and Python. These language are gaining traction on both mobile (Kotlin) and server side (both). Swift has to compete with these if it wants world-domination and they are both a lot easier to use already without adding yet more annotation.

Sorry to be downer.

[Douglas_Gregor]

@Michael_Ilseman

What are the library evolution implications here?

E.g., the actual type can change in a new library version without causing an ABI break, just like un-annotated structs.

• public func foo() -> opaque Collection can change the return type version-to-version without breaking binary compatibility.

Correct.

@Michael_Ilseman

However, does adding @inlinable force the concrete return type to be ABI? I.e. is it possible to have the choice of keeping the return type opaque at compile time for inlinable functions, leaving the following options:

1. @inlinable public func foo() -> opaque Collection
2. @inlinable public func bar() -> @fixedLayout opaque Collection

But, I don't see how it's possible to support #1 and enforce same-type at runtime, unless inits/calls are automagically virtualized requiring any new version's return type to also supports those inits/calls.

Right. This is described in the updated version of the rendered document. @inlinable exposes the function body, which includes the concrete type that underlying the opaque result type. Once you've exposed that, you can't change it resiliently.

@Michael_Ilseman

Have you considered ways to forward an opaque return type from a helper function call, or otherwise constrain multiple function's opaque return types to be the same at runtime? Fine-tuning a library's ABI often involves hand-outlining just the right parts, and without this the library author reverts to concrete return types for all outlined helper functions. This isn't a big deal, just wondering if this was considered.

It's been discussed a little bit here as an "opaque typealias". I think that would allow for fine-tuning.

Doug

[Douglas_Gregor]

@dabrahams

I'm a little concerned about overloading “ _ ” in the way proposed; everywhere else it means a kind of nothingness (no label, no pattern constraint, no name) but here it's being used to refer to a specific thing. It's hard to see that on a continuum with its other uses.

Yeah, I agree that the use of _ here has the wrong connotations. Offline, I've had the suggestion to use opaque, since it's the opaque type we're talking about.

@dabrahams

I'm also concerned about the arguable overloading of “ -> ” and lack of symmetry in this example:

Ah, interesting! I like this better than what's in the proposal now.

@dabrahams

Is it possible to work around that with something like this as long as you were willing to return a nominal type instead of a tuple?

protocol SameElement {
    associatedtype C0 : Collection
    associatedtype C1 : Collection where C1.Element == C0.Element
    var c0: C0 { get }
    var c1: C1 { get }
}
func g() -> opaque SameElement { }

Yes, this works, but if you end up having to invent a protocol for this... it doesn't seem like it's all that much better than defining a new resilient struct or two.

(I'll try to clarify the proposal w.r.t. your other points)

Doug

[masters3d (Chéyo Jiménez)]

I love this.
Some concerns:

I agree with others that this should be some form of typealias or declaration.

I think a better name for this features is something along the lines of proxy for a type.

I don’t think opaque is a good name user facing name because as far as I can tell there is a lot the compiler will know about this type.

This feature reminds me of unknown from typescript whixh is like Swift’s any.

What's new in TypeScript · microsoft/TypeScript Wiki