Opaque result types


(Douglas Gregor) #1

Hello all,

Here's a proposal I've been thinking about for a while (I blame Rust), but which came up again due to the review of SE-0222. Now seems like a fine time to pitch it.

EDIT: The proposal has been significantly revised since this initial posting. The most up-to-date version is always here in rendered form.

Introduction

This proposal introduces the ability to "hide" the result types of specific functions from the caller in public interfaces. Instead of providing a specific concrete type, such functions will return an unknown-but-unique type described only by its capabilities, e.g., a Collection with a specific Element type.
For example, SE-0222 introduces a new standard library type LazyCompactMapCollection for the sole purpose of describing the result type of compactMap(_:):

extension LazyMapCollection {
  public func compactMap<U>(_ transform: @escaping (Element) -> U?)
      -> LazyCompactMapCollection<Base, U> {
    return LazyCompactMapCollection<Base, U>(/*...*/)
  }
}

With this proposal, the standard library could hide the result type, exposing it as "an opaque Colection whose Element type is U:

private struct LazyCompactMapCollection<Base: Collection, Element> { ... }

extension LazyMapCollection {
  public func compactMap<U>(_ transform: @escaping (Element) -> U?)
      -> opaque Collection where _.Element == U {
    return LazyCompactMapCollection<Base, U>(/*...*/)
  }
}

The implementation of compactMap(_:) can still return an instance of LazyCompactMapCollection, but now LazyCompactMapCollection can be private: its identity is hidden from clients, and could change from one version of the library to the next without breaking those clients, because the actual type identity was never exposed. This allows us to provide potentially-more-efficient implementations without expanding the surface area of the library.

Motivation

Libraries in Swift often end up having to expose a number of generic types in their public interface to describe the result of generic operations. LazyCompactMapCollection, noted in the introduction, would be one such type in the standard library, along with existing types like EnumeratedSequence, FlattenSequence, JoinedSequence, and many others (including several Lazy variants). These generic types, which are needed to express the results of generic algorithms (enumerated, flatten, joined), are generally not of interest to users, but can significantly increase the surface area of the library.

Proposed solution

This proposal introduces an opaque type that can be used to describe an opaque result type for a function. opaque types can only be used in the result type of a function, the type of a property, or the element type of a subscript declaration. An opaque type is backed by a specific concrete type, but that type is only known to the implementation of that function/property/subscript. Everywhere else, the type is opaque, and is described by its characteristics and originating function/property/subscript. For example:

func makeMeACollection<T>(_: T.Type)
     -> opaque MutableCollection & RangeReplaceableCollection where _.Element == T {
   return [T]()   // okay: an array of T satisfies all of the requirements
}

The syntax of the "opaque" type is derived from the discussions of generalized existentials. Essentially, following the opaque keyword is a class, protocol, AnyObject, or composition thereof (joined with &), optionally followed by a where clause indicating other requirements. Within that where clause, one can use _ to refer to the opaque result type itself, so _.Element would refer to the Element associated type of the opaque type (which otherwise cannot be named).

A caller to makeMeACollection(_:) can rely on the result type having all of the requirements listed. For example:

var c = makeMeACollection(Int.self)
c.append(17)         // okay: it's a RangeReplaceableCollection with Element == Int
c[c.startIndex] = 42 // okay: it's a MutableCollection with Element == Int
print(c.reversed())  // okay: all Collection/Sequence operations are available

func foo<C: Collection>(_ : C) { }
foo(c)               // okay: unlike existentials, opaque types work with generics

Moreover, these types can be used freely with other generics, e.g., forming a collection of the results:

var cc = makeMeACollection(type(of: c))
cc.append(c)         // okay: Element == the result type of makeMeACollection
var c2 = makeMeACollection(Int.self)
cc.append(c2)        // okay: Element == the result type of makeMeACollection

An opaque result type is not considered equivalent to its underlying type by the static type system:

var intArray = [Int]()
cc.append(a)         // error: [Int] is not known equal the result type of makeMeACollection

However, as with generic type parameters, one can inspect an opaque type's underlying type at runtime. For example, a conditional cast could determine whether the result of makeMeACollection is of a particular type:

if let arr = makeMeACollection(Int.self) as? [Int] {
  print("It's an [Int], \(arr)\n")
} else {
  print("Guessed wrong")
}

In other words, opaque result types are only opaque to the static type system: they don't exist at runtime.

Implementing a function returning an opaque type

The implementation of a function returning an opaque type must return a value of the same concrete type T from each return statement, and T must meet all of the constraints stated on the opaque type. For example:

protocol P { }
extension Int : P { }
extension String : P { }

func f1() -> opaque P {
  return "opaque"
}

func f2(i: Int) -> opaque P {   // okay: both returns produce Int
  if i > 10 { return i }
  return 0;
}

func f2(flip: Bool) -> opaque P {
  if flip { return 17 }
  return "a string"       // error: different return types Int and String
}

func f3() -> opaque P {
  return 3.1419           // error: Double does not conform to P
}

func f4() -> opaque P {
  let p: P = "hello"
  return p                // error: protocol type P does not conform to P
}

func f5() -> opaque P {
  return f1()             // okay: f1() returns an opaque type that conforms to P
}

protocol Initializable { init() }

func f6<T: P & Initializable>(_: T.Type) -> opaque P {
  return T()              // okay: T will always be a concrete type conforming to P
}

These rules guarantee that there is a single concrete type produced by any call to the function. The concrete type can depend on the generic type arguments (as in the f6() example), but must be consistent across all return statements.

Note that recursive calls are allowed, and are known to produce a value of the same concrete type, but that the concrete type itself is not known:

func f7(_ i: Int) -> opaque P {
  if i == 0 {
    return f7(1)                 // okay: returning the opaque result type of f(), similar to f5()
  else if i < 0 {
    let result: Int = f7(-i)     // error: opaque result type of f() is not convertible to Int
    return result
  } else {
    return 0
  }
}

Of course, there must be at least one return statement that provides a concrete type!

Opaque result types can be used with properties. For example:

extension Collection {
  var lazy: opaque Collection where _.Element == Element { ... }
}

For computed properties, the concrete type is determined by the return statements in the getter. Opaque result types can also be used in stored properties that have an initializer, in which case the concrete type is the type of the initializer:

var strings: opaque Collection where _.Element == String = ["hello", "world"]

"Naming" the opaque result type

While one can use type inference to declare variables of the opaque result type of a function, there is no direct way to name the opaque result type:

func f1() -> opaque P { /* ... */ }

let vf1 = f1()    // type of vf1 is the opaque result type of f1()

However, the type inference used to satisfy associated type requirements can be used to give a name to the opaque result type. For example:

protocol P { }

protocol Q {
  associatedtype SomeType: P
  
  func someValue() -> SomeType
}

struct S: Q {
  func someValue() -> opaque P {
    /* ... */
  }
  
  /* infers typealias SomeType = opaque result type of S.someValue() */
}

let sv: S.SomeType     // okay: names the opaque result type of S.someValue()
sv = S().someValue()   // okay: returns the same opaque result type

Opaque result types vs. existentials

On the surface, opaque types are quite similar to existential types: in each case, the specific concrete type is unknown to the static type system, and can be manipulated only through the stated capabilities (e.g., protocol and superclass constraints). For example:

protocol P
  func foo()
}

func f() -> P { /* ... */ }
func g() -> opaque P { /* ... */ }

f().foo()   // okay
g().foo()   // okay

let pf: P = f()   // okay
let pg: P = g()   // okay

The primary difference is that the concrete type behind an opaque type is constant at run-time, while an existential's type can change. For example, assuming both Int and String conform to P, f() could return a value of a different concrete type:

func f() -> P {
  if Bool.random() {
    return 17
  } else {
    return "hello, existential"
  }
}

This means that, for example, an array populated by calls to f() could be heterogeneous:

let fArray = [f(), f(), f()]  // contains a mix of String and Int at run-time

With an opaque type, there is a single concrete type:

let gArray = [g(), g(), g()]  // homogeneous array of g()'s opaque result type

The guarantee of a single concrete type allows opaque result types to compose much better with generics. For example, an opaque result type can make use of protocols with Self requirements and use those values with generic operations like Collection.sort():

func h() -> opaque Comparable { return /* ... */ }

var hArray = [h(), h(), h()]
hArray.sort()   // okay! the Element type is Comparable, and all types are the same

Existentials do not allow such an operation, even with generalized existentials, because two values of the same existential type may have different types at runtime.

Interaction with conditional conformances

When a generic function returns an adapter type, it's not uncommon for the adapter to use conditional conformances to reflect the capabilities of its underlying type parameters. For example, consider the reversed() operation:

extension BidirectionalCollection {
  public func reversed() -> ReversedCollection<Self> {
    return ReversedCollection<Self>(...)
  }
}

ReversedCollection is always a BidirectionalCollection, with a conditional conformance to RandomAccessCollection:

public struct ReversedCollection<C: BidirectionalCollection>: BidirectionalCollection {
  /* ... */
}

extension ReversedCollection: RandomAccessCollection
    where C: RandomAccessCollection {
  /* ... */
}

What happens if we hid the ReversedCollection adapter type behind an opaque result type?

extension BidirectionalCollection {
  public func reversed() -> opaque BidirectionalCollection
      where _.Element == Element {
    return ReversedCollection<Self>(...)
  }
}

Now, clients that call reversed() on a RandomAccessCollection (like an array), would get back something that is only known to be a BidirectionalCollection: there would be no way to treat it as a RandomAccessCollection. The library could provide another overload of reversed():

extension RandomAccessCollection {
  public func reversed() -> opaque RandomAccessCollection
      where _.Element == Element {
    return ReversedCollection<Self>(...)
  }
}

However, doing so is messy, and the client would have no way to know that the type returned by the two reversed() functions are, in fact, the same.

Alternatively, we can introduce syntax to describe the conditional conformances of an opaque result type. For example, we could state that the result of reversed() is also a RandomAccessCollection when Self is a RandomAccessCollection. One possible syntax:

extension BidirectionalCollection {
  public func reversed() -> opaque BidirectionalCollection
      where _.Element == Element
      where Self: RandomAccessCollection -> _: RandomAccessCollection {      
    return ReversedCollection<Self>(...)
  }
}

Here, we add a second where claus that states the conditional requirements (Self: RandomAccessCollection) and the consequence of that conditional requirement (_, the opaque result type, conforms to RandomAccessCollection). One could have multiple conditional clauses, e.g.,

extension BidirectionalCollection {
  public func reversed() -> opaque BidirectionalCollection
      where _.Element == Element
      where Self: RandomAccessCollection -> _: RandomAccessCollection
      where Self: MutableCollection -> _: MutableCollection {
    return ReversedCollection<Self>(...)
  }
}

Here, the opaque result type conforms to MutableCollection when the Self type conforms to MutableCollection. This conditional result is independent of whether the opaque result type conforms to RandomAccessCollection.

Detailed design

Grammar of opaque result types

The grammatical productions for opaque result types aren't straightforward:

type ::= 'opaque' type where-clause[opt] opaque-conditional-requirement*
     |   '_'
     
opaque-conditional-requirement ::= where-clause '->' requirement-list

The first type production introduces the opaque type with its optional where clause and conditional requirements; the second type production introduces the contextual type _ to describe the opaque result type.

The conditional requirements is a set of where clauses, each followed by a requirement-list.

Restrictions on opaque result types

Opaque result types can only be used within the result type of a function, the type of a variable, or the element type of a subscript. However, they can occur within some part of the structure of that type. For example, one can return an optional opaque result type:

func f(flip: Bool) -> (opaque P)? {
  if flip {
    return 1       // concrete type for the opaque result type is Int
  }
  
  return nil
}

Opaque result types cannot be used in the requirements of a protocol:

protocol Q {
  func f() -> opaque P        // error: cannot use opaque result type within a protocol
}

Associated types provide a better way to model the same problem, and the requirements can then be satisfied by a function that produces an opaque result type.

Contextual named types (e.g., .Element) can only be used within the where clause of an opaque result type. Generalized existentials are likely to expand the usefulness of this syntax.

Single opaque result type per entity

As a (possibly temporary) restriction, a particular function/variable/subscript can only contain a single opaque result type, so the following is ill-formed:

func g(flip: Bool) -> (opaque P, opaque P) {  // error: two opaque result types
  return (1, "hello")
}

While it is technically possible to support multiple opaque result types in a given function/variable/subscript, the contextual named type syntax starts to break down. For example, say we want to return two opaque types that conform to Collection but whose element types are the same:

func g() -> (opaque Collection where _.Element: Equatable, opaque Collection) 
    where /* Element types of both opaque types are equivalent? */ {
  // ...
}

The _.Element syntax that works nicely for saying that the Element type of the first Collection is Equatable doesn't allow us to relate that Element type to the second Collection element type. We would need to invent more syntax, e.g., where _.0.Element == _.1.Element.

Uniqueness of opaque result types

Opaque result types are uniqued based on the function/property/subscript and any generic type arguments. For example:

func makeOpaque<T>(_: T.Type) -> opaque Any { /* ... */ }
var x = makeOpaque(Int.self)
x = makeOpaque(Double.self)  // error: "opaque" type from makeOpaque<Double> is distinct from makeOpaque<Int>

This includes any generic type arguments from outer contexts, e.g.,

extension Array where Element: Comparable {
  func opaqueSorted() -> opaque Sequence where _.Element == Element { /* ... * }
}

var x = [1, 2, 3]. opaqueSorted()
x = ["a", "b", "c"].opaqueSorted()  // error: opaque result types for [Int].opaqueSorted() and [String].opaqueSorted() differ

Ambiguity with where clauses

Opaque result types contain optional trailing where clauses, as do function and subscript declarations, leading to a parsing ambiguity. For example:

func sorted<C: RandomAccessCollection>(_: C) -> opaque RandomAccessCollection
    where C.Element: Comparable, _.Element == C.Element {
  /* ... */
}

Is the where clause part of the opaque result type or part of the function itself? The maximal munch principle implies that it should be part of the opaque result type, e.g., that this declaration is equivalent to:

func sorted<C: RandomAccessCollection>(_: C)
     -> (opaque RandomAccessCollection where C.Element: Comparable, _.Element == C.Element) {
  /* ... */
}

rather than being part of the declaration itself:

func sorted<C: RandomAccessCollection>(_: C) -> (opaque RandomAccessCollection)
    where C.Element: Comparable, _.Element == C.Element {
  /* ... */
}

Note that the last of these won't compile, because _.Element does not make sense outside of the opaque result type.

We propose to follow the maximal munch principle here, associating the where clause with the opaque result type, because it is strictly better than the alternative:

  • All constraints of the result type are implied constraints of the function itself, so stating the C.Element: Comparable constraint as part of opaque result type also makes that a constraint on the generic function as a whole.
  • The generic type parameters are already part of the uniqueness criteria for opaque result types, so there is little downside to exposing all of the known capabilities of the generic type parameters in the opaque result type, because the client already has to satisfy those constraints.

To the second point, consider:

func weirdSorted<C: RandomAccessCollection>(_: C)
  -> (opaque RandomAccessCollection where _.Element: Equatable,
                                        _.Element == C.Element)
  where C.Element: Comparable {
/* ... */
}

This could be interpreted to mean that the resulting opaque result type only guarantees that its element type is Equatable, even though the weirdSorted function only works with collections whose element types are Comparable, and the argument and result element types are equivalent. For simplicity, we state that there is no difference between the opaque result types of weirdSorted() and any of the former sorted() examples, i.e., the entire generic signature of the function is used to describe the capabilities of the opaque result type.

Implementation strategy

From an implementation standpoint, a client of a function with an opaque result type needs to treat values of that result type like any other resilient value type: its size, alignment, layout, and operations are unknown.

However, when the body of the function is known to the client (e.g., due to inlining or because the client is in the same compilation unit as the function), the compiler's optimizer will have access to the specific concrete type, eliminating the indirection cost of the opaque result type.

Source compatibility

Opaque result types are purely additive. If they were to be adopted in the standard library, that change could affect source compatibility.

Effect on ABI stability

Opaque result types are an ABI-additive feature. If they were to be adopted in the standard library, it would affect the standard library ABI. This feature can be useful for ABI stability, because it would allow the standard library to hide a large number of detail types (EnumeratedSequence, FlattenSequence, JoinedSequence, and so on) from both source code and binaries, allowing more customization in the future.

Effect on API resilience

Opaque result types are part of the result type of a function/type of a variable/element type of a subscript, which cannot be changed without affecting API resilience.

We could allow an API originally specified using an opaque result type to later evolve to specify the specific result type. The result type itself would have to become visible to clients, and this might affect source compatibility, but (mangled name aside) such a change would be resilient.

Rust's impl Trait

The proposed Swift feature is largely based on Rust's impl Trait language feature, described by Rust RFC 1522 and extended by Rust RFC 1951. There are only a small number of differences between this feature as expressed in Swift vs. Rust's impl Trait as described in RFC 1522:

  • Swift's need for a stable ABI necessitates translation of opaque result types as resilient types, which is unnecessary in Rust's model, where the concrete type can always be used for code generation.
  • Swift's opaque result types are fully opaque, because Swift doesn't have pass-through protocols like Rust's Send trait, which simplifies the type checking problem slightly.
  • Due to associated type inference, Swift already has a way to "name" an opaque result type in some cases.
  • We're not even going to mention the use of opaque in argument position, because it's a distraction for the purposes of this proposal; see Rust RFC 1951.
  • Rust didn't tackle the issue of conditional constraints.

Alternatives considered

Let implementation details continue to leak out of generic libraries.

Doug


Add an `adjacentPairs` algorithm to Sequence
Probably a compiler bug: can't convert Range<Int> to Sequence where Element == Int
SE-0222: Lazy compactMap Sequence
"Strict" inout parameters
Pre-pitch: striding through collections should be easier
Default Implementation in Protocols
Unexpectedly not conforming to protocol
(Matthew Johnson) #2

I love this pitch! It solves an important problem in a very clean way. Thank you! :clap: :clap::clap: :clap:

The pitch includes an example of a var with an opaque type. I assume it is possible to mutate the collection if we tweak the example as follows:

var strings: opaque MutableCollection where _.Element == String = ["hello", "world"]

Is that correct?

I also have a question related to "naming" the opaque type with a protocol. Again using a modified example:

protocol P { }

protocol Q {
  associatedtype SomeType: P
  
  func someValue() -> SomeType
  func  takesSomeValue(_ value: SomeType)
}

struct S: Q {
  func someValue() -> opaque P { // ... }
  func  takesSomeValue(_ value: SomeType) { // ... }
}

protocol R {
  associatedtype SomeType: P
  
  func someValue() -> SomeType
  func someOtherValue() -> SomeType
}

struct T: R {
  func someValue() -> opaque P { // ... }
  func someOtherValue() -> opaque P { // ... }
}

Are the above constructions allowed? My reading of the pitch is that they would not be but it isn't completely clear.

Finally, I am wondering if it might be possible to enhance this feature in the future to correlate opaque types returned by different functions, and if so to receive correlated opaque values as inputs. I don't have concrete use cases in mind at the moment but I immediately thought that might be a useful enhancement. I'm not sure how this would be supported syntactically, but the idea of an opaque typealias came to mind as something that could facilitate that.

Great work on this! Thank you again!


(John McCall) #3

When an opaque type is used in a storage declaration, does the storage have to be read-only? What does it mean for it not to be?

Should opaque types be allowed in typealias declarations?

You have an example where you declare intArray but don't use it.

Can a protocol requirement declare an opaque result type? If so, what does that mean?


(Douglas Gregor) #4

Yes, that's correct.

Your example here:

struct S: Q {
  func someValue() -> opaque P { // ... }
  func  takesSomeValue(_ value: SomeType) { // ... }
}

is well-formed. We would deduce the type of SomeType from someValue(), then use that as the parameter to takesSomeValue(_:).

This question is related to the one that precedes it. In your example here:

protocol R {
  associatedtype SomeType: P
  
  func someValue() -> SomeType
  func someOtherValue() -> SomeType
}

struct T: R {
  func someValue() -> opaque P { // ... }
  func someOtherValue() -> opaque P { // ... }
}

The conformance of T to R is ill-formed because the result types of someValue() and someOtherValue() are considered different, so there is no consistent type that satisfies the SomeType requirement. If there were some way to say that the two functions returned the same opaque type, we could do that here and SomeType would be inferred to that type.

Doug


(Lukas Stabe 🙃) #5

How would this affect the compilers ability to optimise code involving opaque return types? Most methods defined on or returning specialised sequence/collection types for the lazy system are marked inlinable. Is there some way to retain inlinability/zero cost abstractions for methods called on opaque types while making the types themselves invisible?

Edit: I misunderstood the implementation strategy section. So to retain the current optimisability, the lazy types would need to be @usableFromInline (can types be @usableFromInline?), but not part of the visible API?


#6

@John_McCall already mentioned this.

var intArray = [Int]()
cc.append(a)         // error: [Int] is not known equal the result type of makeMeACollection

Is a typo an should read

var intArray = [Int]()
cc.append(intArray)         // error: [Int] is not known equal the result type of makeMeACollection

right?


#7

A few amateurish questions just to clarify, if I may.

This will be legal code then?

//---- Module A
public protocol P {}
private struct Bar: P {}
public func foo() -> opaque P {
    return Bar()
}

//---- Module B
var baz = foo()

If so, what gives type(of: baz)? Would defining the type like var baz: opaque P = … be OK, but var baz: opaque P & Q (in this case) be a compile time error?

Edit: Typos


(Matthew Johnson) #8

Yeah, this is the direction in which I was thinking. Do you have any syntax in mind for doing this? Does the idea of an opaque typealias make sense as a mechanism for doing this? Perhaps that would also support "naming" an opaque type without having to involve a protocol.


#9

I don't know much about this area, so forgive the naïve question. The proposed syntax for where clauses:

opaque Collection where _.Element == U
opaque MutableCollection & RangeReplaceableCollection where _.Element == T

suggests to me an alternative syntax here

_ where _: Collection, _.Element == U
_ where _: MutableCollection & RangeReplaceableCollection, _.Element == T

which, at first glance, aligns with the other meanings of the underscore and makes things seem more uniform to me. It might also be easier to extend to the multiple opaque type case. Did you consider this spelling and decide against it? Perhaps it's too cryptic, or ambiguous, or hard to search for when you encounter it.


(Douglas Gregor) #10

Yes, types can be @usableFromInline. One could make the function @inline and the type @usableFromInline (or @_fixedContents even) to retain optimizability. If we need it, we could attribute the opaque further to let the type identity be known to the optimizer as well, e.g., public func f() -> @_fixedContents opaque P.

Doug


(Douglas Gregor) #11

That's correct, thank you! I've updated the linked document.

Doug


(Douglas Gregor) #12

It's "the opaque result type of A.foo()". It's going to be weird when it shows up in diagnostics, and I hope we can come up with a good way to describe it.

Yes, that would be fine.

Presumably Q is another protocol, in which case it would be a compile-time error because the result of foo() is not known to conform to Q.

Doug


(Douglas Gregor) #13

An opaque type alias would certainly make sense. We would need to somehow get both the opaque type and its binding into the syntax, e.g.,

private struct MyOpaqueCollectionImpl<T> { ... }
public typealias MyOpaqueCollection<T> = opaque Collection where _.Element == T
   = MyOpaqueCollectionImpl<T>

Having two different meanings for = is a little odd, so maybe we could use a different symbol for one of them. This would let us use MyOpaqueCollection<Something> as a type basically anywhere in our APIs. It's known by its capabilities but its underlying type is unknown.

Doug


(Michael Gottesman) #14

I have a few questions about the implementation that I would like to understand: specifically how would this be SILGened/represented in SIL and how would the optimizer chew through this abstraction if the caller is in the same resilience domain? And in the caller how would this be represented in SIL... as a special form of existential box where the type is not dynamic?


(Mark Lacey) #15

One small suggestion: rather than using _ as the name of this opaque type, allow a name for the type, usable only in the where clause, if provided:

public func f<T>(...) 
  -> opaque FungibleCollection : Collection
     where FungibleCollection.Element == T {...}

or perhaps using =:

public func f<T>(...) 
  -> opaque FungibleCollection = Collection
     where FungibleCollection.Element == T {...}

which is a little more like a short-lived typealias.


(Michael Gottesman) #16

I imagine off the top of my head that there are two codegen strategies that lead to either a specialization based approach or an approach that is more similar to how we devirtualize today (i.e. thunk + concrete impl). Specifically:

  1. Specialization: In this case we would codegen a generic function and then the optimizer would specialize/clone the code for optimization purposes. This could have interesting code-size implications.
  2. Since the type is static, we could instead take an approach similar to how we implement devirtualization and create a completely concrete implementation and a resilient thunk. In the same resilience domain, the optimizer would be able to see the body of the thunk and via inlining eliminate it. This IMO would give the smallest code-size and potentially the best performance since we would not have a generic implementation.

That being said I haven't completely thought about this and am not sure how optimizing this fits into broader ideas like generalized existentials (if it does at all).


(Mark Lacey) #17

Do these really need to be the same type, or can they be a set of types with a common supertype?

For example can I return a value with a type and another value with a type that inherits from the other type, or perhaps return an Int? in one place and an Int in the other? What if a literal is returned in one place and a type that is ExpressibleBy... that literal type is returned in another?

What's the motivation behind allowing this?


(Adrian Zubarev) #18

Hi Douglas, first of all I'd like to thank you for this really interesting pitch. I read your proposal and the comments before my post and here is my feedback. I really like the vision of opaque types presented in the proposal, especially the part which adds more flexibility to work with generics. However I have also a few concerns - most syntax-wise though.

To begin with I noticed that this feature overlaps with some parts of the existential world quite heavily, especially when there is a where clause applied. From my perspective this whole idea can be treated as a constraint on existentials in general. The only thing that I could observe in your proposal is that an opaque type is constrained to be of the same type and the dynamic type is not exposed anymore, compared to an existential. Since I'm no compiler developer, I'm in no position to judge the type system, but this is my vision as a daily Swift developer.

// Assuming that `P` and `R` are existentials with possible 
// associated types iff any of them is a protocol or a class

// The following function can return any type that conforms
// to the required protocol or is a subclass of a required
// superclass constraint
func foo() -> P
func bar() -> P & Q

// In the future we should be able to further constrain
// existentials with a `where` clause, which also does 
// not prevent us from combining them even further.
typealias MyP = P where AssociatedType == Int
typealias MyPQ = MyP & Q

// Now I see the `opaque` keyword as a possible constraint
// on existentials.
typealias OpaqueMyP = opaque MyP

// These functions can only return the same type that satisfies
// the requirement.
func baz() -> opaque P
func buz() -> OpaqueMyP
func bum() -> opaque MyPQ

Furthermore I'm not a fan of the proposed _.AssociatedType syntax nor do I like the dozens of where clauses as shown in the original post. Instead I would like you to consider a slightly different approach. David (cc @hartbit) and I were talking about the where clause on existentials, considering the previous work from Austin Zheng, and came to the conclusion that when introducing this feature it should be restricted to be declared on a typealias only. In short: If you want to create an existential that is constrained with a where clause you have use a typealias for that matter. This removes a little of the flexibility but we think that the tradeoff is worth it because:

  • it makes you reason more about your type names
  • it reduces possible ambiguity of multiple where clauses in a single delcaration
  • it forces the developer to a re-usable approach
  • it removes the need of prefix dot .AssociatedType or even the _.AssociatedType of this proposal (which is consistent to protocol MySequence : Sequence where Element == Int { ... } syntax)
  • var something: GoodTypeName is preferred over var something: A & B & C where AssociatedType == SomethingElse

In that sense I think we should force the declaration of a constrained existential and an opaque type into the typealias.

// Instead of this:
var strings: opaque MutableCollection where _.Element == String = ["hello", "world"]

// We would have this
typealias OpaqueMutableStringCollection = opaque MutableCollection where Element == String
var strings: OpaqueMutableStringCollection = ["hello", "world"]

I mentioned before that this pitch feels to me like a constraint over existentials, which raises another question: Why can we introduce opaque types with a where clause before we even have existentials with a where clause?

I have re-written a few of your examples by using a few other missing features to see how I'd prefer the syntax to look like visually and declaratively compared to the original pitch.

extension BidirectionalCollection {
  // In this example the returned type cannot be a `RandomAccessCollection`
  public func reversed() -> OpaqueBidirectionalCollection<Element> { ... }
}

// This existential would potentially eliminate the current struct in the sdlib
typealias AnyBidirectionalCollection<T> = BidirectionalCollection where Element == T

// Creating a new `Opaque*` family of types similar to `Any*` family
typealias OpaqueBidirectionalCollection<T> = opaque AnyBidirectionalCollection<T>

// Missing feature: extending existentials
extension AnyBidirectionalCollection where Element == String {
  public func joined(separator: String) -> String
}

In case of ambiguity of the where clause I had a discussion a while ago where I pitched a different keyword for constraining conditionally a type directly from the declaration.

extension BidirectionalCollection {
  public func reversed() -> OpaqueBidirectionalCollection<Element>
    constraints 
    OpaqueBidirectionalCollection : RandomAccessCollection where Self : RandomAccessCollection,
    OpaqueBidirectionalCollection : MutableCollection where Self : MutableCollection {
    return ReversedCollection<Self>(...)
  }
}

// If there are no other generic parameters to constrain,
//  we could omit the reference to the type and just write
extension BidirectionalCollection {
  public func reversed() -> OpaqueBidirectionalCollection<Element>
    constraints 
    RandomAccessCollection where Self : RandomAccessCollection,
    MutableCollection where Self : MutableCollection {
    return ReversedCollection<Self>(...)
  }
}

One thing that we have to consider is that we lose the ability to conditionally conform the currently exposed types or the returned opaque type to custom protocols. There is no way to inject any further constraints into the above reverse function. It would be a major breaking change and removing some of the current flexibilities until we can extend opaque types ourselves.

extension ReversedCollection : MyProtocol where C : MyProtocol { ... }

// This is a required alternative that must exist!
extension opaque BidirectionalCollection : MyProtocol where C : MyProtocol { ... }

How about something like this?

extension Int : P {}
struct T : R {
  // Uses the same extra keyword `constraints` to avoid ambiguity 
  // with possible `where` clause
  typealias OpaqueP = opaque P constraints OpaqueP == Int
  func someValue() -> OpaqueP { ... }
  func someOtherValue() -> OpaqueP { ... }
}

To sum up I would like to see the evolution in the following order:

  1. where clause for existentials (and opaque types later) allowed only on a typealias
  2. adding opaque types as a constraint over existentials
  3. adding a constraints keyword (or similar) to not create confusion or ambiguity with the where clause
  4. allow extending existentials and opaque types

There are still some questions left:

  • Why is is it not possible to have opaque constants? A result of an query function that returns an opaque type does not need to be always mutable, what do I miss here?

  • Can we combine multiple opaque types?

typealias OpaqueP = opaque P
typealias OpaqueQ = opaque Q

// Is this possible?
typealias OpaquePQ = OpaqueP & OpaqueQ // means `opaque P & Q`

(Gwendal Roué) #19

This is tremendous!!!

The Opaque result types vs. existentials paragraph looks like it is the most important point of the pitch.

If only it were possible to drop the opaque keyword, and with it the distinction between opaque types and existentials, the language would make a major step forward. But I expect that you have all thought pretty hard about it. Maybe opaque types are the best compromise so far, in that they make it possible to implement a much desired feature, while avoiding existential dragons. (Edit: I wrote this paragraph because my second gut reaction (after enthusiasm) was that opaque types look like an implementation detail that leaks into userland).

All my trust and congratulations to @Douglas_Gregor :heart:

I hope we'll have time to ship it in Swift 5, considering the effect on ABI stability :crossed_fingers:!!!


(Gwendal Roué) #20

I hope I didn't totally missed the point below.

If it is true that changing from () -> P to () -> opaque P has no impact (API-wise) on clients, then can we say that opaque types are both:

  • an optimization opportunity for demanding libraries such as a stdlib who care about ABI and performance,
  • a partial implementation strategy for generalized existentials?

If so, opaque types would become a concern for libraries, much less for their clients. We could de-emphasize them, as below:

In Swift 5:

protocol P {}

// implemented as today, with an existential
func f() -> P

// NEW: implemented with opaque types, with all their limitations
// such as all return statements must use the same concrete type.
func f() -> Collection where ...

// NEW: long-term commitment to opaque types ABI and optimization opportunities
func f() -> @opaque P
func f() -> @opaque Collection where ...

And when eventually support for existentials improves, we'll get, in Swift 6:

// same: implemented with an existential
func f() -> P 

// NEW (ABI-breaking, API-compatible)
// implemented with existentials, with limitations of opaque types lifted
func f() -> Collection where ...

// same: long-term commitment to opaque types ABI and optimization opportunities
func f() -> @opaque P 
func f() -> @opaque Collection where ...

?