SE-0244: Opaque Result Types

[Joe_Groff]

@Paul_Cantrell

My suggestion — which is truly only a suggestion — is that this boilerplate may in fact have utility in making opaque types easier to reason about and communicate about, and thus might make them more acceptable to the community.

My concern with making this feature centered on opaque typealiases would be that, at least in many of the cases I can envision, type aliases could end up impeding reasoning about code. Any time you introduce more names, you're introducing more terms the reader has to keep in their head in order to understand the code. Part of the goal of this feature is to allow highly generic libraries to more directly and specifically describe their interfaces; a function being able to return some Collection where <something>.Element == T, as I see it, achieves that goal directly, since it's something you can understand if you already understand what Collection is, whereas if it had to return FunctionSpecificCollection<T>, a reader would still have to Jump to Definition on FunctionSpecificCollection to figure out what that means. That's an improvement over LazyMapCollection<LazyFilterCollection<LazyZip...>> for sure, but IMO still not as direct as could be.

[Moximillian (Mox)]

@Joe_Groff

a function being able to return some Collection where <something>.Element == T , as I see it, achieves that goal directly, since it's something you can understand if you already understand what Collection is, whereas if it had to return FunctionSpecificCollection<T> , a reader would still have to Jump to Definition on FunctionSpecificCollection to figure out what that means.

As an app developer using framework with said typealias (or library author for that matter), xcode (quick help) will helpfully tell not only the name of the typealias but it’s definition as well in the sidebar.
Typealias name will also make it more searchable in the code.

[Joe_Groff]

@Moximillian

As an app developer using framework with said typealias (or library author for that matter), xcode (quick help) will helpfully tell not only the name of the typealias but it’s definition as well in the sidebar.
Typealias name will also make it more searchable in the code.

This is true, though you're still giving up a layer of indirection in Xcode. If this weren't Collection but some other unfamiliar protocol, with the some notation, the protocol itself can have that prime sidebar space, rather than the alias. The definition of the typealias furthermore may dilute the high-level signal it's trying to communicate, since you'll see not only the interface constraints of the opaque typealias but the definition as well.

[Paul_Cantrell]

@Joe_Groff

Part of the goal of this feature is to allow highly generic libraries to more directly and specifically describe their interfaces; a function being able to return some Collection where <something>.Element == T , as I see it, achieves that goal directly, since it's something you can understand if you already understand what Collection is, whereas if it had to return FunctionSpecificCollection<T> , a reader would still have to Jump to Definition on FunctionSpecificCollection to figure out what that means.

That’s fair.

In practice, I expect an API with many functions returning function-specific opaque types would put the opaque typealias next to the function’s definition, in which case the extra indirection wouldn’t be so painful:

protocol Florgleblatzer {
    associatedType Element

    typealias FooCollection: Collection where <something>.Element == T
        = HiddenFooCollectionImplementation<T>

    func foo() -> FooCollection<Element> {
        ...
    }

    typealias BarCollection: Collection where <something>.Element == T
        = ComplexThing<NestedThing<T>>

    func bar() -> BarCollection<Element> {
        ...
    }
}

And to my mind, the ability to get this error message:

cannot assign value of type FooCollection<Int> to type BarCollection<Int>

…and to do this:

let foos: FooCollection<String>  // NB: type annotation necessary for deferred var initialization
if eenieMeenie {
    foos = thinger.foo()
} else {
    foos = whatever.etc().foo()
}

…and for the API author to be able to express this:

protocol Florgleblatzer {
    associatedType Element

    typealias FooCollection: Collection where <something>.Element == T
        = HiddenFooCollectionImplementation<T>

    func foo() -> FooCollection<Element> {
        ...
    }

    func foo(mode: FooMode) -> FooCollection<Element> {
        ...
    }

    func fooWithFrosting() -> FooCollection<Element> {
        ...
    }

    ...
}

…all offset the cost of the extra indirection.

[John_McCall]

Even supposing we don't include opaque typealiases in the first release of this feature, I think it's important to consider them to make sure that the language design is in the right place.

I think there are two basic ways to spell opaque typealiases:

• The obvious spelling is to just allow a typealias to name an anonymous opaque type:

  typealias FilterCollection<Element> = some Collection
  

  The problem with this is that it's still just as unclear how to express the output constraints on the opaque type. Opaque typealiases are fundamentally unlike ordinary typealiases in that they need to distinguish "input" constraints (the generic signature of the typealias declaration itself) from "output" constraints (the guarantees being made of the aliased type). Output constraints can be totally general: there are good reasons to allow any and all of same-type constraints, class bounds (perhaps on associated types), unconditional conformances, and conditional conformances (which of course then also require a set of requirements as conditions). It's probably unreasonable to expect that the anonymous opaque type syntax will ever support all of that; the code would be totally unreadable.

• The other spelling acknowledges that opaque typealiases have more complex language requirements than ordinary typealiases and so blesses them as a different kind of declaration. I think the most reasonable way of doing that is with a modifier, something like this:

  opaque typealias FilterCollection<Element>: Collection = MyFilteringCollection<Element>
  

  I don't want to get side-tracked on the exact spelling here. The important point is that this needs a (contextual) keyword, and it probably shouldn't be a different keyword from the anonymous spelling. some typealias would be terrible, though. To me, that strongly argues for using a keyword like opaque in the anonymous spelling.

[Joe_Groff]

That's a reasonable concern. For better or worse, we do have lots of precedent for different syntaxes for the same thing in different situations (function decls vs. closures, mutating and consuming self vs inout and owned other arguments, static vs class methods, etc.), and in many cases this was motivated by what "reads better" contextually.

[John_McCall]

@Joe_Groff

Part of the goal of this feature is to allow highly generic libraries to more directly and specifically describe their interfaces

I'm not actually sure anonymous opaque types are a good fit for this use case. It seems to me that such a library is too likely to find it useful to give a name to the type, either so that it can be shared between obviously-related functions or just to allow clients to name it so that it can be used in all of the places where today we require a type annotation, like a parameter or a type property. I believe I've seen vague arguments that there could be an evolution path for turning anonymous opaque types into named ones, but I haven't found that convincing, in part because I'm not really sure what it would look like.

I think anonymous opaque types are largely only useful in the much narrower use case of a fairly simple protocol (probably with few or no refinements) with a lot of "combinator" implementations and where you've got a bunch of functions that return values that need to conform to that protocol (but you don't want to or can't use a protocol type). You want those functions to able to expressively apply combinators and then return the result, but the actual type of applying those combinators is absurdly specific and writing it out in the signature would be awful. You get this sort of thing in DSLs a lot — it especially happens in some kinds of C++, but I also know it happens with e.g. lazy in Swift.

[John_McCall]

This is a lot more corner-case than mutating and inout, though, which means it has a much higher bar to reach.

[Joe_Groff]

@John_McCall

I think anonymous opaque types are largely only useful in the much narrower use case of a fairly simple protocol (probably with few or no refinements) with a lot of "combinator" implementations and where you've got a bunch of functions that return values that need to conform to that protocol (but you don't want to or can't use a protocol type). You want those functions to able to expressively apply combinators and then return the result, but the actual type of applying those combinators is absurdly specific and writing it out in the signature would be awful. You get this sort of thing in DSLs a lot — it especially happens in some kinds of C++, but I also know it happens with e.g. lazy in Swift.

This is exactly the sort of situation where generic or "protocol-oriented" programming is at its best, though, and that's why I don't think this will be a corner case for long. I talked about this a bit in an earlier reply; the analogous Rust features gets used heavily with a number of different protocols. On a different note, the future syntactic analogy that I think makes sense for this feature is akin to what Rust did with its impl Trait syntax, allowing it to be used both in argument and in return position, as an alternative to having to name independent type variables. If we go down that road, then the connection between this syntax and opaque typealiases becomes more tenuous, and potentially more misleading as well.

[Joe_Groff]

I also think it'd be good to consider not only the component combinators themselves, but the clients thereof, where this feature would also be useful. If you have a helper method like:

func processThings<C: Collection>(in collection: C) -> <really long composed collection type> {
  return zip(collection.lazy.map { ... }.filter { ... }, someOtherCollection...)
}

then that might be when this feature is its most compelling. It is unlikely that the return type of a helper like this is going to intentionally coincide with other helpers; the explicit return type is likely to be in large part a type-level reiteration of the implementation; and exposing the return type to callers is therefore an ABI and API evolution risk if any part of the helper implementation ends up changing.

[Chris_Lattner3 (Chris Lattner)]

@Joe_Groff

@Douglas_Gregor, @Joe_Groff as the proposal authors, have you considered the approach described above, where this problem is solved by introducing the ability for API authors to "opaquize" the concrete type that a typealias is defined by?

We explored that direction before proposing this, and we don't object to it as a future direction. As you noted, it would largely be an extension of the same underlying implementation. I think that, even if we had opaque typealiases, it will still be the common case when using this feature to have a unique opaque return type per function, and so being required to declare the typealias and keep it in sync with the associated function's implementation would be pure boilerplate. Opaque typealiases are also a strictly more complex feature, since they raise more interesting questions about the visibility of the underlying type, whereas when the opaque type is constrained to a single definition, it is obviously contained to the body of that definition. On balance, we decided that opaque return types are a more fundamental part the "tech tree" for this feature, and that typealiases would make more sense as an extension.

Personally, I don't see one as an extension of the other. I see these as completely different language design approaches, which share an implementation approach.

Look at the 'future directions' section of the proposal. Every single one of these would be different if this feature were about opaque type aliases, and most of them have obvious solutions. The direction the proposal (as written) implies a massive forking of the conceptual design space, and requires significant added complexity to the language surface over time to catch back up with the capabilities that the opaque type alias solution naturally provides.

You are right that the opaque type alias approach would require type aliases to be defined -- that is definitely a tradeoff for the simple case. However, for the general case (which this proposal does not include, but many commenters argue makes it insufficient) the type alias approach seems strictly superior. If this were a feature that was front and center with new Swift programmers I could see the argument for avoiding boilerplate. Given that this is an advanced feature used occasionally by library developers, I personally don't weight the sugar argument highly.

Also, all the controversy about naming is worth considering at a higher level. While it can easily be brushed aside as bikesheding, the fact that we can't really get a good name to describe this is a pretty strong signal that it doesn't align well with the overall design of the language. An "opaque type alias" is something that is pretty simple to understand, and dovetails with the existing resilience model in a very logical way, instead of reinventing (over time as the feature is fully baked) tons of syntax and concepts that only align at the constraint solver's implementation level.

@Joe_Groff

My concern with making this feature centered on opaque typealiases would be that, at least in many of the cases I can envision, type aliases could end up impeding reasoning about code. Any time you introduce more names, you're introducing more terms the reader has to keep in their head in order to understand the code.

The proposal as written exclusively talks about the "let x = method()" and "method().otherThing()" case, and this doesn't affect that.

@Joe_Groff

I don't think it's difficult to produce a better error message. I haven't gotten far enough in the implementation to work on improving diagnostics yet.

It is a super minor point, but my sense is that an opaque type alias approach (where in the impl, you have type alias sugar node wrapping a typevar) would "just work" with almost zero special code. The fact that it falls out of the implementation is another strong design smell that such an approach fits better with the overall design of the language than one that would require a bunch of special case logic to tune and tweak the type checker in specific cases like this.

-Chris

[Joe_Groff]

@Chris_Lattner3

Look at the 'future directions' section of the proposal. Every single one of these would be different if this feature were about opaque type aliases, and most of them have obvious solutions. The direction the proposal (as written) implies a massive forking of the conceptual design space, and requires significant added complexity to the language surface over time to catch back up with the capabilities that the opaque type alias solution naturally provides.

Of the future directions described there, I can only see one that opaque typealiases address (aside from opaque typealiases themselves):

• Opaque typealiases would be of limited use for describing opaque types in structural position; once you have more than one, you may want to impose constraints between them.
• Opaque typealiases would have the exact same naming problem for describing where clause constraints, since with either opaque result types or with opaque typealiases, you need some way to describe the free type variables within the exposed interface of the opaque type. Using the typealias name itself would be as limiting as using a placeholder like _ for the return type case, since it only scales to a single free variable.
• Opaque typealiases do indeed offer an obvious solution for conditional conformances, for which extension notation would be the obvious choice.
• Opaque typealiases are completely independent of "opaque argument types". There is perhaps an alternative factoring of these features, where we have opaque typealiases, and then the "some" sugar is introduced uniformly for arguments (to introduce anonymous generic arguments) and returns (to introduce an anonymous opaque return typealias).

@Chris_Lattner3

The proposal as written exclusively talks about the "let x = method()" and "method().otherThing()" case, and this doesn't affect that.

Those examples are trying to illustrate the use of method, though, not the feature itself. Opaque type aliases benefit the implementation of method() itself, where type inference doesn't help you like it can in the callers to method. I don't think that's a fair summary of the examples, though, since particularly in the introduction we provide some other examples of the kinds of library design this enables. The GameObject example is intended to be an example of where user code would use this feature heavily to interface with a generic library; it isn't intended to be just a niche library feature (and that's definitely not how the analogous feature has ended up in Rust, as I noted before).

I can understand your (and @Paul_Cantrell's) uncertainty about anonymous types, given that they're extremely ugly and problematic in C++ and other substitution-based languages. I think this feature as is has a few important benefits over anonymous types in those other languages. For a C++ lambda, all you really have to describe it is its source location, and because of the nature of C++'s type system, there's not really much you can do to abstract information about the type; as other templates get instantiated on the lambda, all you can do in C++ is snowball the type information into an avalanche of multi-page error messages when things go bad. On the other hand, the feature we're proposing is all about abstraction; it creates a separate entity the type system can reason about, and that entity is strongly tied to its source declaration, making it possible to refer to as "the return type of f" (and hey, in the fullness of time, we could maybe even allow type(of: f) to itself be used as a type). The opaque type hides the underlying implementation type, preventing the type avalanching problem C++ has, and is itself well-typed by its constraints, making it clear what its intended capabilities are.

[Paul_Cantrell]

@Joe_Groff

Opaque typealiases would have the exact same naming problem for describing where clause constraints, since with either opaque result types or with opaque typealiases, you need some way to describe the free type variables within the exposed interface of the opaque type. Using the typealias name itself would be as limiting as using a placeholder like _ for the return type case, since it only scales to a single free variable.

Joe, would you be willing to work out the details of this one in an example? I’d have naively thought that type parameters in an opaque type alias would solve this — but I don’t think I fully understand what you’re describing.

[Joe_Groff]

For an opaque typealias, you need to describe the type as seen from outside, as well as the underlying type. In full generality, there may be multiple structural parts of the type you want to hide, so you need what amounts to another generic signature to describe it:

opaque typealias PartitionedCollection<T: Collection>
  = <U: Collection, V: Collection> (oddValues: U, evenValues: V)
    where T.Element == U.Element, U.Element == V.Element

@orobio did a great job describing this in his "reverse generics" thread, which describes the problem in terms of opaque returns, though opaque typealiases are effectively the same thing as far as type system behavior. Phrasing something like this as two independent opaque typealiases would not be equivalent, because you would lose the ability to describe the constraints between the two elements of the tuple.

[John_McCall]

Right, sorry, the combinator clients were exactly what I was thinking of, and that's an excellent summary of why it matters for them.

[Moximillian (Mox)]

@Joe_Groff

Opaque typealiases would be of limited use for describing opaque types in structural position; once you have more than one, you may want to impose constraints between them.

In case of two named opaque typealiases, why wouldn’t you be able to add additional constraints to the function declaration using the names? Similar to regular generics...

[Moximillian (Mox)]

As @Chris_Lattner3 said anonymous syntax is it’s own design space. For me the full potential of that would be to have a wide reaching shorthand that covers wide variety of types: anonymous structs, anonymous protocols (existentials?), anonymous generics, anonymous typealiases, anonymous associated types and so on (don’t know which ones make sense).
To design that space well, would mean that it’s applied to lot of those types from the start, not just as a byproduct of opaque type feature.

Likewise, ideally I’d like to see opaque types designed within the existing Swift syntax, ensuring it fits well to the language as is, and then later augment it with the general implementation of anonymous types.

[Chris_Lattner3 (Chris Lattner)]

I will take a stab at sketching out a proposal alternative that uses opaque typealiases, so you all have something more concrete to A/B compare.

I don't have a lot of time to devote to this so it will be very sketchy, but hopefully it will be useful for the discussion. I don't see any reason to introduce new keywords and grammar productions to the language here, and opaque type aliases are more general overall than what is being proposed.

[Chris_Lattner3 (Chris Lattner)]

Ok, here is a draft of the document, please take a look.

In comparison to the proposal outlined here, this approach requires no new keywords or grammar productions (just adds a decl modifier, and undisables an existing grammar production in type aliases), it composes properly towards the 'future directions', and doesn't have the unsolved issues that the original proposal has w.r.t. future directions.

it also seems like a very nice thing because API authors can choose to expose exactly what they want, and because these types can be passed and returned, it may also provide a mechanism to solve the latent issue with the C importer, where we don't know how to import opaque types from C while preserving their identity.

[Moximillian (Mox)]

@Chris_Lattner3

Ok, here is a draft of the document , please take a look.

Love it! Such an elegant, understandable and Swifty solution to opaque types.