SE-0244: Opaque Result Types

[Ben_Cohen]

Hi Swift Community,

The review of SE-0244: Opaque Result Types begins now and runs through March 15, 2019.

Reviews are an important part of the Swift evolution process. All review feedback should be either on this forum thread or, if you would like to keep your feedback private, directly to me as the review manager via email or direct message on the forums. If you send me email, please put "SE-0244" somewhere in the subject line.

What goes into a review of a proposal?

The goal of the review process is to improve the proposal under review through constructive criticism and, eventually, determine the direction of Swift.

When reviewing a proposal, here are some questions to consider:

• What is your evaluation of the proposal?

• Is the problem being addressed significant enough to warrant a change to Swift?

• Does this proposal fit well with the feel and direction of Swift?

• If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

• How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

Thank you for contributing to Swift!

Ben Cohen
Review Manager

[xwu (Xiaodi Wu)]

A reasonably scoped proposal in terms of what's in and what's out. I think this would be a good addition.

I do regret that, since we don't have a true bottom type in 'Never', we have to add unreachable code in some circumstances to use opaque types. I understand the underlying limitations, however, but I would suggest a modification:

Allow statements like fatalError() in the absence of other return values cause the return type to be inferred as Never without the extra return. The compiler can simply error if the opaque constraints aren't satisfied. The user is likely to be writing a stub, in which case adding a different return value is fine, but they may wish (where possible) simply to conform Never to the protocol in question.

[anandabits (Matthew Johnson)]

• What is your evaluation of the proposal?

+1 This design looks like a great start on opaque types in Swift. Everything in the future directions looks very nice as well but there is no reason to hold back the initial implementation. The roadmap is clear so we should move ahead with the initial implementation.

• Is the problem being addressed significant enough to warrant a change to Swift?

Yes, this will eliminate the need to explicitly specify verbose type names. It will improve the clarity and simplicity of some designs in a significant way.

• Does this proposal fit well with the feel and direction of Swift?

Absolutely.

• If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

N/A

• How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

I have eagerly followed all of the opaque result type threads and have been looking forward to seeing them implemented.

[Vogel]

-∞

• What is your evaluation of the proposal?

I think it is horrible

protocol A {
    associatedtype B
}

struct C { }

func d() -> opaque A where B: C {
    return ...
}

//just some kind of convenience method that always ends up calling d:
func e() -> opaque A where B: C {
    //other stuff...

    return d()
}

let f = [d(), e()] //OH NO, ERROR, NOT THE SAME TYPES (even though they actually are, but we can't express that.)

• Is the problem being addressed significant enough to warrant a change to Swift?

The proposal is so bad that it doesn't even properly address any problems. Not wanting to expose underlying implementations can be addressed using a simple wrapper struct – without breaking composability.

• Does this proposal fit well with the feel and direction of Swift?

Absolutely not. It weakens the type system, because instances cannot be passed through functions anymore without essentially altering their type.

Currently, I can always build a convenience function that returns the result of some other function, but also does additional work (or not). In general, if I get a hand on some value, I can pass that value around or return it from my function. Breaking this very basic promise seems like a horrible idea to me.

• How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

I have been very active in the thread before this proposal went in review.

[beccadax (Becca Royal-Gordon)]

I'm in favor of the feature and I think the proposal covers enough ground to be a useful addition to the language.

However, the vague syntax suggestions in the "Future Directions" section concern me. I'm worried that the -> some P syntax may be adequate for this proposal's purposes, but may not extend cleanly to uses we envision in the future. Maybe there is some larger scheme we could fit the some syntax into; if so, I'd like to see a sketch of that scheme before we commit to it here. (More thoughts on that in the Protocol<.AssocType == T> thread.)

Otherwise, I support it.

[Tino]

-1

Opaque result types may have their merit, but for the average developer (those guys who write code instead of proposal reviews ;-), it's just increasing the surface of the language. I can't think of any other features that resemble a similar syntax, and imho the discussion thread had too much bikeshedding and too little real alternatives (which might feel less alien).

Maybe the hypothetical ownership-model could be expressed in a similar way, but that is all to vague for me, as I doubt we could ever get rid of some even if we discover something better later down the road (this wouldn't be as critical if only the Evolution process had something like an extended evaluation phase...)

Besides a "bigger picture", I think there might also be some irritating interactions with existing syntax:

open class Test {
    open func foo() -> some Hashable {
        return 1
    }
    // some implies final, doesn't it?
}

What about nil?
As I understand the proposal, it's not possible to have opaque results which are Optional, and this imho would be a strange limitation.

[Nevin]

If we accept the proposal as-is, meaning we do not initially support type-aliases, what will the library-evolution story look like when we add them later?

In particular, if a library has two functions that return the same opaque type, will it be possible for a future version of the library to introduce an opaque typealias for both of them to use?

// ShapeLibrary v1
func foo() -> some Shape { return Square(2) }
func bar() -> some Shape { return Square(4) }

Will it be okay for the next version of the library to have this?:

// ShapeLibrary v2
typealias SomeShape = Square : some Shape

func foo() -> some SomeShape { return Square(2) }
func bar() -> some SomeShape { return Square(4) }

And in the version after that, how about this?:

// ShapeLibrary v3
typealias SomeShape = Rectangle : some Shape

func foo() -> some SomeShape { return Rectangle(2, 4) }
func bar() -> some SomeShape { return Rectangle(4, 6) }

[Joe_Groff]

The formats of the data structures for opaque result types and type aliases would be the same. We'd need some way to describe the fact that an opaque type alias used to be a return type, to be able to forward the existing symbols, but it would be possible to support this evolution.

Similarly, a library could choose to reveal its concrete return type in a later version, as long as it notes that it was previously opaque.

[SlaunchaMan (Jeff Kelley)]

+1. This is one of those things that I haven’t needed often, but would have been super helpful when I did.

[John_McCall]

I certainly hope some doesn't imply final. It should just be a requirement that methods with opaque result types be final.

I still don't like the spelling some.

[Paul_Cantrell]

General +1, with some concern about complexity.

My compliments on this particularly well-written proposal. In my naiveté about the issue, I had many, many questions — and the proposal answered just about all of them all thoroughly and clearly.

---

My only reservation is about further increasing the cognitive surface area of the language. The rules surrounding opaque types are certainly going to be confusing for users — particularly the fact that some P implies a different anonymous opaque type every time it appears. That is not going to always “just work” in a pleasant way. Careful attention to error messages will be crucial here.

I would be happier if there were a way to avoid this construct altogether. Given that people far more knowledgable than me are convinced it is necessary, however, I’m willing to defer to their judgement.

(Aside: why do Java / C# not need this? Is it because JIT compilation can achieve the same performance benefits by transforming interface method invocations to static dispatches, whereas Swift is always forced to go through a witness table when an existential is involved? Or is it something subtler?)

---

FWIW, I rather like the some P syntax, at least in spirit. Reading the code samples in the proposal, I find it helps one reason heuristically about these curious beasts.

As an alternative, I’m tempted to argue in favor of ditching in-situ ad hoc opaque types altogether and only supporting opaque types via typealiases, as in the Future Directions section. The explicit naming of each opaque type could help clear up the confusion surrounding “some P ≠ some P sometimes but not always.” It would give users an explicit type name to use when passing opaque types around. It would also help API authors understand exactly what types they are exposing. However, I can see that the proliferation of named types would be a nuisance.

---

Two questions:

First, what does one do if it becomes necessary to add an explicit type annotation for an opaque type? In other words, suppose I have this code:

var things = values.lazy.map(f).compactMap(g)

I assume this doesn’t work, because I need to specify that things is the particular opaque type returned by compactMap … right?

var things: some Collection<.Element == G> = values.lazy.map(f).compactMap(g)

So what is the explicit type of things? Can I not specify one? What if I want to write a helper function that operates on things, and doesn’t want the performance overhead of using an existential?

---

Second question, which I suspect reflects my ignorance about how ABI stability works. The proposal states:

opaque result types are only opaque to the static type system. They don't exist at runtime.

…but also:

Opaque result types are part of the result type of a function/type of a variable/element type of a subscript. The requirements that describe the opaque result type cannot change without breaking the API/ABI. However, the underlying concrete type can change from one version to the next without breaking ABI, because that type is not known to clients of the API.

How does this happen? Does Swift’s dynamic library loading process stitch up client code to adapt to the now-possibly-different shape of the opaque type? Or are usages of opaque types dispatched through some kind of witness table?

[laposheureux (Aaron L'Heureux)]

I would vastly prefer to see a very literal term. Swift isn't hesitant to call an optional an Optional, an associated type an associatedtype, etc. Why is an opaque type a some and not an opaque?

With regard to how other keywords are sometimes figurative, they are either narratively intuitive (let x = 8 contrasted with var) or require investigation (a ? does not inherently have the connotations of a word). some on the other hand could suggest a limited result set or a mixed result set or this opaque result set, assuming the first time reader even knows that is a thing. It's a really vague word.

I am also concerned with the increase in conditional casting that opaque types will promote. It seems at odds with a strong desire for compilation-based type safety. I'm also concerned with regard to how this potentially diminishes the desire to get to a solution for generalized existentials as this is a different way of dealing with self/associated type requirements, just lacking the type safety of those existentials.

[DevAndArtist (Adrian Meister)]

• What is your evaluation of the proposal?

In general I support the idea of opaque types, but I would prefer opaque keyword instead of some (speaking as a non-native English speaker).

• Is the problem being addressed significant enough to warrant a change to Swift?

Kind of, this feature is great when you want to hide some simple concrete types, at least as it's proposed right now. I certainly will not use it at this stage as in my projects as for me it will require support for typealias and where clauses to be useful.

• Does this proposal fit well with the feel and direction of Swift?

It certainly does.

• If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

N/A

• How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

Followed the discussion pitch and read the proposal.

[DevAndArtist (Adrian Meister)]

@Joe_Groff can you clarify what this exactly means in detail:

Opaque result types are an ABI-additive feature, so do not in and of themselves impact existing ABI. However, additional runtime support is however needed to support instantiating opaque result types across ABI boundaries, meaning that a Swift 5.1 runtime will be required to deploy code that uses opaque types in public API .

• Do I just need to update my project to Swift 5.1 to use this feature, while my minimal deployment target can still be let's say iOS 10?

• Or this whole feature locked down to the OS version that comes with Swift 5.1?

---

If in Swift X we added where clauses to opaque types, would it also require additional runtime support?

[johannesweiss]

+1! Also I love the some Thing spelling.

I think the change in Swift is more than warranted because this is a common case in library/framework code for which we don’t have only bad solutions today. As the proposal shows some of the stdlib’s collection methods really have pretty awful return types (and that’s nobody’s fault because we can’t do better today without penalising performance in many cases).

[marco.masser]

I honestly don’t feel qualified to comment on whether the proposal as written is a good idea long-term, but I stumbled upon three questions when reading it. All are more requests for clarification than anything else.

First, it’s pretty clear to me that the result types of two different functions returning some Equatable have different static types. But is that really a problem if they can statically be inferred to be the same?
Adapting one example from the proposal, this is what I mean:

func getEquatable1() -> some Equatable { ... }
func getEquatable2() -> some Equatable { ... }

let x = getEquatable1()
let y = getEquatable2()
if x == y { // Is this allowed?
  print("Bingo!")
}

Could this (at least theoretically, if not practically) be allowed, at least in certain scopes?
I understand that this cannot work when the functions are defined in a different module than the use of ==, but what about when they are in the same module?
Would that only work when compiling with whole module optimization?
What about when they are in the same file?
Not that I think it would be a good idea to have these differences.

---

Second, there’s a restriction that “the opaque type must be the entire return type of the function, For example, one cannot return an optional opaque result type.”

This is mentioned as being an error:
func f() -> (some P)?

What about this:
func f(flip: Bool) -> some P?

Is that even syntactically valid? P? a.k.a. Optional<P> would be a pretty concrete type, so I would assume that this is not allowed.

But why would it be a problem to return nil from such a function?

---

Third: When opaque result types were first pitched I assumed that they were intended to be included in Swift 5 and the advent of ABI stability. But what is the expected impact of this feature for the standard library now? If opaque result types are introduced in Swift 5.1, all those internal types that could be made opaque can never be made opaque. That pretty much limits opaque result types to future standard library APIs third party libraries, correct?

[ptomaselli (Peter Tomaselli)]

• What is your evaluation of the proposal?

-1

• Is the problem being addressed significant enough to warrant a change to Swift?

Unsure. I initially felt “unqualified” (as others have said) to even comment on this proposal. But perhaps this perspective is useful: as an app developer (as opposed to a framework author), I have not felt a pressing need for a feature like this! There are already a number of unique / weird constructs used in the standard library (afaik), can’t y’all like… I dunno, do some more of that weird stuff?

I am not sure whether or not this will make consuming eg. the standard library more or less ergonomic, to be honest. I have worked with lots folks just starting out with Swift and one thing I always try to lean on is thinking clearly about the types involved with any given computation. An opaque type is not a concrete type, not a protocol type, it’s a third weird, kinda untouchable thing. And there’s gonna be a lot of them on the public API surface. Right? I’m not certain, but I’m concerned that may push a lot of complexity onto beginners.

• Does this proposal fit well with the feel and direction of Swift?

I agree with others that, if we want this, we should lean in to the idea that it is a new feature and spell it opaque — overloading some may scan better, but it’s way too quiet, in my opinion. Also, in my opinion, weakening (to a small extent I suppose) Never is not great. Additionally, the introduction of unutterable types is not Swifty, in my opinion. A feature of Swift I really like is that type inference is there, it works well, but you can always escape from inference hell with an explicit type if you need to. A sudden proliferation of unutterable types (apologies if I misunderstand here) complicate that considerably!

• If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

I have a lot of experience with C# and LINQ, where collection shenanigans are erased using IEnumerable. IEnumerable only exposes read-only methods and as such can be annotated in the type system as covariant. Iterator shenanigans such as lazy mapping and filtering, etc. are therefore always expressed as returning IEnumerables.

I get why we can’t have this right now, but this is a far superior experience to either what we have now in Swift, or to what we’d have after having this proposal, in my opinion.

• How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

Read the pitch thread, read this thread.

[Vogel]

@ptomaselli

I am not sure whether or not this will make consuming eg. the standard library more or less ergonomic, to be honest. I have worked with lots folks just starting out with Swift and one thing I always try to lean on is thinking clearly about the types involved with any given computation. An opaque type is not a concrete type, not a protocol type, it’s a third weird, kinda untouchable thing. And there’s gonna be a lot of them on the public API surface. Right? I’m not certain, but I’m concerned that may push a lot of complexity onto beginners.

You've figured out the main problem with this proposal: It's not very powerful, so you know that you as the app developer won't need it, but merely consume these types from frameworks that you're using, including the standard library, however, frameworks can't actually use it, because it would mess up the composability of their function return values, as my code above shows.

@gwendal.roue

the current stage of design only works for "leaf APIs", I mean results that can't be further processed/composed by client code

So, no one can use this productively.

[nuclearace (Erik Little)]

@Vogel

So, no one can use this productively.

No, as a framework author, this is still a productive addition. There are many a time where I would've liked to hide the concrete type of something from a consumer of my library. Mostly because the concrete type is something that has a lot additional methods/properties/etc, that, because of Swift's access control model and lacking existential support, mean I can't directly hide it behind a protocol.

Also, most of the time I've wanted this kind of feature has been for functions/methods that return results that should immediately be transformed/used.

That being said, I'm all for this proposal. Kudos to the people who helped bring this forward!

[allevato (Tony Allevato)]

I really want to support this fully because I've written libraries where hiding implementation details has not always been possible, and this is a huge step in that direction. In addition to the places in the stdlib that would benefit from this addition, I recently wished that CaseIterable.allCases returned an opaque Collection instead of an array, for performance reasons.

A couple things do give me pause, some of which have already been mentioned here:

• The inability to compose an opaque type into another type is unfortunate, especially in the case of Optional. Purely from an API design point of view, it seems odd that I could return a some P, but not a "some P or nothing". I'd like to know, is this limitation a matter of implementation time/difficulty, and could it be lifted in the future, or is it a fundamental conflict with how the type system works?

• There are a number of places in the stdlib that would benefit and be made more resilient if they were changed to use opaque types instead of custom wrapper types. Would those also be changed as a consequence of this proposal? It seems that for resilience reasons, that might be the last chance to do so.