SE-0244: Opaque Result Types

Evolution Proposal Reviews
81

Thank you for this! I must admit that I read the proposal in its entirety, and only this explanation finally helped me wrap my head around the idea.

5 Likes
82

So... some for the first case and someSpecific for the second?

(I'm half joking, but only half)

83

I'm new here. My apologies if I make any mistakes.

I saw this proposal on the Swift evolution list on github, and joined just so I could contribute.

Happily, I see that @orobio is already saying much the same thing I was planning to say, though I'd prefer a slightly different syntax.

But it seems to me that a lot of people are ignoring what he's really getting at.

To me, the crux of his suggestion is that it inherently includes a name for the opaque type. Without that, it's difficult to make it clear whether two instances of the opaque type are the same or not. And I believe both answers are sometimes what's desired. Also, without a name for the type, it is difficult to include all the things you might want to say in a where clause.

The main problem I have with his syntax is that using a caret "^" is visually lightweight. Opaque types aren't going to be very common, and I would prefer that they stand out visually. (On my first reading of @orobio's suggestion, I failed to notice the "^" at all.)

In addition, if this is in some sense a dual of a regular generic argument, I think it would be easier for people to understand if it looks's both like and unlike regular generics. So I'd suggest something like:

func makeFoo<<T: Foo>>() -> T { /* ... */}

(In fairness, @orobio did mention something a little like this as a possibility, but it seems to have gone by the wayside.)

This would show that it's similar to regular generics, but different from them. That would be my preference.

It would make is easy to implement the usual where clause syntax, to constrain the type in all the usual ways. And it becomes simple to say both:

func makeTwoTheSame<<T: Foo>>() -> (T, T) { /* ... */}

and

func makeTwoPotentiallyDifferent<<T: Foo, U: Foo>>() -> (T, U) { /* ... */}

It also allows:

func convenienceFunction<<Something: Foo>>() -> Something {
    let aFoo = makeFoo()
    return aFoo
}

And, perhaps even better, you can say:

struct SomeType<<T: Foo>> {
    typealias MyFoo T
    static func makeFoo() -> MyFoo { /* ... */}
}

let anotherFoo: Sometype.MyFoo = SomeType.makeFoo()

(I think this will be more commonly used inside a type like this, rather than in a free function.)

I don't show any examples with a where clause, or something like returning two possibly different kinds of collections, which have the same element types, and so forth, but I think it's clear how you could say things like that. It gives the author of a function returning an opaque type the flexibility to be very clear about what is (and is not) being promised.

Now, I currently know nothing about the inside of the compiler, so I don't know how hard this would be to implement, but if this is really just about the same thing as a regular generic, except with the "inside" and the "outside" roles reversed, it doesn't seem to me that it should be unreasonably difficult. (Famous last words!)

I do understand @Chris_Lattner3's preference for something that's easy to search for, but I still prefer this to having any keyword at all. But if it's felt that there absolutely does need to be a keyword, how about something like:

func makeFoo() -> sometype T: Foo { /* ... */}

where a type name must be provided. And then additional uses of the same type could just refer to "T".

<><><><><>

Answering the initial question that started this thread...

I think the general idea is OK if it's extended to always include a name for the type, but without that I think it would paint Swift into a corner. I'd also prefer that it include all the where clause syntax from the start, because otherwise it feels too limited.

As to whether it's really important enough to be worth the change, I'm really not certain. I think folks who have worked on something like the standard library would have a better perspective on that than I do.

I think it fits fairly well with Swift, especially if it used something similar to the syntax I suggested.

I spent several days thinking about this before writing this. I'd think for longer if the deadline weren't fast approaching.

2 Likes
84

What is your evaluation of the proposal?

I think it's good. My two gripes with it are:

  • I'd rather have opaque as the keyword for this feature. It's not that I worry much about the ambiguity with Optional.some, but I'd rather see the feature name aligned with what you see in the source code.

  • I worry about the inability to refer in other code. If you can't name it, it's difficult to store as a variable in a struct or elsewhere, and it's difficult to write wrapper functions without each of them creating a new distinct opaque type.

Is the problem being addressed significant enough to warrant a change to Swift?

I think the feature will be useful enough once further developed. I'm not too sure of the utility without the ability to add constraints to the opaque type. Sure, you can shuffle the elements in an opaque collection, but if you can't read any of them, what's the point?

Does this proposal fit well with the feel and direction of Swift?

I think it will, mostly, once we have constraints. My main reserve is about the various consequences on composability introduced by having anonymous types. The main problem is the difficulty of giving them a name.

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

It feels a bit like opaque pointers in C, commonly used to hide implementation details. But instead of exposing global functions taking an opaque pointer, you expose methods in a protocol.

Opaque types are quite similar to Voldemort types in D. But in D you can still declare a variable of a type that cannot be named using typeof(<some_expression>) so it does not really affect composability.

One common issue with Voldemort types in D is the excessive growth in mangled function names the can happen when combined with templates. I'm not too sure if this applies to Swift or not since the proposal has little to say about name mangling, but I find it likely that generic function specialization that would include specialization of their opaque type could result in similar issues.

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

Participated in the pitch, but mostly skimmed the actual review thread.

1 Like
85

Swift's version of the feature should hopefully not suffer from this because our opaque types are also ABI boundaries. The thing that needs to be mangled (as well as what needs to be reported in diagnostics, etc.) is only "return type of <decl>" rather than the transitive composition of the underlying types.

86

In thinking of opaque types as dual to generics, I wonder if it would be useful to address the "two functions returning the same opaque type" issue with some sort of dual to generic type parameters on types themselves:

// strawman syntax based on what has been discussed:
struct MyCollectionGenerator -> <T: Collection> {
    func giveMeACollection() -> T {
        return [Int]()
    }

    func giveMeACollection(withCount count: Int) {
        return [Int](repeating: 0, count: count)
    }
}

let gen = MyCollectionGenerator()
let collections = [gen.giveMeACollection(), gen.giveMeACollection(withCount: 3)]

This could also assist with composability:

func forwardACollection() -> MyCollectionGenerator.T {
    return MyCollectionGenerator().giveMeACollection()
}
87

How would you make a function with classic generics and this thread's "reverse" generics? (The "func name<T: SomeProtocol1>(_ t: T) -> <U: SomeProtocol2> U" double-generic syntax from earlier in the thread obviously allows this.)

88

I think that's a very logical place to put it and I thought about that as well. However, I feel that it clutters up the return section. Perhaps there are good reasons to do it that way, but having them part of the already existing generic type parameters list seems cleaner to me.

Personally, I feel something like ^ would be enough, but there are similar alternatives that stand out more. Using a second list of parameters, like <<T>>, will create a lot more noise, especially when used together with normal generic parameters:

func makeCollection<T><<C: Collection>>(with element: T) -> C { ... }
89

For people that are still confused about opaque result types, I did an attempt to explain them from the perspective of the earlier mentioned 'reverse generics'.

It would also be helpful if some people with more knowledge could have a look at it and point out any errors.

90

As I've been watching this and thinking about it I've fallen into the camp of being a big +1 on this feature. I feel it will be particularly useful for library authors. I've often wanted and reached for this kind of functionality.

I've read through most of the pitch thread and this thread.

Other languages I've worked with don't have the same issues of working with PATs that Swift has so they can already vend their equivalent types to Collection and other protocols we have in Swift. The either are just non-typed or they can return a generic with a specific type such as this straw man name: Collection<Int>.

I like the future directions of potentially being able to use generics to constrain the returned Opaque Type's associated types etc. But even without that at this time I feel this has enough value it should be added.

I am fine with the use of some but would prefer the use of opaque. I haven't been a fan of the other names I've seen pitched, as I don't feel like they add enough clarity over the simplicity and clarity of opaque.

2 Likes
91
  • What is your evaluation of the proposal?

+1 but largely prefer opaque over some keyword. In most of the discussions it's referred to as an "opaque [return/result] type" not a "some type" which sounds too close to "sum type" anyway.

  • Is the problem being addressed significant enough to warrant a change to Swift?

Yes, even more so when generic constraints and typealiases are added.

  • Does this proposal fit well with the feel and direction of Swift?

Yes, see above.

  • If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

Nothing quite the same. Base class wrapper encapsulation is the closest thing but this seems far more light weight and supports more than just reference types.

  • How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

Read all the things since the first pitch a ways back.

3 Likes
92

One question that still bugs me is related to this, "There are some mitigations for source compatibility, e.g., a longer deprecation cycle for the types or overloading the old signature (that returns the named types) with the new signature (that returns an opaque result type)."

How long of a deprecation cycle could be expected in the former case? We still supported building to iOS clients back to iOS 9 until just recently and I imagine there's apps written in Swift that still go back even before that. 4-5 years from now I imagine plenty of apps will still need to support the version of iOS that Swift 5.0 shipped with.

3 Likes
93

After mulling this over in the back of my head this week, hereā€™s a late-breaking addendum to my earlier review.

I expressed concerns about confusion and cognitive load. I notice this was a common theme in the responses: people find this feature hard to reason about.

Much of the discussion focused on the word some, but perhaps the real problem is anonymity. This proposal creates many mutually incompatible anonymous types that have the same spelling (some Foo) in code.

The problem is perhaps most obvious in this poetically bollixing error message:

cannot assign value of type '(__opaque opaque.(file).g()@/tmp/opaque.swift:8:6)' to type '(__opaque opaque.(file).f()@/tmp/opaque.swift:4:6)'

Iā€™m sure a release version of the compiler could generate a better error message, but the difficulty of expressing that error well may signal the deeper underlying problem that their anonymity makes these types communication-proof: hard to talk about, hard to think about.

An alternative to consider: make the opaque type aliases described under Future Directions the only way to declare opaque types. This gives each distinct opaque type a distinct name, which has several advantages:

  • It makes it easier for API consumers to reason about when and why opaque types are distinct.
  • It makes compiler errors much easier to understand.
  • It helps API authors reason about how many distinct types they are exposing.
  • The current proposal (1) gives no way to express that different functions return the same opaque type, and (2) AFAICT gives no way to pass an opaque type returned from a library to other functions, or even store it in a variable with a type annotation, without using an existential. This rules out many rudimentary refactoring and abstraction techniques, and will be an immediate source of pain. Named types could help address both these problems.
  • This could conceivably address some of the use cases for which people have wanted newtype.

There are disadvantages too:

  • Itā€™s more verbose.
  • It leads to a proliferation of names.
  • It may not serve the future scenario Joe envisions of passing some P as a parameter.

To be clear, Iā€™d still vote +1 for the proposal as is, in deference to others who understand the problem better than I do. I mention this as something to consider if the current proposal just isnā€™t sitting quite right.

15 Likes
94
  • What is your evaluation of the proposal?

After some (healthy?) scepticism, Iā€™ve turned around and like the idea of opaque types. Iā€™m still rooting for generalised existential types but opaque types look like an orthogonal feature to GETs. However, Iā€™m sceptical about the implementation and syntax, especially the terseness.

Until now, Swift has always required providing the full signature of a function declaration. The main reason, if I understand it correctly, has been to mitigate the complexity traps that type inferencing in a language with significant amounts of polymorphism entails. Another reason Iā€™ve read often, however, is code readability. Type inference is great for terse property declarations but functions should be ā€œhuman-parseableā€ without having to look in the body.

The current implementation of ORTs, as I understand it, is that the compiler infers the actual return type from all return statements, i.e, the most specific type over every return expression that matches the ORTs constraints. Itā€™s a more limited type of type inference (the constraints are given) unless the ORT is some Any in which case itā€™s just full-scale type inference.

Iā€™m unfamiliar with the type-checker implementation for ORTs but it seems that this re-introduces the type inferencing problem. Has type checker performance with extended use of ORTs (like in the stdlib) been tested on this?

My greater concern, however, is readibility. I understand consumers of an API donā€™t need to know the concrete type ā€”thatā€™s exactly the point of ORTsā€” but as a library designer Iā€™d very much like to know what types Iā€™m creating and returning in my functions. The compiler guarantees the returned type conforms to the ORT constraints but I want to be sure, as the implementor, that my function creates and returns the right concrete type ā€” for performance reasons, for example.

Itā€™s hard to come up with example for a feature that I havenā€™t been able to use in practice yet but Iā€™m thinking along the lines of a lazy collection type that can be initialised by array literal. Iā€™d declare some Collection but Iā€™d really want LazyFrobulatingCollection<T> instead of [T]. This implementation detail should not matter outside of the function body but it very much matters when Iā€™m writing (and reading) the function. To force this inference, one would just need a single as LazyFrobulatingCollection but this is optional and itā€™s hidden among other lines of code in the body. IMHO, the function declaration is an ideal central place for this information, as it has been until now.

To summarise, I want ORTs to be used as a type hiding mechanism for API clients and not as a way to bring type inference to function declarations. The implementor should still write the type name in full like now but the compiler would hide that information at the API boundary. We still avoid the gigantic type expressions we get from chaining lazy collection operations (which is my first thought when I think of good applications for ORTs).

Iā€™m not sure what is a good syntax to express an ORT and an explicit concrete type at the same time but here goes one (straw-man) proposal:

public struct SomeCollection<Element> {
    public opaque typealias FilterCollection = MyFilteringCollection<Element> where FilterCollection : Collection, FilterCollection.Element == Element
    public func filter(where predicate: ā€¦) -> FilterCollection {
        ā€¦
        return .init(ā€¦)    // It works because compiler knows the concrete FilterCollection
    }
}

An (unintended) consequence is that the same opaque type can be used across declarations on the same concrete type. We can forbid this or we can treat them as unequal across uses but I donā€™t see a problem with allowing multiple declarations to return values that have the same but opaque type to the client. (For API resilience, it wouldnā€™t be possible to split an opaque typealias into two. Merging should work though.)

A variant of my straw-man implementation and syntax is listed as a possible future direction but Iā€™d argue to go straight to that and not allow some in the return type of a method declaration, for the reasons stated above.

  • Is the problem being addressed significant enough to warrant a change to Swift?

Iā€™d argue the issues with deep type expressions and protocol-oriented programming are well-known, even for all the benefits they provide (static typing, optimisation opportunities, etc.).

I welcome every feature to make protocol-oriented programming easier in Swift that doesnā€™t compromise Swiftā€™s strengths. This is one of them, and certainly after a few more refinements.

  • Does this proposal fit well with the feel and direction of Swift?

In its current form it introduces terseness and might make implementations harder to read. Swift strives to make code easy to read, even at the expense of more keystrokes.

Having said that, ORTs make code easier to write for API consumers, even in its current form. While looking up documentation for compactMap, Iā€™d rather see

opaque typealias CompactMapCollection<E> where CompactMapCollection : Collection, CompactMapCollection.Element == E
func compactMap<ElementOfResult>(_ transform: @escaping (Elements.Element) -> ElementOfResult?) -> CompactMapCollection<ElementOfResult>

or even

func compactMap<ElementOfResult>(_ transform: @escaping (Elements.Element) -> ElementOfResult?) -> some Collection where .Element == ElementOfResult

than

func compactMap<ElementOfResult>(_ transform: @escaping (Elements.Element) -> ElementOfResult?) -> LazyMapSequence<LazyFilterSequence<LazyMapSequence<Elements, ElementOfResult?>>, ElementOfResult>

Also, itā€™s a feature orthogonal to any generalised existential type features that might come in a future version of Swift. They donā€™t conflict and have different applications.

  • If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

Iā€™m vaguely familiar with Rustā€™s impl Trait feature but havenā€™t really used it. Swift has some unique ideas around protocol-oriented programming and thus has its own challenges but a lot of features are an amalgamation of features in other languages.

The closest I can think of myself for ORTs is Javaā€™s collection class hierachy. Java hides concrete return types by declaring a superclass or interface return type (like List instead of ArrayList) but these are more like GETs: one can return a LinkedList somewhere else in the method. ORTs provide more capabilities than GETs though by being more restrictive ā€” the SE document specifies some examples. GETs have their own applications.

  • How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

Iā€™ve read the proposal several times and Iā€™ve tried keeping up with the communityā€™s response to it.

6 Likes
95

I've mostly stayed out of this thread, but this is exactly my concern. Swift has gone out of its way to have declarations for types (structural types excepted), which is important to the linkage and compilation model, but also the diagnostic and user model. Specifically, it is very problematic to me that these types cannot be named in a natural way, but their identity matters. This is what makes them different than structural types.

I think this approach would be far better: "Opaque type aliases" as a feature seems like it would solve exactly the same set of problems (and the type aliases can have generic constraints already etc) but would provide a name and identity for the type that can be used across declarations within client code. Also, the "future directions" in this proposal seem like they will add a significant amount of language complexity when/if they happened, whereas they would naturally fall out of opaque type aliases.

I'd strongly +1 that direction over the proposal as is.

-Chris

23 Likes
96

@Douglas_Gregor, @Joe_Groff as the proposal authors, have you considered the approach described above, where this problem is solved by introducing the ability for API authors to "opaquize" the concrete type that a typealias is defined by?

It seems that the type checker implementation details would be effectively the same as your current implementation, but would lead to a much simpler and more extensible user model, that generalizes to your "future directions" in a better way.

-Chris

14 Likes
97

I don't think it's difficult to produce a better error message. I haven't gotten far enough in the implementation to work on improving diagnostics yet. I would like to have it recognize when you're trying to match two independent opaque types and provide a message more like:

error: return types of 'g' and 'f' may have different underlying types
note: 'g()' refers to 'g', declared here
note: 'f()' refers to 'f', declared here
1 Like
98

We explored that direction before proposing this, and we don't object to it as a future direction. As you noted, it would largely be an extension of the same underlying implementation. I think that, even if we had opaque typealiases, it will still be the common case when using this feature to have a unique opaque return type per function, and so being required to declare the typealias and keep it in sync with the associated function's implementation would be pure boilerplate. Opaque typealiases are also a strictly more complex feature, since they raise more interesting questions about the visibility of the underlying type, whereas when the opaque type is constrained to a single definition, it is obviously contained to the body of that definition. On balance, we decided that opaque return types are a more fundamental part the "tech tree" for this feature, and that typealiases would make more sense as an extension.

99

Yes, clearly a better error message. Still a bit puzzling, but light years better.

To be clear, my argument was not that the error message is a deal-killer, but rather that its difficulties might flag more fundamental confusion stemming from these types being anonymous.

My suggestion ā€” which is truly only a suggestion ā€” is that this boilerplate may in fact have utility in making opaque types easier to reason about and communicate about, and thus might make them more acceptable to the community.

3 Likes
100

Indeed. Now, if anonymous opaque types were only used in itā€™s most basic form, and typealiases used always when ā€whereā€ -constraints or other rules were needed, then that would be more palatable. But Iā€™m afraid anonymous opaque types wouldnā€™t be useful enough in that situation. And if anonymous opaque types need the full set of rules, then there needs to be this whole big bag of new weird syntax just because of anonymous types.

I could see more pressing need for having full featured anonymous types if opaque types were highly frequently used in code, but itā€™s not and is it really that hard to maintain a few typealiases?