SE-0244: Opaque Result Types

I was in the same boat, but after some reading up, I believe the difference is as follows. I'll use these functions as an example:

func one() -> Equatable { ... }
func two() -> some Equatable { .... }

The return type of one can be of any type that is Equatable. Different invocations of one may return instances of different types. The function only guarantees it will return an Equatable, not that it will return the same type every time. The return type of two is some type that is Equatable. Unlike one, it's the same type every time; it's just hidden from the user.

As a result, instances returned from one will have to use dynamic dispatch, as the compiler doesn't know the exact type, so the decision has to be made at runtime. Instances returned from two can use static dispatch if the compiler can infer the concrete type from the implementation.

So:

  • If you want a function that can decide which type to use at runtime, use one (this is regular OO polymorphism).
  • If you want to hide the type you return because it's an implementation detail and shouldn't be exposed, but you want to avoid the overhead of dynamic dispatch, use two.

I hope that's a correct summary. If not, someone will surely correct me :slight_smile:

4 Likes

I am surprised the alternatives considered section only includes two names here, when many many others were discussed over on the pitch thread.

I have the same position as before - this is an obscure feature that most people will not see very often, and is not a core part of the language that all users should be expected to know to be effective in Swift. Therefore, it makes sense to use a long (and probably compound) keyword for this concept. Using a short name harms 'web searchability' and reduces the odds that someone (who doesn't already know what is going on) can figure out what they keyword is doing without doing that research.

This is not going to show up pervasively in Swift code, I see no advantage to huffman encoding it down to a 4 character identifier, I'd rather see a 15 character compound name of some sort.

It would also be nice for the proposal to be revised to include all the alternatives that the pitch thread considered.

-Chris

23 Likes

Here are some examples proposed in the pitch thread in use of an opaque return type. I'm not assuring that it's complete or accurate, but it's an attempt to throw things at the wall for discussion.

For those that are having trouble with visual connotations of words like opaque or hazy, remember that's going to happen with any colloquial word before one understands it's purpose and use. If you didn't know lazy was a term-of-art, your brain will try to imply sloth on a lazy Collection of OfficeWorkers. It may take time for one to read code that calls for an opaque Shape rather than an opaqueShape but with use the visual connotations will fall away. On the positive side of a colloquial word or phrase, it should be easy to look up and less likely for conflict with future words Swift might want to reserve for later.

Throwing things against the wall from the pitch thread:

func translucentRectangle() -> opaque Shape { ... }

func translucentRectangle() -> unspecified Shape { ... }

func translucentRectangle() -> anyOld Shape { ... }

func translucentRectangle() -> hazyButSpecific Shape { ... }

func translucentRectangle() -> someSpecific Shape { ... }

func translucentRectangle() -> someConcrete Shape { ... }

func translucentRectangle() -> nonspecific Shape { ... }

func translucentRectangle() -> unclear Shape { ... }

func translucentRectangle() -> arbitrary Shape { ... }

func translucentRectangle() -> someThing Shape { ... }

func translucentRectangle() -> anonymized Shape { ... }

func translucentRectangle() -> nameless Shape { ... }

func translucentRectangle() -> unnamed Shape { ... }

7 Likes

It looks like you missed:

func translucentRectangle() -> doNotLookBehindTheCurtain Shape { ... }
4 Likes

We might as well keep the original ‘opaque’ name. It was fine by me.

Indeed, this is pretty much what opaque types boil down to, is the ability for a function to "output" generic parameters in addition to taking them as inputs. Another way we could conceivably notate this is by putting the output parameters after the ->, as a number of people suggested:

makeFoo() -> <F: Foo> F

The proposal tries to explain it, though the section about it is quite lengthy and I can see how it can be unclear. @Douglas_Gregor, maybe we should go over this section again and try to make it more to the point. @svanimpe's description is a good summary.

I understand the concern about searchability, and I checked—Google does a fine job providing relevant answers for swift some today. (There's a legitimate concern that having two somes could dilute the waters, I agree.) I however don't agree that this feature will remain obscure for long. The equivalent impl Trait feature in Rust is used pervasively; On Github I find 5,000 hits for impl Fn arguments and returns, 1,000 for impl Iterator<Item=...>, and 400 for impl Future alone. Although Swift uses type erasure for function types, obviating the first to some degree, we could deploy this feature for standard library collections as the proposal suggests, and we could well end up with protocol-oriented futures, key paths (and maybe even generic function objects like Rust's, since there are optimizations that enables that our current closure design makes difficult) in the future. Part of the goal of this feature is to make those sorts of libraries easier and more ergonomic to design and use.

As for "Huffman coding" the identifier, I have a couple other reasons I think going for something concise that reads well is a good idea. I'd really like to see this feature eventually extend to also allow generic arguments to be written as some Protocol, the same way that Rust lets you use impl Trait as an argument, since that will make writing a lot of simple generic functions clearer and easier. For this sugar to be worth using, it ought to be at least as concise and readable as the syntax it replaces, and a bulky, obscure keyword would kill that quickly. Furthermore, I also see a bit of a let vs var situation with regard to existential notation. @dabrahams and others have noted that protocol existentials in their full generality quickly become a rather obscure, complex feature, that doesn't necessarily deserve the pride of place it currently has in our type syntax (and that if anything, opaque types as provided by this feature are in fact closer to what you really want a good percentage of the time we currently tell people to use existentials). Rust eventually decided that making existentials undecorated was a bad idea and added the dyn keyword to encourage their use to be explicit. In previous discussions of existential syntax, Any or any seemed like the clear favorite as the modifier keyword for explicitly introducing an existential. Both linguistically, and in the spirit of fairness of not syntactically biasing one style over the other, Some or some strikes me as the corresponding modifier for specific opaque types.

14 Likes

+1 for the concept, but -1 for the syntax because it doesn't allow explicit constraints, and doesn't scale to multiple opaque types. I quite liked the syntax proposed in the original discussion thread by @anthonylatsis (here: Opaque result types - #88 by anthonylatsis) which explicitly named the opaque "thing" so it could be constrained in a where clause. To summarize what I think the linked post was getting at, it would look like:

func foo() -> some T
    where T: Collection, T.Element == Int
{ 
    return [1, 2, 3]
}

This has several advantages. For one, it doesn't require any new syntax for the constraints and instead re-uses the existing where clause that is already familiar. This allows it to scale to multiple types:

func foo() -> (some T, some Q)
    where T: Collection, T.Element == Int, Q: Numeric
{ 
    return ([1, 2, 3], 4)
}

And finally it allows opaque constraints to mix with generic constraints (composability!) which I think will be really important for the opaque type concept in general:

func foo<R>(input: R) -> some T
    where T: Collection, T.Element == R
{ 
    return [input]
}

I would of course support type inference to allow omission of some or all of the above constraints if they can be inferred from the return type:

func foo() -> some T: Collection
{ 
    // T.Element == Int is inferred
    return [1, 2, 3]
}
2 Likes

Since part of the intent of this feature is to provide more control over the interface a function represents, I don't think it would be a good idea to infer constraints that weren't written. For instance, while you may want to expose the Element type of the returned collection, you most likely don't want to expose its Index, since in most cases that'd stick you with one underlying collection implementation in practice.

6 Likes

That's a really good point, and I agree but correct me if I misunderstand but the proposal as written would infer both Index and Element form the return type, wouldn't it?

As written, nothing is inferred; it doesn't yet propose a way to describe constraints on the returned type. Your notation would be one way to do so; other approaches are being discussed in this thread:

I guess I have a fundamental misunderstanding of how this works. From the original proposal:

func makeMeACollection<T>(with element: T) -> some MutableCollection & RangeReplaceableCollection {
   return [element] // ok: an array of T satisfies all of the requirements
}

Does this not infer .Element == T for the underlying type? If not, how can the program actually compile?

I guess I'm worried that since this proposal explicitly does not name the opaque thing, by accepting it we are tacitly vetoing any syntax which seeks to do so. I think the syntax for constraining opaque types needs to be nailed down before accepting a proposal introducing opaque types.

1 Like

It works similar to what you'd get if you passed in a generic argument constrained to <T: MutableCollection & RangeReplaceableCollection> with no Element constraints. You know that the value has an Element type, and that it's consistent among values returned by the same function, but you don't know exactly what type it is.

A number of people suggested mirroring the generic argument syntax on the other side of the <>, which seems nice and uniform with generic arguments:

func makeMeACollection<T>(with element: T) -> <C: MutableCollection & RangeReplaceableCollection> C
   where C.Element == T

and could allow the some syntax to still be used in either argument or return position in common cases where there are no secondary constraints among arguments or results.

4 Likes

I went back to the proposal and found this:

Following the some keyword is a class, protocol, Any , AnyObject , or composition thereof (joined with & ).

It's maybe a degenerate case, but some Any doesn't seem to read too well. :slight_smile:

I notice that list of things that can follow the keyword is more or less the same list that can follow a : indicating type conformance (e.g. struct T: P). If you want to consider something longer than some, what about one of these:

    … -> conformanceTo P
    … -> typeConformingTo P

or something along that line?

I find this extremely compelling.

In general, I am reticent about new syntax or decorators that doesn’t immediately fit into the status quo, because it gives unnecessary ammo to “Swift is impossibly big to learn” criticisms. With this explanation, some feels of-a-part; it will be acceptable in Swift today, composes well with future proposals, and will eventually feel at home as it integrates with (or subsumes) our generics model overall.

I know y’all don’t want to commit to future plans, but some of the thinking here should definitely be explained in the proposal text and/or its Future Directions section. This explanation helped me close the loop on how this would fit into the overall generics picture that I wasn’t getting in the pitch thread. +1

6 Likes
  • What is your evaluation of the proposal?

A general +1 from me. I’m not an expert on the topic but there has been more than one occasions where this feature would have helped.

  • Is the problem being addressed significant enough to warrant a change to Swift?

Yes. In general I think every little step to an improved type system (see the generics manifesto) is good for the language expressivity in the long term. Sometimes I needed to just return an instance of a protocol without really caring about the specific types and it gets cumbersome really quickly.

  • Does this proposal fit well with the feel and direction of Swift?

The proposal expresses the problem really well and it’s really well written. I have to admit that every time I read it I start thinking that it would be nicer to just “-> P” instead of including a new keyword, but after reading it I’m convinced of the reason. Is still tricky to grasp the full theory of this type system functionalities and how it compares with generalised existential types. Luckily the proposal gives good examples and clarifies how this features differ.

That said I prefer opaque over some. While reading the code “some” seems to be too ambiguous to me, it doesn’t really seem so different as “any”. “opaque” seems much more appropriate to me and much more clear. Maybe is because it reminds me to when you refer to opaque return values on C. But I guess I can get used to “some” if native speakers and language experts say is more appropriate.

A concern I have is about the future directions needing more syntax changes. I’m concern we put ourselves in a corner in terms of syntax, and that if we would look into it holistically we may find a better solution. Again, if more knowledgable people says is fine let’s go for it, I just wanted to raise my concern that maybe is worth taking a look at those future directions now.

Somewhat tangent to the proposal but I think is important is the error messages around this feature. Type errors can be really confusing (specially in Swift early days) and it would be unfortunate to go back to that state with this feature, specially since opaque types don’t have names. Even for experienced developers some times parsing error messages gets hard.

Finally, is there any concern about type inference? Sometimes type inference can’t resolve some expressions and adding types helps the compiler, but in this case there won’t be a type to add so maybe the user is stuck? Just throwing it out there.

  • If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

The Rust community has gone trough similar path for their type system. I’m glad the proposal is aware of this and acknowledges their learnings.

  • How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

I’ve been following the pitch thread since it was posted and I’ve read the proposal multiple times. I haven’t tried the toolchain.

Why not simply go with this:

func someFunction<T, U: MutableCollection>(_ t: T) -> U {
    // ...
}

This already has meaning today; it means that the caller gets to pick what U is, and the implementation must be able to return any MutableCollection a caller asks for. An opaque type is different; U is picked by the implementation. It's more like <U> is returned from the function along with the return value:

func someFunction<T>(_ t: T) -> <U: MutableCollection> U
9 Likes

Thank you for this! I must admit that I read the proposal in its entirety, and only this explanation finally helped me wrap my head around the idea.

5 Likes

So... some for the first case and someSpecific for the second?

(I'm half joking, but only half)

I'm new here. My apologies if I make any mistakes.

I saw this proposal on the Swift evolution list on github, and joined just so I could contribute.

Happily, I see that @orobio is already saying much the same thing I was planning to say, though I'd prefer a slightly different syntax.

But it seems to me that a lot of people are ignoring what he's really getting at.

To me, the crux of his suggestion is that it inherently includes a name for the opaque type. Without that, it's difficult to make it clear whether two instances of the opaque type are the same or not. And I believe both answers are sometimes what's desired. Also, without a name for the type, it is difficult to include all the things you might want to say in a where clause.

The main problem I have with his syntax is that using a caret "^" is visually lightweight. Opaque types aren't going to be very common, and I would prefer that they stand out visually. (On my first reading of @orobio's suggestion, I failed to notice the "^" at all.)

In addition, if this is in some sense a dual of a regular generic argument, I think it would be easier for people to understand if it looks's both like and unlike regular generics. So I'd suggest something like:

func makeFoo<<T: Foo>>() -> T { /* ... */}

(In fairness, @orobio did mention something a little like this as a possibility, but it seems to have gone by the wayside.)

This would show that it's similar to regular generics, but different from them. That would be my preference.

It would make is easy to implement the usual where clause syntax, to constrain the type in all the usual ways. And it becomes simple to say both:

func makeTwoTheSame<<T: Foo>>() -> (T, T) { /* ... */}

and

func makeTwoPotentiallyDifferent<<T: Foo, U: Foo>>() -> (T, U) { /* ... */}

It also allows:

func convenienceFunction<<Something: Foo>>() -> Something {
    let aFoo = makeFoo()
    return aFoo
}

And, perhaps even better, you can say:

struct SomeType<<T: Foo>> {
    typealias MyFoo T
    static func makeFoo() -> MyFoo { /* ... */}
}

let anotherFoo: Sometype.MyFoo = SomeType.makeFoo()

(I think this will be more commonly used inside a type like this, rather than in a free function.)

I don't show any examples with a where clause, or something like returning two possibly different kinds of collections, which have the same element types, and so forth, but I think it's clear how you could say things like that. It gives the author of a function returning an opaque type the flexibility to be very clear about what is (and is not) being promised.

Now, I currently know nothing about the inside of the compiler, so I don't know how hard this would be to implement, but if this is really just about the same thing as a regular generic, except with the "inside" and the "outside" roles reversed, it doesn't seem to me that it should be unreasonably difficult. (Famous last words!)

I do understand @Chris_Lattner3's preference for something that's easy to search for, but I still prefer this to having any keyword at all. But if it's felt that there absolutely does need to be a keyword, how about something like:

func makeFoo() -> sometype T: Foo { /* ... */}

where a type name must be provided. And then additional uses of the same type could just refer to "T".

<><><><><>

Answering the initial question that started this thread...

I think the general idea is OK if it's extended to always include a name for the type, but without that I think it would paint Swift into a corner. I'd also prefer that it include all the where clause syntax from the start, because otherwise it feels too limited.

As to whether it's really important enough to be worth the change, I'm really not certain. I think folks who have worked on something like the standard library would have a better perspective on that than I do.

I think it fits fairly well with Swift, especially if it used something similar to the syntax I suggested.

I spent several days thinking about this before writing this. I'd think for longer if the deadline weren't fast approaching.

2 Likes