SE-0244: Opaque Result Types

[Tino]

@Douglas_Gregor

I personally don't feel that we need to re-use existing keywords for this feature: it's a new idea and should get it's own keyword, and our implementation is capable of treating this as a contextual keyword to avoid breaking code.

In general, I agree that re-use of existing terms to minimize the number of keywords doesn't make sense (I wouldn't propose enum instead of some ;-).
But I think as is less ambiguous for those without prior knowledge of Swift, and also for existing users:
At least for superclass-relationships, as can already be used to achieve a transformation like the one needed for opaque types (also I guess the protocol-case is much more important for the pitch).

(I consciously posted in the discussion-thread; I don't mind the text being moved over, but pease don't complain about distraction from the review ;-)

[orobio (Manolo van Ee)]

@koher

I think generic arguments and opaque result types are dual . For example, for the following two functions,

func useFoo<F: Foo>(_ foo: F) { /* ... */ }
func makeFoo() -> some Foo { /* ... */ }

• users of useFoo decide a concrete Foo type and implementers of the function use an abstract Foo type
• implementers of makeFoo decide a concrete Foo type and users of the function use an abstract Foo type

Both of them are counterparts of existential types. They have similar roles for arguments and return values respectively.

@Joe_Groff

func useFoo(_ foo: some Foo) { /* ... */ }
func makeFoo() -> some Foo { /* ... */ }

making the duality more apparent.

Exactly this duality tells me we need to design a counterpart to generics. I don't know if there is an existing term for that. Perhaps 'reverse generics'? I wil use this term for the rest of this post. Please correct me if that doesn't make any sense.

Refining my earlier idea a little bit, using the ^ in the generic parameter list to designate reverse generic parameters:

func useFoo<F: Foo>(_ foo: F) { /* ... */ }
func makeFoo<^F: Foo>() -> F { /* ... */ }

So, to be clear:

• Normal generic type parameters are made concrete by the caller.
• Reverse generic type parameters (prefixed ^) are made concrete by the callee.

Type inference will most of the time do its job, but explicitly typing something as the reverse generic type parameter of makeFoo() could perhaps be done as follows:

// Explicitly typing a variable with the return type of makeFoo():
var myFoo: makeFoo.F = makeFoo()

// Explicitly typing an array of the type returned by makeFoo():
var myFooArray = [makeFoo.F]()
myFooArray.append(makeFoo())

This notation also makes it possible to constrain functions to return the same concrete type, which could make @Vogel happy:

// Returns an opaque collection
func makeCollection<T, ^C: Collection>(with element: T) -> C where C.Element == T {
  return [element]
}

// Returns the same type opaque collection, with Element == String
func makeCollectionWithString<^C>() -> C where C == makeCollection.C, C.Element == String {
  return makeCollection(with: "Foo")
}

// The following will type check correctly:
var myCollection = makeCollectionWithString() + makeCollection(with: "Bar")
print(myCollection) // ["Foo", "Bar"]

Using this notation there is no need for a keyword like 'opaque' or 'some'. I see that @beccadax posted a similar idea in the Protocol<.AssocType == T>... thread and he notes that 'some' could be used as shorthand syntax. In my opinion the following would be the right course of action:

1. Decide on the basic design for a 'reverse generics' system.
2. See how ergonomics can be improved with additional shorthand syntax (if there is a need for it).

I think this will provide a better foundation for future directions.

I'm no language expert, so if I said anything that makes no sense or if I butchered any terminology, please let me know.

[ryansobol]

Opaque types are new to me. As such, one of the aspects if this proposal I’m still having trouble understanding is under what scenarios is an opaque type is preferred over an existential type. Would anyone be willing to write up a brief ELI5 summary?

[svanimpe (Steven Van Impe)]

@ryansobol

Would anyone be willing to write up a brief ELI5 summary?

I was in the same boat, but after some reading up, I believe the difference is as follows. I'll use these functions as an example:

func one() -> Equatable { ... }
func two() -> some Equatable { .... }

The return type of one can be of any type that is Equatable. Different invocations of one may return instances of different types. The function only guarantees it will return an Equatable, not that it will return the same type every time. The return type of two is some type that is Equatable. Unlike one, it's the same type every time; it's just hidden from the user.

As a result, instances returned from one will have to use dynamic dispatch, as the compiler doesn't know the exact type, so the decision has to be made at runtime. Instances returned from two can use static dispatch if the compiler can infer the concrete type from the implementation.

So:

• If you want a function that can decide which type to use at runtime, use one (this is regular OO polymorphism).
• If you want to hide the type you return because it's an implementation detail and shouldn't be exposed, but you want to avoid the overhead of dynamic dispatch, use two.

I hope that's a correct summary. If not, someone will surely correct me

[Chris_Lattner3 (Chris Lattner)]

@sveinhal

I do like the direction this is going, but I feel strongly that the word some is cutesy and unnecessary.

I am surprised the alternatives considered section only includes two names here, when many many others were discussed over on the pitch thread.

I have the same position as before - this is an obscure feature that most people will not see very often, and is not a core part of the language that all users should be expected to know to be effective in Swift. Therefore, it makes sense to use a long (and probably compound) keyword for this concept. Using a short name harms 'web searchability' and reduces the odds that someone (who doesn't already know what is going on) can figure out what they keyword is doing without doing that research.

This is not going to show up pervasively in Swift code, I see no advantage to huffman encoding it down to a 4 character identifier, I'd rather see a 15 character compound name of some sort.

It would also be nice for the proposal to be revised to include all the alternatives that the pitch thread considered.

-Chris

[WilliamMoss]

Here are some examples proposed in the pitch thread in use of an opaque return type. I'm not assuring that it's complete or accurate, but it's an attempt to throw things at the wall for discussion.

For those that are having trouble with visual connotations of words like opaque or hazy, remember that's going to happen with any colloquial word before one understands it's purpose and use. If you didn't know lazy was a term-of-art, your brain will try to imply sloth on a lazy Collection of OfficeWorkers. It may take time for one to read code that calls for an opaque Shape rather than an opaqueShape but with use the visual connotations will fall away. On the positive side of a colloquial word or phrase, it should be easy to look up and less likely for conflict with future words Swift might want to reserve for later.

Throwing things against the wall from the pitch thread:

func translucentRectangle() -> opaque Shape { ... }

func translucentRectangle() -> unspecified Shape { ... }

func translucentRectangle() -> anyOld Shape { ... }

func translucentRectangle() -> hazyButSpecific Shape { ... }

func translucentRectangle() -> someSpecific Shape { ... }

func translucentRectangle() -> someConcrete Shape { ... }

func translucentRectangle() -> nonspecific Shape { ... }

func translucentRectangle() -> unclear Shape { ... }

func translucentRectangle() -> arbitrary Shape { ... }

func translucentRectangle() -> someThing Shape { ... }

func translucentRectangle() -> anonymized Shape { ... }

func translucentRectangle() -> nameless Shape { ... }

func translucentRectangle() -> unnamed Shape { ... }

[Avi]

It looks like you missed:

func translucentRectangle() -> doNotLookBehindTheCurtain Shape { ... }

[hartbit (David Hart)]

We might as well keep the original ‘opaque’ name. It was fine by me.

[Joe_Groff]

@orobio

Exactly this duality tells me we need to design a counterpart to generics. I don't know if there is an existing term for that. Perhaps 'reverse generics'? I wil use this term for the rest of this post. Please correct me if that doesn't make any sense.

Indeed, this is pretty much what opaque types boil down to, is the ability for a function to "output" generic parameters in addition to taking them as inputs. Another way we could conceivably notate this is by putting the output parameters after the ->, as a number of people suggested:

makeFoo() -> <F: Foo> F

@ryansobol

Opaque types are new to me. As such, one of the aspects if this proposal I’m still having trouble understanding is under what scenarios is an opaque type is preferred over an existential type. Would anyone be willing to write up a brief ELI5 summary?

The proposal tries to explain it, though the section about it is quite lengthy and I can see how it can be unclear. @Douglas_Gregor, maybe we should go over this section again and try to make it more to the point. @svanimpe's description is a good summary.

@Chris_Lattner3

I have the same position as before - this is an obscure feature that most people will not see very often, and is not a core part of the language that all users should be expected to know to be effective in Swift. Therefore, it makes sense to use a long (and probably compound) keyword for this concept. Using a short name harms 'web searchability' and reduces the odds that someone (who doesn't already know what is going on) can figure out what they keyword is doing without doing that research.

This is not going to show up pervasively in Swift code, I see no advantage to huffman encoding it down to a 4 character identifier, I'd rather see a 15 character compound name of some sort.

I understand the concern about searchability, and I checked—Google does a fine job providing relevant answers for swift some today. (There's a legitimate concern that having two somes could dilute the waters, I agree.) I however don't agree that this feature will remain obscure for long. The equivalent impl Trait feature in Rust is used pervasively; On Github I find 5,000 hits for impl Fn arguments and returns, 1,000 for impl Iterator<Item=...>, and 400 for impl Future alone. Although Swift uses type erasure for function types, obviating the first to some degree, we could deploy this feature for standard library collections as the proposal suggests, and we could well end up with protocol-oriented futures, key paths (and maybe even generic function objects like Rust's, since there are optimizations that enables that our current closure design makes difficult) in the future. Part of the goal of this feature is to make those sorts of libraries easier and more ergonomic to design and use.

As for "Huffman coding" the identifier, I have a couple other reasons I think going for something concise that reads well is a good idea. I'd really like to see this feature eventually extend to also allow generic arguments to be written as some Protocol, the same way that Rust lets you use impl Trait as an argument, since that will make writing a lot of simple generic functions clearer and easier. For this sugar to be worth using, it ought to be at least as concise and readable as the syntax it replaces, and a bulky, obscure keyword would kill that quickly. Furthermore, I also see a bit of a let vs var situation with regard to existential notation. @dabrahams and others have noted that protocol existentials in their full generality quickly become a rather obscure, complex feature, that doesn't necessarily deserve the pride of place it currently has in our type syntax (and that if anything, opaque types as provided by this feature are in fact closer to what you really want a good percentage of the time we currently tell people to use existentials). Rust eventually decided that making existentials undecorated was a bad idea and added the dyn keyword to encourage their use to be explicit. In previous discussions of existential syntax, Any or any seemed like the clear favorite as the modifier keyword for explicitly introducing an existential. Both linguistically, and in the spirit of fairness of not syntactically biasing one style over the other, Some or some strikes me as the corresponding modifier for specific opaque types.

[fractal]

+1 for the concept, but -1 for the syntax because it doesn't allow explicit constraints, and doesn't scale to multiple opaque types. I quite liked the syntax proposed in the original discussion thread by @anthonylatsis (here: Opaque result types - #88 by anthonylatsis) which explicitly named the opaque "thing" so it could be constrained in a where clause. To summarize what I think the linked post was getting at, it would look like:

func foo() -> some T
    where T: Collection, T.Element == Int
{ 
    return [1, 2, 3]
}

This has several advantages. For one, it doesn't require any new syntax for the constraints and instead re-uses the existing where clause that is already familiar. This allows it to scale to multiple types:

func foo() -> (some T, some Q)
    where T: Collection, T.Element == Int, Q: Numeric
{ 
    return ([1, 2, 3], 4)
}

And finally it allows opaque constraints to mix with generic constraints (composability!) which I think will be really important for the opaque type concept in general:

func foo<R>(input: R) -> some T
    where T: Collection, T.Element == R
{ 
    return [input]
}

I would of course support type inference to allow omission of some or all of the above constraints if they can be inferred from the return type:

func foo() -> some T: Collection
{ 
    // T.Element == Int is inferred
    return [1, 2, 3]
}

[Joe_Groff]

@fractal

I would of course support type inference to allow omission of some or all of the above constraints if they can be inferred from the return type:

func foo() -> some T: Collection
{ 
    // T.Element == Int is inferred
    return [1, 2, 3]
}

Since part of the intent of this feature is to provide more control over the interface a function represents, I don't think it would be a good idea to infer constraints that weren't written. For instance, while you may want to expose the Element type of the returned collection, you most likely don't want to expose its Index, since in most cases that'd stick you with one underlying collection implementation in practice.

[fractal]

@Joe_Groff

Since part of the intent of this feature is to provide more control over the interface a function represents, I don't think it would be a good idea to infer constraints that weren't written

That's a really good point, and I agree but correct me if I misunderstand but the proposal as written would infer both Index and Element form the return type, wouldn't it?

[Joe_Groff]

As written, nothing is inferred; it doesn't yet propose a way to describe constraints on the returned type. Your notation would be one way to do so; other approaches are being discussed in this thread:

Protocol<.AssocType == T> shorthand for combined protocol and associated type constraints without naming the constrained type

Swift's notation for generic constraints generally requires naming the things being constrained; you say that a particular generic parameter conforms to a protocol by naming the generic parameter and the protocol it conforms to, like <C: Collection>, and you put further constraints on its associated types by naming them in a where clause, like C.Element: Equatable. However, opaque result types, generalized existentials, and other conceivable future language features need a way to describe constr…

[fractal]

@Joe_Groff

As written, nothing is inferred;

I guess I have a fundamental misunderstanding of how this works. From the original proposal:

func makeMeACollection<T>(with element: T) -> some MutableCollection & RangeReplaceableCollection {
   return [element] // ok: an array of T satisfies all of the requirements
}

Does this not infer .Element == T for the underlying type? If not, how can the program actually compile?

@Joe_Groff

Your notation would be one way to do so; other approaches are being discussed in this thread: Protocol<.AssocType == T> shorthand for combined protocol and associated type constraints without naming the constrained type

I guess I'm worried that since this proposal explicitly does not name the opaque thing, by accepting it we are tacitly vetoing any syntax which seeks to do so. I think the syntax for constraining opaque types needs to be nailed down before accepting a proposal introducing opaque types.

[Joe_Groff]

@fractal

Does this not infer .Element == T for the underlying type? If not, how can the program actually compile?

It works similar to what you'd get if you passed in a generic argument constrained to <T: MutableCollection & RangeReplaceableCollection> with no Element constraints. You know that the value has an Element type, and that it's consistent among values returned by the same function, but you don't know exactly what type it is.

@fractal

I guess I'm worried that since this proposal explicitly does not name the opaque thing, by accepting it we are tacitly vetoing any syntax which seeks to do so. I think the syntax for constraining opaque types needs to be nailed down before accepting a proposal introducing opaque types.

A number of people suggested mirroring the generic argument syntax on the other side of the <>, which seems nice and uniform with generic arguments:

func makeMeACollection<T>(with element: T) -> <C: MutableCollection & RangeReplaceableCollection> C
   where C.Element == T

and could allow the some syntax to still be used in either argument or return position in common cases where there are no secondary constraints among arguments or results.

[QuinceyMorris]

I went back to the proposal and found this:

Following the some keyword is a class, protocol, Any , AnyObject , or composition thereof (joined with & ).

It's maybe a degenerate case, but some Any doesn't seem to read too well.

I notice that list of things that can follow the keyword is more or less the same list that can follow a : indicating type conformance (e.g. struct T: P). If you want to consider something longer than some, what about one of these:

    … -> conformanceTo P
    … -> typeConformingTo P

or something along that line?

[zwaldowski (Zachary Waldowski)]

@Joe_Groff

Rust eventually decided that making existentials undecorated was a bad idea and added the dyn keyword to encourage their use to be explicit. In previous discussions of existential syntax, Any or any seemed like the clear favorite as the modifier keyword for explicitly introducing an existential. Both linguistically, and in the spirit of fairness of not syntactically biasing one style over the other, Some or some strikes me as the corresponding modifier for specific opaque types.

I find this extremely compelling.

In general, I am reticent about new syntax or decorators that doesn’t immediately fit into the status quo, because it gives unnecessary ammo to “Swift is impossibly big to learn” criticisms. With this explanation, some feels of-a-part; it will be acceptable in Swift today, composes well with future proposals, and will eventually feel at home as it integrates with (or subsumes) our generics model overall.

I know y’all don’t want to commit to future plans, but some of the thinking here should definitely be explained in the proposal text and/or its Future Directions section. This explanation helped me close the loop on how this would fit into the overall generics picture that I wasn’t getting in the pitch thread. +1

[Alejandro_Martinez]

• What is your evaluation of the proposal?

A general +1 from me. I’m not an expert on the topic but there has been more than one occasions where this feature would have helped.

• Is the problem being addressed significant enough to warrant a change to Swift?

Yes. In general I think every little step to an improved type system (see the generics manifesto) is good for the language expressivity in the long term. Sometimes I needed to just return an instance of a protocol without really caring about the specific types and it gets cumbersome really quickly.

• Does this proposal fit well with the feel and direction of Swift?

The proposal expresses the problem really well and it’s really well written. I have to admit that every time I read it I start thinking that it would be nicer to just “-> P” instead of including a new keyword, but after reading it I’m convinced of the reason. Is still tricky to grasp the full theory of this type system functionalities and how it compares with generalised existential types. Luckily the proposal gives good examples and clarifies how this features differ.

That said I prefer opaque over some. While reading the code “some” seems to be too ambiguous to me, it doesn’t really seem so different as “any”. “opaque” seems much more appropriate to me and much more clear. Maybe is because it reminds me to when you refer to opaque return values on C. But I guess I can get used to “some” if native speakers and language experts say is more appropriate.

A concern I have is about the future directions needing more syntax changes. I’m concern we put ourselves in a corner in terms of syntax, and that if we would look into it holistically we may find a better solution. Again, if more knowledgable people says is fine let’s go for it, I just wanted to raise my concern that maybe is worth taking a look at those future directions now.

Somewhat tangent to the proposal but I think is important is the error messages around this feature. Type errors can be really confusing (specially in Swift early days) and it would be unfortunate to go back to that state with this feature, specially since opaque types don’t have names. Even for experienced developers some times parsing error messages gets hard.

Finally, is there any concern about type inference? Sometimes type inference can’t resolve some expressions and adding types helps the compiler, but in this case there won’t be a type to add so maybe the user is stuck? Just throwing it out there.

• If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

The Rust community has gone trough similar path for their type system. I’m glad the proposal is aware of this and acknowledges their learnings.

• How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

I’ve been following the pitch thread since it was posted and I’ve read the proposal multiple times. I haven’t tried the toolchain.

[pvieito (Pedro José Pereira Vieito)]

Why not simply go with this:

func someFunction<T, U: MutableCollection>(_ t: T) -> U {
    // ...
}

[Joe_Groff]

@pvieito

Why not simply go with this:

func someFunction<T, U: MutableCollection>(_ t: T) -> U {
    // ...
}

This already has meaning today; it means that the caller gets to pick what U is, and the implementation must be able to return any MutableCollection a caller asks for. An opaque type is different; U is picked by the implementation. It's more like <U> is returned from the function along with the return value:

func someFunction<T>(_ t: T) -> <U: MutableCollection> U