Opaque result types

Evolution Pitches
224

Referencing my example earlier since I think the post might’ve been missed: is there any reason why an _ is needed? .Element (meaning “the Element associated type of the conformance I just listed) reads better to me than _.Element.

I also still feel that conceptually, “an opaque type that conforms to these protocols under these conditions” (i.e. defaulting to Any and using : for all conformances) makes more sense than “an opaque type of this protocol which happens to also conform to these other protocols under these conditions.”

225

That seems ok. Would you allow/encourage this alternative syntax in other places with where clauses?
E.g.:

extension Collection<Element: Equatable> : Equatable {}
func hasSameElements<C: Collection<Element == Self.Element>>(_ with: C) -> Bool

I suspect there's really only a readability / conciseness benefit in the opaque return type case, in which case I'm not sure an alternate syntax really supports its weight.

226

I think that general shorthand would become more compelling in time, if we later support either generalized existentials or implicit generic parameters from input types. The latter is in a sense the counterpart to opaque result types for arguments; it'd be nice to just write:

func hasSomeElements(_ with: opaque Collection<Element == Self.Element>) -> Bool

or something like that without the explicit generic argument, which is only necessary to provide the type of the argument. At least for read-only arguments, taking an existential or taking a unique generic argument type are isomorphic as well.

@davedelong also suggested the idea of inferring the AssocType == in cases where there's only one associated type, so that Foo<T> means Foo<AssocType == T> which might be nice for simple protocols.

(To be clear, I'm not suggesting we expand the scope of this proposal to include any of the above, only pointing out other places having a shorthand for associated type constraints may become useful.)

227

Hmm... if the underscore is problematic, what about spelling it (Opaque as Collection) instead of (_ as Collection)

I'm not really tied to the word as so much as I am having a consistent place for the explicit underlying type to be defined. Even if we don't implement opaque typealiases right away, I don't want us to block ourselves from doing so in the future (and I would also strongly prefer that the syntaxes compose naturally)

That said, I would argue that as makes sense here because, once returned, the whole expression/pattern DOES act like the type to the right of as. Once returned, it acts like an existential/protocol... just without the usual technical limitations of a PAT.

That said, if using as is a problem, we could come up with a new term that means "pretending to be", e.g:

func foo() -> Int playing Comparable

I care more about the structure and composability than the term.

I liked @Moximillian's idea of re-using Opaque here.

func foo()-> opaque as Collection where opaque.Element == Self.Element
1 Like
228

For one, that wouldn't work for generalized existentials. It seems like we want the same syntax for both features, with some kind of keyword denoting the "opaque" case.

I think we disagree on the goals here. I have no interest in having a place for the explicit underlying type, because it doesn't matter. To the client, it never matters. If the implementor cares, it's really easy to put a

typealias Result = the type I want to return
return <big expression> as Result

in the body.

I disagree, because existential/protocol types do have these limitations, and they won't ever fully go away. I prefer a keyword specifically because we need to call out the differences, because they're important for the client.

As I noted somewhere earlier (within the last... 225... messages...) that opaque isn't a great answer for the opaque.Element case because the same syntax should apply for generalized existentials, should we get those some time.

Doug

229

This is disappointing. Is this true of the feature in general, or just the places it is likely to be used?

If it is the former, and we can't actually swap out implementations behind the scenes (or I was even hoping to keep some internal types internal, and only publicly expose the protocol guarantees), then I don't understand the utility of the feature anymore. Seems like it would just be saving a handful of keystrokes for people working on the standard library. Maybe I am misunderstanding this comment?

4 Likes
230

I'll answer here as well: the Opaque keyword wouldn't work for generalized existentials, which would be unfortunate. We'd have to invent another something like Opaque for the existential case, and it works against the ability to use generic typedefs:

typealias AnyCollection<T> = Collection where _.Element == T

func foo() -> opaque AnyCollection<T> { ... }

Doug

231

If you choose to make things inlinable, you can't swap out implementations behind the scenes, because you've exposed all of the implementation details of your function's implementation. The entire resilience feature is about this tradeoff: there is a performance cost to maintaining flexibility for the future, and various features (@fixedContents, @inlinable) let you trade off that flexibility for better performance.

Doug

1 Like
232

…but to make it clear, if the function is not inlinable then swapping out the implementation is perfectly fine.

6 Likes
233

If you choose to make things inlinable and you're ABI stable, that is. Which very few libraries need to be. The standard library is a pretty rare case in this regard – it's a highly generic, highly performance-sensitive library, yet needs to be ABI stable. This really is quite a niche set of requirements.

I think there's some confusion from the fact that the motivating example came from the standard library. Most libraries (or non-library code in large code bases maintained by many people) either won't be ABI stable, or will be able to make their types resilient, and so can choose to inline and still use this feature. Under those circumstances, this feature gives you a big win in terms of source stability, in that you can swap out the old type for new without worrying about your callers having hard-coded the type on their side.

Even with the standard library being ABI stable, there would still be wins for users, not authors, of the std lib.

Take the LazyCompactMapCollection example. Suppose you were writing something like the luhn algorithm on credit card numbers. For one step you have a string with digits and spaces and you want to turn it into a collection of actual digits, lazily. So you write a function for it:

func digits(ccnum: String) -> ??? {
  return ccnum.lazy.compactMap {
    Int(String($0))
  }
}

What goes into ??? is the somewhat monstrous LazyMapCollection<LazyFilterCollection<LazyMapCollection<String, Int?>>, Int>. To figure that out, you have to assign the value to variable and right-click it, then cut and paste what you find (assuming you're using Xcode... if not, good luck!). And if you composed it with more lazy methods, then it gets really ridiculous. Much nicer to make it opaque Collection where _.Element == Character. If that's already what all the lazy methods return, all the better to guide you towards that.

If the standard library were to have had this from the beginning, it wouldn't even need to expose these different lazy types from map, filter etc. They would still need to be @inlinable internal, so still in the ABI and not changeable after the fact, but users wouldn't have to know about their existence and they wouldn't need to appear in the documentation. That's a big simplification for new users unfamiliar with the fact that these different types don't matter for most purposes. Making that change now would be source breaking, so probably not worth it at this point, but would have been nice if we'd done it from the start.

But if you aren't ABI stable, you get an even bigger win! You can change lazy.compactMap to return a completely different type, like SE-222 suggested, and you no longer break source when you do. This would have been huge during the development of the std lib prior to ABI stability, and I'm pretty certain it will be huge for the many libraries and codebases that exist today and want to increase their use of generics over time without breaking source whenever they need to refactor. Libraries like NIO, or many libraries found in the compatibility suite today, could certainly benefit from this.

14 Likes
234

Thank you. This was a really helpful explanation!

1 Like
235

Well, there's also the fact that a function parameter list must have parentheses. That's a lot less subtle.

236

Oh, also bear in mind this goes the other way. If you have an Array as your computation result, you should strongly prefer returning it over an opaque value. It is really nice, as a user, to be given an array. Currency types are important things. Yes, it's a strong contract. But if there's no other reasonable implementation, don't make your user have to jump through more hoops for the sake of some future flexibility.

For example, sorted should return an [Element]. The possible micro-optimizations of (maybe someday) returning a funky opaque type from an out-of-place sort are vastly outweighed by the inconvenience you'll be foisting on your users who would much rather you give them a standard named type.

6 Likes
237

I just had some thoughts relating to what I might expect from opaque types:
If we had Apps A, B and Libraries C, D where:

// Library C
protocol P { /* … */}
struct S: P { /* … */}
func foo() -> S { /* … */}
func bar() -> S { /* … */}

// Library D 
import C
extension S: Q { /* … */}
func foo() -> S as Q { /* … */}
func bar() -> S as P { /* … */} // this one may not work

// App A
import D
// call foo()
// maybe: call bar()

// App B
import C
import D
// call foo(), maybe: get C.foo due to inverse overloading
// call bar(), maybe: get C.bar due to inverse overloading

Finally, I would like to point out that, depending on the semantics of opaque types, there already is a 'function' that works somewhat like this: Swift.type(of:). At compile time, all that is known is that it is a MetaType, but at runtime it is the type of the passed-in value.

239

This use case is what a named newtype feature would be for. Hmm… suppose we use newtype instead of opaque and when it appears in a return type that means it's a new (and thus unique) unnamed new type?

4 Likes
240

I recommend reading my comments on newtype earlier in this thread. I don't think it's what we want for opaque result types, because it has too many limitations.

Doug

241

Ah, well that's kind of unfortunate - I figured most of the benefit of lazy would be in the runtime performance (not eagerly copying, etc), so maybe inlining and specialising wouldn't be such a massive win.

As for ReversedCollection, it has a custom Index type which exposes the underlying Collection's index, so we'd need to introduce a new protocol to make it opaque. Basically trading one type for another.

protocol TransformedIndex: Comparable { // or whatever...
  associatedtype Base: Collection
  var base: Base.Index { get }
}

extension BidirectionalCollection {
func reversed() -> opaque Collection where _.Element == Element, 
                                           _.Index: TransformedIndex,
                                           _.Index.Base == Self
}

If we're going to use angle-brackets for where clauses, I would much rather we just went with Any<Collection where Element == Self.Element>. The Any<X where ...> syntax is the most easily-readable and expressive way to write and existential that I've seen so far. When considering generalised existentials and all the places you might want such a thing, it already implies that the underlying type is not fixed.

e.g. when I read [Any<Collection where Element == Int>], I understand it as already implying that each element may have a different underlying type. I can insert any collection to it, etc.

For opaque types, I would be happy if we used Some<X where...> or Opaque<X where...>. We could also use Any<X where ...> with an @attribute which tells us more about the axes the underlying type-identity depends on - e.g. @identity(Foo.bar<Int>) Any<X where ...> tells you that the existential could contain any X, but is always the same for calls for Foo.bar when the function's generic parameter is Int.

2 Likes
242

This is all subjective, of course, but to me Collection<Element == T> would also be a much more straightforward way of describing the existential type, in line with how most other languages describe their analogous interface types.

1 Like
243

What I mean is that if you have, say, a stored property...

struct Thing {
  var data: Collection<Element == Int>
}

It isn't immediately clear whether this is a specific type of Collection (i.e. an opaque Collection), or whether I could replace its value with my own, custom Collection. Maybe several times, each with different types.

struct Thing {
  var data: Any<Collection where Element == Int>
}

is much more obvious to me. But of course, you're right - it is subjective.

244

If the goal is to reduce the amount of characters to write, then there's obvious choice here. But if you want someone unfamiliar with the code to understand what the code does, I think the Any<...> approach or something like that goes a long way to make things easier (and it also reads better as a sentence).

Now, I don't mind having shortcuts or shorthands for power users. What I wouldn't want to see is shorthand being the only way to do it.

1 Like