Opaque result types

[brendon (Brendon Justin)]

unspecified conveys this without implying anything about the underlying type:

func makeMeACollection<T>(_: T.Type) -> unspecified Q: MutableCollection & RangeReplaceableCollection where Q.Element == T {
    return [T]()
}

[Vogel]

But if it weren't possible, that would mean that we'd break a very strong guarantee of the type system. Currently, I can always build a convenience function that returns the result of some other function, but also does additional work (or not). In general, if I get a hand on some value, I can pass that value around or return it from my function. Breaking this promise seems like a horrible idea to me.

[Vogel]

@Jumhyn

let f = [d(), e()] //OH NO, ERROR, NOT THE SAME TYPES (even though they actually are, but we can't express that.)

There have been proposed syntaxes that would allow this sort of behavior, either via a typealias or some other annotation of the return type, so that something like this would be legal:

typealias opaque Q: A where B: C = MyConcreteType

func d() -> Q {
    return ...
}

func e() -> Q {
    return d()
}

func f() -> e().ResultType {
    e()
}

let f = [d(), e(), f()] // OK!

Does this address your concerns?

This would work but also defeat the purpose of the proposal, I might as well just return MyConcreteType here. The "opaqueness" of the typealias wouldn't really achieve anything anymore.

[Jumhyn (Frederick Kellison-Linn)]

I don’t see how this defeats the point of the proposal. It allows you to hide MyConcreteType as an implementation detail of whatever type implements these methods. That two methods return the same concrete type doesn’t affect that you may not want to expose that type beyond saying “it’s a concrete Collection-conforming type with Int elements.”

[Vogel]

@Joe_Groff

Yeah, it wouldn't be appropriate to use this feature if the fact that d and e return the same type is important. If this were expanded out to allow for opaque typealiases, you could share an alias between these two definitions. It's hard for me to see how the standard library using opaque types for collections would affect its current level composability, since the use of separate concrete wrapper types already in practice means that you'll get different types for different computations. You can't put [x.lazy.map(...), x.lazy.filter(...)] into an array either without forcing both elements to a common concrete collection type, and you could do the same if the return types of map and filter were opaque.

True, but I could create two functions that both end up calling Array.lazy.filter(...).map(...) and then afterwards use them together like so: [functionOne(), functionTwo()]. This works currently and would only work with the opaque typealiases.

Now, whenever I write some kind of function that might be called somewhere else (be it in the standard library or somewhere else), I never know whether someone will eventually use the same pattern (two functions that call my functions and then combining the results of those functions into a collection etc.) or one of infinitely many other similar patterns.

So, I have to use the opaque typealias thing. I can never write a function with the opaque keyword in the signature because of this issue. Only the typealias works.

And at that point, the typealias is not a powerful feature anymore. The very idea of getting rid of the "identity" of the type has failed. I'm using my concrete typealiases now. Why not just return the actual type at this point?

But at the very least, this means that the use of opaque in function signatures should not be allowed.

[Vogel]

@Jumhyn

I don’t see how this defeats the point of the proposal. It allows you to hide MyConcreteType as an implementation detail of whatever type implements these methods. That two methods return the same concrete type doesn’t affect that you may not want to expose that type beyond saying “it’s a concrete Collection -conforming type with Int elements.”

I think that if the definition your type looks like this:

struct MyConcreteType: Collection {
    typealias Element = Int
    
    //... only public / internal declarations to satisfy Collection requirements, nothing else
}

Then it is very obvious that that's the case.

[Jumhyn (Frederick Kellison-Linn)]

@Vogel

I think that if the definition your type looks like this:

struct MyConcreteType: Collection {
    typealias Element = Int
    
    //... only public / internal declarations to satisfy Collection requirements, nothing else
}

Then it is very obvious that that's the case.

Right, but no matter how obvious it is, right now there’s no way to prevent that implementation detail from leaking into client code. The caller of lazy.compactMap really doesn’t care or even want a LazyCompactMapCollection<Int>—they just want a Collection of Ints.

[bjhomer]

Perhaps this is mixing keywords too much, but what about:

func foo() -> private Collection { }

It returns a specific collection, but you don't get to know exactly what kind it is because it's private.

Unfortunately, then people would probably want to use internal and public as well, and those probably don't have logical corollaries here.

[Vogel]

@Jumhyn

Right, but no matter how obvious it is, right now there’s no way to prevent that implementation detail from leaking into client code. The caller of lazy.compactMap really doesn’t care or even want a LazyCompactMapCollection<Int> —they just want a Collection of Int s.

As stated above, this would still require an opaque typealias. Now you're confronted with the problem of naming the typealias. How do you name it?

[Jumhyn (Frederick Kellison-Linn)]

@bjhomer

Perhaps this is mixing keywords too much, but what about:

func foo() -> private Collection { }

I actually really like that, aside from the fact that internal, fileprivate, and public don’t have good corollaries. The language in the original proposal and discussion since motivates this language:

@Douglas_Gregor

The implementation of compactMap(_:) can still return an instance of LazyCompactMapCollection , but now LazyCompactMapCollection can be private

@Douglas_Gregor

These generic types, which are needed to express the results of generic algorithms ( enumerated , flatten , joined ), are generally not of interest to users, but can significantly increase the surface area of the library.

@Douglas_Gregor

An opaque type is backed by a specific concrete type, but that type is only known to the implementation of that function/property/subscript.

@gwendal.roue

...opaque types look like an implementation detail that leaks into userland

@allevato

I'm happy to see more work being done to address leaking of implementation details.

Hiding "implementation details" is exactly what access control modifiers are meant for, and this is a sort of access control, or at least analogous. I'm interested if others think that this overloading of private would be too confusing. "Swift private return type" seems like I would be easy enough to search for documentation on, especially if this feature is released under that name to canonize it.

[Jumhyn (Frederick Kellison-Linn)]

@Vogel

As stated above, this would still require an opaque typealias.

Not if we adopt one of the proposed syntaxes for referencing another method's result type (#resultType(...), methodName(_:).ResultType, etc.).

@Vogel

Now you're confronted with the problem of naming the typealias . How do you name it?

It would be up to the author of the methods to come up with a suitable name that encompasses what they want.

[gwendal.roue (Gwendal Roué)]

@Jumhyn

Vogel:

Now you're confronted with the problem of naming the typealias . How do you name it?

It would be up to the author of the methods to come up with a suitable name that encompasses what they want.

Yes, but Vogel was trying, with his question, to show that there is then no specific advantage using opaque type, compared, say, to a typealias for the hidden type. In both cases, you end up with something concrete:

// what's the difference in the end?
typealias MyResult = SpecificBlah<Foo<Bar>, ...>
opaque typealias MyResult = BlahProtocol where ...

func myFunction() -> MyResult { ... }

And I fully support Vogel's questioning. I'm quite concerned as well.

Yet, opaque types allow the library author to expose a behavior instead of a concrete type. This makes it possible, for the library author, to change the concrete type, without breaking source-level compatibility. That's quite useful and interesting for library authors.

[Vogel]

@Jumhyn

As stated above, this would still require an opaque typealias .

Not if we adopt one of the proposed syntaxes for referencing another method's result type ( #resultType(...) , methodName(_:).ResultType , etc.).

Also problematic and not nice, and often impossible especially in the here often mentioned, overload-heavy world of sequences and collections. #resultType(of: LazySequence.compactMap(_:)) for example is ambiguous.

@Jumhyn

Now you're confronted with the problem of naming the typealias . How do you name it?

It would be up to the author of the methods to come up with a suitable name that encompasses what they want.

It was a trick question. Obviously, the best name for the opaque typealias that wraps, for example, a LazyMapSequence would be, well, LazyMapSequence (If there was a better name, that name would always also be a better name for the concrete type in the first place). Which really shows off how useless it is to even have it.

[Jumhyn (Frederick Kellison-Linn)]

@gwendal.roue

Yet, opaque types allow the library author to expose a behavior instead of a concrete type. This makes it possible, for the library author, to change the concrete type, without breaking source-level compatibility. That's quite useful and interesting for library authors.

This is precisely the difference, though. The same would hold true for an opaque typealias.

[Jean-Daniel]

@Joe_Groff

One alternative that comes to mind is some . This has the right connotation, that the function returns a value of some type, that's still specific, and it nicely complements our use of Any* in dynamically type-erasing containers. some is however already used as one of the case names for Optional (although it's rare to need to spell it out explicitly because of Optional's sugar). How does some sound? If anyone has other ideas, I'd like to hear them too.

Maybe we can use kindof. This was already used by Obj-C to define a return type that represent a type that is not properly defined but is guarantee to be a subclass of the defined type.
This is close enough to the semantic here. You don't really now what the return type, but you know what it looks like.

[gwendal.roue (Gwendal Roué)]

Yes, yes. But Vogel is attempting at showing that the current stage of design only works for "leaf APIs", I mean results that can't be further processed/composed by client code without "mandatory typealias hacks" that hinder the proposal. (S)he brought very valid concerns. Let's hear them.

[Vogel]

@gwendal.roue

Yet, opaque types allow the library author to expose a behavior instead of a concrete type. This makes it possible, for the library author, to change the concrete type, without breaking source-level compatibility. That's quite useful and interesting for library authors.

This a valid use case, I just think that it requires much less language-level intervention. In most cases, I think it would be the best solution to just write a small struct that wraps a single private instance variable containing the actual implementation.

The bulk of the work here that all this opaque stuff would really be helpful with, is forwarding protocol requirements from the wrapper type to an instance variable instead of having to do things like this lots of times:

func x() -> Y {
    return wrapped.x()
}

Maybe we can find a way to make the compiler synthesize these things for you if you tell it that the type that you're creating is some kind of wrapper type. That would solve the problem of making this convenient without cluttering the type system and making it more awkward.

[Lantua]

I understand @Vogel’s concern that the opaque type isn’t a first class citizen in Swift, but I think it’s more of a problem with protocol’s existential (We have [[Int]] but not [Collection<Int>]) not unique to opaque type.

I may get it mixed up here, and that PE and opaque are two different things, but it seems that opaque type doesn’t create more problem, just make more pronounced the old one.

[gwendal.roue (Gwendal Roué)]

@Vogel

This a valid use case, I just think that it requires much less language-level intervention. In most cases, I think it would be the best solution to just write a small struct that wraps a single private instance variable containing the actual implementation.

But it would then fail addressing the original intent of the proposal. Quoting the OP:

@Douglas_Gregor

For example, SE-0222 introduces a new standard library type LazyCompactMapCollection for the sole purpose of describing the result type of compactMap(_:) :

extension LazyMapCollection {
  public func compactMap<U>(_ transform: @escaping (Element) -> U?)
      -> LazyCompactMapCollection<Base, U> {
    return LazyCompactMapCollection<Base, U>(/*...*/)
  }
}

With this proposal, the standard library could hide the result type, exposing it as "an opaque Colection whose Element type is U :

private struct LazyCompactMapCollection<Base: Collection, Element> { ... }

extension LazyMapCollection {
  public func compactMap<U>(_ transform: @escaping (Element) -> U?)
      -> opaque Collection where _.Element == U {
    return LazyCompactMapCollection<Base, U>(/*...*/)
  }
}

Isn't it one goal of the proposal, to reduce the number of exposed types?

[gwendal.roue (Gwendal Roué)]

@Lantua

I understand @Vogel’s concern that the opaque type isn’t a first class citizen in Swift, but I think it’s more of a problem with protocol’s existential (We have [[Int]] but not [Collection<Int>]) not unique to opaque type.

This was my initial understanding as well. Later I understood that opaque types allow much more efficient compilation than existentials. This is because unlike existentials, an opaque type is only opaque at the API level, but completely transparent for the compiler, which knows exactly which concrete type is hidden.