Opaque result types

(Douglas Gregor) #61

I'd state this a bit differently. I think we shouldn't be afraid to introduce a keyword when we are introducing a new kind of "thing". But we should be wary of introducing any new "thing", especially if it's very similar to something that already exists in the language.

I think your Opaque<...> and Any<...> suggestions fit nicely together. I have two concerns. First, the Any<...> was discussed extensively as part of SE-0095 a while back, and at this point I don't think we should go back and change it. Second, Opaque<...> wouldn't be a general-purpose type you could use anywhere: it is syntactically limited to result types, which makes opaque feel (to me) more like a modifier on the result type than a full-fledged type in the type system.

type(of:) returns the underlying concrete type, at run time.

Doug

(Douglas Gregor) #62

I've been thinking about this for a bit, and the second bullet is really what gets to me. That syntax I conjured up to describe conditional conformance is rather awful. It's completely non obvious and quite verbose. @dabrahams improved on it a bit, but it's verbose by necessity.

I think the right course forward for this proposal is to introduce opaque type aliases, which give a name to the opaque type so that it can be reused/tweaked. That provides a natural way to describe the conditional conformance, so we can eliminate the awful conditional-conformance-in-the-result-type syntax from the proposal.

Doug

(Douglas Gregor) #63

I'll ignore all of off-topic stuff from your post, but I want to respond to this part here. Generic protocols can replace associated types (some things would get worse, some things would get better), don't change much about existentials (the same problems of type identity remain), or generalized existentials (some trivial cases get simpler; the general case remains about the same), and don't solve the problems opaque result types solve (which are primarily about type identity).

I've seen this same line about generic protocols being "all we need" to simplify the language a number of times, enough that I think a significant number of people believe it despite the complete lack of a testable design that would achieve the stated wins. A handful of small examples doesn't cut it---we're talking about something that could radically reshape the way we work with generics, as well as the standard library. If you want to talk design of generic protocols, that's great... for another thread, but please refrain from claiming that "feature X will fix everything" unless you're prepared to provide sufficient detail to evaluate said claim.

Doug

(Anthony Latsis) #64

@Douglas_Gregor Do I understand correctly that an opaque type can be expressed as follows?

func foo<Opaque: Collection>() -> Opaque where Opaque.Element == Int {...}

If so, do we really need a new modifier and additional specific underscore syntax to refer to ATs?

(Brent Royal-Gordon) #65

The signature you list would allow the caller to choose a type for Opaque, and it could be any Collection where Element == Int. This feature has the callee choose one type; it’s just that its choice is hidden.

(Anthony Latsis) #66

You (the caller) actually can't choose a type for obvious reasons. All you can do is use exactly Collection where Element == Int, which will be possible once opened existentials land.

(Howard Lovatt) #67

The Tim Cook quote etc. was to remind people of the big picture; which should take precedence, which should be at the forefront when making changes.

But they do in languages like Kotlin/Java/Scala, it is one of their main uses. Consider:

protocol IteratorProtocol<Element> {
    func next() -> Element
}
protocol Sequence<Element> {
    func makeIterator() -> IteratorProtocol<Element>
    ...
}

The type that makeIterator returns is not stated and can be a private type (it commonly is in Java/Kotlin/Scala), i.e. it provides opacity.

Surely Java/Kotlin/Scala represent a thoroughly tested and complete design? I just don't buy the argument that Swift is so different from these languages that despite the solution known to work well in these languages it can't possible work in Swift. As further evidence, Scala provides both generics and associated types and the latter is rarely used.

Sure the compiler/runtime in Java/Kotlin/Swift has to work hard to optimise the code, but the compilers for Java/Kotlin are probably quicker than Swift (Scala is probably a similar speed to Swift) and the resultant code from all three executes as quickly, therefore proving that it is possible. But obviously generic protocols with the associated optimisations are significant work to implement.

(Brent Royal-Gordon) #68

No, I’m saying that you can write a function with exactly this signature right now, and the caller can write as Array<Int> or as Set<Int> or whatever else they want right after it, and foo() just has to figure out how to return the type the caller specifies. This proposal has a different behavior, so it needs to have a different syntax.

(Anthony Latsis) #69

Could on elaborate a bit on how func foo() -> opaque Collection where _.Element == Int {...} would be different? If the actual type is public, you might just guess it by casting, but if it's private the cast will never succeed.

(Adrian Zubarev) #70

The whole purpose of opaque types is to hide the underlying type, but allow it to work with generics.

(Brent Royal-Gordon) #71

It’s different because the compiler will diagnose it as a syntax error—probably an unknown type attribute, but I’m on my phone right now, so I can’t test it. Your code will be accepted by the current compiler; Doug’s will he rejected. That means we can’t change the meaning of your code without breaking source compatibility.

(Anthony Latsis) #72

I understand it isn't valid syntax, I am asking about the difference between the semantics of my signature and the one Doug proposes. In other words, why doesn't a generic return type - the underlying type is hidden, but it can be guessed by casting iff it's accessible - satisfy the concept of an opaque type.

(Adrian Zubarev) #73

You can count me to the folks that want generic protocols in Swift, but I also want HKT, opaque types, all existential features and even more missing generic features. However I disagree that generic protocols solve the current problem nor every other problem. Every generic feature solves a specific subset of issues. Sure some of them do overlap, sometimes a lot (opaque types vs. existentials), however they still allow different solutions to be found to different problems. My only concern about discussion regarding generic protocols is that if we don‘t stop hijacking every topic about generic features that the core team will be so annoyed that they reject it alltogether. I don‘t want that to happen so I encourage everyone to calm down about that particular feature and wait until it‘s time to bring it to the table.

(Anthony Latsis) #74

Doug means that changing the syntax from ATs to generic protocols doesn't cross out the problems we currently have, the generics manifesto and the roadmap for the type system in general, which is quite different from that of Java and even Kotlin. You're talking about syntax, Doug is about implementation (for Swift).

(Thomas Roughton) #75

That’s not quite the point of opaque types. Consider if the function returned an opaque Equatable; an existential type doesn’t help since you don’t know the Self requirement (so the == method needs a dynamic type check as it would in e.g. Java). With an opaque type, you know that the particular type returned from a particular method will always have the same associated types; as such, you can safely use ==, or create an array of that type, or anything else that you'd be prevented from doing without generalised existentials or other generics features.

In general, I'm +1 on this proposal. I do think the syntax is ugly and cumbersome for conditional conformances, but if that's hidden behind typealiases for the user then I think it fits with progressive disclosure, and I can't think of anything better. I certainly think the idea of opaque types is better than leaking out names like LazyMapFilterCollection to the user, and I think opaque types may end up being more readable since they only expose the information about the type that the user is likely to care about – what you can do with it given the protocols it conforms to.

(Xiaodi Wu) #76

Given func foo<Opaque: Collection>() -> Opaque where Opaque.Element == Int {...}, the caller can indeed choose the return type.

One does this by using the type coercion operator as, which is widely misunderstood:

let x = foo() as [Int]

This does not involve any casting whatsoever because as is not a casting operator here: it merely specifies the return type. The function foo must return a value of whatever type the user so chooses as long as the choice conforms to the stated constraints. That is, the author of the function chooses the constraints and the user chooses the type.

An opaque return type means that the author of the function chooses a single return type not disclosed to the user but guaranteed to conform to the stated constraints. That is, the author of the function chooses both the constraints and the type.

(Randy Eckenrode) #77

Not a fan of the proposed syntax. If the issue with existentials is that sometimes we do care about the Self requirement, then could an alternative be to treat it as an associated type of the protocol? For example:

/// Returns an existential
func f() -> Comparable { /* ... */ }

/// Returns an anonymous type conforming to Comparable
func g() -> Comparable.Self { /* ... */ }

let fArray = [f(), f(), f()]
f.sort() // This doesn’t work

let gArray = [g(), g(), g()]
g.sort() // But this does

That doesn’t address conditional conformances for existential types. I’m not a fan of the proposed syntax. Maybe being more verbose (e.g., Self: MutableCollection implies _: MutableCollection versus Self: MutableCollection -> _: MutableCollection) would be better?

(Thorsten Seitz) #78

I don't understand. I would think that the callee chooses the type because it calls the constructor creating the result value.
Just to be sure I did verify this (with a generic class instead of a protocol because we do not have existential yet) and got a cast error as expected:

Could not cast value of type '__lldb_expr_21.Hidden<Swift.Int>' (0x114ac1e10) to '__lldb_expr_21.Other<Swift.Int>' (0x114ac4e88).

(Tomáš Znamenáček) #79

This is about the same case, isn’t it?

func foo<Opaque: Collection & ExpressibleByArrayLiteral>() -> Opaque where Opaque.Element == Int {
    print(String(describing: Opaque.self))
    return []
}

let x: Array<Int> = foo() // prints “Array<Int>”
let y: Set<Int> = foo() // prints “Set<Int>”

And you can verify using the print statement that the Opaque type is chosen by the caller.

(Thorsten Seitz) #80

Ah, thanks! I think I'm beginning to understand the problem now :-)