SE-0244: Opaque Result Types


(Mox) #121

I may be simplifying a lot, but...

Opaque types is a compiler feature without hard requirements on big syntax / language changes.

Anonymous types is a big syntax/language design topic with minimal impact to compiler aside from parsing (as it’s just a shorthand).


(Thomas Krajacic) #122

This looks incredibly clean and Swifty. Easy to understand and reason about.
Yes please!


(Michel Fortin) #123

I was not warm to the idea of anonymous types, but Chris's draft proposal express why much better than I did. You just tweak a bit the syntax for a typealias and everything magically falls into place with the existing language facilities.

Whereas with anonymous opaque types you'd have to invent a new syntax for everything: declaring the constraints, using the type for arguments in a non-generic function, declaring a function (generic or not) that returns the same opaque type, declaring tuples, arrays, collections, optionals using that opaque type. Without a way to express all of this, a type being anonymous makes it significantly less usable.


(Goffredo Marocchi) #124

Really clear explanation Chris, makes the topic fall so much better into place... it makes a lot more sense: the kind of proposal I hope we can agree to go forward with.


(Adrian Zubarev) #125

I might have missed here something but what is the point restricting the opaque types to be public? If that would be the ultimate goal of the whole feature, then I don't think I would still support it as I personally want to use some opaque types internally. Making internal API that only requires the opaque feature set public is a no-go for me. Also I would be highly disappointed if this feature would be merged into an access modifier like we did in the past with open.


(Michel Fortin) #126

I think there is a case to be made for internal opaque typealias when the concrete type is fileprivate or private.


(Adrian Zubarev) #127

I don't view opaque types to work on access level like that. You can have public types erased by the opaque type from the perspective of the API caller. I think it's just a useless restriction to make opaque types restricted to public API or to such access hierarchies. It just opens up a new can of worms.


The oversimplification of opaque types would be: opaque types are concrete types that follow certain constraints, where only these constraints are exposed to the caller and the actual underlying types are erased. Since the compiler knows more then the caller, we can make it work with generics.

There is no need mix this with access level.


(Joe Groff) #128

Thanks for writing this up. This is pretty much what we had in mind for opaque typealiases as well. However, by declaring Rust's impl Trait irrelevant, I'm afraid you're missing some of the point of this feature. I understand the uncomfortableness about anonymous types, and the desire to plan for the general case. However, Rust's impl Trait design is the strongest prior art I know of for a feature like this, and Rust programmers use the feature heavily in both user and library code, and these issues as far as I can tell are simply not a problem in practice; I've asked. Doug and I did our homework here; we thoroughly explored the idea of typealiases before writing up this proposal, and found them wanting for a number of reasons:

  • They lose one of the primary benefits of the Rust feature, which is the ability to directly describe a function's return interface without the indirection of another name. Furthermore, this feature is competing directly with existential types in that "hide the concrete return type" role; we know existentials are inadequate in that role for all the reasons the proposal outlines, but if it takes a second decl, a restatement of the concrete underlying type, and coming up with a new name to take advantage of opaque types, then existentials will still look syntactically attractive as an alternative.
  • Opaque type aliases introduce new design and implementation complications, since the relationship between the opaque and underlying type becomes visible across multiple declarations; naively representing this as a bidirectional implicit conversion in the type checker would be bad for performance and predictability. Implementing any sort of "these types are sometimes the same, sometimes different" feature is quite difficult once you get into the details. Furthermore, because an opaque type alias is an independent decl, returning an opaque typealias would lose the ability for a function to use a nested type to completely hide its implementation type.
  • Opaque type aliases don't make the "where clause" problem any easier in the general case.

Looking at the bigger picture, we're also trying to lay groundwork for other language improvements that don't have anything directly to do with return types. The "anonymous type" design problem still exists in the language at large and influence the design of other future language features like generalized existentials. We also foresee the sugar syntax being useful as a shorthand for generic arguments. We'll likely need a more expressive syntax for describing constraints on anonymous types for generalized existentials, and the sugar syntax we're proposing has a useful generalization, we're aiming for something that achieves the most important subset of abstracting return types while planning for a syntactic analogy with generalized existentials and clearer overall generics notation going forward.


(Mox) #129

While opaque types are by now pretty fully baked feature (preferably @Chris_Lattner3:s version), we are barely scratching the surface of design space for anonymous types.

Yet we are piggypacking a major design decision on top of opaque types without actually going through all the places it has impact.

So essentially the argument is ”Rust solved it, we copy it”, rather than truly writing a separate proposal for anonymous types and deciding what is the swift’s version of that design space.

Even if anonymous opaque types is the ultimate solution for some use cases, the first version of opaque types doesn’t have to include it. You get the opaque functionality in v1, but need to have a bit patience for the ultimate solution until the anonymous syntax is fully baked.


(Paul Cantrell) #130

@Chris_Lattner3: Your proposal sketch is much what I was vaguely envisioning, and seeing it spelled out, it really does make more sense. Speaking as somebody who is unfamiliar with this feature in other languages and barely understands the underlying implementation concerns, it’s much easier to form a casual mental model of this alternative proposal.

It’s simply more comprehensible; the answers to my questions are all there in the syntax. I appreciate Swift’s “progressive disclosure” aesthetic, and this alternative fits that aesthetic better.


Joe’s uneasiness makes me uneasy too. Two of my tried and true Swift heuristics are “Joe Groff knows best” and “Chris Lattner knows best,” and these competing proposals put them in conflict!

@Joe_Groff, to some of your concerns:

I’m highly sympathetic to this effort to make the easy the best thing. However … in practice, if people simply want to hide a return type and (perhaps thoughtlessly) use an existential to do so, what concerns other than performance are there?

I think of Knuth’s optimization razor, and wonder if the mental burden is worth it for performance benefits alone. Certainly there are other places where Swift (unlike Rust) chooses convenience, simplicity, progressive disclosure, and low friction over performance, and the “high performance, high mental burden” escape hatch is there for those who need it.

The cases the proposal mentions where there is benefit beyond performance to guaranteeing that the same function returns the same (but hidden) concrete type seem arcane to me. In the common case, existentials seem perfectly adequate for type hiding, other than the slight runtime overhead. But I assume I’m missing something.

I’m not convinced I would want the relationship to be bidirectional, even within a module. My thoughts echo what @DevAndArtist said: from my language user’s point of view, opaqueness is about implementation hiding. That doesn’t only apply at module boundaries.

This matches my intuition of what opaque should mean. If I were using an internal opaque typealias, I would still expect it to help me isolate pieces of internal code from each other. I would not object to having to use as — or even as! — to unwrap an opaque typealias to its underlying type; I’d even expect that.


Joe’s concerns about future directions for generic syntax trouble me the most. Generic type declarations and where clauses are already hard to read for those who don’t spend all day thing about them. We want good expressiveness, sugar, decomposition mechanisms, and self-disclosing syntax to tame these beasts.

In my limited knowledge of ML-family languages, it seems that it’s best practice to decompose massive type declarations into small, separate building blocks. The C++ aesthetic of jamming complex types inside of larger type declarations has always sat poorly with me; nobody likes a 20-line function signature. I wonder if typealias or similar could become a standard tool for decomposing complex generic types? Perhaps one could adjust Chris’s proposal with this future in mind?


(Chris Lattner) #131

Good point, I didn't consider that, it makes perfect sense to allow them within a module when the defining type is private. I'll update the draft, thanks!


(Adrian Zubarev) #132

Why do we need to mix access level hierarchies here still? As far as I'm concerned, I should be able to write an internal opaque type alias that erases already a publicly exposed type, there is no need for the concrete type on the rhs from = to be private at all. If that was true then I could not make Array into an opaque type, at least this is how I understand your pitch.


(Chris Lattner) #133

Sorry for that, I changed the writing from "Irrelevant" to say this proposal is not based on impl traits. The goal of the proposal is not to take a feature from Rust, it is to solve a set of API modeling problems in the Swift world. While it makes sense to learn from other communities, our goal isn't to just take their features forward, it is to learn from and adapt ideas from lots of places and find the best thing that fits with the language.

You two are super smart, but none of that homework entered the writeup of the proposal in the alternatives considered section, so none of the feedback during the review period was able to consider and discuss it. If this was explored in the alternatives section, I would feel that the community was given the opportunity to look at both alternatives and consider them on their relative merits. As it is, I (and apparently most of the community) didn't know that this is something that was a worthwhile part of the design space.

Sure, I can see how that is the primary benefit of the Rust feature, but it is almost completely irrelevant to the goals of the proposal. This is why I don't think the Rust feature is particularly relevant here - I don't think Rust even has a stable ABI to consider or the concerns around that. These are very different worlds.

To restate my understanding of the feature, I believe that the primary goals is to not export implementation internal types, eg allowing them to be private. This reduces ABI surface area and clutter. The LazyMapSequence still needs to be written one way or the other, it just shouldn't need to be exported from the stdlib.

I interpret your argument as saying that this is actually primarily a syntactic sugar feature intended to reduce the amount of typing that has to be written manually. If that really is your goal then I (and I suspect many others in the community) would be generally against the feature in the first place. My personal rationale is that it is a very complicated and invasive feature as proposed with a lot of unbaked edge cases - I assume it will add a bunch of technical debt and complexity to the compiler as well.

It has been a long time since I've worked on the constraint solver, but that isn't how I would model it. I would treat them as a new type of nominal type, e.g. OpaqueNominalType, which allows you to put all the generic constraints and everything else on that type. This would require compiler changes for sure (both proposals do) but the pathways in the compiler are already pretty well plumbed through in an abstracted way due to GenericTypeDecl and NominalTypeDecl having multiple subclasses already.

I don't understand what you mean, can you please elaborate and provide an example? Are you talking about the sugar aspect of this?

Ok, I for one am not getting that from the proposal as written. It sounds like you have a large number of motivations that aren't captured in the proposal. I also don't understand the comparison to existentials here, they are different features (dynamic vs static duals of each other) as I mention in a comment further downthread from here.

In any case, to me (and as others have mentioned frequently), this proposal feels super half baked and complicated. What is the driving motivation for pursuing it now? If generalized existentials are somehow the motivation, why not develop those (a manifesto would be nice) and show how this plugs in?

-Chris


(Chris Lattner) #134

FWIW, I don't see this feature as relating to existentials at all. Existentials allow many different types to be dynamically returned from a function call, the only unifying principle is that they conform to a protocol.

// This is ok with existentials so long as Int and String conform to P.
func f() -> P {
  if .. {
    return 42
  } else {
    return "foo"
  }
}

The motivation for this feature, as far as I understand it, is that there is a specific concrete type that is always returned, it is just opaque. The proposal as written is very clear about this, each instance of "identity" in the proposal is talking about related issues.


(Chris Lattner) #135

FWIW @Paul_Cantrell, your post is what made all this click for me :-)

-Chris


(Paul Cantrell) #136

Yes. The similarity is that both opaque types and existentials let a function force its callers to interact with its return value only through a protocol. I believe that Joe’s concern is that people will choose existentials simply because the syntax is shorter and it superficially serves the same “implementation hiding” purpose, even when opaque types work better. I’m looking for more clarity about what “works better” means in the situation where either would work.

Thanks, I’m glad it was helpful!


(Joe Groff) #137

Swift lazy collections definitely suffer this same problem today, and now that I think about it, named opaque typealiases would not really help with it. Using named opaque typealiases, you can achieve the library-centric goal of hiding the ABI and API surface area of the concrete types, but it does nothing about the user-centric goal of simplifying the presented types you get from using the API.
If lazy map and filter were implemented using opaque typealiases, they might look like this:

extension Collection {
  opaque typealias LazyMapCollection<U>: Collection = _LazyMapCollection<Self, U>
  func map<U>(_ transform: (Element) -> U) -> LazyMapCollection<U>

  opaque typealias LazyFilterCollection: Collection = _LazyFilterCollection<Self>
  func filter<U>(_ predicate: (Element) -> Bool) -> LazyFilterCollection
}

which means that, as you compose operations, the composed typealiases still accrete:

func myHelper<C: Collection>(_ collection: C) -> C.LazyMapCollection<String>.LazyFilterCollection.LazyMapCollection<String> {
  return collection.map { ... }.filter { ... }.map { ... }
}

Of course. The goal here isn't to blindly copy the feature. We see patterns of Swift code today suffering from many of the same problems, and there could be even more similar situations in the future, so we think a similar solution is appropriate. There have been many times when users have asked me "why doesn't func foo(x: InputProtocol) -> OutputProtocol work? And since the compiler tells me to use OutputProtocol as a generic constraint, why doesn't func foo<T: InputProtocol, U: OutputProtocol>(x: T) -> U do what I want to hide the underlying type of U?"

It's reasonable to challenge the notion that this is a useful feature at all. IMO, if we don't think anonymous opaque return types are a feature that is valuable for Swift, I would personally still judge named opaque typealiases as even less valuable. Having to name the typealias definitely has advantages, but it also compromises what I see as many of the core benefits of what we're proposing.

That's fair, we could have elaborated further on the tradeoffs. However, we did cover typealiases as a future direction, and they were also discussed in the pitch thread. I don't think it's fair to say we ignored them completely.

I see it as both—this is both an abstraction feature for library ABIs, and also a sugar feature for user code. The implementation has if anything significantly cleaned up the compiler's handling of archetypes; once I modeled opaque types as a new kind of archetype, the implementation almost completely fell out from our existing generics model. (As part of the refactoring, opened existential archetypes are also much more fully baked, which will make generalized existentials easier to support in the future.) I honestly don't see many uncovered edge cases in the implementation; the implementation model is built to accommodate named typealias declarations and fully generalized constraints on the opaque type, and the semantics should be obvious; it's almost entirely a syntactic issue.

That's fair as well. I can revise the proposal to more fully elaborate on these related directions, and give fairer airtime to the alternatives.


(Joe Groff) #138

Sorry, I presented an example in an earlier reply to @Paul_Cantrell:


(Yuta Koshizawa) #140

It seems so beautiful.

// generics
func useAnimal<A: Animal>(_ animal: A) { /* ... */ }
// opaque argument types (sugar of ^)
func useAnimal(_ animal: some Animal) { /* ... */ }
// existential (argument) types
func useAnimal(_ animal: any Animal) { /* ... */ }

// "reverse generics"
func makeAnimal() -> <A: Animal> A { /* ... */ }
// opaque result types (sugar of ^)
func makeAnimal() -> some Animal { /* ... */ }
// existential (result) types
func makeAnimal() -> any Animal { /* ... */ }

// opaque types with constraints
let sequence: some Sequence<.Element == Int> = [2, 3, 5]
// generalized existentials
let sequence: any Sequence<.Element == Int> = [2, 3, 5]

(Paul Cantrell) #141

Yuta, something that confuses me immensely about this idea of opaque parameters:

As a consumer of this API, how do I know what Animal I can pass to this function? I can’t just pass any Animal; it’s not any.

In other words, what ___ makes this code compile?

func useSomeAnimal(_ animal: some Animal) { ... }

class Habitat {
  var someAnimal: ___

  func useIt() {
    useSomeAnimal(someAnimal)
  }
}