Opaque result types


(Mox) #203

Similar to @Dante-Broggi's post, I would be totally fine with having the generated interfaces (e.g. Developer documentation, .swiftinterface, Jump to Definition) erase the named types (and maybe have _ there or something else). But in the actual source code that developers write, have the named types there (or at least option to use them as @Jon_Hull is asking).

So in the lines of " opaque O: Collection where O.Element " when written in code, is turned into:
" opaque _: Collection where _.Element " when exposed in generated interface.


(Tino) #204

I also strongly dislike the use of the underscore, so just to revive discussion about alternatives, here is one I haven't seen yet:

func makeMeACollection<opaque Result: MutableCollection & RangeReplaceableCollection, T>(_: T.Type)
     -> Result where Result.Element == T {
   return [T]()   // okay: an array of T satisfies all of the requirements
}

This would introduce opaque result types as generic arguments that aren't specified at the call site, but rather in the function that uses them (maybe a name like unique or fixed would be a better fit here).

Maybe this syntax could even be reused for other use cases, e.g. covariance and contravariance.


(David Waite) #205

First, I'd almost be okay if the feature is only accessible in the form of opaque type aliases, except that requiring opaque types to be fully specified would complicate use within generic functions (such as in a lazy collection's map function)

Second, (to me) this feel like an additional constraint on existentials, saying this type represents a consistent, concrete type (a leaf, if it weren't for subclassing). This allows both use of self functions and potential compiler optimizations around memory and dispatch.

More of a bike-shed recommendation on painting technique than a recommendation of color; the first usable instance of generalized existentials might be via this opaque mechanism, and a name indicating it is consistent or concrete might be better than opaque (since this is an extra constraint on usage of protocols, which already provide opaqueness through their abstraction of the underlying type)


(Kenny Leung) #206

This may sound crazy, but how about this: Create a special do-nothing generic type called OpaqueType, and use it as a wrapper (like Optional). At the implementation site, you keep doing what you have been doing, except you wrap the return result in OpaqueType. So your example turns into this:

extension LazyMapCollection {
    public func compactMap<ElementOfResult>(
        _ transform: @escaping (Elements.Element) -> ElementOfResult?
        )
        -> OpaqueType<
              LazyMapSequence<
                 LazyFilterSequence<
                    LazyMapSequence<Elements, ElementOfResult?>
                 >,
                 ElementOfResult
              >
           >
    {
            return self.map(transform).filter { $0 != nil }.map { $0! }
    }
}

The tools and compiler will just refuse to show you the generic contents inside OpaqueType when you use it.


(Vincent Esche) #207

It’s unclear to me how generic protocols have anything to do with this and by that how they could possibly be inferior. Existentials and generic protocols serve a completely different and orthogonal purpose. One provides a means of type erasure for PATs, the other a means for multiple protocol conformances of a single type.

Yes, one could try to implement Collection<T> using generic protocols and hence quickly realize that it’s a garden path. But that’d be a bit like trying to screw in a lightbulb using boxing gloves, quickly realizing that it’s a lost cause and henceforth arguing that one should abandon boxing gloves entirely.

Both are false dichotomies.

Generic protocols as a concept are as orthogonal to associates types as generic types are to type aliases (as type members). Neither of them is inferior to the other. Instead they solve different problems and are even more powerful when combined.

As such generic protocols can be combined with associated types to great effect (see protocol Multiplication<T> in linked post) and could even be used together with existentials or opaque types.


(Joe Groff) #208

By far the most common thing people want "generic protocols" in Swift for is as notation for existentials with associated type constraints, though, not independent-multi-parameter type classes. Although Rust uses generic protocol notation for the latter, it's not a given that Swift would. There are other threads discussing this; I think it's a distraction to bring it up here. The motivation of opaque return types is implementation hiding without type erasure, and the comparisons to generalized existentials are only to highlight why these are different features.


(Douglas Gregor) #209

The _.Element thing is not my favorite, but I feel like the alternatives we're coming up with to avoid it are pulling us away from the purpose of opaque result types. The point of the opaque P syntax is that you don't write or care about the type, neither as the implementor nor as the client.

I'm not thrilled with the "hole" in the syntax (_ as Collection) as a way to express opaque result types, for several reasons:

  • The use of as is misleading here: when an expression or a pattern involves as, the whole expression/pattern acts like the type to the right of the as. Here, that type is an existential type (Collection), not an opaque type.

  • Because we don't have an equivalent to _.Element, we rely entirely on spelling the underlying concrete type somewhere.

    Doug


(Greg Titus) #210

I agree that the alternative syntaxes proposed so far don't really seem significantly better. But I'm also not a fan of _ representing an unnamed thing that we care about, rather than an unnamed thing that we're ignoring.

How about repurposing Type? It's already reserved, but only in a member position (after a .), and it reads fairly well:

func reversed() -> opaque Collection where Type.Element == Element

You also have the additional semantic reinforcement that this is a single concrete type that we're talking about.


(Joe Groff) #211

I think introducing Protocol<AssocType == ...> and Protocol<AssocType: ...> sugar, and simply not allowing the opaque type to be mentioned outside of that sugar, would probably be sufficient for most use cases. The former is a long-overdue generally useful improvement in and of itself. If the _ is the syntactic sticking point for a lot of people here, we could nicely sidestep that whole problem.


(Dave Abrahams) #212

I'd like to clarify this slightly, because the details here matter and explain why I think existentials add a lot of undesirable complexity.

Dynamically, it's true that the returned value is something that conforms to P, exactly the same way that it might be true if the declaration was func existential() -> Any. You can try to down-cast this thing to some actual P via existential() as? SomeP, for example. It goes a little further, because (today) if P has requirements, you can use those too. But statically, it's not really a P, because you don't have access to all of P's interface.

struct Y : P {}

extension P {
    func f(_ x: Self) {
        assert(type(of: x) == type(of: self)) // valid assumption
    }
}

existential().f(existential()) // error member 'f' cannot be used on value of protocol type 'P'; use a generic constraint instead

The compiler can't allow that call because you could be passing a Y to f when Self is X.

Because we don't currently support “generalized existentials” this problem mostly hides in the margins: you have to write a generic function or use Self in an extension to see it. But if we generalize existentials it will become much more glaring, because basic features of a protocol's API will be unavailable on its existentials. For example, an existential Collection of Ints can't be indexed, even though indexing is a fundamental part of the Collection API.

So, I'm sure many people will disagree with me, but personally I'm not convinced that generalized existentials are a good thing for the language overall. In contrast, opaque result types solve a lot of real problems without introducing more of this “it's P but it's not a P“ craziness.


(Greg Titus) #213

My sticking point there is Collection<Element == Element>. Not even the repeated Element part so much, although it emphasizes what really bugs me, which is that I expect == to be symmetric, always. Instead, this is really a key: value setup, but we can't always use : because it means conforms-to. So I'd be ok here if we could use a different 'operator'. Collection<Element := Element> ? Unfortunately, now I've introduced something that doesn't exist anywhere else in Swift.


(Douglas Gregor) #214

Yes, and also: how do a I refer to the Self type? That's the role the _ is playing in the current pitch.

Doug


(Joe Groff) #215

We could say that name lookup in the brackets is always from the context of the type being constrained by the protocol constraint, making you qualify the RHS, e.g. Collection<Element == Self.Element>. That'd also address this issue:

since you could write opaque Foo<X == Y> to constrain the opaque type's X and Y associated types to be equal.


(Mox) #216

Might not be popular, but I'd still prefer verbosity over multiple meanings to what _ is. For example by using Opaque keyword instead:

func reversed() -> opaque Collection where Opaque.Element == Element

The advantage of the above being also that you can read it as proper sentence: " an opaque collection where the opaque type's element is equal to (Collection's) element


(Ben Cohen) #217

To clarify: making various standard library methods return opaque types will not help avoid ABI lock in.

The places where this would be appropriate to use, such as lazy.map, reversed etc, are highly performance sensitive and so required to be fully inlinable and specializable. So they are still baked into the ABI even if, to the user, they're opaque.


(Jordan Rose) #218

I think introducing Protocol<AssocType == ...> and Protocol<AssocType: ...> sugar, and simply not allowing the opaque type to be mentioned outside of that sugar, would probably be sufficient for most use cases.

How do you conditionally provide conformances using this syntax?


(Joe Groff) #219

That question is orthogonal to how you spell conformances relative to the opaque return type. Using the <> constraint sugar works well with @dabrahams' proposed multiple return type syntax, if we do something like this, since it syntactically separates the resulting constraints on the return value from the input constraints.


(Dave Abrahams) #220

It's a subtle difference, but there's a comma before all but the first -> in my proposal


(Matthew Johnson) #221

Ahh, that's very subtle - I didn't notice it. I don't have any better ideas, but it certainly seems like this syntax will confuse anyone not already familiar with the feature.


(Jordan Rose) #222

More on conditionally-available conformances: the more I think about it, the less happy I am if conditional conformances can't be added in future versions of the library.

// I picked a syntax for this example; I'm not endorsing it.
func produceACollection<Element>(fromSingleElement element: Element) ->
  opaque C: Collection
where
  C.Element == Element,
  C: Equatable if Element: Equatable

Oops, I forgot to mention C: Codable if Element: Codable. Can I add that in the next version of my library? If I were using a distinct type, it'd be possible to do that, but here it seems kind of iffy. Is the ABI for this going to be:

swift_type_t *
getTypeFor_$_produceACollection(
  swift_type_t *Element,
  swift_witness_table_t * _Nullable Element_Equatable, // Element: Equatable
  swift_witness_table_t * _Nullable *out_Equatable); // C: Equatable

or

swift_type_t 
getTypeFor_$_produceACollection(
  swift_type_t *Element);

swift_witness_table_t * _Nullable
getConformanceFor_$_produceACollection_$_Equatable(
  swift_type_t *Element,
  swift_witness_table_t * _Nullable Element_Equatable);

? The latter is more flexible, but the former is probably way more efficient across library boundaries.