Opaque result types

Understood.

I was thinking if there were enough valid use-cases, we could create a syntax that means "I explicitly want the compiler to infer and fill in this type at compile time". For example:

let x:??? = a.returnsAnInt() 
//same as let x = a.returnsAnInt()

func foo() -> ??? { //Compiler would infer String here
   return "foo"
}

Then the syntax would normally compose from the two concepts:

func bar() -> ??? as Comparable {...}

It would also work like Scala's "I'll think of a type/name later" placeholder because it wouldn't error until compile time when the type can't be inferred...

But if it isn't useful much beyond this single use case, then it doesn't make sense...

Edit: Ok, because of the Scala thing, I talked myself into checking whether this is something people want in a separate thread.

Similar to @Dante-Broggi's post, I would be totally fine with having the generated interfaces (e.g. Developer documentation, .swiftinterface, Jump to Definition) erase the named types (and maybe have _ there or something else). But in the actual source code that developers write, have the named types there (or at least option to use them as @Jon_Hull is asking).

So in the lines of " opaque O: Collection where O.Element " when written in code, is turned into:
" opaque _: Collection where _.Element " when exposed in generated interface.

2 Likes

I also strongly dislike the use of the underscore, so just to revive discussion about alternatives, here is one I haven't seen yet:

func makeMeACollection<opaque Result: MutableCollection & RangeReplaceableCollection, T>(_: T.Type)
     -> Result where Result.Element == T {
   return [T]()   // okay: an array of T satisfies all of the requirements
}

This would introduce opaque result types as generic arguments that aren't specified at the call site, but rather in the function that uses them (maybe a name like unique or fixed would be a better fit here).

Maybe this syntax could even be reused for other use cases, e.g. covariance and contravariance.

First, I'd almost be okay if the feature is only accessible in the form of opaque type aliases, except that requiring opaque types to be fully specified would complicate use within generic functions (such as in a lazy collection's map function)

Second, (to me) this feel like an additional constraint on existentials, saying this type represents a consistent, concrete type (a leaf, if it weren't for subclassing). This allows both use of self functions and potential compiler optimizations around memory and dispatch.

More of a bike-shed recommendation on painting technique than a recommendation of color; the first usable instance of generalized existentials might be via this opaque mechanism, and a name indicating it is consistent or concrete might be better than opaque (since this is an extra constraint on usage of protocols, which already provide opaqueness through their abstraction of the underlying type)

2 Likes

This may sound crazy, but how about this: Create a special do-nothing generic type called OpaqueType, and use it as a wrapper (like Optional). At the implementation site, you keep doing what you have been doing, except you wrap the return result in OpaqueType. So your example turns into this:

extension LazyMapCollection {
    public func compactMap<ElementOfResult>(
        _ transform: @escaping (Elements.Element) -> ElementOfResult?
        )
        -> OpaqueType<
              LazyMapSequence<
                 LazyFilterSequence<
                    LazyMapSequence<Elements, ElementOfResult?>
                 >,
                 ElementOfResult
              >
           >
    {
            return self.map(transform).filter { $0 != nil }.map { $0! }
    }
}

The tools and compiler will just refuse to show you the generic contents inside OpaqueType when you use it.

It’s unclear to me how generic protocols have anything to do with this and by that how they could possibly be inferior. Existentials and generic protocols serve a completely different and orthogonal purpose. One provides a means of type erasure for PATs, the other a means for multiple protocol conformances of a single type.

Yes, one could try to implement Collection<T> using generic protocols and hence quickly realize that it’s a garden path. But that’d be a bit like trying to screw in a lightbulb using boxing gloves, quickly realizing that it’s a lost cause and henceforth arguing that one should abandon boxing gloves entirely.

Both are false dichotomies.

Generic protocols as a concept are as orthogonal to associates types as generic types are to type aliases (as type members). Neither of them is inferior to the other. Instead they solve different problems and are even more powerful when combined.

As such generic protocols can be combined with associated types to great effect (see protocol Multiplication<T> in linked post) and could even be used together with existentials or opaque types.

4 Likes

By far the most common thing people want "generic protocols" in Swift for is as notation for existentials with associated type constraints, though, not independent-multi-parameter type classes. Although Rust uses generic protocol notation for the latter, it's not a given that Swift would. There are other threads discussing this; I think it's a distraction to bring it up here. The motivation of opaque return types is implementation hiding without type erasure, and the comparisons to generalized existentials are only to highlight why these are different features.

3 Likes

The _.Element thing is not my favorite, but I feel like the alternatives we're coming up with to avoid it are pulling us away from the purpose of opaque result types. The point of the opaque P syntax is that you don't write or care about the type, neither as the implementor nor as the client.

I'm not thrilled with the "hole" in the syntax (_ as Collection) as a way to express opaque result types, for several reasons:

  • The use of as is misleading here: when an expression or a pattern involves as, the whole expression/pattern acts like the type to the right of the as. Here, that type is an existential type (Collection), not an opaque type.

  • Because we don't have an equivalent to _.Element, we rely entirely on spelling the underlying concrete type somewhere.

    Doug

I agree that the alternative syntaxes proposed so far don't really seem significantly better. But I'm also not a fan of _ representing an unnamed thing that we care about, rather than an unnamed thing that we're ignoring.

How about repurposing Type? It's already reserved, but only in a member position (after a .), and it reads fairly well:

func reversed() -> opaque Collection where Type.Element == Element

You also have the additional semantic reinforcement that this is a single concrete type that we're talking about.

2 Likes

I think introducing Protocol<AssocType == ...> and Protocol<AssocType: ...> sugar, and simply not allowing the opaque type to be mentioned outside of that sugar, would probably be sufficient for most use cases. The former is a long-overdue generally useful improvement in and of itself. If the _ is the syntactic sticking point for a lot of people here, we could nicely sidestep that whole problem.

3 Likes

I'd like to clarify this slightly, because the details here matter and explain why I think existentials add a lot of undesirable complexity.

Dynamically, it's true that the returned value is something that conforms to P, exactly the same way that it might be true if the declaration was func existential() -> Any. You can try to down-cast this thing to some actual P via existential() as? SomeP, for example. It goes a little further, because (today) if P has requirements, you can use those too. But statically, it's not really a P, because you don't have access to all of P's interface.

struct Y : P {}

extension P {
    func f(_ x: Self) {
        assert(type(of: x) == type(of: self)) // valid assumption
    }
}

existential().f(existential()) // error member 'f' cannot be used on value of protocol type 'P'; use a generic constraint instead

The compiler can't allow that call because you could be passing a Y to f when Self is X.

Because we don't currently support “generalized existentials” this problem mostly hides in the margins: you have to write a generic function or use Self in an extension to see it. But if we generalize existentials it will become much more glaring, because basic features of a protocol's API will be unavailable on its existentials. For example, an existential Collection of Ints can't be indexed, even though indexing is a fundamental part of the Collection API.

So, I'm sure many people will disagree with me, but personally I'm not convinced that generalized existentials are a good thing for the language overall. In contrast, opaque result types solve a lot of real problems without introducing more of this “it's P but it's not a P“ craziness.

5 Likes

My sticking point there is Collection<Element == Element>. Not even the repeated Element part so much, although it emphasizes what really bugs me, which is that I expect == to be symmetric, always. Instead, this is really a key: value setup, but we can't always use : because it means conforms-to. So I'd be ok here if we could use a different 'operator'. Collection<Element := Element> ? Unfortunately, now I've introduced something that doesn't exist anywhere else in Swift.

Yes, and also: how do a I refer to the Self type? That's the role the _ is playing in the current pitch.

Doug

We could say that name lookup in the brackets is always from the context of the type being constrained by the protocol constraint, making you qualify the RHS, e.g. Collection<Element == Self.Element>. That'd also address this issue:

since you could write opaque Foo<X == Y> to constrain the opaque type's X and Y associated types to be equal.

1 Like

Might not be popular, but I'd still prefer verbosity over multiple meanings to what _ is. For example by using Opaque keyword instead:

func reversed() -> opaque Collection where Opaque.Element == Element

The advantage of the above being also that you can read it as proper sentence: " an opaque collection where the opaque type's element is equal to (Collection's) element

3 Likes

To clarify: making various standard library methods return opaque types will not help avoid ABI lock in.

The places where this would be appropriate to use, such as lazy.map, reversed etc, are highly performance sensitive and so required to be fully inlinable and specializable. So they are still baked into the ABI even if, to the user, they're opaque.

1 Like

I think introducing Protocol<AssocType == ...> and Protocol<AssocType: ...> sugar, and simply not allowing the opaque type to be mentioned outside of that sugar, would probably be sufficient for most use cases.

How do you conditionally provide conformances using this syntax?

That question is orthogonal to how you spell conformances relative to the opaque return type. Using the <> constraint sugar works well with @dabrahams' proposed multiple return type syntax, if we do something like this, since it syntactically separates the resulting constraints on the return value from the input constraints.

It's a subtle difference, but there's a comma before all but the first -> in my proposal

Ahh, that's very subtle - I didn't notice it. I don't have any better ideas, but it certainly seems like this syntax will confuse anyone not already familiar with the feature.