Opaque result types

As an alternative syntax: how about something like:

extension BidirectionalCollection {
  public func reversed() -> opaque 
                       : BidirectionalCollection where .Element = Element,
                       : RandomAccessCollection where Self: RandomAccessCollection,
                       : MutableCollection where Self: MutableCollection {
    return ReversedCollection<Self>(...)
  }
}

func makeMeACollection<T>(_: T.Type) -> opaque 
                : MutableCollection & RangeReplaceableCollection 
                  where  .Element = T {
   return [T]()
}

The idea is that the return type is just opaque, and it's assumed to be of Any type. That can be refined with a list of conditional or unconditional conformances; all are specified using : Type, and there can also be a where clause containing conformances (where Self : RandomAccessCollection), same-type requirements (where Self.Element == String), and associated types (using .AssociatedType = ConcreteType).

I also toyed with the idea of having an if clause for conditional conformances, so you could do something like:

extension BidirectionalCollection {
  public func reversed() -> opaque 
                       : BidirectionalCollection where .Element = Element,
                       : RandomAccessCollection where .Element = Element if Self: RandomAccessCollection,
                       : MutableCollection where .Element = Element if Self: MutableCollection {
    return ReversedCollection<Self>(...)
  }
}

although that gets a little confusing since it's not clear when reading whether the 'if' refers to the associated types (e.g. that opaque.Element == Self.Element only when Self : RandomAccessCollection) or the conformance (e.g. that opaque : RandomAccessCollection only when Self : RandomAccessCollection).

As another option for keywords: you could replace the where before the associated types with whose and then use == instead of =, saving where exclusively for conditional conformances, although you still run into the issue where it's not clear when reading what the condition applies to (even though it would be clear in the grammar of the language).

Note that I've used .Element = Element in a few places; you could spell that more explicitly as .Element = Self.Element.

1 Like

This would break composabillity, because:

protocol A {
    associatedtype B
}

struct C { }

func d() -> opaque A where B: C {
    return ...
}

//just some kind of convenience method that always ends up calling d:
func e() -> opaque A where B: C {
    //other stuff...

    return d()
}

let f = [d(), e()] //OH NO, ERROR, NOT THE SAME TYPES (even though they actually are, but we can't express that.)
1 Like

I can agree with you on that. I think opaque P does read nicer in that exact use case, but it quickly starts to need more info in more complex contexts (forcing weird things like _.Element and a new form of typealias).

I guess my main concern is that this be a modular concept which can be applied wherever it makes sense without having to alter it's syntax or the syntax of what it is interacting with. I would also want the ability to be explicit about the type in the original use-case when I want to be.

One note: Just because the type shows when written in this form: Int as Comparable, doesn't mean that it is visible to anything external. The world sees it as this Opaque Comparable thing. It is only inside the function that we see it as Int.

Crazy idea! What if we were to spell it:

let a:opaque P = Int as P

That is, the external signature (and thunk if needed) is spelled 'opaque P', but 'Type as P' is how you create such things explicitly. You can't create 'opaque P' directly (except using as) because it doesn't contain the necessary info, but you can infer it from somewhere that does have that info. Thus foo()-> opaque P is you asking for it to infer, but you could still do foo()-> Int as P explicitly wherever you needed/wanted to.

I'll keep thinking about Conditional Conformances.

To be clear, I really like the idea of Opaque types... I just want us to find a compossible/reusable syntax for it that will allow us to unlock it's full potential over time.

If we have a functions like:

typealias MyCollection<T> = [T] as MutableCollection & RangeReplaceableCollection where Self.Element == T
func makeMeACollection<T>(_: T.Type) -> MyCollection<T> {
   return [T]()   // okay: `MyCollection<T>` is `Array<T>`
}

// or
func makeMeANumeric() -> Int as Numeric {
   return 5
}

The generated interface could be:

typealias MyCollection<T> = _ as MutableCollection & RangeReplaceableCollection where Self.Element == T
func makeMeACollection<T>(_: T.Type) -> MyCollection<T> 

func makeMeANumeric() -> _ as Numeric
1 Like

@Douglas_Gregor: Ah, I see your point.

Assuming it is only used for existentials, opaque types, and typealiases of either, what do you think of when syntax? It reads really clearly, but it does burn a keyword (but only in that position)... aka I couldn't have a type named when.

I do also have other ideas for where when could be used that rely on the upcoming compile time static code. Namely, you could conditionally run a piece of code at compile time based on the exact type. That is beyond the scope of the feature we are talking about here, however.

Understood.

I was thinking if there were enough valid use-cases, we could create a syntax that means "I explicitly want the compiler to infer and fill in this type at compile time". For example:

let x:??? = a.returnsAnInt() 
//same as let x = a.returnsAnInt()

func foo() -> ??? { //Compiler would infer String here
   return "foo"
}

Then the syntax would normally compose from the two concepts:

func bar() -> ??? as Comparable {...}

It would also work like Scala's "I'll think of a type/name later" placeholder because it wouldn't error until compile time when the type can't be inferred...

But if it isn't useful much beyond this single use case, then it doesn't make sense...

Edit: Ok, because of the Scala thing, I talked myself into checking whether this is something people want in a separate thread.

Similar to @Dante-Broggi's post, I would be totally fine with having the generated interfaces (e.g. Developer documentation, .swiftinterface, Jump to Definition) erase the named types (and maybe have _ there or something else). But in the actual source code that developers write, have the named types there (or at least option to use them as @Jon_Hull is asking).

So in the lines of " opaque O: Collection where O.Element " when written in code, is turned into:
" opaque _: Collection where _.Element " when exposed in generated interface.

2 Likes

I also strongly dislike the use of the underscore, so just to revive discussion about alternatives, here is one I haven't seen yet:

func makeMeACollection<opaque Result: MutableCollection & RangeReplaceableCollection, T>(_: T.Type)
     -> Result where Result.Element == T {
   return [T]()   // okay: an array of T satisfies all of the requirements
}

This would introduce opaque result types as generic arguments that aren't specified at the call site, but rather in the function that uses them (maybe a name like unique or fixed would be a better fit here).

Maybe this syntax could even be reused for other use cases, e.g. covariance and contravariance.

First, I'd almost be okay if the feature is only accessible in the form of opaque type aliases, except that requiring opaque types to be fully specified would complicate use within generic functions (such as in a lazy collection's map function)

Second, (to me) this feel like an additional constraint on existentials, saying this type represents a consistent, concrete type (a leaf, if it weren't for subclassing). This allows both use of self functions and potential compiler optimizations around memory and dispatch.

More of a bike-shed recommendation on painting technique than a recommendation of color; the first usable instance of generalized existentials might be via this opaque mechanism, and a name indicating it is consistent or concrete might be better than opaque (since this is an extra constraint on usage of protocols, which already provide opaqueness through their abstraction of the underlying type)

2 Likes

This may sound crazy, but how about this: Create a special do-nothing generic type called OpaqueType, and use it as a wrapper (like Optional). At the implementation site, you keep doing what you have been doing, except you wrap the return result in OpaqueType. So your example turns into this:

extension LazyMapCollection {
    public func compactMap<ElementOfResult>(
        _ transform: @escaping (Elements.Element) -> ElementOfResult?
        )
        -> OpaqueType<
              LazyMapSequence<
                 LazyFilterSequence<
                    LazyMapSequence<Elements, ElementOfResult?>
                 >,
                 ElementOfResult
              >
           >
    {
            return self.map(transform).filter { $0 != nil }.map { $0! }
    }
}

The tools and compiler will just refuse to show you the generic contents inside OpaqueType when you use it.

It’s unclear to me how generic protocols have anything to do with this and by that how they could possibly be inferior. Existentials and generic protocols serve a completely different and orthogonal purpose. One provides a means of type erasure for PATs, the other a means for multiple protocol conformances of a single type.

Yes, one could try to implement Collection<T> using generic protocols and hence quickly realize that it’s a garden path. But that’d be a bit like trying to screw in a lightbulb using boxing gloves, quickly realizing that it’s a lost cause and henceforth arguing that one should abandon boxing gloves entirely.

Both are false dichotomies.

Generic protocols as a concept are as orthogonal to associates types as generic types are to type aliases (as type members). Neither of them is inferior to the other. Instead they solve different problems and are even more powerful when combined.

As such generic protocols can be combined with associated types to great effect (see protocol Multiplication<T> in linked post) and could even be used together with existentials or opaque types.

4 Likes

By far the most common thing people want "generic protocols" in Swift for is as notation for existentials with associated type constraints, though, not independent-multi-parameter type classes. Although Rust uses generic protocol notation for the latter, it's not a given that Swift would. There are other threads discussing this; I think it's a distraction to bring it up here. The motivation of opaque return types is implementation hiding without type erasure, and the comparisons to generalized existentials are only to highlight why these are different features.

3 Likes

The _.Element thing is not my favorite, but I feel like the alternatives we're coming up with to avoid it are pulling us away from the purpose of opaque result types. The point of the opaque P syntax is that you don't write or care about the type, neither as the implementor nor as the client.

I'm not thrilled with the "hole" in the syntax (_ as Collection) as a way to express opaque result types, for several reasons:

  • The use of as is misleading here: when an expression or a pattern involves as, the whole expression/pattern acts like the type to the right of the as. Here, that type is an existential type (Collection), not an opaque type.

  • Because we don't have an equivalent to _.Element, we rely entirely on spelling the underlying concrete type somewhere.

    Doug

I agree that the alternative syntaxes proposed so far don't really seem significantly better. But I'm also not a fan of _ representing an unnamed thing that we care about, rather than an unnamed thing that we're ignoring.

How about repurposing Type? It's already reserved, but only in a member position (after a .), and it reads fairly well:

func reversed() -> opaque Collection where Type.Element == Element

You also have the additional semantic reinforcement that this is a single concrete type that we're talking about.

2 Likes

I think introducing Protocol<AssocType == ...> and Protocol<AssocType: ...> sugar, and simply not allowing the opaque type to be mentioned outside of that sugar, would probably be sufficient for most use cases. The former is a long-overdue generally useful improvement in and of itself. If the _ is the syntactic sticking point for a lot of people here, we could nicely sidestep that whole problem.

3 Likes

I'd like to clarify this slightly, because the details here matter and explain why I think existentials add a lot of undesirable complexity.

Dynamically, it's true that the returned value is something that conforms to P, exactly the same way that it might be true if the declaration was func existential() -> Any. You can try to down-cast this thing to some actual P via existential() as? SomeP, for example. It goes a little further, because (today) if P has requirements, you can use those too. But statically, it's not really a P, because you don't have access to all of P's interface.

struct Y : P {}

extension P {
    func f(_ x: Self) {
        assert(type(of: x) == type(of: self)) // valid assumption
    }
}

existential().f(existential()) // error member 'f' cannot be used on value of protocol type 'P'; use a generic constraint instead

The compiler can't allow that call because you could be passing a Y to f when Self is X.

Because we don't currently support “generalized existentials” this problem mostly hides in the margins: you have to write a generic function or use Self in an extension to see it. But if we generalize existentials it will become much more glaring, because basic features of a protocol's API will be unavailable on its existentials. For example, an existential Collection of Ints can't be indexed, even though indexing is a fundamental part of the Collection API.

So, I'm sure many people will disagree with me, but personally I'm not convinced that generalized existentials are a good thing for the language overall. In contrast, opaque result types solve a lot of real problems without introducing more of this “it's P but it's not a P“ craziness.

5 Likes

My sticking point there is Collection<Element == Element>. Not even the repeated Element part so much, although it emphasizes what really bugs me, which is that I expect == to be symmetric, always. Instead, this is really a key: value setup, but we can't always use : because it means conforms-to. So I'd be ok here if we could use a different 'operator'. Collection<Element := Element> ? Unfortunately, now I've introduced something that doesn't exist anywhere else in Swift.

Yes, and also: how do a I refer to the Self type? That's the role the _ is playing in the current pitch.

Doug

We could say that name lookup in the brackets is always from the context of the type being constrained by the protocol constraint, making you qualify the RHS, e.g. Collection<Element == Self.Element>. That'd also address this issue:

since you could write opaque Foo<X == Y> to constrain the opaque type's X and Y associated types to be equal.

1 Like

Might not be popular, but I'd still prefer verbosity over multiple meanings to what _ is. For example by using Opaque keyword instead:

func reversed() -> opaque Collection where Opaque.Element == Element

The advantage of the above being also that you can read it as proper sentence: " an opaque collection where the opaque type's element is equal to (Collection's) element

3 Likes