Opaque result types

(Jon Hull) #141

I am a bit behind on reading this thread, so forgive me if this has been proposed, but what about having an explicit definition of what the actual type is (so it doesn't have to be inferred from returns) and explicitly stating the part which is exposed externally.

Another way of putting it is that opaque marks a specific type, but it only guarantees some properties of that type will stay the same for ABI purposes:

opaque SpecificType exposing Collection where ...

You could also use a typealias like so:

typealias MyType = opaque SpecificType exposing Collection where ...

With some careful thought, we might be able to expand the usefulness of opaque types beyond just return types. Any method on the opaque type which returns Self should still be of the same type (even if we aren't quite sure what that type is). Because the underlying type is available to the compiler, it could determine if two opaque types are the same in different methods... meaning that you could get something as a return value, modify it with something that returns Self, and use it in some sort of setter method that expects the same opaque type. All of this is made easier if we use a typealias to give a public name to that opaque type. (Note: Since typealiases can be inside a type at times where a generic conflict doesn't allow the actual type being opaqued to be namespaced within the type, this can lead to much simpler names: e.g. Index vs. LazyBidirectionalBlahBlahBlahIndex)

I actually think something like Collection Index Types might be prime candidates for this expansion:

struct MyCrazyCollection<Element>:Collection {
    typealias Index = opaque Int exposing Numeric 

You would still refer to it as MyCrazyCollection.Index, but instead of knowing it was an int, you could only call the properties of Numeric on it. Note: this isn't limited to returns. It can be used anywhere the index is normally used.


(Jon Hull) #142

If people are worried about the conciseness of having two keywords, we could lose opaque (though I agree it is a great term) and do something like:

func foo() -> Int as Numeric  //This is actually returning an Int, but the outside world only sees it as Numeric

or as people have been suggesting allowing a local name

public func foo() -> MyPrivateCollection as C:Collection where C.Element == String

Note: The type on the left could be a private type, since only the part on the right is exposed externally.

The mental model here would be: I am actually returning type X, but I am only telling the world about some of it's properties (and the world will be limited to using it in that way). This means that the part on the left side of as can be swapped out without affecting ABI as long as everything on the right stays the same.

This could also be used with typealias:

typealias MyType = Int as Numeric

• Specifies the exact type being used without fixing it in the ABI (or exposing it "public"-ly)
• Very concise syntax
• Can be used not just for return types as shown in my last post

• Yet another magic use for as (though I believe it shouldn't conflict with current uses since the lhs is a concrete type)

To come back to the original example, it is still a bit ugly/complex, but I think it is just as readable (if not more) than the version with the opaque keyword and _.Element:

private struct LazyCompactMapCollection<Base: Collection, Element> { ... }

extension LazyMapCollection {
  public func compactMap<U>(_ transform: @escaping (Element) -> U?)
      -> LazyCompactMapCollection<Base, U> as C:Collection where C.Element == U {

For something like an Index, where you want to be able to set things as well, you would use a TypeAlias with it:

struct MyCrazyCollection<T:Equatable>:Collection {
    typealias Element = T
    typealias Index = Int as Numeric  //We are free to change Int to anything conforming to Numeric in the future
    func index(of element: Element) -> Index //This returns Int as Numeric
    func object(at index: Index) -> Element //This takes Index aka "Int as Numeric"

//We can take a returned Index (aka Int as Numeric) and use it in a parameter (potentially modifying it using Numeric methods returning Self)
let a: MyCrazyCollection<String> = ["a","b","c"]
let idx = a.index(of: "b")
let elem = a.object(at: idx)

//But we can't pass an Int
let intIdx: Int = 7
let elem = a.object(at: intIdx) //ERROR: type Int is not expected type MyCrazyCollection.Index

(Pierre Lorenzi) #143

I don't think opaque is clear, we could use hidden, it seems closer to me to the meaning "this is a constant type I don't have visibility on".

typealias SomeType = hidden T: MutableCollection where T.Element == String

I'm not sure but if we want to avoid a typealias, we may have to declare the type in the local scope to use it:

let value: hidden SomeInteger: FixedWidthInteger = someFunction()
if SomeInteger.bitWidth == 32 { ... }

(Amir Abbas Mousavian) #144

@Douglas_Gregor does it pave road for factory initializers?

(Jordan Rose) #145

Factory initializers pretty much work already today for classes; there's just no syntax for them. (Semantically, they're non-inherited initializers that can't be called with self.init or super.init.) They're not blocked by this proposal; someone just needs to go pick a syntax, implement it, and write it up as its own pitch.

(Greg Titus) #146

Keyword bike shedding: I don't think the hiddenness of the result type is the important part for understanding (on the library users side rather than the authors side), since the actual type is available at runtime. And since libraries (especially the stdlib) will have so many more users than authors, I think the keyword would be more descriptive if it helped make clear that the type is a constant, singular type, in order to reduce confusion that it might either be a generic type (due to having a where clause) or an existential (due to also spelling a protocol name).

If it didn't conflict with the Optional case, I think some would have been an ideal keyword:
compactMap<U>() -> some Collection where _.Element == U

compactMap<U>() -> particular Collection where _.Element == U, perhaps?
compactMap<U>() -> specific Collection where _.Element == U, maybe?

(Chéyo Jiménez) #147

What about using the word trait? My other suggestions would be aspect or proxy

(TJ Usiyan) #148

One reason is that you can change the type vended by your library without breaking client code, in many cases.

(Douglas Gregor) #149

I've updated the proposal's introduction to bring this concern first and foremost, and make the "keeping these types private" argument second.


(Alejandro Martinez) #150

It's not that I grasp 100% this proposal but I would hesitate on using words like trait or aspect because in other programming context that has other meanings (interfaces + default implementations + provided storage).

I personally think that opaque is good as it reminds of C opaque pointers or structs.

(Alejandro Martinez) #151

Hi! I've been following the proposal for a bit now and read it a couple of times. Before I thought I understood what existential mean but now I'm not so sure anymore :sweat_smile: I would love if somebody could provide (or point to) some easier explanations about what they are, how they compare to this proposal and with the current state of Swift. Reading against the section of the proposal gives some light in to the matter but I feel like is so subtle that I would like more info about it.

From a simplistic view the example of LazyMapCollection. compactMap could just have a return type of Collection where .Element == ElementOfResult. My question is why is that not the same? what does opaque add? Aren't we just saying that compactMap returns a collection anyway?

I understand that the proposal tries to explain this so if the answer is read it again I will understand :grinning:

(Douglas Gregor) #152

I'm working on a revision of my proposal that will try to make this clearer, and will post again when I've pushed the updated proposal.


(Alejandro Martinez) #153

Thanks that would be amazing ^^

I see the problem and I'm in favour of fixing it of course, and this sounds like a great solution. I'm just curious and want to understand better the topic :blush:

Cheers ^^

(Chéyo Jiménez) #154

Sure. I think I'd be more in favor of this feature with a name like opaquetype to go along the lines of typealias and associatedtype

(Greg Titus) #155

'Existential' is from predicate logic. It's a statement that says that a thing exists.

So this Swift code:

protocol P {}
func existential() -> P

Will return a thing that conforms to P. It can be any sort of P (any concrete type at all that conforms to P). In a way, it's proof that an object can be made that is a P - thus the tie in to the idea from logic.

The code above will compile and run and work fine, but if your protocol P has an associatedtype in it, you'll get an error message instead because of limitations in the compiler and because there are semantic difficulties with figuring out how you'd actually work with such a thing when you don't necessarily know what the associated type(s) are. Working out these issues has been discussed as "generalizing existentials", thus the comments you'll see here on this forum about how everything will be sunshine and roses as soon as we get "generalized existentials".

Where returning an existential is logically saying "here exists one", a generic function is saying "all":

func generic<T>(t: T) -> T { return t }

For all types T, if you give me a t of that type, I'll return you something of that type. And, of course, with where clauses you can restrict that to some subset of types, but with generics the idea remains that if you have <T: Collection> and then -> T that I'm saying that this will work with all Collections.

Existentials: Writer of the function decides what the concrete type is, and it can be different every time. Caller just knows it is a thing that exists that is a P.
Generics: Writer of the function makes it work for all (of some subset of) types. Caller picks which particular type for each call.

Now these opaque types are like existentials in that the writer of the function decides what the concrete type is.
The difference is that it must be the same type every time.

And also these opaque types are a bit like responsibility-flipped generics, where the function writer picks the type, and the caller has the generic-esque responsibility of dealing with all (of some subset of) returned types.

And the result is that for a certain class of problem, you get most of the benefit you'd get from generalized existentials, but the implementation complexity and issues are much easier to reason about by making the rule 'one type and always the same type' instead of 'any possible existing type'.

(Alejandro Martinez) #156

THANKS :heart:️ This helps me get a better idea of it ^^

I guess my question remains with

Existentials [...] Caller just knows it is a thing that exists that is a P.

opaque types are like existentials [...] The difference is that it must be the same type every time.

What does this rule give you when at the call site you just know you receive something that conforms to the protocol? As the caller just knows that it works with something conforming to P, what's the advantage or reason to need to make sure that the underlying type is always the same?

(Nobody1707) #157

For one thing, because opaque types are always the same, if the opaque type is Comparable you can safely store it in an array and sort it. Any protocol with same type constraints or associated types will just work as an opaque type, whereas generalized existentials need extra work to get access to all of the requirements, and it's not clear that all of that work can be automated.

(Douglas Gregor) #158

As proposed, I have updated the proposal with a revised and expanded discussion of the differences between opaque result types and (generalized) existentials.


(Douglas Gregor) #159

Hello all,

I've been revising the proposal based on the discussions here. The most up-to-date proposal is here, which includes a number of changes since my original version:

  • Improved introduction describing the two main use cases ("I don't want to write the type" and "I don't want to expose the type publicly")
  • Revised and expanded discussion of the differences between opaque result types and (generalized) existentials
  • Slightly improved syntax for conditional conformance
  • Opaque type aliases as a possible future direction
  • Discussion of properties and subscripts

Thanks for all the feedback so far!


Scala like placeholder for types
(Amir Abbas Mousavian) #160

I prefer concrete keyword over opaque.