Can I also stress that a huge advantage of this feature is that it also reduces the scope of possible uses of your API. As an example in SwiftNIO, we hide several of our underlying data structures behind existentials: users never hold concrete EventLoop or Channel types, they hold the existential versions of those types. This allows us to strongly suggest to users that they should operate on these types generically, rather than concretely, making it easier for users to migrate their objects to alternative I/O models.
While in some cases we'd still have to resort to exposing existentials, it would be useful if we could express more featureful types in some cases. In particular we've had a strong desire to express the relationship between EventLoops and Channels, as not all EventLoops can host all Channels. But this is not really possible to do when users only hold the existential.
I bring this up to note that SwiftNIO does not care about the scope of the ABI: we'd be inclined to make any method that returned an opaque type @inlinable, to give the compiler optimisation opportunities. What we want to do is to avoid users expressing their code in terms of specific implementations, and instead to express it in terms of behaviours.
With that said, naming the type is not an extra burden for us, so most of the proposals thus far would work for the use-cases we've considered.
In the argument case, you can in fact pass anything that you could pass as any Animal. A type that conforms to Animal works by binding the implicit generic argument, and a type-erased Animal can also be passed, by opening the dynamic type.
Ahhhh, a light bulb finally went on. Is there any semantic difference between the two, then? Or is this essentially an optimization / code generation flag?
It's exactly the same relationship you have between <T: P> (T) and (P) today, and the differences come down to covariance and contravariance. For a purely input argument, accepting an existential is isomorphic to accepting an independent generic argument; in the former case, you can accept any type that conforms to P by type erasure, and in the latter case, you can accept an already type-erased P by reopening its generic argument. The differences come into play with inout and out arguments. With inout arguments, the difference between inout some P and inout /*any*/ P would be that the latter is allowed to change the dynamic type of the value, by replacing it with a new existential value, whereas the former, since it represents a fixed generic argument, cannot change the dynamic type, but can only update the argument with a new value of the same type. For function outputs, you have the differences between opaque types and existentials that the proposal describes; an opaque output type is always a consistent static type, whereas an existential output type can vary.
The section of the proposal comparing opaque types and existentials tries to capture some of the differences. As a concrete type, opaque types compose better with other language facilities than existential types. For example, given:
protocol P { ... }
func opaqueP() -> some P { ... }
func existentialP() -> /*any*/ P { ... }
Here are a few quick differences:
If I have a function takesP, I can only call it with opaque() and not with existential():
func takesP<T: P>(_: T) { ... }
takesP(opaqueP()) // okay
takesP(existentialP()) // error: "P does not conform to P"
You would have to extend existentials to allow them to conform to their own protocols to remove this limitation.
Existentials cannot involve protocols that have associated types, e.g.,
protocol P {
associatedtype Element
func getElement() -> Element
}
With opaque types, opaqueP().getElement() returns the Element of opaqueP()'s result type, which is a type we can reason about in the static type system.
With existential types, what type does existentialP().getElement() return? Statically, we have no idea, because existentialP() could return a value of any type, that has any Element type. The best static type we can provide for existentialP().getElement() is Any.
An earlier attempt at generalized existentials tried to approach this by tying the identity of the type to a particular let. For example,
let ep = existentialP()
let epElement = ep.getElement() // has a type "ep.Element", i.e., it's the element type of the value `ep`
Of course, this approach doesn't work if you have any possibility for mutation:
var mutableEp = existentialP()
let epElement1 = mutableEp.getElement() // type mutableEp.Element
mutableEp = existentialP()
let epElement2 = mutableEp.getElement() // problem: *also* type mutableEp.Element, but could be different from epElement1
There isn't an answer here that "just works" to give epElement1 and epElement2 distinct-but-reasonable-sounding types. The enhanced-existential proposal I linked won't give the expression mutableEp.getElement() any type beyond Any.
Existentials cannot involve protocols that have constraints described with Self. For example, perhaps P is (or inherits from Equatable):
protocol P: Equatable { ... }
The expression opaqueP() == opaqueP() is well-typed and reasonable.
The expression existentialP() == existentialP() is ill-formed, because the two calls to existentialP() can return values with different dynamic types. The == implementations don't support having the two parameters have different dynamic types.
Again, the enhanced existentials proposal contains some ideas for dealing with this by opening the type, so one could perform that == with:
let a = existentialP()
let b = existentialP()
if let openedB = b as? a.Self { // dynamically cast b to the type of a
print(a == openedB) //
} else {
print("different types")
}
Out of the box, opaque types behave more reasonably in the type system than existentials. As noted above, you can generalize existentials to somewhat account for this---but it's a complicated design space and you need to handle all of the issues above to get to the point where you can meaningfully return (say) an existential Collection and have the result be usable. I doubt they will even be ergonomic because of (3) above, the need to manually check whether two existentials have the same type dynamically.
For this reason, I somewhat regret that we gave existentials the privileged syntax (just P), rather than something more explicit (like 'any P') that made it clear that we're dealing with a more complicated abstraction. But we're in the position where they do have the privileged syntax, so it's important that the more appropriate feature for hiding a result type---opaque result types---be something that's not too onerous syntactically. some P fits that bill; a separate one-off opaque typealias does not.
Thank you for this write up side note. I was completely confused by the âreverse geneticsâ terminology that was being brought up to describe this feature. The opaque typealias approach looks like automatic type erasure? If so that is something I could use as a normal user and it has been something that I had to deal with before in swift surprisingly in not so complicated interactions that expected to work.
Given that typealias donât allow where clauses now, I would suggest that this feature be itâs own âopaquetypeâ or âinterfacealiasâ in the same way that we have âassociatedtypeâ be its own thing.
I went fairly far down the path of designing opaque type aliases, and for me they fell into this mid-point between opaque result types as-proposed and a more powerful forwarding abstraction.
Opaque type aliases are too heavy for the first motivating example in the proposal:
That opaque type alias CompactMapCollection is difficult to write in the first place. We gained no clarity by writing it out, and we really can't even come up with a name that's more meaningful than "the result of compactMap."
The same information is there, but without the extra declaration or the ugly type that needs to be synchronized with the body of compactMap. Note that in both cases I'm assuming we have the where syntax to add additional constraints (on the Element of the hidden type)---you need that in both SE-0244 and the opaque typealias proposal, although it's not outlined or used in the latter. I feel like Greg Titus made this argument well back in the pitch phase; it's part of what made me realize how much we were giving up in simplicity of use when the design started shifting toward opaque type aliases.
On the other hand, it would be nice if it were easy to have a named type that I can implement in terms of another type, but hide my implementation details. An opaque typealias would let me do that in one way, but it means that my named type can only be an alias for some other type---it can't have it's own identity.
Back in the pitch for this proposal, the question "is an opaque type alias really newtype?" came up. newtype is a Haskell construct that creates a new, unique type whose implementation is provided by another type. If you had such a thing in Swift, it would look a lot like opaque type aliases: you would define a new name, provide the (hidden) underlying type, and forward along some protocol conformances to state its initial capabilities. However, unlike with opaque type aliases, you could write an extension on the newtype that would only apply to the new type and not the underlying type, make it conform to protocols on its own, etc.
But once you look at newtype, you realize that it's not all that different from declaring a new struct with a single stored property (the underlying value) and then forwarding conformances along. @anandabits has a proposal draft for protocol forwarding that demonstrates this (it predates the switch to Discourse, so the formatting is a mess) that I find compelling. For example, it would help you implement a Meters type as a struct containing a Double, and then forward the FloatingPoint APIs so you get a unique type that let's you perform numeric computation with just a few lines of code.
For me, opaque type aliases fall into this uncanny middle ground between "lightweight way to hide a result type that's hard to write and reason about" and "useful feature for building new types." It solves neither problem well, but if it's there people will reach for it and not get what they want.
I'd say having a type that cannot be spelled out is just as uncanny, in a different way.
If the goal is to not have to spell out precisely what the concrete return type of a function is, then I can see these two approaches working.
Allow opaque typealias with inferred concrete type:
// note how the concrete type is not specified
typealias MyResult<T>: some Collection where Element == T
// In the same file:
// concrete type of MyResult is inferred from the returned type
func myFunction<T>(_ value: T) -> MyResult<T> {
return [value, value, value]
}
Or give it a name after the fact with #returnType(of: __function__):
// returning an opaque type
func myFunction<T>(_ value: T) -> some Collection where _.Element == T {
return [value, value, value]
}
// Anywhere in any module:
// aliasing the anonymous type
typealias MyResult<T> = #returnType(of: myFunction<T>)
The main difference is #2 lets you get away without creating a typealias by allowing anyone anywhere to create one. In #1 the typealias is mandatory but still easy to spell out because its concrete type is inferred.
That's an interesting point. Can you write an extension for an opaque type? My intuition tells me it should be allowed, but that's only for debate if you can somehow name the type.
FWIW, I was working on a new draft of the forwarding proposal with a modified design that incorporated the feedback I received in that thread. I set it aside when it became clear that this feature would not be considered for Swift 3. I am still interested picking it back up again someday (but I would need a collaborator to work on implementation).
This would be a reasonable thing to add. I had the same general feeling; if you need to name these things, in the common case I foresee there's really no better name than "return type of ". This would fit easily into the implementation model for this proposal because the opaque return types are only syntactically "anonymous"; in the compiler representation, they still have a unique identity.
This wouldnât help the protocol forwarding aka type erasure use case. I know there is a notion than those should be separate but my swift mental model breaks down when thinking about type aliasing a return type that is magically opaque.
Could the proposal address Anonymous types vs Type Erasure?
The ability to pass an opaque type into a existential looks to me like type erasure. Opaque Typealias makes that much easier. As a naive user this type erasure side effect is more useful to me than being able to declare anonymous opaque return types but I may be missing something.
That's an error today, but in principle it doesn't have to be. This could be allowed to work by "opening" the existential and passing the dynamic type as the binding for T.
#1 splits the source of the type inference from the type being inferred, which I'd prefer we don't do. Our only example of non-local inference in the language (associated type inference) has proven to be fairly confusing, and I'd rather we not introduce another case like this.
#2 seems like a reasonable extension, so yeah, I'd prefer that.
Doug
No, you cannot write an extension for an opaque type, because you don't know what type you're extending. Extending a (non-opaque) type alias works because you can see the underlying type that you're extending.
I think my mental model for an opaque type was an auto-generated wrapper struct with the original type as a private member. But it doesn't really work like that if you're able to see the underlying type at runtime. I agree it does not really make sense to allow extensions in this case.
I find approach #1 makes things clearer especially when the opaque type has constraints. It's true non-local type inference isn't that obvious. But that's mostly a concern for the implementer; the user of the function never has to deal with this type inference. The user has to decipher the function signature however, and I feel mixing constraints for the opaque return type with constraints for the parameters is pretty messy. The typealias approach (type-inferred or not) more clearly separates the two in my opinion.
The core team has considered the feedback from the review so far and has decided the proposal should be returned for revision .
This proposal and implementation lay important groundwork, but it is clear from the review that work described in the "future directions" section is necessary to make this a complete and well-integrated language feature. There are also similarities to and differences with another feature of generalized existentials.
In order to better consider this proposal as one of a series of potential additions to the language, the core team is recommending that the proposal authors put together a manifesto outlining in more detail the full arc of these features, similar to the generics manifesto.
The authors expect to be able to present such a document soon, and rework this proposal to fit into that context, at which point we will re-open the review.
For the manifesto, would be great to see:
A) an introduction and definition of Opaque Types in as language native syntax as possible (such as opaque type aliases) which would 1) make the feature more easily understandable, 2) allow to focus on feature, not new syntax, 3) make it easier to infer future directions for the feature, such as where clauses etc.
B) a separate section discussing shorthands, that, as discussed, are not limited only to Opaque Types, but also relate to generalized existentials and maybe others.
This would also allow the future discussion to focus on each topic separately, even if the shorthand might be the most used / preferred syntax for opaque types in practise.
Now that Joe has written up his thoughts on future generics UI improvements, the review has resumed on a new thread. Please direct further feedback on the proposal there.