Turns out that that's unfortunately necessary. If we use an opaque typealias, that becomes a concrete exposed type – because while normal typealiases are just syntactic sugar for some other type, an opaque typealias would actually have an identity itself (aka declaring two opaque typealiases with the exact same body would explicitly not make them equal).
So in order to actually reduce the amount of exposed types, we would have to go back to using the inline syntax from the first post that you've shown:
extension LazyMapCollection {
public func compactMap<U>(_ transform: @escaping (Element) -> U?)
-> opaque Collection where _.Element == U {
return LazyCompactMapCollection<Base, U>(/*...*/)
}
}
But, as we've since found out, this breaks composabiliy. So we need to allow this other proposed syntax to refer to our type from somewhere else:
#resultType(of: LazyMapCollection.compactMap(_:))
Well.... that's also an exposed type. And if we agree that there are situations where we might actually use this type, even if not suuuuper often, then we need to know about its existence.
So we haven't really reduced the number of exposed types
And maybe that's just acceptable. Maybe it's okay to have some more types around. I think that that might just be acceptable. Swift has a strong and powerful type system already and maybe reducing the number of types that exist just isn't always the primary goal.
I am interested in that. While I think that opaque types can't replace existentials either way (more efficient or not, they are just different feature-wise), I wonder what about existentials it is that you think would be expensive during compilation?
FWIW,I like private too, if it works without creating syntactic ambiguities. But it would be nice to hear objections, if anyone thinks it's a lousy idea — it's too easy to consider one side only, when deep in bikeshedding.
(Also FWIW, I don't see why this necessarily implies anything about using other access control keywords in the same position, but internal [i.e. a type that's opaque outside its module] might actually be an interesting concept to pursue.)
While private makes this opaque type concept somehow easier to understand, there’s slight semantic wrinkle in it. A regular private variable is completely hidden to outside code. The private (opaque) return type is only partially hidden; it’s visible to outside as ”type of thing with these rules”, only the concrete type is fully hidden.
I think this illustrates the problem with opaque types quite nicely, actually - which is that without an identifier to refer to them by, they are only useful for repeated invocations of exactly the same function, all of which must occur in a single local function.
That's quite unfortunate, because it seems to me that the main point of using an opaque type over a generalised existential is to allow reasoning about the underlying type (e.g. resolving Self for Equatable/Comparable).
That's only true if you care about the exact type being the same among multiple values. If you're composing the type into other generic wrappers, then the opaque type helps refine your interface to only specify the interesting aspects of the return type.
It's much more performant for a non-resilient interface where the caller actually knows what type is being returned by the implementation. This should be true within a library as well as with source libraries.
It should be slightly more performant even for a resilient interface because it avoids the potential boxing overhead of an existential.
An opaque result type on a concrete function (but not e.g. a protocol requirement) can be strengthened to a concrete type later without losing binary compatibility (as long as this is declared in some way so that entrypoints will be emited for the use of older callers), whereas an existential result type will always have the existential representation and therefore not be binary-compatible.
Yes but I think users will hit that wall very quickly. You can only keep the chain of generic wrappers going for so long; eventually somebody is going to care beyond the life of a single function call (and you can't store an opaque value without erasing to Any, because you can't spell it), or is going to construct two identical opaque values from slightly-differing call paths.
I would like to see some more comprehensive examples of where this is useful. I mentioned earlier that I don't think ReversedCollection couldn't use it without something like a Wrapper protocol as its Index, which doesn't seem like big win to me:
protocol Wrapper {
associatedtype Wrapped
var wrapped: Wrapped { get }
}
// (Add conditional conformances for Wrapper: Equatable, Comparable, etc...)
extension BidirectionalCollection {
func reverse() -> opaque BidirectionalCollection where Index: Wrapper, Index.Wrapped == Self.Index, Element == Self.Element
}
@John_McCall by 'non-resilient interface', do you mean something like an App bundle, where the binary libraries are all shipped together? Because wouldn't that be just as optimisable in either case? I mean in the sense that a dynamic library which is bundled with your app can almost be treated like a static library.
Sure, but in the cases we're talking about applying opaque types, it's already extremely unlikely that you'd end up with the exact same pile of wrappers from multiple computations, or that it happening by coincidence is an intentional, maintainable aspect of the code. This in and of itself can be a feature, since it can encourage refactoring the code to sink divergent branches into the structure of the wrapper (and thereby still get the full benefits of laziness and generic specialization). Worst case, existentials, type erasure, or forcing to a common currency (like Array for collections) are still on the table.
In theory, yes, you're absolutely right: given the same information necessary to know the concrete return type assigned to an opaque return type, an optimization could equally well recognize that a return value was always produced by erasing the same type and so either rewrite the signature to avoid boxing or just assume that the returned box contained a value of that exact type. It's just that that optimization falls out in a more immediate way with opaque return types.
I agree with your argument about conditional conformances, though: for collection protocols specifically, it would be hard to use this feature because you wouldn't be able to propagate reasonable conditional conformances. On the other hand, that's something I can't imagine would be expressible at all with generalized existentials.
How do you avoid boxing if the concrete type can be exchanged?
I wonder why not? As long as this is also declared in some way so that entrypoints will be emited for the use of older callers, I feel like that could work just fine for existentials, too.
I've used these guarantees of the type system a lot of times before. Creating an array is unacceptable for Collections if you end up consuming them in a way that shouldn't require the whole lazy collection to be resolved (such as searching for the first element that satisfies a condition etc.).
Type erasure in its current form doesn't work well with conditional conformances, because it requires an exponential amount of type-erasure wrappers for combinations of protocols (2^n wrapper types for n protocols and their combinations)
And if "just cast your opaque type to an existential" would be the solution (which I still find way worse than passing around a LazySequence), then at the very least, existentials would need to be implemented first. After all, Swift is being used in production, and it's not acceptable to just have a gap of a few years where this then-necessary workaround isn't possible yet.
Unfortunately, almost all examples of the usefulness of opaque types that I've seen here are using Collection protocols specifically!
I think that that's an important thing to consider in general: When don't we have a need to potentially propagate lots of conditional conformances? Most of the time, we probably want that to be possible.
I have highly generically based API because it involves lots of statically known information that is passed around through the generic types and conditional conformances. For example I have some something that looks like this and is exposed outside the library to its users (even if it's internal library):
extension CameraSettingEndpointInterface where Base : CommonCameraSettingInterface {
public subscript(cameraSetting: Base.CameraSetting) -> ArchivableFlowContainer<
CharacteristicContainer<
CameraSettingCharacteristic,
ObserveReadWriteConstraint<Camera.Setting.Data, UInt8>
>
> { get }
}
That API contains a lot of information that I can work with to build other generic abstractions around it. Most of that however could be hidden from the library users and rewritten as an opaque type and not an existential because it needs to work with generics API.
The interesting part for the user would be only this:
opaque ORWCharacteristicEndpointContainer where
ConversionValue == Camera.Setting.Data,
WriteValue == UInt8
I also have a type called SimulatedCore, which we use to create a simulated version of the hardware component of our product. There we simulate read/write operations which requires pattern matching of the input/output types (similar to what I showed above) which really is painful to type as I need to match the exact type for pattern matching to trigger correctly. I hope that one day I could switch to existentials there similar to the opaque type I wrote above. Since I have more than 50 of such types, and growing, you can imagine that it becomes hard to maintain at some point.
The problem for my codebase is real and I need both features, existentials and opaque types.
Just to throw more paint colors out there to consider, I'd recommend unspecified or something long like that. There is no need to Huffman encode this to a short identifier. This will come up infrequently in the APIs, so it is more important for it to be communicative. It may also be worth considering a compoundKeyword of some sort.
The same way we do with resiliently-sized value types and the type parameters of generic functions: we pass around references to appropriately-sized memory. If the compiler doesn't know statically how much memory to allocate, it has to do a dynamic allocation — but that allocation can still come from the stack rather than the heap (if that's appropriate for how the value is being used).
You're right that the binary-compatibility problems could be finessed away. However, changing a function to return a concrete type rather than an existential type comes with much more significant source-compatibility risks, the most obvious being with code like:
var c = someCollectionOfInts()
if c.count < 3 { c = [1, 2, 3] }
In contrast, code that was previously correct with an opaque type is likely to have the same overall semantics with a concrete type even if overloading behavior changes subtly.
I think Collection is pretty uncommon in both the complexity of its hierarchy of protocol refinements and the importance of propagating the most accurate refinement. The use cases we imagine for opaque types are simpler, more like having a Shape protocol together with a variety of functions that create shapes which don't want to be locked into exactly which Shape they return. But I agree that the proposal needs to demonstrate examples of these better.
Interesting, that does makes sense, I didn't know that the compiler already did this with generics. But I feel like even if the memory isn't boxed, method calls / member access would have to be (they would have to use dynamic dispatch). Or is there a way around that too?
True, but ultimately it is still very much possible for opaque types too, as mentioned by you in regards to "overloading behavior":
protocol A {
func b<C>(d: C)
}
struct E: A {
func b<C>(d: C) {
print("Unspecific function")
}
func b(d: Int) {
print("Specific function")
}
}
func f() -> opaque A {
return E()
}
f().b(d: 1)
Should print "Unspecific function". Now let's replace the signature:
func f() -> E
And it prints "Specific function".
Ultimately, either of these are really just cases of "nothing is truly source-compatible", because the accessor generation boils down to adding an overload which can always cause these issues.
Opaque types being oh so slightly better at avoiding it, while still very much allowing it, is in my opinion not enough of an argument for them.
We could theoretically statically dispatch the protocol requirements on the opaque type by creating functions that forwarded to the concrete implementations for all of them. That would have terrible code-size costs, though.
We tend to discard that kind of overloading-based argument when considering source compatibility because (1) the examples tend to be highly artificial and (2) it's a kind of argument that can be wielded against essentially any change to the standard library at all, even things as apparently innocuous as adding new methods to existing concrete types (what if the programmer had their own extension that provided an identically-named method with different semantics and slightly more general parameter types such that the new method was preferred?).
In contrast, returning a concrete type instead of an existential seems like a much more plausible source of problems.
