+1 on everything including naming and meta-type stuff.
Review Summary
- What is your evaluation of the proposal?
+1 to the feature, although I have some significant disagreements with the proposal text.
- Is the problem being addressed significant enough to warrant a change to Swift?
Definitely. Generics are too difficult for new developers, and this syntax change helps clarify one aspect of it. I believe we need to radically rethink the entire language model around generics.
- Does this proposal fit well with the feel and direction of Swift?
I believe it does.
- If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
Swift's generics system is quite unlike other languages I've used, so I can't comment.
- How much effort did you put into your review? A glance, a quick reading, or an in-depth study?
I've thought about this issue a great deal, and posted about this problem many times in the past. Unfortunately, I wasn't able to participate in the pitch thread; it was open for only 3 weeks (which, when you consider the scope of this change, is astounding), at the end of the year when everybody is busy with their own projects.
There are a lot of proposals floating around right now, so clearly the team at Apple is also trying to get their projects done or meet some milestones before the new year. The rest of us are, too! But we don't generally have entire work-days to devote to swift-evolution!
I've mentioned it several times in the past, and I'll continue mentioning it: we don't give nearly enough time to analyse proposals of this scale, and I find the lack of consideration for what is traditionally a busy time of year to be disrespectful to the entire Swift community.
At least in this case, one person who (at least in their opinion) has something to contribute, was not able to participate in earlier phases because of the rushed schedule. I urge the core team to change this part of the process. I think the community would be better served taking time to devise a thorough plan for the future of generics, rather than hastily pushing through one-off proposals.
The Proposal
Again, limited time for a discussion of this deep and intricate topic, so I'm just going to speed through the proposal mention the parts I disagree with.
Existential types in Swift have significant limitations and performance implications. Some of their limitations are missing language features, but many are fundamental to their type-erasing semantics.
Some of their limitations (the biggest limitations, in fact) are due to the compiler being stubbornly literal about how it interprets code using existentials. Most of these issues could be cleared up by teaching the language model that, very very very often, value-level and type-level abstraction are one and the same thing.
Basically the only operation which splits the two is assignment. For quite literally everything else (including mutating methods), it's not possible to change the value's type.
func useExistential(p: P) {
generic(p: p, value: ???) // what type of value would P.A be??
}
There is actually an obvious answer to this - it is a value whose type may be coerced to the associated type of the value currently in p.
Syntactically, the cost of using one is hidden, and the similar spelling to generic constraints has caused many programmers to confuse existential types with generics. In reality, the need for the dynamism they provided is relatively rare compared to the need for generics, but the language makes existential types too easy to reach for, especially by mistake.
There is a deeper question here - why does the language pretend that this dynamism exists, even in contexts where it quite clearly does not?
There is no dynamism in the useExistential function above. p never changes its type. The optimiser knows this (which is why the mentioned costs are actually not as bad as one might expect), but the language model does not. Existentials are just artificially handicapped because our model of generics in the language is focussed on being explicit about differences which don't exist 90+% of the time.
any Anyandany AnyObjectare redundant.AnyandAnyObjectare already special types in the language, and their existence isn’t nearly as harmful as existential types for regular protocols because the type-erasing semantics is already explicit in the name.
I find it strange that Any and AnyObject are the only existential types which do not require the any keyword. It seems more cohesive to spell Any as any, and AnyObject as any Object.
I see the argument about churn, and I'm not convinced. We could still accept Any and AnyObject during a transition period, and IMO having these existentials (probably the most commonly-used existentials) stick with the legacy syntax creates an even more complex and confusing language model. Wasn't this proposal all about making things simpler and clearer for developers? Consistency would help.
There's already going to be a lot of churn. Why is it not worth changing these, but it is worth changing every other existential across the entirety of all code written in Swift?
The existential metatype, i.e.
P.Type, becomesany P.Type. The protocol metatype, i.e.P.Protocol, becomes(any P).Type. The protocol metatype valueP.selfbecomes(any P).self
To be honest, I don't think anybody's going to remember this; what they're going to do is awkwardly fumble around with the syntax until the darn thing just compiles! And I'll be right there with them. The whole .Protocol, .Type and .self stuff just barely makes sense as it is, and this change makes the differences even more subtle - (any P).self vs any P.self vs (any P).Type vs any P.Type.
I actually think this part of the proposal is a regression compared to what we have today.
Extending existential types
Please no. Can we remove this "Future Direction"? It's not necessary, and it's far from uncontroversial.
See Lifting the "Self or associated type" constraint on existentials - #138 by Karl for more information.
2 Likes
I did start a discussion back in October about the usability of generics as a follow-up to a generics vision document that was posted over two years ago. In my discussion thread, it was made very clear by many community members that the place we should start is clarifying existential types, and I took that feedback very seriously. @Jumhyn 's comment from that thread sums it up clearly:
This is true for function parameters, but it is not for stored instance properties. A stored let -constant with existential type cannot change dynamically for a single instance of the enclosing type, but there isn't much you can do with that assumption because the underlying type of the property can certainly change across instantiations of that type, which means the entire type must be parameterized on that underlying type of its stored property. This is a much bigger semantic difference than using an existential type, and I do not think introducing semantic differences between stored properties with existential type and all other kinds of storage is a good idea for the programming model.
The type system differences apply to functions, too. Here's a contrived example:
protocol P {}
func generic<T>(_ arg1: T, arg2: T) { ... }
func existential(arg1: any P, arg2: any P) {
generic(arg1, arg2)
}
The above code is valid because arg1 and arg2 have the same static type. This is not true if you treat each any P as an independent type parameter, which is what the semantics would need to be in order to pass two different value types conforming to P at the call-site.
There are also ABI differences between type parameters and existentials, and we can't change the ABI of a function based on how its parameters are used in the body. For all of these reasons, I firmly believe treating existentials as type parameters where possible is best left as an optimization.
I think a much better approach is for programmers to be explicit with any, and in the future, we could consider "defaulting" a plain protocol name to mean some instead of any, and encourage developers to think of the plain protocol name as an implicit type parameter rather than an existential type, as outlined in the future directions.
I'm not totally happy with Any and AnyObject in this proposal either, but I don't know that any alone as a replacement for Any works. It doesn't allow you to distinguish between the existential and singleton metatype unless we keep .Protocol around. If we make today's Any only usable as an existential type, we would be removing a feature from the language -- a way to explicitly write an unconstrained requirement -- which is arguably not useful, but it does change the fact that this migration is entirely mechanical (even the corner cases around typealias are mechanical by splitting the typealias into two). Perhaps it's possible to evaluate the likelihood that somebody out there has written some Any in their code. And maybe it's fine if the change isn't entirely mechanical, but I think the source change is a lot more acceptable if a tool can do it for you.
I'm completely open to more strongly considering any Value and any Object. My primary hesitation is that Value isn't the best name for an empty protocol composition.
A future direction is not a commitment or POR, nor does it mean a potential feature is uncontroversial. I do think that existential opening -- something you've frequently advocated for in other discussion threads -- would greatly alleviate the need for this feature in the majority of use cases, but I believe there are still rare use cases where an extension on an existential type would be useful. In any case, it's just a future direction.
5 Likes
It is one thing that we should clarify, but it would be significantly easier to do that if we had an idea where the pieces are going to land. And I don't count Joe's (now very old) post; it's a starting point, but I don't agree with having a solution just presented to us - it needs to be a discussion, and a bit more recent.
As I said, I believe we need more than just superficial tweaks - the reasons why the generics model is so difficult to learn go beyond just syntax.
I don't agree with this at all. The fact that the type cannot change is not an assumption; it is vital information. And there is plenty you can do with that information! Let's say I have a stored existential Collection:
struct Foo {
let bytes: /*any*/ Collection where Element == UInt8
func frobnicate() {
var idx = bytes.startIndex // <- Could do this, for one thing
while idx < bytes.endIndex { // <- We know typeof(idx) == typeof(bytes).Index, for another thing.
...
}
}
func frob2(from: bytes.Index) { // <- No reason we couldn't do this, either.
}
}
Is there anything semantically invalid about this? I can't see it. So no, I don't believe you "must" parameterise on the type -- it's a thing you may do, but it isn't and shouldn't be required.
And if the compiler can see that I create Foo containing both Array<UInt8> and Data, there is no reason it couldn't specialise the entire type and all of its members. Those specialisations may be type-erased as they escape in to broader contexts, but that's neither here nor there IMO. I have the option to then say - "no, I want the specific type used by each Foo to be propagated throughout all contexts", and introduce a type-level parameter.
That example is only valid because the compiler doesn't complain about "self-conformance" for protocols without associated types. Introduce an associated type and it immediately breaks, and even with current proposals and all the designs I've seen discussed over the last several years, there would be no resolution other than to entirely restructure your code.
EDIT: Oh, I see the generic type is unconstrained. Sure, that can still work. I don't see any reason why generic types conforming to a protocol without associated types/Self couldn't be coerced in to the same (existential) type if the type is not constrained to conform to P.
IMO, this is more like a bug in the existing model (or at least a strange quirk), rather than a feature. I don't think it invalidates the idea at all.
There wouldn't be any ABI differences if existentials as function input parameters ceased to exist ;)
This is precisely why I think a broader discussion on the future of the generics model is warranted. We're tiptoeing around in the dark right now. Is this entire effort going to be limited to only mechanical changes which can be made automatically? Or do we want to design an overall system which is more intuitive than the current one?
I don't really mind about the name, but I don't think "clarifying" existential syntax should result in us having 2 existential syntaxes.
I mean, either it adds to the proposal or it doesn't. If it adds to the proposal, it is given some extra weight which IMO it doesn't deserve. If it doesn't add to the proposal, it's noise. Why add noise?
Could Instance work?
1 Like
Beyond issues with coming up with a good name to replace Any, that such a change would necessarily imply every constraint T: Any and U: AnyObject would need to be rewritten should give us pause.
Indeed, while I readily subscribe to the rationale for the overall proposal that it's important to distinguish between P and any P for all the reasons outlined above, I'm not entirely sure that the same can be said for Any and AnyObject. Requiring users to rewrite their class constraints as U: Object—after promoting the spelling U: AnyObject over U: class just a few Swift versions ago [*]—isn't helping people write better code.
For this reason I think the proposal as it is strikes a sensible balance in this area (although I don't know that we necessarily need to warn about redundancy in any Any). Any change in this area should, I think, be the subject of further, separate study.
[*] On that note, though, since bikeshedding is happening anyway, let me throw in another thought:
Since U: class is still valid Swift, we could work in the other direction and spell the existential type any class. What would the counterpart be, then, to Any? Well...
4 Likes
It's not an issue that the type of bytes is opaque in the implementation. The reason why Foo would need to be parameterized on the underlying type for bytes is because of the type system implications for uses of Foo itself. In today's Swift, any instance of Foo has the same static type. This means that different instances of Foo can be used interchangeably, and because the type determines the layout of an object, this means that Foo(bytes: Data(...)) and Foo(bytes: Array(...)) need to have the same layout, which necessitates the dynamic storage that existentials provide. Otherwise, Foo must be parameterized on the underlying collection type, in which case you can no longer use arbitrary instances of Foo interchangeably. Boxing up an instance of Foo itself in order to allow interchangeable uses would impose all of the same issues with existential types onto users of Foo, so this strategy isn't really gaining anything.
I would much rather such a coercion be explicit, and this proposal does provide a way to do that. In that case, the example becomes exactly what's in the future direction to always treat plain protocol names as implicit type parameters, which I personally am very interested in pursuing.
I don't think it's a quirk. These semantics are fundamental to existentials versus generics. They provide different kinds of abstraction, which has implications on the type system. It's impossible to unify these two concepts, although I do agree with you that many uses of existential types today could be replaced with generics.
We cannot completely remove existentials as function input parameters from the ABI precisely because Swift is ABI stable. If we change the ABI of resilient APIs that take in existential types, we'd still need to emit the old version because compiled code needs to continue to work against newer versions of the library, and back deployed code will still need to use the old ABI.
2 Likes
We certainly would be gaining a great deal by that! It means that Foo is still effectively generic - even without being parameterised - and that its layout, as well as its internal computations, can be specialised together as a unit.
In many circumstances, there wouldn't even be any need to box up instances of Foo, as the compiler should be able to tell which version you're creating. The need to box only emerges once that instance escapes in to broader contexts where that information isn't as clear.
func frob() {
let data: Data = ...
let theFoo = Foo(bytes: data)
// 'theFoo' doesn't need to be boxed - not here, and not anywhere this function is inlined.
// It may need to be boxed if it wanders too far from this point, but even then,
// internally it will be specialised and we just dispatch calls to the correct variant.
}
But even in that case, you'd only need one dynamic call to the precise specialisation of Foo. Once you're inside Foo's domain, you'll be running code specialised to Array<UInt8>, Data, etc.
What I'm getting at here is that, today, (and even with these any and some proposals) the language model considers value and type abstraction to be fundamentally different, and we require users to learn and struggle with these concepts. And as you can see, in so many situations, that is just unnecessary.
If we're serious about fixing generics, that's what we need to be fixing. Reducing your exposure to the differences between these concepts, so that new developers don't even need to care that it's a thing.
What I mean by a quirk is that it totally falls apart if you add any level of realistic complexity to the example. It's fun, but it's not worth holding the language model hostage to support that very specific case (which would still be possible BTW).
Yeah I get that. For sure, legacy binaries are their own thing and need to continue to work. I don't see why that wouldn't be possible - the compiler already emits special code for interacting with resilient types.
When I say "existential function parameters would cease to exist", I mean you wouldn't be able to write that function. It may still be possible to receive an existential as an unconstrained function parameter:
func frob(_ a: any Collection) {
// Error: this is isomorphic with a generic function.
// There is quite literally no reason for 'a' to be an existential.
}
func frob2<T>(_ a: T) {
// 'a' could be anything - including an existential perhaps, why not?
}
Just as the current existential syntax will coexist with any P in 5.6, Any and AnyObject could stick around even in Swift 6 mode for source compatibility reasons if that's considered important.
But we should have a consistent story for the new generics syntax - these should be expressible as any X, where X is TBD.
The more I think about it, the more I like the idea of using Any as existential only.
Any.ProtocolbecomesAny.Type.Any.TypebecomesAnyType, a new builtin/typealias.- Generic constraints
T: Any- constraint is removed. This is a safe change that preserves semantics and can be done mechanically. (I'm surprised, but people are actually doing this. I'm curious what is the purpose of this.) - Protocol composition
Any & P-> similarly,Anyis removed, and it becomesany P. Also a mechanical change. (Most of the usages seems to belong to unit tests of Swift compiler and GitHub - Polidea/SiriusObfuscator, but couple times used in normal code as well.) some Any- according to sourcegraph.com it is used only in Swift compiler tests. IMO, it is ok to not support this. As a workaround, if someone will ever need this, they can declare their own empty protocol. Note that they need to conform to this protocol only one single type which is actually returned from the function.
Introduction of AnyType and no support for some Any is a price I'm willing to pay to avoid horrendous any Any or introducing inconsistency in the language.
I'm convinced enough to suggest this as a change to the proposal. @hborla, WDYT?
+1 in favour of this direction of disallowing some Any and some AnyObject for now. It keeps the mental model simple (Any works like existential typealiases, and vice-versa) and would allow for redefining Any as a typealias to the existential for a new top type (any Thing) as a future direction, if the need arises.
This is a nonstarter—in fact, earlier, I wrote a follow-up to my review detailing exactly why. Your suggestion would break Sendable back-deployment, because Sendable has to be usable as a generic constraint when it is aliased to Any for compatibility reasons. One can expect that future marker protocols will likely be back-deployed using the same feature.
I disagree completely. We should do exactly as much to avoid any Any as is proportional to the harm that it causes. And the harm caused by any Any is essentially zero, particularly since this proposal would continue and even favor the existing spelling Any. A separate proposal can look into renaming the pros and cons of renaming Any so as to avoid the repetitive nature of any Any.
2 Likes
Okay, folks, this is a review thread, not a discussion thread. If you’ve got something to say about the proposal, make a convincing argument in your feedback. If you think someone’s actually misunderstood something, as in factually incorrect, you can respectfully point that out; but this is not a venue for telling other people that their opinions are bad.
14 Likes
+1. This is an important step towards addressing a common misconception of Swift developers, especially those coming from an object-oriented programming background. The proposed change will make the language easier to teach and easier to learn.
Yes. The status quo for existentials is the single most common case of confusion/frustration for developers moving to Swift from an object-oriented background.
I think so.
With apologies to the Lispers, one of my favorite maxims on programming language design is that things that are the same should be spelled the same and things that are different should be spelled differently.
The proposed change hits the sweet spot on this. any P is sufficiently similar to some P that it invites the opportunity to teach/learn the difference. I think the alternative of Any<P> is too similar to regular generic types and would invite developers to assume they understood it. That effectively moves the misconception around rather than encouraging developers to confront it.
I read each version of the proposal and the entirety of the pitch thread, even the bits about existential metatypes.
8 Likes
Would it make sense to smooth over the exceptions of Any and AnyObject and do something like this, and then suggest the old spellings be deprecated? I'm not even sure if this makes sense.
typealias Any = any Value
typealias AnyObject = any Object
You would need to drop the any from both type aliases, because the type should remain be usable as a constraint.
protocol P {}
typealias AnyP = any P
// `T` is a concrete type conforming to `P`
func foo<T: P>()
// `T` is a concrete sub-type of the protocol existential `any P`
func bar<T: AnyP>() // error: maybe okay in the future
Dropping any from the type aliases brings us back to square one - to any Any. The whole discussion around this is about having Any usable as an existential.
Normally Any is not used as a constraint, because it is a no-op (I'm still catching up on Sendable use case and marker protocols). AnyObject is used as a constraint, but AFAIU, discussion is based on assumption (based on what data?) that AnyObject as existential dominates. So while having AnyObject as a convenience typealias for existential, you would still have to write <T: Object> for constraints.
Actually, AnyObject as a convenience on top of any Object has little value, so in case of AnyObject just renaming it into Object without any typealiases is viable. But formula "Object" = "AnyObject" - "Any" prefix, does not transfer well to Any. Applying it gives "Any" - "Any" prefix = "".
The issue can be potentially be avoided mechanically. If we would say that each type prefixed with Any (including the bare Any) play the role as an existential box which holds some other value then it would be okay to actually force the any keyword via the typealias.
typealias Any = any Value
typealias AnyObject = any Object
For any illegal usage such as <T: Any> the compiler could provide a simple fixit to swap Any with Value or whatever the name we would get instead.
2 Likes
I appreciate the response here, and the updates to the proposal to address the Any<P> suggestion a bit more. While I still think that Any<P> would be clearer in a vacuum, in the context of future proposed extensions to both any P and some P, I am more convinced of the benefits of the any P spelling.
5 Likes
The point this proposal is addressing is specifically the fact that the bare P of protocol P {...} can be two different things, depending on where they are placed in the code.
- an existential (
func f(p: P)) - a generic constraint (
func f<T: P>(t: T))
These two things are not interchangeable.
Whereas subclasses of a superclass are interchangeable in that anywhere the superclass is referenced, a subclass could have been referenced instead. They are both types, and anywhere one type is legal in the grammar, so is the other.
8 Likes