[Pitch 2] Light-weight same-type requirement syntax

There is zero doubt it's worth solving this problem. At this point however I stated my position loud and clear that I do not support the approach taken for this particular use case. Using Chris' words from the quote, I think the proposed solution is "mortar", but I would rather support an alternative solution which is probably way more complex, requires more time to develop and nail down correctly however it'd be a "brick" in such analogy.

8 Likes

No, generalized opaque result type constraints are fully modeled in the compiler implementation. The only missing piece is a generalized surface syntax, but the syntax design is a very difficult problem to solve, and despite the comments here, I do not believe there is one obvious syntax for it. I’ve already stated above that I’m writing up all of the idea that I know of — there are at least 4 different options and I’m hoping the community will surface more — for a new discussion thread, because it’s a problem worth solving in addition to what’s proposed here. I hope to post that discussion today or tomorrow.

Remember that this proposal is not just about opaque result types. I find this proposal much more broadly applicable to opaque parameters now that it has been accepted, where this feature is purely sugar for what you can already express today. This feature is not a replacement for a general syntax for opaque result type constraints. I would appreciate feedback on the merits of this proposal with that framing.

On a separate note, there’s a lot of speculation based on false extrapolation of comments going on above, and I find it unhelpful and disrespectful. The general theme of the recent generics proposals is improving the learnability of the generics system, primarily for novice programmers and others who shouldn’t have to dive so deeply into the generics system to write basic generic code. I’d like to ask everybody to please keep your comments on the topic of the technical merits of the proposal.

9 Likes

There is plenty of doubt. The use case for general constraints on opaque result types has not been demonstrated, unlike the very tangible need to constrain the element of a collection. Instead, it is being taken as a priori that there must be a use case. This discussion would benefit greatly from real examples of how opaque result types could be constrained in other ways and the expressive power this would bring. Maybe there are loads of good examples, but we need to see them to know for sure.

Again, this does not mean the feature should not or cannot be added eventually. But there is no urgent need, because it is not clear there is any need.

Putting aside the unnecessarily emotive language: in what way does it "break the neck" of the language user to introduce this feature in this way? You are making a claim that somehow not introducing generalized opaque result constraints hurts specific kinds of users, but do not provide any supporting evidence for this claim. Actual code examples would be really helpful here.

Bear in mind when thinking about those examples that opaque result types is almost entirely a feature for the production of frameworks. Not just ABI stable frameworks but also packages that want to remain source stable, avoiding a semantic version bump, whilst updating implementation details that would otherwise leak out in the form of a concrete type.

When you are purely a user with control of both sides of an interface, the need to use opaque result types is very marginal (it can sometimes be used to loosely couple code but in general brings as much hassle as benefits). So it is certainly true that improving this feature is being improved specifically to allow for better frameworks. But the claim that this is, in turn, hurting users needs substantiation.

There was the example quoted below. Maybe it got an answer, but I could not find it.

There was also a request for clarification about subclasses (does Collection<Base> accept [Derived] or not?), which did not have any reply.

6 Likes

It is difficult to ignore that a very specific phrase is appearing every single time without fail, and it very much suggests that you view this through the narrow lens of opaque types used in result position only. Firstly, that is a strange position for review manager of the recent opaque parameters proposal to take, and secondly, it seems to be directly contradicted by the proposal's author:

It's difficult to know what to make of this. There seem to be mixed signals about its intended utility.

9 Likes

Thank you for pointing this out - I think this is a good observation. I completely agree that evolving a generic signature from the sugared form to the more expressive form is important, and this will become a crucial part of the learning curve for concepts like type parameters and associated types.

The need to transform a signature when adding associated type requirements is an inherent problem introduced by the opaque parameters feature that was just accepted. I don't think that's a bad thing - from my teaching experience, showing novice programmers a "desugaring" transformation is a very illuminating part of the learning curve, because it shows that this seemingly-complicated thing they have to approach now is something they've already been writing in their code and have internalized what it means. To me, the implication here is that this desugaring transformation should be surfaced to the user, and it must be discoverable. My opinion is this should be surfaced through a diagnostic fix-it, and I'm also a big fan of this sort of transformation via the refactoring engine, as Gwendal suggested, in both directions (sugared → desugared, and desugared → sugared when possible).

Further, I think this proposal introduces that cliff at exactly the right point. You can start writing the extremely common constraints using this syntax. I do believe that when you're working with wrapper or container protocols like Collection, the constraint you want to write the vast majority of the time is on the Element type, and most conformances to such protocols are generic over that same associated type, which is why I believe the concept of a "primary" associated type is valuable. This proposal allows expressing same-type, conformance (and layout), and supertype constraints on primary associated types, so all of the fundamental constraint types are covered. The first is the core part of the proposal, and the others fall out of structural opaque types. So, Collection<Int> expresses a same-type constraint, Collection<some Numeric> expresses a conformance constraint (albeit with an extra type parameter, but that only makes a difference for those who care about extra type parameters in their ABI), and Collection<some MyClass> expresses a supertype constraint.

When the programmer wants to constrain a different associated type, that's when they really need to learn more about associated types and how they are different from type parameters. For basic constraints, that difference doesn't really matter, because you can constrain both forms of type parameters in the same way.

3 Likes

My point here was an attempt to not fixate the discussion on opaque result type constraints, because this proposal is also very valuable for opaque parameters, which are already "just sugar". Personally, I think input type parameters are more common and therefore this feature will be more heavily used in parameter position. Perhaps that is just because opaque result types are fairly limited today, but I still think it's more common to abstract generic code into a function rather than impose generic code onto callers. In any case, the intended utility is for any opaque type, whether in parameter or result position.

I think it's a stretch to claim this is a common use case but it's certainly a real one that might come up. But note, you used an example of a paramter not a result type. And so this problem has a solution in the language today, and we are just talking about whether there's a big leap from your first example to your second and whether the sugar should be adjusted to improve the latter at the expense of the former (which we can discuss more, my view is no).

(incidentally, I think this proposal also improves the second example, which could probably be written func doSomething<C>(_ items: C) where C: Collection<Int>, C.Index: Hashable)

But the main contention here is about opaque result types and expressivity. Your example should be about those. And if it was, it would fall apart. Because an API author is not likely to return a collection that is opaque yet with a hashable index. "But what if the user who is returned this opaque result type needs the index to be hashable" you might say. And this is a reasonable thing to ask, but the fact is the API author is not going to speculatively make their collection indexes hashable just on the off chance some user is going to need that.*

That isn't really unique to this feature though: this is just the standard downside of information hiding. When returning an opaque result type you are choosing to reserve flexibility at the expense of your users benefitting from (or, to put it another way, tightly coupling to) implementation details of the specific returned type. But this is already the case. For example, an API might return a Collection<Int> when really the user needs a RandomAccessCollection<Int>, or even specifically need or want an Array. The framework might even be returning one, opaquely, but the user doesn't get to benefit from it. That's the trade-off the framework author makes when they choose to minimize their interface.

* Here is a attempt to describe a case where this might happen: There are two developers working on the same codebase, one maintaining an app and the other a supporting framework dedicated to the app. The app developer finds they need to hash an index, but the framework developer has made it an opaque collection to keep themselves decoupled from the app developer. They discuss it and agree that the framework start constraining their index type to be hashable. This scenario is plausible to me but still not very compelling.

1 Like

Again - I think there is confusion here.

Opaque parameters are, IMO, by far the most common expression of this proposed feature. Result types are also very interesting, and enable new expressive capabilities, but when we consider the teaching and approachability side, parameters are possibly more important.

2 Likes

I think Ben and I are saying the same thing (in different ways); this proposal is NOT * about adding expressivity to opaque result types. It happens to do that in a limited way, but that is not the goal of the proposal.

EDIT: sorry, one word really changed the meaning of this comment :sweat_smile:

1 Like

The argument is circular: generic result types are uncommon because they are limited.

Again, sorry for insisting, but some Publisher<Int, Never> would be happily welcomed by all Combine users, who are not charmed (euphemism) by eraseToAnyPublisher. We're pretty happy SwiftUI was privileged with some View, but there are many other apis that are looking for improvements.

And if this is not "the goal of the proposal", I'm sorry but it is very much related. The pitch is entirely focused on a single primary associated type, and has a very short "future directions" section. The PersistentSortedMap example in the pitch text is not clear.

7 Likes

The proposal has been updated to include multiple "primary" associated types.

9 Likes

A framework vendor probably won’t have much incentive to express any constraints on their opaquely returned Collection’s associated Index type. They’ll probably just return some Collection<Int> even though they could also make guarantees about other associated types. I agree.

But a framework vendor probably do want to express constraints on their opaquely returned Publisher’s associated Error type, and return e.g some Publisher<Int, Never>.

Ah. Disregard my comment ^

Yes, it looks like some Publisher<Int, Never> is included in the pitch. That's what I'm inferring now.

1 Like

The challenge here is that this proposal is introducing two things: new language expressivity in the case of result types, and sugar for a common use case for parameter types.

They both rely on the same syntax, and as such it's appropriate to introduce them through the same proposal IMO. But it means the discussion needs to avoid crossing the streams when using arguments for or against the two parts.

In the case of the sugar, it is all about the benefit the sugar brings, as everything you can write with it you can write today. You can argue that the sugar is entirely unmerited, or you can argue that the sugar doesn't go far enough and other things should also benefit from that sugar. It comes down to judgement calls, feelings about aesthetics and how common particular use cases are. In your example, you are outlining a big leap from one form to another, and (I think) using that to say that Collection<Int> shouldn't be so well sugared, and should instead be Collection<.Element == Int> because then it's less of a leap to Collection<.Element == Int, Index: Hashable>. Is that the gist of it?

When talking about generic result types, we have a very different discussion because we are now talking not just about sugar, but about increasing expressivity of opaque result types – allowing you to return opaque types you cannot return today. So it is not just about the aesthetics and ergonomics of the sugar.

It is in this context that I am laying out my claim: opaque result types with arbitrary constraints are probably so marginally useful that it is a very low priority to add them to the language. By contrast, being able to return an opaque type constrained by a primary associated type such as Collection<Element> is a clear and immediate need for anyone producing source- or ABI-stable SDKs or packages (or even private frameworks inside an app, when many developers are working on its codebase), so should be assigned a high priority to unblock those developers.

As @jayton says above:

(I would be very interested in knowing if it is most or if it is actually all... the introduction of compelling use cases might shift my view of the priorities)

This does not rule out the addition of arbitrary constraints in the future. But the syntax for them is going to be tricky (both the leading dot and the named placeholder syntax have signficant downsides), and I don't believe working on this should block delivering the feature that does have a clear need for many developers.

4 Likes

I'm sorry but I don't follow swift evolution and haven't been a part of the core team since middle of last year. I don't have enough context to have an informed opinion here.

-Chris

6 Likes

I would refer Chris' comment, as quoted by @DevAndArtist, about building a language with bricks vs mortar, which I think is a good analogy.

Expressing it in terms of "doesn't go far enough" or "not common enough" does not fully capture the argument. It's about the principle of building a language from a collection of ad-hoc special cases, and whether or not that actually achieves the stated goal of being easier to learn (I think @rauhul's comment is illustrative):

No. The argument I've made against Collection<Int> and in favour of Collection<.Element == Int> is not just that the former fails to scale to other associated types, but also that it fails to scale to most other protocols.

The post from earlier contains a couple of examples to illustrate it. After the arguments have been laid out in such detail, I can only suggest that you read them. There is no "gist".

Okay, well I disagree. The examples have been laid out multiple times, by multiple contributors:

I don't believe we should allow n 'primary' associated types but not arbitrary constraints. They are almost identical, but with one slight difference: the former requires enormous source churn for library authors, who must now predict when this syntax should be allowed to work, all for the sake of a syntax which is so terse that it quickly becomes non-intelligible. The latter is more compatible, more flexible, and clearer at the point of the use.

8 Likes

See above – this example (as well as others like a DictionaryProtocol for dictionary types) was addressed in the proposal as examples of protocols specifying multiple primary associated types. This was a great example of feedback on the pitch giving a clear and important use case the proposal didn't cater to. The solution of allowing multiple primary associated types fits very neatly into the proposal, which was amended to cover this case.

The dividing line here is between associated types that are about the essence of the protocol (the "primary" associated types... perhaps we can find a better name, though that name doesn't actually appear in the language in the current proposal so does not need to be set in stone), versus associated types that are just part of the implementation mechanics of a protocol.

The primary types are the "essential" types. Element in the case of Collection, the Value and Error types in the case of Publisher, the Key and Value types for DictionaryProtocol, the Scalar value for SIMD. One way to spot these is that they almost always match the generic placeholders for the concrete implementations. So Array<Element>: Collection, Dictionary<Key,Value>: DictionaryProtocol, SIMD3<Scalar>: SIMD.

Then there are "supporting" associated types. Types that need to vary by implementation, and need to be used in the implementation of methods. Collection.Index is the most commonly encountered one. SIMD.MaskStorage would be another. There must be an associated type to link together the type used by startIndex, endIndex, subscript and so on. But for most use cases, it can remain opaque.

My contention is constraints on these "secondary" types, which are less common but do come up when used on parameters, are going to be extremely uncommon, maybe even to the point of almost entirely unused, on opaque result types. It is counter examples to this that I am looking for.

8 Likes

I know that applying this change to standard library types is in ‘future directions’, but I have a question about whether a particular direction of evolution is feasible.

One particular usability problem I have with generics is this: collection methods often have to return a specialised type from their methods (eg various Iterator structs, Publishers.FlatMap etc in Combine, LazyPrefixWhileSequence and other lazy views). This means that if I want to understand what I can do with the return value, I need to then go to that type and look at what protocols it adopts, then also wonder if there is any additional API it offers on top of these protocols . It looks like this proposal would allow eg flatMap to return some Publisher<P, Failure>, which would be a lot clearer. Is this correct? If it is, this then leads to 2 questions

  1. Will this work from a performance point of view? My understanding is that the compiler is able to better optimise these cases because it knows the types. I’m guessing this isn’t an issue, because SwiftUI uses opaque return types, but would like to check.

  2. Is it possible to migrate from the current situation with nominal return types to opaque return types? I realise that this would be source breaking so would need to wait for Swift 6 - but is it even possible?

Could the same notion of primary (or positional/indexed) types also be used for existentials? Eg have any Collection<Int> instead of (or even replacing) AnyCollection<Int>. I think this also would be helpful in replacing a bunch of nominal types where the names actually work to obscure what is important.

3 Likes