SE-0346: Lightweight same-type requirements for primary associated types

pyrtsa · 2022-03-15T17:25:04.833Z

gwendal.roue:

Why should the designer of Publisher choose Never instead of Error as the default Failure type?

Quick thought: for the same reason as why we don't annotate every non-throwing ordinary function with nothrows before the result type signature's ->.

Jon_Shier · 2022-03-15T19:37:24.881Z

And also because the only other available default, Error, would be a terrible default.

Ben_Cohen · 2022-03-15T22:02:32.604Z

benlings:

Conditional conformances . Some of the concrete return types (e.g. AsyncMapSequence , LazyMapCollection have conditional conformances. Will this be possible with opaque return types?

Not with Lazy in its current form. You still need to surface enough information to drive the conditional conformance. In the case of LazyMapCollection, that conditional conformance is driven by a generic parameter Base, the collection the lazy wrapper wraps. When that wrapped collection is bidirectional, so is the lazy collection.

If you just returned an opaque some Collection<Element> that happened to be a LazyMapCollection<[Int]> then the information to drive the conditional conformance is not available to the caller (even if the compiler sometimes "knows" it for optimization purposes, if the function is inlinable), so no bidirectional conformance.

This is kind of a fundamental conflict. You cannot both keep secrets, and provide functionality based on those secrets. There has to be some give and take. There's a simpler example of this: if you are returning an [Int] but you actually return some Collection<Int> you're holding back some really useful information from the caller. They would probably love to have an Array, or at very least some RandomAccessCollection<Int>, but in order for you to preserve future flexibility to return something different, you're not giving information of which they could otherwise take advantage.

A middle ground could perhaps be to create a protocol that provides just enough information to drive the conditional conformance, while reserving other flexibility:

extension Collection {
  var lazy: some LazyCollection<Self>
}

// A protocol that provides lazy versions of common
// Collection operations.
protocol LazyCollection<
  Base: Collection
>: Collection where Element = Base.Element, Index = Base.Index {
  var base: Base { get } // unwrap the laziness of the collection
  func map(...) -> some LazyCollection<Self>
}

and then, with one additional language feature we don't yet have, do something like this:

// note this next part is not currently valid Swift,
// you can't retroactively conform protocols to other
// protocols, conditionally or otherwise
extension LazyCollection: BidirectionalCollection 
  where Base: BidirectionalCollection { }

This would allow you to get the conditional conformance benefits without exposing concrete implementations like LazyMapCollection , LazyFilterCollection.

Note that unlike Collection, this chooses a whole collection, Base, as the primary associated type for LazyCollection, not just the Element type. This allows for the unwrap operation to return the original eager collection, and follows the pattern we've a few times where primary associated types usually match generic arguments of concrete types implementing the protocol.

Of course, this is quite a lot of effort to go to purely to hide the type implementation details of things like LazyMapCollection. It may well not be worth it, especially as the Lazy types in the standard library are fragile so you don't get the ABI benefits, only the "hide the types" benefit. And the nesting e.g. some LazyCollection<some LazyCollection<some LazyCollection<[Int]>>> still gets exposed. But it may be more compelling in other similar cases.

lorentey · 2022-03-16T21:37:44.842Z

Observation while drafting the adoption of this feature in the stdlib: having all associated types defined using consistent syntax is really valuable. It would be a shame to lose this consistency.

The associatedtype declaration serves as a useful place to attach API documentation, declaration modifiers like the (undocumented) override attribute, and it's just nice to have all of the associated types defined in the same way.

Example 1: Sequence (Doc Strings)

(This is the same issue as @allevato has raised above.)

We currently have:

public protocol Sequence {
  /// A type representing the sequence's elements.
  associatedtype Element

  /// A type that provides the sequence's iteration interface and
  /// encapsulates its iteration state.
  associatedtype Iterator: IteratorProtocol where Iterator.Element == Element
}

This is nice and symmetric, everything is consistent.

As proposed, this would now look like:

public protocol Sequence<Element> {
  /// A type that provides the sequence's iteration interface and
  /// encapsulates its iteration state.
  associatedtype Iterator: IteratorProtocol where Iterator.Element == Element
}

We no longer have an obvious place for Element's documentation.

Of course, we can still put it on the name itself, but this makes the declaration less readable:

public protocol Sequence<
  /// A type representing the sequence's elements.
  Element
> {
  /// A type that provides the sequence's iteration interface and
  /// encapsulates its iteration state.
  associatedtype Iterator: IteratorProtocol where Iterator.Element == Element
}

(Sometimes the doc string will have multiple paragraphs of text. Having it interrupt the protocol declaration would be a shame.)

One solution would be to integrate it into the doc string of the protocol itself in some way. This would work, but it makes the primary associated type even less similar to the rest of the associated types:

/// <Long discussion on what Sequence is all about>
///
/// - Associated type Element: A type representing the sequence's elements.
protocol Sequence<Element> {
  /// A type that provides the sequence's iteration interface and
  /// encapsulates its iteration state.
  associatedtype Iterator: IteratorProtocol where Iterator.Element == Element
}

Example 2: Collection (Associated Types with Modifiers)

Collection refines Sequence, and it wants to have the same primary associated type, Element.

In the current version of the stdlib, Collection redeclares Element with the (undocumented!) override modifier, to help associated type inference:

public protocol Collection: Sequence {
   // FIXME: Associated type inference requires this.
  override associatedtype Element
}

There is no obvious place to put such a modifier in the primary position:

public protocol Collection<Element>: Sequence {
}

Of course, we don't need such a modifier if Collection<Element> has the same semantics as the override associatedtype Element declaration. Which leads us to the last example:

Example 3: RangeReplaceableCollection (Inherited Associated Types)

RangeReplaceableCollection refines Collection, without explicitly overriding Element.

public protocol RangeReplaceableCollection: Collection {
   ...
}

If we want it to have the same primary associated type, we'll want to write:

public protocol RangeReplaceableCollection<Element>: Collection {
  ...
}

However, will Element here refer to the existing associated type that was inherited from Sequence, or will it introduce a new associated type with the same name?

Recomended Solution

My preferred solution would be to leave the existing associatedtype declarations as is. The bracketed syntax is a nice way to designate one of the associated types as the primary, but it should not define the associated type; it should just reference it.

public protocol Foo<Bar> { // error: Unknown associated type 'Bar'
}

For cases like Sequence, we would be able to keep the doc string where it belongs:

protocol Sequence<Element> {
  /// A type representing the sequence's elements.
  associatedtype Element

  /// A type that provides the sequence's iteration interface and
  /// encapsulates its iteration state.
  associatedtype Iterator: IteratorProtocol where Iterator.Element == Element
}

For cases like Collection, we wouldn't need to figure out a new way to attach modifiers:

public protocol Collection<Element> {
   // FIXME: Associated type inference requires this.
  override associatedtype Element

  /// A type that provides the sequence's iteration interface and
  /// encapsulates its iteration state.
  associatedtype Iterator: IteratorProtocol where Iterator.Element == Element
}

For cases like RangeReplaceableCollection, it would be clear that the declaration of the primary associated type isn't going to break ABI by adding a new associated type to the protocol:

public protocol RangeReplaceableCollection<Element>: Collection {
  ...
}

lorentey · 2022-03-16T22:02:00.494Z

I can't let this passage go unnoticed:

However, duplicating the associated type declaration in this manner is still an error-prone form of code duplication, and it makes the code harder to read. We feel that this use case should not unnecessarily hinder the evolution of the language syntax. The concerns of libraries adopting new language features while remaining compatible with older compilers is not unique to this proposal, and would be best addressed with a third-party pre-processor tool.

(Emphasis mine)

Library authors should not be considered second class citizens. Their concerns ought to be considered just as important as the concerns of the clients of said libraries. Library maintainers are people too; there is a limit to how much pain they are willing to suffer.

Forcing us to reach for preprocessing tools means that we're going to lose basic modern software engineering features, like intelligent code completion / syntax highlighting, structural editing, etc. etc. This will lead to lower quality output.

It will also make library code be even less similar to regular Swift code, making contributions that much more difficult.

(That said, most Swift packages have the option to simply bump their required toolchain version in a minor release, which resolves this pain.)

John_McCall · 2022-03-17T15:56:41.413Z

6 posts were split to a new topic: Language downgrade tool for source libraries

DevAndArtist · 2022-03-17T11:05:30.445Z

lorentey:

public protocol RangeReplaceableCollection<Element>: Collection {
  ...
}
However, will Element here refer to the existing associated type that was inherited from Sequence , or will it introduce a new associated type with the same name?

If anything then this should look like this:

public protocol RangeReplaceableCollection<Element>: Collection<Element> {
  ...
}

This would align with the declaration side of generic classes which like to forward its subclass' generic type parameter to the generic superclass.

John_McCall · 2022-03-17T15:54:00.384Z

I don’t want to derail this thread with an extended discussion about a downgrade tool. I’ve moved some of these posts to a new discussion thread.

I will say here that, while the Core Team understands the plight of source library maintainers, we are uncomfortable with accepting a restriction that all declaration syntax must be designed around the ease of #ifing it.

Srdan_Rasic · 2022-03-19T20:01:24.818Z

Is the problem being addressed significant enough to warrant a change to Swift?

Very much so! I'd like to adopt AsyncSequence in various projects instead of other reactive solutions, but the ergonomics just make it impossible, as depicted in the motivational section of the proposal, without falling back to a boilerplate type-erasure solution, like Combine does.

swhitty · 2022-03-21T22:18:02.479Z

Overall +1 , Big improvement on cognitive load when reading and writing generic functions.

Regarding @rethrows — I don’t think it has been explicitly addressed in the proposal or this thread and it probably needs to be.

hborla · 2022-03-21T23:06:42.815Z

swhitty:

Regarding @rethrows — I don’t think it has been explicitly addressed in the proposal or this thread and it probably needs to be.

This proposal does not provide a solution for writing effects constraints for rethrowing protocols. This is a general problem with both opaque result types and generic code that wants to generalize over an effect, independent of primary associated types. John's write up here outlines the limitations on the current form of rethrows and possible language solutions to lift those limitations:

Precise error typing in Swift Discussion

Introduction When error handling was added to Swift in 2015, it intentionally did not allow the expression of precise error types: either a function cannot throw, or it can throw any Error. The error-handling rationale I wrote, which uses an awful lot of words to describe most aspects of the design, gives this one pretty short shrift: There are many kinds of error. It's important to be able to recognize and respond to specific error causes programmatically. Swift should support easy pattern-m…

In my opinion, this problem should be solved as part of the rethrowing protocols proposal (or possibly subsumed by a "typed throws" proposal), which hasn't gone through a formal review yet. Depending on the language design for generalizing rethrowing protocols, effects constraints could either compose with opaque result types via keyword, e.g. (strawman!) some throws(ErrorType) AsyncSequence<Element> or via normal same-type constraints. For example, a precise error associated type could become a primary associated type of AsyncSequence, e.g some AsyncSequence<Element, MyError>. If the latter is the solution, we might want a way to default the constraint to Never, similar to how not writing throws means the default is non throwing.

So, the important thing to figure out for this proposal is whether the current design leaves room for these future language features. I can think of a few implications:

This proposal does not preclude the ability to add an effects specifier to opaque result types via keyword, e.g. some throws P.
If the model for rethrowing protocols transitions to use an associated type, we probably want to leave room for defaulting the constraint on a primary associated type. To me, that suggests that the proposal should remove the ability to add a default type witness for a primary associated type in angle brackets via =. If effects constraints are deemed not common enough to warrant lifting the error associated type to be "primary", perhaps that's motivation for generalized opaque result type constraints, which is also not precluded by the proposed design.
Currently, the default effect for code that's generic over a rethrowing protocol conformance is throws. If we want the syntax some AsyncSequence<Element> to default to non-throwing, we should not adopt primary associated types for AsyncSequence until there is a way to express that default. This proposal does not include adopting primary associated types in the standard library (that will be a separate proposal and discussion) so this proposal doesn't preclude that either.

DevAndArtist · 2022-03-22T07:35:38.696Z

hborla:

If we want the syntax some AsyncSequence<Element> to default to non-throwing, we should not adopt primary associated types for AsyncSequence until there is a way to express that default.

I feel like this example isn't a great fit for the problem being discussed. In my opinion some AsyncSequence<Element> should equal some AsyncSequence<Element, Error> and not some AsyncSequence<Element, Never>. This would be symmetrical to the fact that non-throwingness would require explicit Never or the explicit absence of throws if you look at non-throwing functions etc., while the regular throws without a concrete error type will very likely be the short form for throws(Error). On top of that AsyncSequence in particular would require an associatedtype Failure: Error = Error extension to avoid any breakage.

John_McCall · 2022-03-22T08:30:01.252Z

I think you have that backwards. In general in Swift, things default to not throwing and you have to write something extra to allow them to throw. That is a key part of our effects design: if you default to allowing an effect (like mutating, throws, or async), you have an immediate problem where lots of code and abstract interfaces unintentionally declare that they allow the effect, which can be difficult or impossible to clean up later. In contrast, if you have to opt in to declaring an effect, it’s very likely that you’ll quickly converge to opting in at all the right places, because even a basic implementation will fail to compile otherwise.

That corresponds to types like Task defaulting to an error type of Never. Naively, at least, it seems like AsyncSequence ought to be the same way. I agree that the fact that we’d need to treat bare AsyncSequence as throwing complicates this. Still, that just confirms to me the wisdom of leaving this open as Holly suggests so that we can consider it holistically with whatever language approach we decide on for throwing and non-throwing conformances.

DevAndArtist · 2022-03-22T08:38:05.947Z

John_McCall:

Naively, at least, it seems like AsyncSequence ought to be the same way.

I don't disagree with your reasoning and what Holly was previously analyzing. I just wanted to highlight that AsyncSequence is a bit of a special example that in my opinion does not fit well to describe this problem. Let me put this differently. Yes I agree that if AsyncSequence would already have an associated type for Failure, then it would make total sense to have some AsyncSequence<Foo> the same as some AsyncSequence<Foo, Never>. However since this isn't the case, and as far as my understanding goes on the future extension of this protocol with the missing Failure associated type, we have to provide a default which would mirror the todays behavior, which makes the bare some AsyncSequence a throwing type. Therefore I think it's a bit of an unlucky situation we have now. If I'm totally wrong on my expectations, then feel free to ignore my conversation regarding this particular issue.

Nobody1707 · 2022-03-22T14:09:52.871Z

DevAndArtist:

or the explicit absence of throws

I don't think an absence is explicit. In fact, I think a quality expressed by the absence of some trait is the definition of implicit.

Tino · 2022-03-25T22:05:08.123Z

stephencelis:

To look at this another way, I did a search for "generic protocols" (as opposed to "generic traits") and found mostly questions with accepted answers of "you want to use associated types" and articles about how to use protocols with associated types.

Well, that is all what we have, so it's no wonder no one suggests using a feature that does not exist.
On the other hand, the query "relation over co-equal types" was not even meant to be about a existing or hypothetical feature, but about the concept.

stephencelis:

I think when most folks want

Sorry to pick a specific example — there are some more, and I don't think speculation about what people might want is a good argument. On the other hand, there a quite some people who definitely want generic protocols, and I would not assume that this is not an informed opinion.

Tino · 2022-03-25T22:25:24.625Z

There has been some demand for examples in this thread, and I realized one compelling case is still missing (or maybe I just did not see it... or it is not that compelling ):

The whole family of Any-types (AnyIterator, AnyPublisher...) and manual conversion is just needed because the necessary generic parameter can't be applied to a protocol.
If we had that capability, all those glue types could be removed, and instead all actual types could (automatically) conform to generic protocol. That protocol would not have associated types, so you would not suffer from their limitations.
That is, you could define and use properties like

let numbers: AnyIteratorProtocol<Int>
// ....
   numbers = someIntArray.makeIterator()

hborla · 2022-03-25T23:02:50.136Z

The primary reason why AnyIterator<Element> et al is necessary to manually write as a generic struct is because you cannot write an existential type any IteratorProtocol<Element> where the Element associated type is constrained and known statically. Extending this exact feature of primary associated type constraints to existential types, which is a future direction of this proposal, would allow that to be expressed without manually writing a type-erasing wrapper type that preserves the primary associated type. There is still an issue that the existential type any IteratorProtocol<Element> does not itself conform to IteratorProtocol, but existential opening solves many use cases that would otherwise need conformances for existential types.

1-877-547-7272 · 2022-03-30T14:51:13.352Z

What is your evaluation of the proposal?

+1.

Is the problem being addressed significant enough to warrant a change to Swift?

Yes — this is a nice ergonomic change that makes generics easier to read and write, and takes a significant step towards allowing generalized existentials.

Does this proposal fit well with the feel and direction of Swift?

Yes. The syntax here is familiar because it's reused from generic types. Some people on here have opposed this syntax on the basis that this would be the most obvious syntax for generic protocols. However, generic protocols have been considered an unlikely addition to the language since before the release of Swift 3, in part due to the fact that associated types can usually be used for the same thing. (Case in point: String instances are able to be used as a Collection<Character>, Collection<UTF8.CodeUnit>, Collection<UTF16.CodeUnit>, or Collection<Unicode.Scalar> with the help of a few wrapper types.)

Alternative syntaxes like Collection<.Element == String> are pointlessly verbose — it would be like having to use Array<.Element == String> to refer to an array of strings.

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

I have not used any languages with a similar feature.

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

I've been following discussions relating to the feature and I've read the proposal itself.

One question: would it be an ABI-breaking change if a library compiled in library evolution mode were to replace a function that has the signature

func secondElement<C: Collection>(of collection: C) -> C.Element

with a function that has the signature

func secondElement<E>(of collection: some Collection<E>) -> E

? The second version of this function looks much clearer to me, but LibraryEvolution.rst states that you're not allowed to reorder generic parameters in a function. What's the story for existing libraries?

Slava_Pestov · 2022-03-31T14:15:56.716Z

1-877-547-7272:

One question: would it be an ABI-breaking change if a library compiled in library evolution mode were to replace a function that has the signature
func secondElement<C: Collection>(of collection: C) -> C.Element
with a function that has the signature
func secondElement<E>(of collection: some Collection<E>) -> E
? The second version of this function looks much clearer to me, but LibraryEvolution.rst states that you're not allowed to reorder generic parameters in a function. What's the story for existing libraries?

Yes, it would be ABI breaking. The first function's generic signature is

<C where C : Collection>

The second is

<E, C where C : Collection, C.Element == E>

The calling conventions differ -- the first one passes a single type metadata for C followed by a witness table for C : Collection, the second one passes two type metadata for E and C, followed by the witness table for C : Collection. Generic signature minimization does not delete generic parameters, even though in this case we could theoretically recover the type metadata for E from C.Element (that is, looking up the associated type metadata for Collection.Element from the witness table for C : Collection).