Pitch: Suppressed Associated Types With Defaults

Hi all,

I’m seeking feedback on a new language feature. I’ve included the introduction and proposed solution below, see Full Pitch Link for all the details.

Introduction

An associated type defines a generic type in a protocol. You use them to help define the protocol's requirements. This Queue has two associated types, Element and Allocator:

/// Queue has no reason to require Element to be Copyable.
protocol Queue<Element>: ~Copyable {
  associatedtype Element
  associatedtype Allocator = DefaultAllocator

  init()
  init(alloc: Allocator)

  mutating func push(_: Element)
  mutating func pop() -> Element
  // ...
}

The first associated type Element represents the type of value by which push and pop must be defined.

Any type conforming to Queue must define a nested type Element that satisfies (or witnesses) the protocol's requirements for its Element. This nested type could be a generic parameter named Element, a typealias named Element, and so on. While the type conforming to Queue is permitted to be noncopyable, its Element type has to be Copyable:

/// error: LinkedList does not conform to Queue
/// note: Element is required to be Copyable
struct LinkedList<Element: ~Copyable>: ~Copyable, Queue {
  ...
}

This is because in SE-427: Noncopyable Generics, an implicit requirement that Queue.Element is both Copyable and Escapable is inferred, with no way to suppress it. This is an expressivity limitation in practice, as it prevents Swift programmers from defining protocols in terms of noncopyable or nonescapable associated types.

Proposed Solution

The existing syntax for suppressing these default conformances is extended to associated type declarations:

/// Correct Queue protocol.
protocol Queue<Element>: ~Copyable {
  associatedtype Element: ~Copyable
  associatedtype Allocator: ~Copyable = DefaultAllocator

  init()
  init(alloc: consuming Allocator)

  mutating func push(_: consuming Self.Element)
  mutating func pop() -> Self.Element
}

Now, LinkedList can conform to Queue, as its Element is not required to be Copyable. The associated type Allocator is also not required to be Copyable, meaning the DefaultAllocator, which is used when the conformer doesn't define its own Allocator, can be either Copyable or not. Similarly, stating ~Escapable is allowed, to suppress the default conformance requirement for Escapable. Unless otherwise noted, any discussion of ~Copyable types applies equivalently to ~Escapable types in this proposal.

Defaulting Behavior

Swift's philosophy behind defaulting generic parameters to be Copyable (and Escapable) is rooted in the idea that programmers expect their types to have that ability. Library authors choosing to generalize their design with support for ~Copyable generics will not impose a burden of annotation on the common user, because Swift will default their extensions and generic parameters to still be Copyable. This idea serves as the foundation of the proposed defaulting behavior for associated types.

Here is a simplistic protocol for a Buffer that imposes no Copyable requirements:

protocol Buffer<Data>: ~Copyable {
  associatedtype Data: ~Copyable
  associatedtype Parser: ~Copyable
  ...
}

Recall the existing rules from SE-427: Noncopyable Generics. Under those rules, a protocol extension of Buffer always introduces a default Self: Copyable requirement, since the protocol itself doesn't require it.

By this proposal, default conformance requirements will also be introduced if any of a protocol's primary associated types (those appearing in angle brackets) are suppressed. For Buffer, that means only a default Data: Copyable is introduced, not one for the ordinary (non-primary) associated type Parser, when constraining the generic parameter B to conform to Buffer:

// by default,  B: Copyable, B.Data: Copyable
func read<B: Buffer>(_ bytes: [B.Data], into: B) { ... }

Unlike a primary associated type, an ordinary associated type is not typically used generically by conformers. This rationale is in line with the original reason there is a distinction among associated types from SE-346:

Primary associated types are intended to be used for associated types which are usually provided by the caller. These associated types are often witnessed by generic parameters of the conforming type.

The type Buffer is an example of this, as users often will build utilities that deal with the Data generically, not Parser. Consider these example conformers,

struct BinaryParser: ~Copyable { ... }
struct TextParser { ... }

class DecompressingReader<Data>: Buffer {
  typealias Parser = BinaryParser
}

struct Reader<Data>: Buffer {
  typealias Parser = TextParser
}

// by default,  Self.Copyable, Self.Data: Copyable
extension Buffer {
  // `valid` is provided for both DecompressingReader and Reader
  func valid(_ bytes: [UInt8]) -> Bool { ... }
}

If ordinary associated types like Buffer.Parser were to default to Copyable, then the extension of Buffer adding a valid method would exclude conformers that witnessed the Parser with a noncopyable type, despite that being an implementation detail.

(Link to continue reading the detailed design)

14 Likes

I'm in favor of this design, because it continues the goals of SE-0346. Specifically, it allows an intermediate Swift user to think of extension Array { } and extension Sequence { } as very similar things, even though one is an extension of a concrete type and the other an extension on a protocol.

We already assign a default behavior to extension Optional { }Wrapped is assumed to be Copyable. Not just because changing that would have been highly source breaking, but because extending a generic type is a beginner-intermediate feature, whereas learning about ~Copyable types is an intermediate-advanced feature.

We should do the same for protocols (must, in fact – doing anything else would be very confusing for everyone). And if we do hope to do things like enable Sequence<Element: ~Copyable> in future, this will be mandatory to avoid a source break (just like it was with Optional). But it's also the smoother learning curve for Swift users who want to write sequence algorithms but haven't learned about noncopyable types yet.

Of course you can take this further and say all associated types should default to copyable in extensions, but the detailed design goes into why that is not really practical. Defaulting the primary associated type is the sweet spot that unblocks nearly all use cases while keeping protocols similar-seeming to concrete types.

6 Likes

This doesn’t feel particularly consistent to me. I don’t inherently grok the distinction between primary and non-primary associated types when it comes to copyability. The treatment I would consider without having to understand all of this reasoning is that copyable is assumed anywhere it isn’t explicitly suppressed. This might make writing non-copyable logic in extensions more verbose, but it already is.

The “this associated type should always be assumed non-copyable” feels like a separate thing to me. Maybe an attribute on the protocol or something.

2 Likes

+1 on the pitch. I think this is filling an important gap for the ~Copyable and ~Escapable story in Swift. I have been writing a lot of code recently that is leveraging the existing experimental feature for suppressed associated types and I am really looking forward for it becoming a stable feature.

I also agree with the proposed approach regarding the defaulting logic for ordinary associated types since developers are often not even aware that those exist and it would put an additional discovery burden on them.

5 Likes

Makes sense to me. Ship it!

2 Likes

For those looking for a nightly toolchain to try this out, look for one from ‘main’ as of Dec 4th or newer.

SE-0346 says

Adding a primary associated type list to an existing protocol is a source-compatible change.

With this proposal as it stands, that would no longer be true:

protocol P {
    associatedtype T: ~Copyable
}

extension P {
    func f() { ... do something with T; we can't copy it ... }
}

struct Noncopyable: ~Copyable {}

struct S: P {
    typealias T = Noncopyable
}

func example() {
   S().f() // OK
}

Now add a PAT:

protocol P<T> {
    associatedtype T: ~Copyable
}

extension P {
    func f() { ... do something with T; it's now assumed to be Copyable ... }
}

struct Noncopyable: ~Copyable {}

struct S: P {
    typealias T = Noncopyable // still fine
}

func example() {
   S().f() // not fine; `f` doesn't exist because it's constrained to `where T: Copyable`.
}

The distinction between primary and non-primary associated types in the proposal doesn't make any sense to me, honestly. I think all associated types should be assumed Copyable in all extensions unless explicitly stated otherwise.

9 Likes

I believe the next step in this plan is to pitch an Iterable protocol or similar, which is like Sequence except the Iterator yields borrowed Elements and is itself non-Escapable. It wouldn’t make sense for an extension of Iterable to require an Escapable Iterator by default, because none of them will be in practice.

I personally feel the earlier version of the pitch was best (no default behavior for associated types, just lift the restriction) but I guess that’s not a very popular approach.

I would echo this concern: that adding a primary associated type can be done retroactively was a key tentpole of the design in SE-0346. I do not think reneging on this promise can be done lightly or without evidence that it is actively harmful to existing code. On a practical level—recall that, for a number of protocols in the standard library, we deferred adding primary associated type(s) because we weren't sure which ones should be primary, pending additional experience with the feature. We would have to make a final decision in the same Swift release in which this pitched design is implemented; otherwise, we would effectively be making a final decision never to add a primary associated type for those protocols.

On a philosophical level, I also disagree with the characterization of non-primary associated types as "implementation detail" in contrast with primary associated types. There is nothing about the Index type of a Collection, for example, that is any more "hidden" for users than the Element type: for example, even beginners become very quickly aware that you can't subscript a String with an Int. Having one but not the other as a primary associated type may be a fair signal that a user will not often miss parameterizing their existential type any Collection by Index; but it does not follow by that fact alone Index is less salient to someone writing or reading any useful generic extension on Collection: so I do not think that the case has been adequately made for this pitched design.

13 Likes

I just wanted to clarify that the only change that is source compatible is going from zero primary associated types, to one or more. Once you have at least one primary associated types, you cannot add more without breaking source compatibility.

6 Likes

This proposal still keeps it simple: “any generic types appearing in angle brackets are assumed Copyable/Escapable, unless suppressed”. It creates a consistency with generic structs, classes, and enums:

protocol P<T> {
  associatedtype T: ~Copyable
  associatedtype U: ~Copyable
}

struct S<T: ~Copyable>: P {
  typealias U = // ...
}

extension S {} // T: Copyable inferred, U stays as-is
extension P {} // T: Copyable inferred, U stays as-is
3 Likes

Hold on. This proposal doesn’t break any promises SE-346 made. If you’ve decided to suppress an associated type, you now should also decide whether it’ll be a primary associated type. That’s it.

If your protocol already has stated its primary associated types, then SE-346 already doesn’t let you change that without source break.

To be clear, no, we don’t have to do this. After this proposal, you can still make those associated types primary without source break. You can then later suppress that primary associated type’s Copyable or Escapable requirement, without further source break.

If, in the year 2030, I vend the following protocol:

public protocol P {
  associatedtype Q: ~Copyable
}

...and then in the year 2031, I decide I want to add a primary associated type list:

public protocol P<Q> {
  associatedtype Q: ~Copyable
}

...this is a source-breaking change for my clients.

Yet SE-0346 promises: "Adding a primary associated type list to an existing protocol is a source-compatible change." It is at odds with the requirement that you're pitching: "If you’ve decided to suppress an associated type, you now should also decide whether it’ll be a primary associated type." Both cannot be true for protocol P—we can have the rule in SE-0346, or we can have the rule pitched here.

The source evolution limitations engaged by the rule pitched here would be limiting for adoption of primary associated types: where the author of a protocol has diligently generalized it to support noncopyable associated types, it is ironically those types which benefit most from generalization (plausibly those that are semantically the most important ones for clients) which they can then no longer subsequently declare as actual primary associated types if they want to maintain source compatibility.

But we cannot adopt the feature you are pitching—which, yes, I'm making an assumption that we do want to adopt the feature in the standard library—without either first or simultaneously concluding which associated types must be primary. If we adopt the feature you pitch, we cannot later then add an associated type list without source break: that is the promise in SE-0346 which will no longer hold.

1 Like

Is this a plausible scenario, though? The reason SE-0346 promised source compatibility to adopters of primary associated types was to allow the stdlib to declare that, eg, Sequence.Element was primary. This was useful because of course Sequence predated the introduction of primary associated types.

Now that the feature exists, it’s probably not as common for someone to decide an existing protocol ought to have a primary associated type after the fact, especially not a new protocol introduced after SE-0346, like your hypothetical protocol from the year 2030.

If this ever became an issue, Kavon’s feature could be generalized further to explicitly state the defaulted associated types in the event this list differs from the list of primary associated types.

What if be more concerned about is if someone had a plausible example of a protocol where the list of primary associated types differed from those that ought to default to Copyable and Escapable within an extension.

3 Likes

Yes, and not merely plausible but I'd argue both likely and desirable to support.

That the currently pitched feature suppressing Copyable is being introduced in a later version of Swift than were primary associated types doesn't mean that protocols themselves will tend to evolve in that order. Of course, we must also consider what design makes sense in terms of a viable migration path from past versions of Swift; however, I do not read the rationale for the source compatibility guarantee in SE-0346 to be entirely or even principally about adopting the feature at a historical point in time. Rather, I think it would be the right design on its own merits even if it (and this feature) were both present from the get-go in Swift 1.0, because it accurately reflects how primary associated types are discovered for a protocol:

Whether an associated type ought to be primary is not always obvious right away and can require a judgment call possibly based on usage experience. This has been demonstrated by the back-and-forth discussion and ultimate deferred adoption of primary associated types in parts of the standard library (as an example), which by now is not limited by lack of a language feature but lack of certainty about what's most desirable for the specific protocols in question.

By contrast, my impression is that whether a protocol can and should accommodate a noncopyable associated type can be known with more certainty ab initio based on the protocol's (and associate types') semantics: if it were expressible in Swift 1.0—a counterfactual situation, but isomorphic to the situation of a newly created protocol in future Swift—it is the retroactive suppression of Copyable that would be an implausible scenario, whereas I can see good reason why the retroactive adoption of a primary associated type list after Copyable has been suppressed should be supportable.

6 Likes

I think extension Iterable where Iterator: ~Escapable is the price we all agreed to pay back when we added ~Copyable.

Perhaps the need to state the negative bound could be exchanged for a need to state the positive bound in Swift 7 or 8.

Perhaps in the short term, we could add an @extensionsSuppress(Copyable, Escapable) directive to the protocol and/or associated type to allow opting into that behavior by default, to kind of have our cake and eat it?

2 Likes

Personally, I think this would be the best option, even in the long term. The reason why protocols like Copyable and Escapable need to be explicitly suppressed, aside from backward compatibility, is to support progressive disclosure. So naturally, we'd make an associated type implicitly conform to Copyable and Escapable in generic code if we want people to use those associated types in generic code without first having to learn the concepts of non-copyable and non-escapable types, and if we think it's likely that they'll use those associated types in ways that require copyability and escapability in practice.

I think it's likely that it just comes down to a subjective judgment of what the progressive-disclosure sequence is or should be.

For an Index: ~Escapable type, arguments could be made either way. One could argue that we should allow people to use indices in generic algorithms on collections without making them learn about non-escapable types. But one could also argue that people are likely to just use for loops on generic collections until they've reached a point where they already learned about non-escapable types. Furthermore, one could argue that even if people use indices in generic algorithms on collections before they understand non-escapable types, they're unlikely to run into problems in practice, such as by passing indices to APIs that require escapability.

Another complication is that there are actually two ways people could run into issues. One is what was described earlier, which is that without an implicit Index: Escapable requirement, people would run into issues if they write a generic algorithm that requires indices to be escapable. But on the other hand, if there is an implicit Index: Escapable requirement, then people would run into issues if they try to use their generic algorithm on a collection whose indices are not escapable. So if we want to avoid making people learn about non-escapable types too early, we'd have to figure out which scenario is more likely.

I think the answer is more certain for an Iterator: ~Copyable & ~Escapable type, because it's unlikely that people would work with iterators directly, and even less likely that they'd try to copy iterators or make them escape. And like Slava Pestov said, most existing Iterable types would likely have non-escapable Iterator types, making the second kind of scenario much more likely than the first kind.

I suppose a caveat is that we wouldn't want something like @extensionsSuppress to itself be something that makes it unnecessarily difficult to learn Swift, at least not if that outweighs its benefits.

5 Likes

I like this direction (though it’s not just extensions but other forms of generic signature, so might need to bikeshed the naming). We already have some precedent for this type of thing with @ViewBuilder protocol methods, where you can opt-in to automatically having properties be inferred as @ViewBuilder like SwiftUI’s body.

I assume the standard library adoption discussions you’re referencing are here and here. It does seem like the source of uncertainty that led to the deferral for several sets of related protocols was based on predictions about future language features. That feels like a level of reluctance and forecasting that is unique to the standard library. But I do see your point about how usage experience is part of the discussion when making an associated type primary.

I don’t think that choice has been made just yet; it’s partly why SE-427 left-out the suppression of associated types.

On one end of the spectrum, having zero defaulting for primary associated types was pitched and ultimately decided it creates inconsistencies in the language, makes it impossible to adopt in pre-existing protocols without source breaks, and generally is the wrong default behavior for progressive disclosure.

On the other end of the spectrum, defaulting every associated type is not a panacea. In terms of positives, the freedom to delay picking your primary associated types, as blessed by SE-346, would extend to suppressed associated types too. But there are several downsides. Consider this protocol defined by some library author:

public protocol Queue<Element>: ~Copyable {
  associatedtype Element: ~Copyable
  associatedtype Allocator: Alloc & ~Copyable = DefaultAllocator
  
  var alloc: Allocator { read modify }
}

public struct DefaultAllocator: Alloc, ~Copyable {}

The Allocator is an implementation detail and there’s a default type DefaultAllocator that conformers will automatically use to satisfy the Allocator, if they don’t want to implement their own.

With “everything is defaulted” idea, users of the library, writing their own simple conforming types, can’t make use of a bare extension of Queue:

class LinkedList<Element>: Queue {
  var alloc = DefaultAllocator()
}

extension Queue {
  func clear() { alloc.reset() }
}

LinkedList<Int>().clear() // error: referencing instance method 'clear()' on 'Queue' requires that 'DefaultAllocator' conform to 'Copyable'

That’s because LinkedList is relying on the DefaultAllocator provided by the library that isn’t Copyable (for correctness), yet the extension is implicitly limited to types where Allocator: Copyable.

The DefaultAllocator is meant to be part of the progressive disclosure story for this protocol Queue, because people otherwise just need to store the noncopyable DefaultAllocator and call methods on it, but the default-everything–in-extensions harms this strategy. In this case, it really is that only Element needs to default to Copyable, since that’s the main associated type users care about (hence why it’s primary).

It also is impossible to pass this LinkedList as an existential, because that too would assume Allocator: Copyable. There is a way to suppress defaults for the primary associated types of an existential, using a generic type parameter that suppresses its conformance:

func ok<E: ~Copyable>(_ s: any Queue<E>) {}

But there is no support at all for constraining the non-primary associated types of an existential in Swift today, i.e., there is no any Queue<.Allocator == A>for some A: ~Copyable.


My proposal is trying to find the right point on this spectrum that is still easy for people to reason about and is the “right thing” for most protocols. Primary associated types seemed to fit the bill.

Does anyone who dislikes my proposed defaulting for only primary associated types care to comment on any of the alternative designs?

1 Like