SE-0437: Generalizing standard library primitives for non-copyable types

Evolution Proposal Reviews
,
1

Hello, Swift community.

The review of SE-0437: Generalizing standard library primitives for non-copyable types begins now and runs through June 4th, 2024.

Reviews are an important part of the Swift evolution process. All review feedback should be either on this forum thread or, if you would like to keep your feedback private, directly to me as the review manager, either by DM or email. When contacting the review manager directly, please put "SE-0437" in the subject line.

Trying it out

If you'd like to try this proposal out, you can download a toolchain supporting it from Swift.org; the most recent Trunk Development (main) snapshots support the feature.

What goes into a review?

The goal of the review process is to improve the proposal under review through constructive criticism and, eventually, determine the direction of Swift. When writing your review, here are some questions you might want to answer in your review:

  • What is your evaluation of the proposal?
  • Is the problem being addressed significant enough to warrant a change to Swift?
  • Does this proposal fit well with the feel and direction of Swift?
  • If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
  • How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

More information about the Swift evolution process is available at

swift-evolution/process.md at main · apple/swift-evolution · GitHub

Thank you,

John McCall
Review Manager

20 Likes
3

I have to admit, I'm getting a little tangled up in language, for instance:

Generalizing these constructs for noncopyable types does not fundamentally change their nature -- an UnsafePointer to a noncopyable pointee is still a regular, copyable pointer, conforming to much the same protocols as before, and providing many of the same operations. For example, given this simple noncopyable type Foo :

Now, I assume since we're defining this struct for example purposes we can assume it's not Copyable, but over the last few months I've been reading ~Copyable as "not necessarily Copyable". I wonder if this kind of language scales?

The proposal seems to use the binary Copyable / not Copyable rather than ~Copyable referring to the absence of a conformance. Should that change?

1 Like
4

All of these constructs (e.g. Optional) continue to work with copyable types. By going from:

Optional<T> to Optional<T: ~Copyable>

we really are saying that T is "not necessarily Copyable", rather than saying that T is definitely not Copyable.

4 Likes
5

Any concrete type, with all generic parameters provided, is either copyable or noncopyable, at least at run time. “Not necessarily copyable” is something that applies to generics, either a generic parameter, or a generic type that may or may not have a conditional Copyable conformance provided.

This doesn’t invalidate your point that it’s easy to confuse these things, but the proposal text you quoted is correct as written.

9 Likes
7

The proposal says that the top-level map will not be included in this initial transformation because the transform argument makes sense as either (borrowing T) -> U or (consuming T) -> U, but then proposes including Optional.map<U: ~Copyable>(_ transform: Wrapped -> U?) -> U? without solving the problem of whether the first argument to the closure should be borrowing or consuming. Does this mean Optional.map’s closure effectively defaults to borrowing its first argument?

1 Like
8

Thumbs up on the changes to Optional, Result and other unsafe and managed buffer types. It seems pretty common sense to me, including all the take and consuming changes – seems necessary around the concept. As indicated, it does seem to flow on from earlier SE proposals in the same manner. I do wonder how compiler errors are going to look in these cases (I still expect I'll be confused the first time I'm told the left side of ?? was reused after being consumed).

The effect of these types alone will be pretty narrow.

On the way to generalizing the Standard Library's current sequence and collection abstractions, we'll also need to implement a variety of alternatives to the existing copy-on-write collection types, Array, Set, Dictionary, String, etc, providing clients direct control over (runtime and memory) performance: consider a fixed-capacity array type, or a stack-allocated dictionary construct.

This is the problem I (and I'm sure most others) want to see solved. Especially an array-like type that can't be accidentally copied – although that seems separate to this proposal which is focussed on the element type, rather than the container.

All I can really say is that I hope these changes are walking down that path but it's difficult to casually see the intermediate steps.

Edit Oh, wait, does the Hypoarray concept work as a non-copyable container, even when the element is copyable? If that's the case, then I think I see better how these things are going to work out.

2 Likes
9

The map function shown in the proposal is in an extension without the relaxing ~Copyable constraint, so that method will only be available for optionals where the wrapped type is copyable. If you’re working with a noncopyable optional, you won’t have access to map or flatMap, and will need to perform that transformation manually.

2 Likes
10

In the detailed design, Optional is "conditionally copyable", and the default conformance is explicitly suppressed:

enum Optional<Wrapped: ~Copyable>: ~Copyable {
//                                 ^^^^^^^^^

whereas Result is "conditionally noncopyable", and the default conformance seems to be implicitly suppressed:

enum Result<Success: ~Copyable, Failure: Error> /*: ~Copyable */ {
//                                              ^^^^^^^^^^^^^^^^

Are these intentional differences, or should the detailed design and implementations be updated?

3 Likes
11

Thanks all for this, it's really impressive to be able to think through this alongside the folks that have clearly been working on this for a while, and who will be working on this for years to come. What an amazing amount of work.

Everything proposed here makes a lot of sense to me, and feels like a very reasonable first step towards revising the standard library.

I think everything here fits with my understanding and use of Swift, and the borrowing views seem like a particularly good direction to go somewhere down the line.

12

This is an unintentional discrepancy -- both forms should use the same notation.

enum Result<Success: ~Copyable, Failure: Error>: ~Copyable { ... }
extension Result: Copyable /* where Success: Copyable */ {}

(This text needs to avoid relying on any implicit ~Copyable inference, pending the outcome of SE-0426. Spelling out the conditional conformance makes the intention clear.)

2 Likes
13

Yes; this proposal is an important step towards first-class Standard Library support for noncopyable container types. Those are crucial parts of the Swift performance predictability roadmap.

Hypoarray is indeed an illustration for a noncopyable container that is able to hold both copyable and noncopyable elements. I included it specifically to demonstrate how these generalizations can be used in practice. Note that Hypoarray is just a fews level away from Array -- it does not support ObjC bridging and it does not implement copy-on-write semantics, but it is still dynamically allocated and it implicitly resizes its storage. This is a useful construct, of course, but some use cases will want to tie things down even further.

(Hypoarray also serves as motivation for subsequent work: the proposal makes it possible to build a rough-but-usable sketch, but we are several interim proposals away from making these into a coherent enough abstraction to consider adding to/near the stdlib. Each subsequent step will let us refine these prototypes, until they are ready.)

2 Likes
14

This proposal updates withExtendedLifetime. That gives us an opportunity to fix two pain points the programmers have been complaining about.

  1. In most cases, the syntax that we recommend is:
    defer { withExtendedLifetime(obj) {} }.
    This usage was not anticipated when the API was designed and it's proving to be very confusing. Understandably, people are unsure whether the operation has any effect when an empty closure is passed. We should add the natural non-closure-taking form:
public func extendLifetime<T: ~Copyable>(_ x: borrowing T) {
  Builtin.fixLifetime(x)
}

And begin recommending this API instead in most cases. For example:

  defer { extendLifetime(obj) }
  weak var ref = obj
  _ = ref!

Note that we don't want to introduce a new name for this API, which is just a simpler form of withExtendedLifetime. we only want to drop the with prefix in the common case of no nested closure.

  1. The current API is incompatible with concurrency. Programmers have been asking us to fix this since concurrency was first introduced. We should add an async overload:
public func withExtendedLifetime<T: ~Copyable, Result: ~Copyable>(
  _ x: borrowing T,
  _ body: () async throws -> Result
) async rethrows -> Result {
  defer { extendLifetime(x) }
  return try await body()
}
11 Likes
15

SE-0437 has been accepted.

4 Likes