[Pitch] Noncopyable Standard Library Primitives

Hey all!

As part of our series of enhancements on the Roadmap for improving Swift performance predictability, please consider the pitch linked below. It is aiming to retrofit some fundamental generic constructs in the Standard Library with much-needed support for noncopyable types.

This is a large document detailing lots of tiny API updates, generalizations and additions. I'm looking forward to all feedback!

SE-0427 allowed noncopyable types to participate in Swift generics, and introduced the protocol Copyable to the Standard Library. However, it stopped short of adapting the Standard Library to support using such constructs.

The expectation that everything is copyable has been a crucial simplifying assumption throughout all previous API design work in Swift. [...] Fully rethinking the Standard Library to facilitate working with noncopyable types is not going to happen overnight: it is going to take a series of proposals. This document takes the first step by focusing on an initial set of core changes that will enable building simple generic abstractions using noncopyable types.
[...]
This proposal concentrates on two particular areas: low-level memory management and generalized optional types. We propose to modify some preexisting generic constructs in the Standard Library to eliminate the assumption of copyability. [...]

(Note: up until the point when this turns into an actual proposal, I expect this to be a living document that may receive ad-hoc revisions and amendments as needed, without notice. The gist interface provides access to earlier revisions.)

32 Likes

This is very well thought through, kudos.

I worry a bit about not considering Error up front. If Error is ever going to become ~Copyable, Result's API would change again. On the other hand, whether Error should permanently require Copyable seems worthy of discussion.

I'm not sure extracting(_:) is the best name for the new UnsafeBufferPointer method, since it could be interpreted as extracting the elements to a new place, but I don't immediately have any alternate suggestions besides sliceAndRebase or similar.

Your example for Hypoarray has borrowElement in one place and readElement in another.

There exists a second variant of withExtendedLifetime whose function argument is passed the entity whose lifetime is being extended.

Should the extended-lifetime-entity be borrowed for this variant as well?

For now, we instead recommend explicitly binding memory and using Unsafe[Mutable]Pointer operations.

This is a bit unfortunate since usually the opposite is recommended; while it's not the C way, it's harder to screw up (not that PointerSanitizer has come to be…). I understand that they need some specific design though, since they are now more like loadAndMove(as:…).


Some of the specific Alternatives Considered I think should be noted:

  • The existing Optional.map is equivalent to borrowingMap at the ABI level, and thus we could just have map and consumingMap. With a bit of contrivance, we could probably also have map and borrowingMap. I don't think that's better than what's proposed, but it's worth pointing out.

  • Purely in naming: the Rust equivalent of exchange is std::mem::replace.

7 Likes

Thank you for the proposal!

I think unsafelyUnwrapped should just become consuming (if that doesn’t break ABI, e.g. by keeping the old entry point around under a different name but with the old silgen name). The intention of force unwrapping is usually similar to what .move() does in Pointer types. Copyable code would also likely benefit from these improvements.

Also, I don’t think we should introduce .extracting in buffer pointers just because of current limitations with Collection. As the proposal acknowledges, work on Sequence and Collection is expected with future proposals. If we deem this an essential function for early adopters, I think we should create an underscore-prefixed version and hide it under the upcoming/experimental feature flag.

2 Likes

We currently do not have consuming getters, they are only borrowing at the moment.

2 Likes

I concur - extracting seems quite misleading.

Can a subscript overload not be added? I recognise that does add some ambiguity in some cases, for Copyable Elements, but perhaps the language can be extended to specify a default overload to use when it's not otherwise clear? (that would be handy more generally, anyway - overloading on return type is very powerful but currently a bit awkward to actually use)

I think consuming getters could be useful in the future, but that’s beside the point. Perhaps until we have a better solution we should avoid the awkwardness of having a method and getter with two almost identical names and a very subtle semantic difference. Especially for a type as ubiquitous as Optional, with which a lot of users interact, this could cause a lot of confusion.

3 Likes

@_disfavoredOverload?

1 Like

consuming/borrowing annotations are not involved in overload resolution [...] So we're left with the idea of using consuming and borrowing as naming prefixes

While I'm not fond of using prefixes to resolve ambiguity, it might be necessary under the current circumstances. Should the type system specifically recognize borrowing and consuming annotations in overload resolution, similarly how inout is treated today?

The precedent of introducing consumingMap or borrowingFlatMap could lead to a proliferation of similarly named functions, like borrowingFilter and consumingReduce, resulting in an excess of new boilerplate.

3 Likes

Can we also rename (i.e. deprecate the old method and introduce a new method with the new name) all methods on Unsafe[Mutable][Raw][Buffer]Pointer that start with move to consume? This would be inline with the naming of the consume operator.

2 Likes

I'm excited to see this.

I wonder, though - since, as the proposal says, "Unsafe buffer pointers gain much of their core functionality from the Collection protocol", why don't we defer tackling them in the standard library until we've loosened the requirements on Collection?

I mean, they won't have any Collection algorithms either, will they? No .map, .filter, .contains? And they won't work with any user-defined algorithms, so they'll just have very basic memory operations. That's all stuff you can do yourself pretty easily by maintaining the pointer's location + capacity.

And for the earliest of adopters who need this anyway, even without Collection, we could provide a single-file implementation of the above. Then we won't need to commit things like .extracting to the standard library's API forever.

9 Likes

I think having three versions of map for the majority of optional types would be unfortunate. It has been said this noncopyable types are to be an advanced feature, rarely touched by most users, but this will confront them with the feature everytime they look at autocomplete. I also don't think it'd be that valuable for users of non-copyable types to have different prefixed maps instead of having a conventional default ownership rule for map and other transform operators.

Has an ownership-specifying view type been considered? Something like ConsumedOptional constructed with a consumed property or something with its own map that is consuming. This would be similar to Rust, where you use different iterator conversions for different ownerships. The example would instead be this:

let w: Wrapper<File>?
print(v.consumed.map { try $0.close() })
8 Likes

It seems unlikely to me that we would want to generalize ErrorProtocol; my expectation is that all errors need to be able to get caught by and propagated through existing code that isn't aware of noncopyable types. (Boxing them could get us there, though, I suppose.)

If noncopyable errors ever become a thing, I see no reason we couldn't generalize Result in a separate followup proposal, if and when it's time to do so. (For existing projects that use Result, these generalizations aren't disruptive changes -- no existing API gets deprecated, and (hopefully) there will be no need to migrate anything or to rewrite any existing code. Unless of course someone wants to adopt the new features.)

However, I think it's important to highlight that this is definitely not going to be the final proposal that messes with these types. I don't expect Optional & Result would cease evolving after the changes in this pitch ship.

In fact, we already know for sure that this is not their final form!

While this proposal is targeting Swift 6, over on Swift's main development branch non-escaping types are starting to take shape, too. Once they become a thing, we'll eventually need to have a second round of (smaller-scale) generalizations to allow Optional and Result to hold those, too. Ultimately, these types will need to become not just conditionally copyable, but also conditionally escaping.

Jumping from where we are in Swift 5.10 directly to that target would be way too large a leap: it makes more sense to go step by step. This pitch is opening new frontiers that are worthwhile on their own right. I think it's good to allow some time to explore this new space and get comfortable with its properties, if only to better understand why we need to take the next steps.

4 Likes
Tangent on non-copyable Errors

Non-copyable errors definitely work a lot better with typed throws, yeah. The classic Error type is effectively any Error & Copyable, so anything with just plain throws could be considered to be throws(any Error & Copyable). But even if you wanted to lift that to throws(any Error & ~Copyable) I think you could do it with the boxed representation we use today, and just not allow consuming it via downcast. Which isn't perfect, it needs a lot more ownership features to make it useful, but I wouldn't necessarily say we'd never do it.

And since we did make the choice to have Result tied to Error, unlike Rust, people are eventually going to want to have non-copyable Errors, because the alternative is making their own Result (or worse, Either).

4 Likes

Reading buffer.extracting(5..<10) to mean "copying or moving of (potentially noncopyable!) elements at positions 5, 6, 7, 8, 9 into some new storage, conjured into existence in some vague, unspecified way" is certainly one possible way to interpret this call.

However, I actually think this is close enough to the actual meaning that to me it confirms that this is a good name for this operation, and it will not cause any real confusion.

I don't think anyone will be surprised when they find out what extracting actually means. (When I encounter a new API, I do this "finding out" part by quickly looking up its API docs.)

To be honest, I don't think it is at all sensible to expect that API names will eliminate all room for misinterpretation. Absolutely every name can be and will be misunderstood by somebody.

For example, the name "slice" is similarly misleading, as we could've applied it equally well to a myriad different constructs. But I don't think that matters one bit -- Slice is nevertheless an eminently wonderful name for our specific, copyable "view into a subsequence of elements of another collection" construct.

Of course, an important goal of these pitches is to explore alternatives to the proposed names, and to argue about the semantics of each operation, including whether or not we need to provide it at all.

But can we please not fight against having more names like Slice?

Even longer, poorly edited rant on this subject

The whole idea that names need to precisely explain the concepts that they're labeling feels absurd to me. It feels like it's a twisted misreading of Swift's API Design Guidelines; I think it's a menace and I believe we need to stop pretending it's an actual requirement.

All it achieves is that it encourages labeling things with definitional/explanatory phrases, which (in my view) is an extremely poor naming scheme for concepts that we will need to routinely use and mention.

I'm wearing a pair of things we call shoe; we aren't calling them coveringForTheFootWithASturdySoleNotReachingAboveTheAnkle.

Good names are easy to remember! Good names roll off the tongue! Good names do not make any attempt to define what they are labeling -- they merely vaguely gesticulate towards the general direction of the thing. (I consider String and Thread to be top-notch names, but they provide precious little (if any) actual indication of the constructs they're symbols for. The name "byte" is even better; it is brave enough to get weird, to great success.)

There is this thing called "learning" that humans are apparently pretty good at. If I encounter the word "gigue" in an article, the context may give me some vague clue about what it means. If I want to be sure, I simply look it up in a dictionary. Afterwards, there is a good chance I'll remember it, and I won't need to look it up every time I see it.

</rant>

With that off my chest, concrete suggestions for alternatives are of course very welcome. It is also fair to suggest that some of the operations aren't necessary.

✻ ✻ ✻

For what it's worth, I chose "extract" as the root of the name here simply because it was the most obvious name I could think of. This is the label I've always been using for operations that build a new container out of parts of another, especially another of the same type. (Random examples: _NativeSet.extractSubset, Rope.extract) Using the word "extract" in this sense is very familiar to me. Do y'all have a more obvious label for this operation? (Apologies, but I don't think sliceAndRebase works.)

All too often, I find myself having to construct an Unsafe[Mutable]BufferPointer from a range of items inside a larger buffer, and I always felt that UnsafeMutableBufferPointer(rebasing: foo[i..<j]) was way too clumsy a spelling for it.

Given that I do not expect that it will be possible to generalize U[M]BP's existing slicing subscript for noncopyable elements*, I jumped on the opportunity to provide a direct operation for this very common operation. Unlike slicing, this operation does make sense for buffers of noncopyables, so given its universal usefulness, it seems worthy of a good name.

Footnote on the impossibility of slicing containers of noncopyables

The concept of a "collection slice", as embodied in the standard Slice type and (in particular) the elegant slicing notation foo[i..<j] is heavily relying on copyability: the slice physically owns a full copy of the collection, and it uses this copy to implement both read-only and mutating accesses, all within a single type.

var array = Array(0 ..< 20)
print(array[0 ..< 10]) // borrowing-style read-only access
array[0 ..< 10].sort() // ostensibly in-place mutation (actually not!)
array[0 ..< 10] = array[5 ..< 15] // overlapping range assignment

This universal form of slicing will not carry over to containers of noncopyable elements, at least not without a complete overhaul and/or reduction of its expressive power.

For noncopyable types, Slice does not seem to be a viable abstraction. (To be precise, the Slice type could in theory be generalized into a consuming noncopyable slicing construct. I strongly suspect we will not do that -- if we decide that the idea of a consuming slice has merit, it would be a better idea to build it from scratch, free from nightmarish ABI/source compatibility issues. We will know better soon, once we become ready to talk about noncopyable container abstractions.)

We aren't ready to talk about which aspects of slicing will merit figuring out a noncopyable generic solution. FWIW, I suspect it would be desirable to at least implement a reusable/generic borrowing slice construct.

</footnote>

Omitting this operation altogether is an idea worth considering, too, despite its usefulness.

Over the years, UnsafeBufferPointer has crept into a bunch of public API, from Collection.withContiguousStorageIfAvailable to Array(unsafeUninitializedCapacity:initializingWith:), and this has effectively turned it into a sort of universal interface type that it has no right to be.

In the foreseeable future, we are planning to introduce a small family of non-escapable Span (née StorageView) constructs that we expect will gradually evict UnsafeBufferPointer into the darkest, deepest layers of data structure implementations and similar low-level contexts, where it belongs. I think it would be somewhat rude to deprive these layers of a standard way of extracting sub-buffers -- but it would not be the end of the world.

The copyable case is not a deprecated afterthought -- it remains (and I expect it will indefinitely remain) the primary programming model in Swift, at least in the higher-level layers.

Therefore, I don't think using the foo[i ..< j] notation for the extracting operation would be a viable option. That spelling is reserved for the existing slicing subscript; in Unsafe[Mutable]BufferPointer, this subscript must continue to return a copyable Slice. We cannot mess with that. Existing code is sacred.

But a labeled subscript like foo[extracting: i..<j] would perhaps be a viable option. It is subtly hinting at capabilities that this operation doesn't really provide, which is why I suggested a member function for it.

Well, that attribute has a tendency to turn previously working code into a pile of "the compiler is unable to type-check this expression in reasonable time" errors. It is also not a principled solution: it is more of an emergency patch, and it remains underscored for a reason.

But, assuming it would technically work in this case, I suppose the idea is that we'd prefer the existing copyable subscript, and mark the new subscript implementation with this attribute.

What type would this new subscript operation return?

Insisting on preserving the foo[5 ..< 10] syntax would mean that we'd need to implement index sharing in the noncopyable case, too: the first item of the resulting slice would need to be positioned at index 5. (This is not very negotiable: we cannot have the same notation on the same type mean completely different indexing behavior based on whether or not Element is statically known to be copyable. Element may even be a conditionally copyable type, like Optional or Result!)

There is no existing type we could use to implement such indexing; consequently, we'd need to introduce a pair of new types specifically dedicated to representing slices of a noncopyable buffer pointer.

We'd also end up having to duplicate all the slice APIs from SE-0370. That's an immense number of new APIs, only to get back to the original, clumsy UnsafeBufferPointer(rebasing: foo[5..<10]) spelling!

I don't think this one operation is worth all that, however important it is.

Beware, unsafelyUnwrapped is not the standard force-unwrapping operation -- this is the obscure, unsafe variant of it that does not check that the optional actually contains a value. (Triggering undefined behavior if the value isn't there, rather than a guaranteed trap.)

The actual (safe) force-unwrapping operation is the special form x!, which is hardwired into the language. As of today, it always consumes the optional, but that is a (hopefully temporary) bug -- it is intended to either consume or borrow the optional, depending on usage context.

There is a possibility that we will be able to define properties that similarly support consuming and borrowing access in the future, thereby allowing us to implement this same adaptive behavior using just the existing unsafelyUnwrapped property.

Accordingly, I have removed the addition of an Optional.unsafeUnwrap() function from the pitch, and I added a passage in the Future Work section about this todo item. Thanks for pressing on this!

4 Likes

Does this still apply to unsafe buffer-pointers? As I understand it, they remain copyable even when their element type is non-copyable, because they do not own a full copy of the collection - ownership is managed manually by the programmer.

I don't think we need a full discussion about slices to evaluate this proposal (or if we do those parts should be deferred until it comes time to fully tackle Collection), but I agree that it seems we will need more expressive capabilities than the language has right now. borrow and inout (mutable borrow) variables will probably be required for us to express slices of a non-copyable collection.

2 Likes

Good question! The answer is that Unsafe[Mutable]BufferPointer is a critical part of Swift's low-level memory management infrastructure; ripping out these types would leave us with an incoherent model.

  • The point of unsafe buffer pointers is that they represent a reference to an entire region of available memory, as a single unit. They are the standard way of providing low-level access to memory regions between Swift functions. This is their core purpose: they are Swift's official abstraction for a region of direct memory.

    Passing a naked base address and a separate count (or start/end pointers etc) is not an adequate substitute for this abstraction!

So that's my high-concept abstract answer. Some more down-to-earth answers:

  • The withUnsafeTemporaryAllocation family of functions use UnsafeMutableBufferPointer to expose the regions they allocate, so without UMBP, we would not be able to allocate noncopyable types this way.

  • We are working on prototyping noncopyable variants of our existing container types (inside the stdlib, in swift-collections and elsewhere). UBP/UMBP is a core part of many of the existing implementations, and the generalizations I'm proposing here help us share internals between the various flavors. (Most data structure implementations have an internal construct for operating on some fixed-capacity storage node abstraction -- this is often based around an UMBP. I want to reuse as much existing code as possible: the target of this exercise is to finalize the upcoming container model & set its conventions, not to spend months on developing new implementations from scratch, or to waste effort on adding a myriad custom reimplementations of UBP in every such project.)

    In my view, generalizing UnsafeBufferPointer therefore unblocks work on the new container model. There is no reason to delay landing it, but I see good reasons to do it now.

My point with the sentence you quoted was that UBP relies on Collection's concept of indices to allow its operations (like initializeElement(at:) or indeed the indexing subscript) to refer to its parts. However, it's okay to decouple this concept of indices from Collection itself: the operations are useful even if we cannot use them in generic algorithms yet.

(There are also correctness issues with UBP's Collection conformance, but that's water under the bridge at this point. (A collection type that allows undefined behavior when accessing an element at a valid index is arguably not a real collection.) We'll try to avoid such issues with the upcoming safe replacement alternative Span.)

I do see some merits in potential arguments that the index navigation operations (startIndex, endIndex, index(after:), distance(from:to:) etc.) do not really carry their weight, and so we should leave them conditional on Element: Copyable for now. I don't think they hurt though, and as we'll need an Index type, we might as well just carry these over, too. (If only to avoid rewriting some existing code that happens to use them.)

Yes, sadly it does. You're right in that UnsafeBufferPointer continues to be copyable, so Slice can make a copy it, exactly like before. But the issue actually isn't with that; rather, the problem is that Slice requires its Base to conform to Collection, and Collection requires Element to be copyable (by inheriting it from Sequence).

  • The copyability requirement on Sequence.Element cannot be lifted without breaking pre-existing generic functions over Sequence -- retroactively generalizing an associated type is a source breaking change. (Existing generic functions implicitly get an Element: ~Copyable clause when they get recompiled with such a generalized protocol definition. Existing code generally does not expect to work with such types, and is likely to run into produce compile-time errors.)

    Additionally, and probably even more frustratingly, in ownership terms, Sequence is shaped like a consuming iterator, while Collection is deeply tied to borrowing semantics. While IteratorProtocol has a distinctly consuming shape, most of our collection types implement iterators closer to borrowing iterators, which makes trying to disentangle these concepts a particularly frustrating puzzle. (At the moment, there is a fair chance that we'll end up adding a separate set of protocols dedicated to noncopyable iterables and container types, to lift the shackles of compatibility.)

    There is some faint glimmer of hope that we'd be able to resolve both of these major issues in the future. If and when these hopes somehow do come true, then we'll be able to revisit UBP and generalize its slicing subscript. Meanwhile, the types do carry their weight even without sequence/collection algorithms.

  • An alternative idea would be to leave Sequence/Collection as is, and somehow generalize the standard Slice type to only force its Base type to be Collection if it also happens to be Copyable.

    struct Slice<Base: ~Copyable>: ~Copyable {
      var _startIndex: Base.Index
      var _endIndex: Base.Index
      var _base: Base
    }
    
    extension Slice: Copyable /*where Base: Copyable*/ {}
    
    extension Slice: Collection where Base: Collection {
      ...
    }
    

    This doesn't appear to be viable:

    • (It's unclear to me if such a transformation would preserve the ABI of the existing type declaration. Let's be optimistic and assume that it does, though.)
    • The unconstrained Base type would not have an Index. I'm not sure there is a straightforward way to solve this. (Perhaps there is a way to insert a new protocol above Collection, but it raises too many potential complications to consider it. We try not to resort to adding ad hoc public protocols as a stopgap solution in the stdlib -- they cause trouble.)
    • This would hit the same ABI issue with its stored property _base as we've hit with ManagedBuffer.header. (This can potentially be resolved in the future.)

    I don't see much value in generalizing Slice before Collection, though. It would indeed allow us to keep the slicing syntax for UBP, but there is a danger that generalizing slice too early would end up making generalizing Collection even more difficult later. Again, I don't think the ability to slice buffers is worth that risk.

Adding a small shortcut operation to UBP that extracts a buffer from part of another continues to seem like the most pragmatic option to me.

(Thanks very much for pressing on this! The idea of generalizing Slice was one I have not considered before. It made me really excited for a moment -- although sadly it didn't last. :cry: Anyway, it's been useful to crystallize some of these thoughts.)

4 Likes

This is a worthy idea! Thanks for raising it.

(I considered these earlier this year, but perhaps I dismissed the idea too quickly -- it's definitely worth a more careful look. A direct precursor idea that we were very sad to ultimately give up on was the notion of spelling the memory orderings on struct Atomic using this view syntax -- as in foo.relaxed.load().)

The key general weakness of these views is that a naive implementation of this would require two or three additional "view" types per noncopyable construct (one each for consuming, borrowing, and perhaps even mutating access, respectively). That's a lot of types!

But we could in theory reduce the cost of that by providing generic structs that can work on every type. By having shared ConsumingView/BorrowingView/MutatingView generic constructs, we could perhaps have every type use the same views.

Click here to expand a draft implementation written in a hypothetical future Swift version

Beware, the below code is not written in today's Swift. (And probably not in any future Swift, either -- I'm simplifying some details, and I'm making assumptions that may not be valid.)

@frozen
struct ConsumingView<Base: ~Copyable>: ~Copyable {
  var base: Base
  init(_ base: consuming Base) { self.base = base }
}

@frozen
struct BorrowingView<Base: ~Copyable>: ~Copyable, ~Escapable { // Not a thing yet
  var base: Ref<Base> // Not a thing yet
  init(_ base: borrowing Base) dependsOn(base) -> Self { // Not a thing yet
    self.base = Ref(base)
  }
}

@frozen
struct MutatingView<Base: ~Copyable>: ~Copyable, ~Escapable { // Not a thing yet
  var base: MutableRef<Base> // Note a thing yet
  init(_ base: inout Base) dependsOn(base) { // Not a thing yet
    self.base = MutableRef(&base)
  }
}

(Note: I'm showing a mutating view as an illustration; I'm not suggesting the idea has merit.)

One immediate problem is that BorrowingView and MutatingView need to be non-escapable types that rely on stored borrows and stored inouts (represented by the hypothetical library constructs Ref and MutableRef here); none of these are are a thing yet. We cannot define these today, but we might be able to do so later, albeit likely using some different syntax. (ConsumingView is possible to implement today, though!)

Next, we need to define the view properties on Optional (and Result etc.):

extension Optional {
  var consuming: ConsumingView<Self> {
    consuming get { ConsumingView(self) } // Not a thing yet
  }

  var borrowing: dependsOn(self) BorrowingView<Self> { // Not a thing yet
    read { try yield BorrowingView(self) }
  }

  var mutating: dependsOn(self) MutatingView<Self> { // Not a thing yet
    modify { try yield MutatingView(&self) }
  }
}

Clearly, borrowing & mutating continue to depend on languages features we don't have yet; but unfortunately this time they're joined by consuming too, as consuming getters aren't a thing yet, either. (AIUI, these features are all on the predictable performance roadmap, although they may not land at the same time.)

Finally, we need to provide extensions that provide the specific operations for Optional.

extension<T: ~Copyable> ConsumingView where Base == Optional<T> { // Not a thing yet
  consuming func map<U: ~Copyable, E: Error>(
    _ transform: (consuming Base.Wrapped) throws(E) -> U
  ) throws(E) -> U? {
    switch consume base {
    case .some(let y):
      return .some(try transform(y))
    case .none:
      return .none
    }
  }
}

extension<T: ~Copyable> BorrowingView where Base == Optional<T> { // Not a thing yet
  borrowing func map<U: ~Copyable, E: Error>(
    _ transform: (borrowing Base.Wrapped) throws(E) -> U
  ) throws(E) -> U? {
    switch base {
    case .some(borrowing y):
      return .some(try transform(y))
    case .none:
      return .none
    }
  }
}

extension<T: ~Copyable> MutatingView where Base == Optional<T> { // Not a thing yet
  mutating func map<U: ~Copyable, E: Error>(
    _ transform: (inout Base.Wrapped) throws(E) -> U
  ) throws(E) -> U? {
    switch &base { // Not a thing yet
    case .some(inout y): // Not a thing yet
      return .some(try transform(&y))
    case .none:
      return .none
    }
  }
}

Such extensions are almost within reach, although IIRC generic extensions aren't a thing yet, either, so we'd need to resort to vaguely disturbing workarounds of this sort:

extension ConsumingView {
  consuming func map<T: ~Copyable, U: ~Copyable, E: Error>(
    _ transform: (consuming Base.Wrapped) throws(E) -> U
  ) throws(E) -> U? 
  where Base == Optional<T> {...}
}
var foo: Optional<Foo> = Foo(value: 42)
print(foo.borrowing.map { $0.value }) // 42
foo.mutating.map { $0.value += 1 }
print(foo.consuming.map { $0.value }) // 43
// foo is now gone

Of these three potential views, consuming is the closest to becoming a workable reality today -- we "just" need to allow computed properties to come with getters that consume self, and (ideally) a way to spell generic extensions.

borrowing potentially requires coroutine-based in-place read accessors; it definitely requires non-escapable types with lifetime dependency annotations, as well as stored borrows (or a library-level, non-escaping Ref equivalent to them).

mutating probably requires coroutine-based in-place modify accessors; it definitely requires non-escapable types with lifetime dependency annotations, in-place switch statements over inout bindings, as well as stored inout borrows (or a MutableRef).

(I presented mutating.map for completeness; I'm not promising that I'd propose to expose a variant of Optional.map that can mutate its wrapped item. But in-place mutating greedy transformations are an interesting direction to explore, too.)

So while these are tantalizingly close, unfortunately we cannot define any of these views in Swift 6 yet.

If we want to explore this direction, I suppose we could delay the addition of Optional.map/.flatMap and Result.map/.flatMap! Of course, there is no guarantee that these will be possible to express anytime soon, or that we won't run into some difficulty I'm not seeing yet. We may need to wait a while, and ultimately we end up choosing something more like the originally proposed solution.

3 Likes

I don't personally think this would be a good idea.

First, move is a not a bad description of what these operations actually do. Second, as a package maintainer, I found the previous assignupdate terminology change to be largely pointless but incredibly disruptive, and I would not personally wish to force anyone to relive that experience.

These unsafe pointer operations are used in incredibly fragile code with subtle considerations. Every time we force people to touch such code, we are inviting mistakes.

I think the current names are fine; we should stick to them.

5 Likes

Thanks for this explanation and draft implementation!

Since the copyable view is possible (or closest to being so), could the plain map, flatMap be borrowing? @jrose mentioned that would be ABI preserving. I don't think there's an obvious borrowing vs consuming default, but one way I can rationalize defaulting to borrowing is it doesn't invalidate the original value. The counter argument is Rust defaults to consuming I think?

I suspect we'd also want the existing functions to generalize to consuming behavior by default, if we end up going this way. Beyond the notational benefits, the consuming view doesn't carry as much weight as borrowing and mutating views would. (The BorrowingView and MutatingView concepts are pretty much the same as the Ref/MutableRef types they contain in the draft above, and there is no actual reason to keep them separate. There is no such thing as a "consuming ref"; it seems better to avoid inventing it just to fit the notation.

2 Likes