That syntax could be misleading, though, because external to the method the only thing that matters is that it's consuming. How exactly the value gets consumed is an implementation detail. Since other changes to the method modifier among borrowing/consuming/mutating are ABI- and potentially-API-breaking, using a different name there could imply a bigger external behavior break than is really in play.
7 Likes
The example of using forget in the pitch didn't seem to me like it solved a problem that was very difficult to solve in other ways. For example, the FileDescriptor type in the example could use a sentinel value (e.g. fd == 0 or fd == nil) to signal to deinit that the handle is already closed and therefore shouldn't be closed a second time. In either approach, the programmer must remember to do something in the close() function to inhibit the deinit logic. Is the main benefit of forget in the file descriptor example that complexity in the implementation of deinit can be avoided? I do think that has value, but I'm wondering if I'm missing something else that makes forget a truly necessary tool.
1 Like
I remember that, pre-1.0, Rust had experimented with strictly "linear types" (which require explicit consumption, as opposed to the "affine types" that Rust 1.0 ended up with, and which we're proposing here, that get implicitly consumed), and found them to be severely unergonomic and uncomposable. I can definitely foresee situations where there is no one "default" way to consume a resource, and forcing an explicit choice would be good, but it seems like those cases should be the exception rather than the rule.
We've done a pretty good job in Swift so far of keeping it possible to "make invalid states unrepresentable". You could use a sentinel state, but then the entire implementation of FileDescriptor needs to be prepared to possibly handle being in that state, and clients then might need to be able to check whether the descriptor is valid, and so forth.
6 Likes
I wasn't around to see that, but I can believe it. I'd like to re-pitch this slightly differently: deinit overloads more importantly as an alternative to a forget keyword rather than as a way to get linear types. A public deinit gets you the behavior as currently spelled in the proposal (anybody can drop the value), and forget can be expressed in terms of less-than-public deinit methods, with availability rules that don't need to be explained to anyone already familiar with Swift's access modifiers. Removing the way to end an object without calling any deinitializer seems like a win to me.
How does forget work if self contains other non-copyable types? Are they transitively forgotten or is their deinit run? I would imagine the latter, but it's easy to think it's the former and always going through a deinit might make the answer more intuitive.
2 Likes
From talking with Andy about the current implementation, I think we'll want to restrict the ability to forget to types that would be trivial if not for their noncopyability for now. Eventually we want to get to the point where, when you forget a type with nontrivial fields, you can selectively consume those fields, by forwarding their ownership elsewhere. Any remaining fields you don't consume would have their individual destructors run, so forget would only be skin-deep.
6 Likes
Oh hi, you're poking the ol' move-only bear again! The mem::forget stuff is interesting! 
Here are some useful links:
-
The Pain Of Linear Types In Rust - an article I wrote in the early days of the Swift team looking into "move-only types", discussing why "true linear types" (as opposed to the "affine types" that Rust has) are a huge mess. Also a look into what I originally believed to be the generics-migration story for Swift adopting move-only types back in 2017 (good call to punt on generics, Joe
). -
Rustonomicon - Leaking - official documentation (also by me) on how to deal with the Possibility of leaking (mem::forget) when desiging safe abstractions around unsafe code, and some examples of APIs that had to be adjusted in The Leakpocalypse⢠(where Rust came to terms with the fact that mem::forget could be emulated in safe code and so may as well be marked safe To Be Honest).
-
std::mem::ManuallyDrop a wrapper type that prevents the compiler from automatically dropping a value, allowing the developer to either forget it or drop it when they think is the right time. In essence, this type is inverted mem::forget: everything is forgotten by default, and you need to manually not-forget it.
-
Two Memory Bugs From Ringbahn - some cases where incorrect mem::forget usage resulted in double-drops (relevant to the checking stuff you added).
23 Likes
The docs for mem::forget cite Rc as a way to invent forgetting in safe code without it, but is there a way in Rust to invent forget for a lifetime-dependent value (which Rc seems like it can't since it would impose indefinite lifetime on the thing being refcounted)? Indiscriminate forget has come up as a wrinkle in answering the question "are locks movable without storing them in an out-of-line cell". Apple's os_unfair_lock for instance doesn't have any lingering OS resources when it's not in a locked state, so its unique owner is free to move the OS_UNFAIR_LOCK_INIT bit pattern around in memory to give the lock away. It only needs to be pinned in memory while locked, which can be handled by ensuring locking always happens within the lifetime of a borrow on the lock. That works great unless some jerk mem::forgets the MutexGuard that would normally unlock it, leaving the lock in a locked state while statically appearing to be un-borrowed again while the OS still has guns pointed at its current address, leading to hijinx if the presumptive owner of the lock moves it in this state. It seems to me that the "if it didn't exist, you could invent it" argument doesn't hold as strongly for lifetime-dependent type families like Mutex and MutexGuard, but maybe I'm missing the trick to invent it in this case and it's all hopeless.
1 Like
Apologies I am a bit sick so this might be slightly incoherent.
The Leakpocalypse⢠really was about reference-counted cycles. Also that Mutex and MutexGuard were never actually the problem because if you mem::forget the lock just gets stuck "locked" and the program deadlocks (although your example is an interesting wrinkle!). It's actually exceptionally rare to encounter the issue, to create A Leakpocalyse⢠Type you must:
- Create a type with some non-trivial validity invariant (
Array) - Create another type which mutably borrows (
inout) Array and violates that invariant (Drain) - Argue that it's ok for the invariant to be transiently violated because:
a. The mutable borrow held byDrainprevents anyone from touchingArrayand observing the violated invariant
b. The mutable borrow expiring impliesDrain's destructor ran
c.Drain's destructor repairs/enforces the invariant
All of this is valid except 3b, because you can write code like (pardon my rusty pseudo-swift):
// Make the contents of this non-POD if you want real UB
var arr = [1, 2, 3, 4];
// Start consuming the values out of the array
{
var drain = arr.drain();
drain.next();
// At this point the Array is invalid, as index 0 is logically deinit
// But it's ok, `arr[0]` should be a compiler error because
// `arr` is `inout` borrowed by `drain`!
// ... now leak it with a strong refcounted cycle (Node is a class)
var node1 = Node();
var node2 = Node();
// At this point the compiler sees that node1 holds the borrow
// that drain held, so now we still can't access arr until
// node1 is gone
node1.value = drain;
node1.other = node2;
node2.other = node1;
}
// OK! drain and node1 are both DEFINITELY gone, it's impossible
// for the program to access them (correct!) so the borrow can be
// released (correct?) and we know the dtor ran (incorrect!!!)
// Reading deinit memory!!!
print(arr[0]);
You can do this kind of thing because Rust would type these nodes as like Node<Drain<'a, T>>, and anywhere a Node goes the compiler infects with the 'a borrow (or perhaps more accurately, expands 'a to cover those parts of the program, and the borrow follows). Basically refcounting isn't really indefinite. The compiler can follow all the places the strong references get to, and it knows when they all go away then the borrow can go away (anything that could really smuggle things forever like globals or passing things across non-scoped threads requires T: 'static, the forever lifetime, which would make the borrow never expire).
I'm afraid I can't really extrapolate to what this looks like in Swift. Like, classes were always going to be a nightmare for noncopyable types. I'm not really sure what it Means to put a noncopyable type into a class, since taking an inout borrow of that field is unknowable to some other code that happens to also have a reference to that class instance. This is why Rc/Arc in rust enforce immutable access to their contents (no inout), which you need to claw back with dynamic "interior mutability" types like RefCell or Mutex.
1 Like
We have an existing scheme for dynamically enforcing exclusive access to classes, globals, and other dynamically shared-mutable things, which we're also planning to use here. So every class ivar is in essence wrapped in a RefCell.
To be clear it's not impossible to build a version of Rust that says "building your own safe version of mem::forget is unsound (and any API which can be used to do so is Also Unsound)". We just decided that on balance it's not worth the effort.
Someone actually tried to fork rust (or at least, the stdlib) to do exactly that in response to The Leakpocalypseâ˘, although obviously that didn't get far politically.
One approach is to introduce a LeakSafe marker trait (analogous to Send/Sync) and then make Rc<T> require T: LeakSafe. Types like Drain would opt out of LeakSafe, and be disallowed from being put in refcounted cycles.
Another approach would be to introduce a marker trait that prevents interior-mutability types from being put in Rc, making cycles impossible (really restrictive...).
Another another approach would be to mandate no borrows stored in classes (equivalent to Rust requiring 'static). This one seems perhaps the most viable for Swift, since you've already embraced restrictions on where borrows can go. This would Suck for collections (an "Array of references" is pretty common), but maybe CoW emulated-value-semantics types can manually opt into it? Maybe that's sound?
1 Like
This looks mostly like a self-contained step towards fully usable non-copyable types.
The only concern I have is if we want to instead have @noncopyable type to imply ?Copyable on used/declared generic parameters.
2 Likes
Was consideration given to using NonCopyable as a marker protocol, rather than @noncopyable as an attribute? I didn't see it discussed in the "Alternatives Considered". It seems like Swift has recently preferred to express such features with protocols, rather than attributes. (For example, see Sendable.)
6 Likes
I think this is implicitly answered by:
NonCopyable is not a constraint, it is the lack of a Copyable constraint. It doesn't make sense to think of it like a protocol in any other context, since a generic parameter or protocol that allows for noncopyable types to conform would not require the type to be noncopyable, it allows the type to be noncopyable, and would still accept copyable types.
It could make sense to adopt a "don't require this constraint" syntax like Rust's ?Trait, so you'd write struct Foo: ?Copyable or something like that to opt the type out of satisfying the Copyable constraint. Then eventually you'd also be able to write foo<T: ?Copyable> to specify a generic parameter that doesn't have to be copyable and so on. "Copyable" as a constraint still doesn't quite make sense to think of as a protocol, though, because it's a fundamental trait of the type, and can't be retroactively added or have multiple independent conformances like a protocol can, and it's not a "marker protocol" either because it has a fundamental runtime ABI impact on the type. I think it would end up being more akin to a layout constraint like AnyObject, which is also implicitly "conformed to" by all types that satisfy its requirements (that is, single-refcounted-pointer types like classes and class existentials without witness table requirements), and cannot be explicitly or retroactively conformed to by types that don't fundamentally satisfy that constraint.
4 Likes
Is âUniqueâ is an accurate synonym for âNonCopyableâ? Thatâs a positive constraint.
I also donât understand the conclusion that copyable types would always satisfy NonCopyable type parameters. struct T<U: NonCopyable> doesnât allow any copyable types to substitute for U.
There isn't any fundamental that prevents a copyable type from being used as a noncopyable one. The implementation just wouldn't use the copy functionality. It's like talking about "types that don't have a foo() method" when you have a protocol Foo { func foo() }. We could make up a different requirement for "must-be-unique", but we'd have to decide what the important properties of types that satisfy that requirement are, and what sorts of code you'd write that takes advantage of those properties. It still wouldn't be something that noncopyable types just automatically satisfy, because nothing prevents a noncopyable type from implementing some manual-copy API, or being used to represent a handle to a shared resource that the client is not allowed to propagate but might have handles elsewhere in the program or OS.
1 Like
Thatâs precisely what I would want the @noncopyable attribute to do: prevent var x = MyNonCopyable(); let y = x. Generalized, this is func withHandle<Handle: NonCopyable, T>(_ handle: borrowing Handle, perform closure: (borrowing Handle) rethrows -> T) { closure(handle) }. Another name for withHandle could be ensuringUniquelyOwned.
I presume MyNonCopyable() is concretely a non-copyable type, so let y = x would in fact move x into y rather than copy it. When we generalize the generics system to include noncopyable types, then withHandle can't assume Handle is copyable, and it would act as a noncopyable type within the body of the function. So the function would not be able to introduce copies of whatever Handle is, but callers can still pass in copyable types for the Handle that they can copy themselves.
You may also be interested in the @noImplicitCopy attribute, which we're implementing to suppress implicit copying on values that are of copyable type.
1 Like
It is useful to be able to write generic code that works on an arbitrary type, copyable or not. Many basic data structures and algorithms do not innately require copying values and so can naturally be generalized to work with non-copyable types. For correctness, code like that has to work generically with the type as if it were non-copyable. When given a copyable type, it effectively promises not to (locally) copy it.
From that perspective, copyability is a positive constraint, just like an ordinary protocol constraint. For example, if Array were generalized to allow non-copyable types, it would still be conditionally copyable:
extension Array: Copyable where Element: Copyable
(You might think that you could do the same with NonCopyable as a positive constraint, and in simple cases this does make sense:
extension Array: NonCopyable where Element: NonCopyable
But the basic direction of the logic is wrong, as you can see when you consider a similar generalization of Dictionary:
extension Dictionary: NonCopyable where Key: NonCopyable, Value: NonCopyable
This is wrong; it is making Dictionary non-copyable if both of the key and value types are non-copyable, but in fact Dictionary needs to be non-copyable if either argument is non-copyable. Similar logic applies to e.g. inferring non-copyability for structs. This is a clear sign that the direction of the constraint is backwards.)
13 Likes
How does the compiler know that itâs OK to pass a @noncopyable type to such a function? Just because a parameter is marked borrows doesnât mean the function canât copy it internally. There must be some other element of the functionâs type signature that the compiler can check at the call site, no?