UnsafePointer Sendable should be revoked

Andrew_Trick · 2021-09-09T18:09:05.829Z

Can anyone think of a reason that the family of UnsafePointer types should be Sendable?

I'm guessing this was done for convenient ObjC API migration. That may be sufficient motivation, but that argument needs to be put forward. Aside from that, making UnsafePointer Sendable is clearly wrong from both a logical and practical standpoint. Logically, UnsafePointer directly violates Sendable's protection against shared mutable state. As a practical matter, a Sendable UnsafePointer causes Sendable to be inferred for any wrapper that safely manages ownership, but naturally does not provide exclusive access to the memory.

This means programs will end up with race conditions without any source level indication of the danger.

Conceptually, sharing UnsafePointers across actors requires a strategy to avoid shared mutable state. Either through uniqueness or shared immutability. At the very least, that strategy should be made explicit.

// The underlying memory may only be accessed via a single
// UniquePointer instance during its lifetime.
struct UniquePointer<T> : @unchecked Sendable {
  var pointer: UnsafeMutablePointer<T>
}

// The underlying memory may only be accessed via other instances of
// SharedPointer during the lifetime of a SharedPointer instance.
struct SharedPointer<T> : @unchecked Sendable {
  var pointer: UnsafePointer<T>
}

In the not too distant future, move-only language features will allow the compiler to enforce both of these strategies.

Karl · 2021-09-09T18:52:53.997Z

I agree that it’s probably unwise, especially with automatic checking/derivation, but it was part of the proposal: https://github.com/apple/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md#adoption-of-sendable-by-standard-library-types

Unsafe(Mutable)(Buffer)Pointer : these generic types unconditionally conform to the Sendable protocol. This means that an unsafe pointer to a non-concurrent value can potentially be used to share such values between concurrency domains. Unsafe pointer types provide fundamentally unsafe access to memory, and the programmer must be trusted to use them correctly; enforcing a strict safety rule for one narrow dimension of their otherwise completely unsafe use seems inconsistent with that design.

jrose · 2021-09-09T19:03:20.935Z

Yeah, I agree that the derivation aspect tips it the other way. Rust has pretty much the same feature and they make pointers non-Sendable by default for exactly that reason.

[R]aw pointers are, strictly speaking, marked as thread-unsafe as more of a lint. Doing anything useful with a raw pointer requires dereferencing it, which is already unsafe. In that sense, one could argue that it would be "fine" for them to be marked as thread safe.

However it's important that they aren't thread-safe to prevent types that contain them from being automatically marked as thread-safe. These types have non-trivial untracked ownership, and it's unlikely that their author was necessarily thinking hard about thread safety.

(Send and Sync - The Rustonomicon)

I wish we had realized this sooner, but it’s probably still worth fixing.

Paul_Cantrell · 2021-09-09T19:31:30.929Z

Counterargument: It says “unsafe” in the name.

Countercounterargument: You don’t see that name at the call site.

If not for a revocation of the conformance, this does seem to me to require some kind of compile-time diagnostic, and/or some feature that puts the word “unsafe” in the contexts where concurrency safety decisions matter.

xwu · 2021-09-09T19:40:36.660Z

jrose:

I wish we had realized this sooner, but it’s probably still worth fixing.

It did not escape “our” collective attention. I explicitly pointed this out during the original discussions, even the derivation part—more than once—but it was decided to proceed in this manner.

But, I’m still glad it’s getting attention again; better late than never. Thanks for fleshing out the full argument @Andrew_Trick.

jrose · 2021-09-09T19:53:49.666Z

You’re right. I wish I had supported it sooner and that the core team had evaluated the concerns raised by you and others sooner.

Andrew_Trick · 2021-09-09T20:00:30.050Z

@xwu Thanks for pointing out the original discussion. You're right. I'm not raising any new concern here. I started this topic because I wanted to see the final justification explained before we actually enable enforcement.

Two points I'd like to add

Revoking Sendable before enforcement is enabled is reversible. IIUC we can always add '@unchecked Sendable' later if there turns out to be a major usability problem. If the usability concern is only speculative, then it doesn't makes sense to irreversibly introduce definite risk.
I disagree with the characterization of any "Unsafe" API as "broadly unsafe". The programmer must be aware of precisely every aspect of unsafety. The obvious requirement for the UnsafePointer family is the need to separately guarantee memory lifetime. And, obviously subscripting a non-buffer type has no bounds check. Type safety is uncompromised. The reason pointers are not Sendable is that they have reference semantics, which actually has nothing to do with the "Unsafe" prefix. Sendable's job is to stop programmers from accidentally passing around things with reference semantics even when it's obvious that those things have reference semantics.

Douglas_Gregor · 2021-09-15T16:23:08.239Z

Andrew_Trick:

Revoking Sendable before enforcement is enabled is reversible. IIUC we can always add '@unchecked Sendable' later if there turns out to be a major usability problem. If the usability concern is only speculative, then it doesn't makes sense to irreversibly introduce definite risk.

Given that Swift 5.5 is not really enforcing Sendable, I do think we have a window in which we can revoke this conformance without causing undue source compatibility breaks.

Andrew_Trick:

I disagree with the characterization of any "Unsafe" API as "broadly unsafe". The programmer must be aware of precisely every aspect of unsafety. The obvious requirement for the UnsafePointer family is the need to separately guarantee memory lifetime. And, obviously subscripting a non-buffer type has no bounds check. Type safety is uncompromised. The reason pointers are not Sendable is that they have reference semantics, which actually has nothing to do with the "Unsafe" prefix. Sendable's job is to stop programmers from accidentally passing around things with reference semantics even when it's obvious that those things have reference semantics.

This is a great argument, and one I should have given more thought to earlier. I think we should revoke these conformances. Apologies to @xwu and others for not thinking about this bit deeply enough, and thanks to @Andrew_Trick for making the argument so well.

Doug

Andrew_Trick · 2021-09-15T18:53:36.100Z

Thanks @Douglas_Gregor. Can this be reviewed by core team as an amendment to:
SE-0302 Sendable and sendable closures](swift-evolution/0302-concurrent-value-and-concurrent-closures.md at main · apple/swift-evolution · GitHub)

Or are there upcoming Sendable proposals we can combine it with?

Or as it's own SE?

PR:

github.com/apple/swift

Unsafe*Pointer types should not be Sendable.

apple:main ← atrick:revoke-pointer-sendable

opened 02:34AM - 09 Sep 21 UTC

atrick

+7 -21

To send them across actors, they need to be wrapped in an '@unchecked Sendable'… type. Typically such a wrapper type would be be responsible for ensuring its uniqueness or immutability. Inferring Sendability for arbitrary types that contain Unsafe*Pointers would introduce race conditions without warning or any explicit acknoledgement from the programmer that the pointer is in fact unique.

Douglas_Gregor · 2021-09-15T20:50:39.677Z

Andrew_Trick:

Thanks @Douglas_Gregor. Can this be reviewed by core team as an amendment to:
SE-0302 Sendable and sendable closures](https://github.com/apple/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md)

Given that this part of SE-0302 is implemented in Swift 5.5, I suspect that the Core Team will want to treat this as a separate proposal rather than an amendment.

Andrew_Trick:

Or are there upcoming Sendable proposals we can combine it with?

There are a few other Sendable-related bits that I think need we need to consider and could be bundled together:

There is no spelling for "this type is explicitly not Sendable", to suppress implicit inference of Sendable for internal/private/fileprivate types.
The hard requirement that key paths cannot be formed with non-Sendable captured values is too strict, because it makes key-paths useless with some types. We should probably have key path literals produce a KeyPath<...> & Sendable type when their captures are all Sendable.
We likely want a -require-explicit-sendable option to produce warnings about any public type in a module that is not marked as Sendable or non-Sendable, so library authors can make sure they don't forget about Sendable for the types they vend.

Doug

xwu · 2021-09-15T20:57:23.092Z

Douglas_Gregor:

There is no spelling for "this type is explicitly not Sendable ", to suppress implicit inference of Sendable for internal/private/fileprivate types.

I had thought there was discussion about marking an explicit Sendable conformance as unavailable?

Douglas_Gregor · 2021-09-16T16:57:29.154Z

xwu:

I had thought there was discussion about marking an explicit Sendable conformance as unavailable?

I brought it up in the thread on staging Sendable checking, with this suggestion:

public struct MyType : @available(unavailable, *) Sendable {
  var x, y: Double
  // some day we may add some shared state here
}

Doug

lorentey · 2021-09-23T23:37:45.897Z

Andrew_Trick:

The reason pointers are not Sendable is that they have reference semantics

I don't want to derail this thread, but I'm getting increasingly confused by what these terms mean. What exactly makes UnsafePointer have "reference semantics"? Why would reference semantics preclude "sendability"?

Does ManagedAtomic<Int> in the swift-atomics package have "reference semantics"? If so, does that mean it should not be Sendable? (It is safely transferrable across concurrency domains.)

Does this extension give Int "reference semantics"? Is it in conflict with its Sendable conformance?

var storage: [String] = ...
extension Int {
  var pointee: String { 
    get { storage[self] }
    set { storage[self] = newValue }
  }
}

(How can we possibly infer sendability by just looking at a type's stored properties, without analyzing its operations?)

Andrew_Trick · 2021-09-23T23:59:49.369Z

lorentey:

Andrew_Trick:

The reason pointers are not Sendable is that they have reference semantics

I don't want to derail this thread, but I'm getting increasingly confused by what these terms mean. What exactly makes UnsafePointer have "reference semantics"? Why would reference semantics preclude "sendability"?

I could be wrong about any of this, but here's how I use the terms.

In this context, "reference semantics" means shared mutable state. In particular, access to storage reachable via the value can result in mutation visible across executors.

Pointers have a special kind of references semantics that doesn't involve "object references". Maybe that's the confusion?

lorentey:

Does ManagedAtomic<Int> in the swift-atomics package have "reference semantics"? If so, does that mean it should not be Sendable ? (It is safely transferrable across concurrency domains.)

Atomics absolutely have reference semantics. They are a special category of explicitly "shared mutable" types though, so they are Sendable as long as their payload is Sendable.

lorentey:

Does this extension give Int "reference semantics"? Is it in conflict with its Sendable conformance?
var storage: [String] = ...
extension Int {
  var pointee: String { 
    get { storage[self] }
    set { storage[self] = newValue }
  }
}
(How can we possibly infer sendability by just looking at a type's stored properties, without analyzing its operations?)

That would only give Int reference semantics if that extension were actually callable from multiple executors, which it should not be.

So, reference semantic things can be Sendable under various conditions:

immutability
uniqueness
mutable-if-uniqueness
shared mutability

...but all these properties must be transitively guaranteed across the object (or pointer) graph reachable from the value

tera · 2021-09-24T00:46:19.038Z

Andrew_Trick:

That would only give Int reference semantics if that extension were actually callable from multiple executors, which it should not be.

...but all these properties must be transitively guaranteed across the object (or pointer) graph reachable from the value

that's just hard-to-impossible to guarantee.. the function can be in a third party code, you integrated that code and audit its sources, all good at that point no mutable global access. next revision of that third party code introduces a mutable global access as nothing in the function contract (signature) prevents it doing so... (forget third party.. it can be your team member doing this change (forget your team member, that can be yourself, a few months later)). as in the other thread i believe such traits must be reflected in the function signature (or derived from the parent type if not specified at the function level) and checked by the compiler.

Terje · 2022-01-25T10:43:49.651Z

I was about to shoot myself in the foot on this.
Will this be revoked in the future? I hope so but I couldn't find any recent posts or proposals on this.
I hope it was not too inappropriate to post this as a friendly reminder.

Lantua · 2022-01-25T12:08:47.399Z

github.com

apple/swift-evolution/blob/main/proposals/0331-remove-sendable-from-unsafepointer.md

# Remove Sendable conformance from unsafe pointer types

* Proposal: [SE-0331](0331-remove-sendable-from-unsafepointer.md)
* Authors: [Andrew Trick](https://github.com/atrick)
* Review Manager: [Doug Gregor](https://github.com/DougGregor)
* Status: **Implemented (Swift 5.6)**
* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0331-remove-sendable-conformance-from-unsafe-pointer-types/53979)
* Implementation: [apple/swift#39218](https://github.com/apple/swift/pull/39218)

## Introduction

[SE-0302](0302-concurrent-value-and-concurrent-closures.md) introduced the `Sendable` protocol, including `Sendable` requirements for various language constructs, conformances of various standard library types to `Sendable`, and inference rules for non-public types to implicitly conform to `Sendable`. SE-0302 states that the unsafe pointer types conform to `Sendable`:

> `Unsafe(Mutable)(Buffer)Pointer`: these generic types _unconditionally_ conform to the `Sendable` protocol. This means that an unsafe pointer to a non-Sendable value can potentially be used to share such values between concurrency domains. Unsafe pointer types provide fundamentally unsafe access to memory, and the programmer must be trusted to use them correctly; enforcing a strict safety rule for one narrow dimension of their otherwise completely unsafe use seems inconsistent with that design.

Experience with `Sendable` shows that this formulation is unnecessarily dangerous and has unexpected negative consequences for implicit conformance.

Swift-evolution thread: [Discussion thread](https://forums.swift.org/t/unsafepointer-sendable-should-be-revoked/51926)

## Motivation

This file has been truncated. show original