UnsafePointer Sendable should be revoked

Evolution Discussion
1

Can anyone think of a reason that the family of UnsafePointer types should be Sendable?

I'm guessing this was done for convenient ObjC API migration. That may be sufficient motivation, but that argument needs to be put forward. Aside from that, making UnsafePointer Sendable is clearly wrong from both a logical and practical standpoint. Logically, UnsafePointer directly violates Sendable's protection against shared mutable state. As a practical matter, a Sendable UnsafePointer causes Sendable to be inferred for any wrapper that safely manages ownership, but naturally does not provide exclusive access to the memory.

This means programs will end up with race conditions without any source level indication of the danger.

Conceptually, sharing UnsafePointers across actors requires a strategy to avoid shared mutable state. Either through uniqueness or shared immutability. At the very least, that strategy should be made explicit.

// The underlying memory may only be accessed via a single
// UniquePointer instance during its lifetime.
struct UniquePointer<T> : @unchecked Sendable {
  var pointer: UnsafeMutablePointer<T>
}

// The underlying memory may only be accessed via other instances of
// SharedPointer during the lifetime of a SharedPointer instance.
struct SharedPointer<T> : @unchecked Sendable {
  var pointer: UnsafePointer<T>
}

In the not too distant future, move-only language features will allow the compiler to enforce both of these strategies.

12 Likes
2

I agree that itā€™s probably unwise, especially with automatic checking/derivation, but it was part of the proposal: https://github.com/apple/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md#adoption-of-sendable-by-standard-library-types

  • Unsafe(Mutable)(Buffer)Pointer : these generic types unconditionally conform to the Sendable protocol. This means that an unsafe pointer to a non-concurrent value can potentially be used to share such values between concurrency domains. Unsafe pointer types provide fundamentally unsafe access to memory, and the programmer must be trusted to use them correctly; enforcing a strict safety rule for one narrow dimension of their otherwise completely unsafe use seems inconsistent with that design.
3

Yeah, I agree that the derivation aspect tips it the other way. Rust has pretty much the same feature and they make pointers non-Sendable by default for exactly that reason.

[R]aw pointers are, strictly speaking, marked as thread-unsafe as more of a lint. Doing anything useful with a raw pointer requires dereferencing it, which is already unsafe. In that sense, one could argue that it would be "fine" for them to be marked as thread safe.

However it's important that they aren't thread-safe to prevent types that contain them from being automatically marked as thread-safe. These types have non-trivial untracked ownership, and it's unlikely that their author was necessarily thinking hard about thread safety.

(Send and Sync - The Rustonomicon)

I wish we had realized this sooner, but itā€™s probably still worth fixing.

5 Likes
4

Counterargument: It says ā€œunsafeā€ in the name.

Countercounterargument: You donā€™t see that name at the call site.

If not for a revocation of the conformance, this does seem to me to require some kind of compile-time diagnostic, and/or some feature that puts the word ā€œunsafeā€ in the contexts where concurrency safety decisions matter.

1 Like
5

It did not escape ā€œourā€ collective attention. I explicitly pointed this out during the original discussions, even the derivation partā€”more than onceā€”but it was decided to proceed in this manner. :man_shrugging:

But, Iā€™m still glad itā€™s getting attention again; better late than never. Thanks for fleshing out the full argument @Andrew_Trick.

9 Likes
6

Youā€™re right. I wish I had supported it sooner and that the core team had evaluated the concerns raised by you and others sooner.

5 Likes
7

@xwu Thanks for pointing out the original discussion. You're right. I'm not raising any new concern here. I started this topic because I wanted to see the final justification explained before we actually enable enforcement.

Two points I'd like to add

  1. Revoking Sendable before enforcement is enabled is reversible. IIUC we can always add '@unchecked Sendable' later if there turns out to be a major usability problem. If the usability concern is only speculative, then it doesn't makes sense to irreversibly introduce definite risk.

  2. I disagree with the characterization of any "Unsafe" API as "broadly unsafe". The programmer must be aware of precisely every aspect of unsafety. The obvious requirement for the UnsafePointer family is the need to separately guarantee memory lifetime. And, obviously subscripting a non-buffer type has no bounds check. Type safety is uncompromised. The reason pointers are not Sendable is that they have reference semantics, which actually has nothing to do with the "Unsafe" prefix. Sendable's job is to stop programmers from accidentally passing around things with reference semantics even when it's obvious that those things have reference semantics.

12 Likes
8

Given that Swift 5.5 is not really enforcing Sendable, I do think we have a window in which we can revoke this conformance without causing undue source compatibility breaks.

This is a great argument, and one I should have given more thought to earlier. I think we should revoke these conformances. Apologies to @xwu and others for not thinking about this bit deeply enough, and thanks to @Andrew_Trick for making the argument so well.

Doug

12 Likes
9

Thanks @Douglas_Gregor. Can this be reviewed by core team as an amendment to:
SE-0302 Sendable and sendable closures](swift-evolution/0302-concurrent-value-and-concurrent-closures.md at main Ā· apple/swift-evolution Ā· GitHub)

Or are there upcoming Sendable proposals we can combine it with?

Or as it's own SE?

PR:

10

Given that this part of SE-0302 is implemented in Swift 5.5, I suspect that the Core Team will want to treat this as a separate proposal rather than an amendment.

There are a few other Sendable-related bits that I think need we need to consider and could be bundled together:

  • There is no spelling for "this type is explicitly not Sendable", to suppress implicit inference of Sendable for internal/private/fileprivate types.

  • The hard requirement that key paths cannot be formed with non-Sendable captured values is too strict, because it makes key-paths useless with some types. We should probably have key path literals produce a KeyPath<...> & Sendable type when their captures are all Sendable.

  • We likely want a -require-explicit-sendable option to produce warnings about any public type in a module that is not marked as Sendable or non-Sendable, so library authors can make sure they don't forget about Sendable for the types they vend.

    Doug

4 Likes
11

I had thought there was discussion about marking an explicit Sendable conformance as unavailable?

1 Like
12

I brought it up in the thread on staging Sendable checking, with this suggestion:

public struct MyType : @available(unavailable, *) Sendable {
  var x, y: Double
  // some day we may add some shared state here
}

Doug

2 Likes
13

I don't want to derail this thread, but I'm getting increasingly confused by what these terms mean. What exactly makes UnsafePointer have "reference semantics"? Why would reference semantics preclude "sendability"?

Does ManagedAtomic<Int> in the swift-atomics package have "reference semantics"? If so, does that mean it should not be Sendable? (It is safely transferrable across concurrency domains.)

Does this extension give Int "reference semantics"? Is it in conflict with its Sendable conformance?

var storage: [String] = ...
extension Int {
  var pointee: String { 
    get { storage[self] }
    set { storage[self] = newValue }
  }
}

(How can we possibly infer sendability by just looking at a type's stored properties, without analyzing its operations?)

1 Like
14

I could be wrong about any of this, but here's how I use the terms.

In this context, "reference semantics" means shared mutable state. In particular, access to storage reachable via the value can result in mutation visible across executors.

Pointers have a special kind of references semantics that doesn't involve "object references". Maybe that's the confusion?

Atomics absolutely have reference semantics. They are a special category of explicitly "shared mutable" types though, so they are Sendable as long as their payload is Sendable.

That would only give Int reference semantics if that extension were actually callable from multiple executors, which it should not be.

So, reference semantic things can be Sendable under various conditions:

  • immutability
  • uniqueness
  • mutable-if-uniqueness
  • shared mutability

...but all these properties must be transitively guaranteed across the object (or pointer) graph reachable from the value

4 Likes
15

that's just hard-to-impossible to guarantee.. the function can be in a third party code, you integrated that code and audit its sources, all good at that point no mutable global access. next revision of that third party code introduces a mutable global access as nothing in the function contract (signature) prevents it doing so... (forget third party.. it can be your team member doing this change (forget your team member, that can be yourself, a few months later)). as in the other thread i believe such traits must be reflected in the function signature (or derived from the parent type if not specified at the function level) and checked by the compiler.

16

I was about to shoot myself in the foot on this.
Will this be revoked in the future? I hope so but I couldn't find any recent posts or proposals on this.
I hope it was not too inappropriate to post this as a friendly reminder.

17
1 Like