Is Sendable necessary for closures passed into asynchronous scope functions?

Using Swift
1

I've been pondering upon the following code for a function.

public extension Database {
    func transaction<T>(_ closure: @Sendable @escaping (Database) async throws -> T) async throws -> T {
        try await self.transaction { db -> EventLoopFuture<T> in
            let promise = self.eventLoop.makePromise(of: T.self)
            promise.completeWithTask{ try await closure(db) }
            return promise.futureResult
        }.get()
    }
}

This is actual code from the Vapor framework.
Here is the URL for reference: https://github.com/vapor/fluent-kit/blob/main/Sources/FluentKit/Concurrency/Database%2BConcurrency.swift

In this transaction function, the argument closure has the @Sendable and @escaping attributes.
I am wondering if both could be removed.

In particular, the @Sendable attribute implies that the types passed down to the closure must be of Sendable,
which imposes quite strict limitations.
Hence, if we could omit it, that would be much more convenient.

The reason I think it could be removed is because,
while this closure is indeed passed onto the event loop thread,
by the time it leaves the transaction function,
it is awaiting EventLoopFuture.get(),
ensuring that the function invocation of closure is complete.

In other words, there is no chance of closure being simultaneously called from two different threads.

Indeed, theoretically since the eventLoop type is abstracted to any EventLoop,
implementing a way to store the closure passed to completeWithTask into a global variable or something similar is possible,
but that would clearly be unnatural behavior for EventLoop and EventLoopFuture and I don't think there's anything to worry about.

Furthermore, I think @escaping can be eliminated for the same reason. The closure doesn't actually escape.

Could the above thoughts be right?
I'd love to hear thoughts from someone with more insight to weigh in.

For reference, the specific implementation would look as follows:

public extension Database {
    func transaction<T>(_ closure: (any Database) async throws -> T) async throws -> T {
        try await withoutActuallyEscaping(closure) { (closure) in
            let closureBox = UncheckedSendableBox(closure)
            return try await self.transaction { db -> EventLoopFuture<T> in
                let dbBox = UncheckedSendableBox(db)
                let promise = self.eventLoop.makePromise(of: T.self)
                promise.completeWithTask {
                    let db = dbBox.value
                    let closure = closureBox.value
                    return try await closure(db)
                }
                return promise.futureResult
            }.get()
        }
    }
}

And I believe this thought could be generalized.
I refer to these functions that take a value and allow for flexible processing using a closure, as scope functions.
Such scope functions can indeed be asynchronous, however, even when they're asynchronous,
in my opinion, as long as the closure execution in the scope functions is done serially,
they don't necessarily have to be @Sendable or @escaping.

What are your thoughts about this?

2

Unfortunately, no, they cannot.

Sadly this is not what the concept of Sendable implies. Sendable is required any time we pass the value across concurrency domains, which we do here.

@escaping is a different kettle of fish. promise.completeWithTask requires the closure to be @escaping, as you may not immediately await it, but as noted here conceptually this is safe for Vapor to omit. The risk there is that this code becomes somewhat subtle to understand, but with careful code commenting it could plausibly be safe to handle.

1 Like
3

Thank you for your comment.

Sadly this is not what the concept of Sendable implies. Sendable is required any time we pass the value across concurrency domains, which we do here.

I understand that the concept of Sendable implies that a value can cross concurrency domains,
and I acknowledge that this code is indeed crossing those domains.
However, Sendable checks are static, and because there are limitations on a compiler's analytical capabilities,
it may conservatively treat safe situations as errors.

The following Sendable proposal contains useful information for contemplating this issue.

Proposal URL: https://github.com/apple/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md

According to this, the following implementation patterns are considered conformable with Sendable:

  • Value types, provided all the values they hold are Sendable
  • Immutable Classes
  • Internally Synchronized Reference Types

For value types, the values are copied when sent, so they are not shared.
Immutable classes are not mutable, so there's no issue there.
Synchronized reference types, as they are synchronized, cannot be accessed at the same time.

Is the purpose of the three examples ultimately to avoid
"simultaneous access to shared mutable state" ?

Additionally, in the "Transferring Objects Between Concurrency Domains" section,
it is written that even if unsynchronized raw pointers are passed,
it can still be safe as long as the sending side does not touch the value after sending it.

This also suggests that there is no problem as long as sharing is not performed.

This fourth pattern, unlike the other three, does not depend on the nature of the type implementation,
but on how the value is handled, which is under the control of the user.
This is why current compilers cannot statically determine its safety.

And in the case of the body in this transaction,
it's exactly the situation where the sender doesn't touch the value after passing it on.

Based on the above thought, since body isn't shared,
even if it's not statically proven to be Sendable (i.e., type-checked),
shouldn't it be able to cross boundaries in this situation?

On the other hand, if this reasoning is incorrect,
under what specific execution orders could race conditions or memory corruption potentially occur?

4

This problem is well-understood, but the solution isn't to drop the Sendable requirement in the current language, it's to make it so that the compiler can infer that it isn't needed at all. See [Pitch] Safely sending non-`Sendable` values across isolation domains for an example of how the compiler might grow this capability.

I agree with you that this usage pattern is probably safe, so long as we are excluding behaviours that are unsafe in async code anyway, but I'd still be reluctant to remove the annotation at this time.

1 Like
5

Another option is to ignore "sendability" by marking whatever you need to pass @unchecked Sendable.

Sendable is unfortunately an inflexible and weak system that disallows many safe patterns that are possible with standard lock usage. Sendable patterns also make code more complex, because closures break control flow.

1 Like
6

The ideal would indeed be for the compiler to be able to infer.
I will look into the thread you pointed out. Thank you.

7

Yes, it is easy to mark the type being passed as @unchecked Sendable.

However, if we could ignore the sendability of transaction,
instead of making the type we're passing Sendable,
I would want to modify transaction so it does not require sendability.

This is because changes to a type have much broader impacts.
Problems would arise when passing such a type to other functions that truly require sendability.

8

Another option is to use a wrapper:

struct UnsafeTransfer<Value>: @unchecked Sendable {
    var value: Value
}