Difficulty designing a static requirement due to Sendable & SE-0412

Using Swift
1

Hello,

In this post I share some hesitations I'm facing when designing/adapting apis for strict concurrency checks and in particular SE-0412.

I'll first expose the problem, based on a real example so that this post is not too abstract.

I'll then explore the various solutions I could identify, and describe how each one of them has undesired consequences.

Finally, I'll ask language designers what is their opinion on the topic.

The problem

I have two public protocols and one support type that is Sendable:

/// A type that can enter the SELECT clause of an SQL query.
public protocol SQLSelectable { }

/// A type that is tied to a database table and
/// builds database queries.
public protocol TableRecord {
  static var databaseTableName: String { get }
  static var databaseSelection: [any SQLSelectable] { get }
}

public struct Column: Sendable { }

You can see the static properties of TableRecord in effect below:

struct Player: TableRecord, ... { }

// SELECT * FROM player
//        ^      ^ Player.databaseTableName
//        Player.databaseSelection
let players: [Player] = try Player.fetchAll(db)

Enter a user who customizes their record type:

struct Player: TableRecord {
  // Intent: SELECT id, name FROM player
  // instead of SELECT * FROM player
  static let databaseSelection: [any SQLSelectable] = [
    Column("id"), Column("name")
  ]
}

Finally, enter SE-0412, which creates a warning in user's code:

:warning: Static property 'databaseSelection' is not concurrency-safe because it is not either conforming to 'Sendable' or isolated to a global actor; this is an error in Swift 6

The solutions

User defines the stored property as nonisolated

The user can make the static property nonisolated:

 struct Player: TableRecord {
-  static let databaseSelection: [any SQLSelectable] = [
+  // This is safe because `Column` is Sendable.
+  nonisolated(unsafe) static let databaseSelection: [any SQLSelectable] = [
     Column("id"), Column("name")
   ]
 }
  • :+1: This removes the compiler warning.
  • :-1: The user have to change the code of their record definition (SE-0412 is not, strictly speaking, source-breaking, but it creates churn).
  • :-1: The user uses an unsafe language feature.
  • :-1: The user either writes a comment explaining why this code is safe nevertheless, either leaves some code that will raise a concern in the mind of all future readers.
  • :-1: The rigorous user has to spend time convincing theirselves that the code is indeed safe.
  • :-1: The library looks ill-designed.

User turns the stored property into a computed property

The user can replace the let stored property with a var computed one:

 struct Player: TableRecord {
-  static let databaseSelection: [any SQLSelectable] = [
-    Column("id"), Column("name")
-  ]
+  static var databaseSelection: [any SQLSelectable] {
+    [Column("id"), Column("name")]
+  }
 }
  • :+1: This removes the compiler warning.
  • :-1: The user have to change the code of their record definition (SE-0412 is not, strictly speaking, source-breaking, but it creates churn).
  • :-1: The library looks ill-designed.
  • :-1: The var computed property instantiates a lots of arrays, instead of a single one with the stored let property.

Library makes SQLSelectable Sendable

The library modifies SQLSelectable so that it is Sendable:

-public protocol SQLSelectable { }
+public protocol SQLSelectable: Sendable { }
  • :+1: This removes the compiler warning.
  • :+1: The user does not have to change the code of their record definition.
  • :-1: The library has introduced a Sendable requirement without good justification. Not only this requirement is not needed, but it creates churn inside the library in order to solve the concurrency warnings created by this extra conformance.
  • :-1: The library has to remove support for NSString and NSData as selectable values. Not a big deal, but a possible inconvenience for users who deal with some legacy ObjC code.
  • :-1: Users who define mutable classes that conform to SQLSelectable have to make them Sendable, which is not a trivial task. This is even more vexing, considering that the library did not need SQLSelectable to be Sendable in the first place (as mentioned above).

Library has databaseSelection require Sendable values

The library modifies databaseSelection so that it requires Sendable values:

 public protocol TableRecord {
-  static var databaseSelection: [any SQLSelectable] { get }
+  static var databaseSelection: [any SQLSelectable & Sendable] { get }
 }

The user modifies their code accordingly, so that the compiler sees the customization:

 struct Player: TableRecord {
-  static let databaseSelection: [any SQLSelectable] = [
+  static let databaseSelection: [any SQLSelectable & Sendable] = [
     Column("id"), Column("name")
   ]
}
  • :+1: This removes the compiler warning.
  • :-1: The user have to change the code of their record definition, otherwise the default implementation (SELECT *) kicks in.
  • :-1: The library has introduced an unneeded Sendable requirement without good justification.
  • :-1: (An old complain) the language does not see that protocol requirements have changed, and won't guide the user towards the proper resolution.
  • :-1: The mention of [any SQLSelectable & Sendable] (a mouthful) in user's code is mandatory, so that the compiler does not miss the customization point.
  • :-1: The library looks verbose.
  • :-1: The library has to mention this change in the upgrading guide.
  • :-1: Some users will not read the upgrading guide and will wonder how to solve the initial warning.

Isolating databaseSelection to a global actor

This is a no go. The library users enjoy parallel database accesses, as well as synchronous database accesses, two features that are too precious to give up. AFAIK, those features are unsupported by global actors.

Conclusion

Did I miss a solution that has no (or less) drawbacks?

Of course, I wish the compiler would understand that this code is safe:

public protocol SQLSelectable { }

public protocol TableRecord {
  static var databaseTableName: String { get }
  static var databaseSelection: [any SQLSelectable] { get }
}

public struct Column: Sendable { }

struct Player: TableRecord {
  // Safe, because the actual values ARE sendable.
  static let databaseSelection: [any SQLSelectable] = [
    Column("id"), Column("name")
  ]
}

What is your opinion? And in particular, do you think the compiler could learn that the above construct is safe?

9 Likes
3

I’m no longer a language designer, and certainly not for async, but I want to point out that the warning is correct:

struct Evil: TableRecord {
  static let cursed: NSMutableString = .init("original")
  static let databaseSelection: [any SQLSelectable] = [Self.cursed]
}

// Thread A
Evil.cursed.append("modified")
// Thread B
db.fetchAll(Evil.self)

I think you’ve laid out your options pretty well, limited as they are by “protocol requirements must exactly match in type”. (Otherwise, the common case could provide an array of Column or String.) But perhaps a current language designer will have more to add.

4

SQLSelectable essentially represents an abstract SQL computation, with the most common case being a single column but potentially being essentially anything that you could write in the list immediately after SELECT? I’m confused why that wouldn’t always be a Sendable value type.

5

Yes, @jrose, the warning is correct when you assign a non-sendable value to the existential. I'll post a pitch below, and would be interested in your feedback.

I think you’ve laid out your options pretty well, limited as they are by “protocol requirements must exactly match in type.

Thank you :-)

6

Yes, exactly! Users frequently select columns, but they also select *, values, subqueries, sql function calls, columns from joined tables, anything that fits behind SELECT.

I’m confused why that wouldn’t always be a Sendable value type.

I can make it Sendable, but I currently do not have to. Practically, SQLSelectable is a protocol that contains a single requirement, the building of an opaque Sendable type that can safely fly in database concurrency:

public protocol SQLSelectable {
    var sqlSelection: SQLSelection { get }
}

public struct SQLSelection: Sendable { ... }

Not making SQLSelectable Sendable it a nicety for the users. They can extend the library with new selections (or expressions, etc.) that fit their needs without caring about Sendable. Don't let me be misunderstood: my point is not to foster non-Sendable type. My point is to avoid adding Sendable requirements that can be avoided, because not all user types can easily be made Sendable. And we don't want to foster slammed @unchecked Sendable on non-sendable types.

EDIT: missing in my initial exposition is the fact that SQLSelectable is not only used for defining the default columns of a record.

Users can also customize the selection:

// SELECT name FROM player
func fetchNames(_ db: Database) throws -> [String] {
  try Player
    .select(Column("name")) // func select(_ : any SQLSelectable...)
    .fetchAll(db)
}

// SELECT player.*, COUNT(...) FROM player LEFT JOIN ...
Player.annotated(with: Player.teams.count)...

The last example involves an aggregate that uses a COUNT(...) selectable.

SQLSelectable fulfils many use cases.

2 Likes
7

It looks like it would possible.

The pitch could be named...

Inference of Sendable for existentials and existential containers

Motivation

The compiler is not able to detect that some existentials of non-Sendable protocol that wrap Sendable values can not create Sendable violations:

// Program 1
protocol P { }
struct S: P, Sendable { }
func f(_ x: Sendable) { }

func demo() {
  let s = S()
  
  // ⚠ Warning: type 'any P' does not conform to the 'Sendable' protocol
  f(s as any P)
  f([s as any P])
  f(["hello": s as any P])
}

// Program 2
protocol Requirements {
    static var value: any P
    static var array: [any P]
    static var dict: [String: any P]
}

struct Demo: Requirements {
  // ⚠ Static property is not concurrency-safe because it is not
  // either conforming to 'Sendable' or isolated to a global actor;
  // this is an error in Swift 6
  static let value: any P = S()
  static let array: [any P] = [S()]
  static let dict: [String: any P] = ["hello": S()]
}

The compiler emits Sendable violation warnings in both programs above. This is overly conservative.

The first program is safe, because:

  • the p variable is constant, declared with let.
  • the value visible through the existential interface is Sendable: it is s.

The second program is safe, because:

  • the globals are constant, declared with let.
  • the value stored in the global contains existential value(s) that wrap Sendable values.

There no way to use or reaffect those variables in a way that would create a Sendable violation. They have all the qualities of Sendable values.

The first program above does not quite look like a real program. But the second is quite similar to the example given in the original post of this thread. It demonstrates an undesired interaction of existentials with SE-0412 Strict concurrency for global variables.

Sketch of a solution

Inspired by SE-0414 Region based Isolation and SE-0418 Inferring Sendable for methods and key path literals, the compiler can assert the safety of the above program, by infering existentials to be any P & Sendable.

This addresss the first program:

// No warning
f(s as any P /* & Sendable */)
f([s as any P /* & Sendable */])
f(["hello": s as any P /* & Sendable */])

In order to address the undesired interaction with SE-0412 and protocol requirements, we need something more.

Indeed, infering Sendable as described above would miss the protocol requirement, and fail the developer intent which is to fulfill a customization point:

struct Demo: Requirements {
  // Misses the `any P` requirement
  static let value = S()            // inferred as S
  static let value = S() as any P   // inferred as any P & Sendable
  
  // Misses the `[any P]` requirement
  static let array = [S()]          // inferred as [S]
  static let array = [S() as any P] // inferred as [any P & Sendable]
  
  // Same miss for the dictionary
}

Declaring the type of static properties in order to target the protocol disables the Sendable inference:

struct Demo: Requirements {
  // No longer infered as any P & Sendable: warning due to SE-0412
  static let value: any P = S()
  
  // No longer infered as [any P & Sendable]: warning due to SE-0412
  static let array: [any P] = [S()]
  
  // Same miss for the dictionary
}

To address this, the compiler only uses the Sendable inference of existentials when considering the assignment to the constant global:

struct Demo: Requirements {
  // value, of type `any P`, is assigned to a `any P & Sendable`: OK.
  static let value: any P = S()
  
  // array, of type `[any P]`, is assigned to a `[any P & Sendable]` which is
  // sendable because arrays of sendables are sendable: OK.
  static let array: [any P] = [S()]
  
  // Same inference for the dictionary
}
8

Making that work for your use case would require special knowledge of Array. I’m not saying it’s completely off the table, but it’s a lot more restricted than I think you’re giving it credit for.

9

I was indeed considering if it would be possible to design a Container<T>, that is Sendable when T is Sendable, but could still ruin the Sendable inference of existentials I outlined above. If this is the case, then I'm indeed asking if a special case could be done for Array - and I'm not holding my breath :sweat_smile: I mean, I won't insist. I have exposed my motivations. Now others can consider and pounder if they're relevant and expand outside of my own personal needs.

EDIT:

For example, I was trying to build a funny Container like this one:

struct Container<T> {
  var nonSendable: NonSendable?
  init(values: [T]) where T: Sendable { }
  init(values: [T]) { nonSendable = NonSendable() }
}
extension Container: @unchecked Sendable where T: Sendable { } // Correct

But I think the code below is still safe, and the logic exposed in the pre-pitch above still holds:

// Container<any P> assigned to a value inferred as
// Container<any P & Sendable> which is sendable due
// to the conditional conformance: OK for "enhanced SE-0412"
static let global: Container<any P> = Container(values: [1])
10

my totally naïve, outsider’s perspective is that this is your best option, and the impact on users is less than you are imagining, because one would already have to make the property computed if you wanted to make the conforming type generic. (i personally avoid static let specifically to preserve the freedom to genericize a type later.)

1 Like
11

I have probably hyperbolized the downsides of various solutions, because, well, if I do not take care of the library users, who will?

I'd really like to avoid user code change that can be avoided, for multiple reasons.

Some people don't read the migration guide when they upgrade to the next major version, and the compiler warning provides no hint at the possible resolution (replacing the stored property with a computed one). This means that some users will open issues, and I will have to spend time providing the answer.

I don't want to be the author of a library that creates warnings with perfectly valid user code. There are advantages in computed properties, that you have outlined, but who am I to force users to use a particular code style? static let exists, is relevant our particular context, and there is no reason to discourage users from using it. Good ergonomics is adapting to users, not the other way around.

My real best option, in the current state of the language, is to make SQLSelectable Sendable. This will remove the warning in user's code, without change. This will create a little churn in the library implementation, but that's something I can handle. This will force some users to make Sendable their custom selectable type, but they should be rare. I can see how one could consider that, after all, this is not a big deal. But this won't remove the fact that the language could be enhanced in a way that would resolves all the problems described in the OP. I'm not in position to judge whether this enhancement is necessary or not.

3 Likes
12

When checking Sendable for arguments that cross an isolation boundary, the compiler already looks through implicit conversions to the underlying argument type, and the call is fine as long as the underlying value type is Sendable. If a global variable is immutable (meaning it also must have a value assigned), why wouldn't we be able to do the same thing?

5 Likes
13

Yeah, I was about to ask something similar—isn't the 'special knowledge' required here the already-special covariance that Array admits on its generic parameter? I can't immediately see a way to break this if the underlying initialization expression is of Sendable type. It feels like allowing an immutable global to be initialized from a value of Sendable type would be a reasonable relaxation of SE-0412, unless I'm missing something.

Edit: AFAICS this is essentially isomorphic to the following (which compiles without warning) except that in the direct assignment case the type conversion causes us to lose track of the underlying value's Sendable information:

struct Demo2: Requirements {
    static let s = S()
    static var value: any P { s }
}
2 Likes
14

I worry a little about adding inference for an explicitly-typed global variable based on its value, because if someone ever wants to change the value, they may suddenly see warnings. Now they potentially have an API contract they're supposed to uphold, but it's not the one they thought they were promising.

(Limiting this to non-public types might be good enough for a library like this, but that's not really a general answer.)

2 Likes
15

I don't think we should change the inferred type of an explicitly typed variable; I just think we should suppress the concurrency warning if the underlying value is Sendable because there's no potential for data races even if the type of the variable is not Sendable.

4 Likes
16

Yeah, but the type is supposed to be the contract you want. If I wanted to add a dynamic string to that array in my next release, I wouldn't find out until then that I can't.

5 Likes
17

That's not an issue with the contract, though, is it? This suggestion wouldn't result in library authors accidentally 'leaking' a Sendable promise where they don't mean to, it would just push off the implementation error to the point where they actually try to use a non-Sendable value in a way that would admit data races. I don't see how this is fundamentally different from the similar Demo2 situation I had above, modified slightly to emphasize value as the ‘contract’ and s as the ‘implementation’:

struct Demo2: Requirements {
    private static let s = S()
    public static var value: any P { s }
}

if S itself, or the value returned from value one day becomes non-Sendable, then we'll get diagnostics just the same, but only because we really are trying to do something that is concurrency-unsafe.

2 Likes
18

If we inferred the array literal as having Sendable type and then converted it to an non-Sendable type, sure. That is not what’s happening here, though; the array is being inferred from context as having Array<any SQLSelectable> type. So the special knowledge would be that those are equivalent as far as sendability goes: if you construct a non-Sendable array with elements that are Sendable-in-fact, the array will be Sendable-in-fact. It is not generally true of non-Sendable types that constructing them with Sendable arguments makes them actually Sendable, though.

I don’t know how to make what you’re saying about covariance into a concrete rule — something like that we could have inferred a Sendable type for the array element and the converted it? Is that something we can actually decide retroactively? And presumably that wouldn’t really be the rule we’d want, because it would rule out heterogeneous arrays (e.g. of Any) where all the elements happen to be differently Sendable.

3 Likes
19

I've just met a similar issue with Decoder.userInfo, declared as [CodingUserInfoKey: Any]. The property does not create any problem per se. But any application that wants to store some userInfo statically will face the same problematic interaction with SE-0412 as the one described in this thread. Despite all involved types being Sendable, the existential Any ruins everything:

// stdlib CodingUserInfoKey is Sendable
private let myKey = CodingUserInfoKey(rawValue: "context")!

struct MyType: Decodable {
    // ⚠ Static property is not concurrency-safe because it is not either
    // conforming to 'Sendable' or isolated to a global actor;
    // this is an error in Swift 6
    static let blueContext: [CodingUserInfoKey: Any] = [myKey: "blue"]
    static let redContext: [CodingUserInfoKey: Any] = [myKey: "red"]

    /// Configure decoding with MyType.blueContext or redContext.
    init(from decoder: any Decoder) throws {
        // decode according to context
    }
}

Such code is not irrealistic. If a type configures its decoding from userInfo, it is reasonable that this type exposes the userInfo it understands as static members. Those dictionaries are part of its public api.

This time, it is not possible for the library author to make their type Sendable ;-) The only possible solutions are in userland (make the static property nonisolated, or computed).

This is one more example where safe user code has the compiler emit a false positive concurrency warning.

And this is also an illustration of the limit of the "just make the protocol Sendable" position:

  • There's no protocol that could turn Any into a Sendable value.
  • The standard lib does not need, and thus does not want to make userInfo Sendable (ignoring ABI stability in this argument).

Putting the burden on the shoulders of the library, as in "any type defined or used in a library should be Sendable, for the sake of ergonomics, just in case users would like to declare an immutable global of this type" is not tenable.

Putting the burden on the shoulders of the user, as in "users should just avoid stored immutable globals and declare them as nonisolated(unsafe) (1), or computed (2), or global-actor isolated (3)" is not tenable either. (1) is not to be fostered, (2) does not work with globals that are heavy to build, (3) has strong runtime implications, possibly undesired.

If libraries and users are not at stake, we're left with the language itself.

It just happens that users need globals whose type is not under user control. This does not only happen when some protocol has a static requirement (the focus of the beginning of the thread). When the type of an immutable global is not sendable, it would be nice if the user could store a known-sendable value into it, without compiler warning.

2 Likes
20

Okay, I see what you're saying—let me try to explain the intuition I'm reaching for here, though I'm not 100% sure it's actually formalizable.

The cases I'm primarily interested in are ones that could be rewritten likeI’ve done with Demo2 above: we have a static let of some non-Sendable supertype N, as well as an initialization expression which, if pulled out into an un-annotated static let of its own, would 'naturally' be inferred as some type S: N, Sendable.

In this situation, I feel like what I'd want the compiler to do is look at the initializer expression, and see that the last conversion step drops Sendable in order to store the value in a variable of type N, so the initialization is actually fine. In the Array case, where we have a variable of type [N] and an initialization expression, I'd want type checking to essentially 'hold on' to any Sendable for as long as it can. So rather than doing an elementwise S -> N to solve the array literal, we'd prefer a solution which does an elementwise S -> N & Sendable conversion, so that the final step in the conversion for storage is [N & Sendable] (which is itself Sendable) to the storage type [N].

Even if this 'delay dropping Sendable' logic doesn't totally work for complex initializer expressions, it would provide a trapdoor for users to guide typechecking. I could write:

static let array: [N] = [S()] as [S] // or 'as [N & Sendable]'

to ensure that we'll only find solutions where the final conversion really does drop Sendable.

Is the workaround I posted above suitable as a stop-gap? I.e., store the Sendable value in an appropriately-typed static let and then return through the computed non-Sendable static var?

2 Likes
21

Thanks for saying this. It is also the kind of idea I was contemplating at the very end of the mini-pitch above.

struct Demo2: Requirements {
    private static let s = S()
    public static var value: any P { s }
}

Thank you for mentioning this other way to silence the warning. It is a computed property, but it avoids the recomputation of the value :+1:

To answer your question (is it suitable?), let's put my language-designer hat [1]. I'd be concerned by the fact that the compiler diagnostic on the plain static let would have to explain that there are four possible ways to solve the warning, while I'm perfectly aware that this warning can be a false positive, and draw the developer in an unneeded obscuring of their program. In a LSG meeting, I'd try to have the problem 1. acknowledged, and 2. bumped from "nice to have eventually" to "required rather soon in the lifetime of Swift 6, as all other identified compiler features that we know are needed for the Swift 6 mode to have the slightest chances of a success, and are actively working on at present time."

  1. which I'm not :rofl: ↩