SE-0302 (second review): Sendable and sendable closures

Since we've already severely limited Sendable's abilities by making it a "marker" protocol, the biggest action-at-a-distance hazard I see with implicit conformances to it is name lookup. If we also say that extension Sendable extensions are not allowed to declare members, that should contain that effect, so that modifying a type's layout does not cause members to flit in and out of existence everywhere else. At that point, I think most of the consequence Sendable conformance appearing or disappearing ought to be contained to first-order "does not conform" errors in call sites that require sendability.

1 Like

I'm referring to a user attempting to send a value across a concurrency boundary and receiving an error because no implicit Sendable conformance was provided. This could be due to a member of a member of a member, i.e. arbitrarily distant from the actual type the user attempted to send. @Chris_Lattner3 and I discussed this in one of the earlier threads.

That's not really any different that the errors users see with automatic Codable or Hashable conformance, so it doesn't seem to make anything worse. Those errors can identify which property of a property (ad infinitum) breaks the conformance, so it should work for this case as well. Improvements to the diagnostic would help in all cases as well.

1 Like

It is actually quite different because Equatable conformance has to be explicitly added so you get the error in the expected location.

I would think the diagnostic could point wherever we want, starting at the point at which you try to pass it between contexts, then to the type, then to the properties preventing the conformance. It's not really any different than trying to decode a type that has no Decodable conformance.

1 Like

I think it makes a lot of sense that a “marker” protocol can’t make new APIs appear on types, even via extensions. (Users could still, of course, create a protocol refining a marker protocol and have those types adopt that conformance.)

If this is agreed, then it simplifies our reasoning greatly regarding the possible risks/benefits of implicit conformance to Sendable:

What’s the possible harm of implicit conformance where the author hasn’t thought about it? If the rules are sound surrounding which value types can conform via the “checked conformance” route, the only drawback I can see is that future changes to the type could accidentally take away the conformance. In that case, an end user could always add it back with an “unchecked” conformance. This seems strictly better than the alternative of not conforming at all.

2 Likes

Finally chiming in here, trying to earn a bit of my (IMO entirely undeserved, but you all are very kind) authorship credit:

I like this form of the proposal. The high-level mental model is easy enough to grasp heuristically, and the fiddly details — while potentially surprising in context — do all follow naturally from that high-level model.

Speaking to the specific questions in this follow-up review:

  • Sendable is a better name than Concurrent. The latter might imply that multiple concurrency domains can simultaneously use a shared value of the type; the former makes it clear that it is transferring, not sharing, that is safe.

    I do worry a bit that Sendable is an over-broad name. Sending where, across what boundary? Between processes? Over the network?

    A yet-clearer name would be ActorSendable, though that wrongly limits the meanings to actors instead of all forms of concurrency. An even-yet-clearer-but-unacceptably-ugly name would be CrossConcurrencyDomainSendable (yikes). This rabbit hold seems like a bad one; I am willing to accept that, like Codable and @escaping, a potentially more general word takes on a very specific meaning with no further qualification because of how common that one specific meaning is. Given how pervasive the term would be, I’m sure a web search for “swift sendable” would turn up answers for confused new users. Sendable thus strikes me as acceptable.

    I’ll float one possible alternative: AsyncSendable. Here it is in context:

    struct Person : AsyncSendable {
      var name: String
      var age: Int
    }
    
    @propertyWrapper
    struct UnsafeTransfer<Wrapper> : @unchecked AsyncSendable {
      ...
    }
    
    actor MyContactList {
      func filteredElements(_ fn: @asyncSendable (ContactElement) -> Bool) async -> [ContactElement] { … }
    }
    

    Advantages:

    • It’s clear that the name refers to concurrency, not other kinds of sending.
    • It ties to the async keyword with which this proposal interacts.
    • It perhaps improves searchability.
    • It’s short.
    • It still loosely respects Rust’s term of art.

    Marginally better? Perhaps. As I wrote above, Sendable is acceptable; this is not something I have Strong Feelings™ about.

  • Being able to mark individual properties @unchecked instead of the entire type is a very compelling idea, and we should consider it sooner rather than later. Having one nuisancy property spoil the safety checking for an entire type is a recipe for future bugs: a large type gets marked @unchecked Sendable because of one dark corner, then another developer comes along later and adds features to another part of the type, not spotting the @unchecked at the top of the file and expecting Swift’s usual level of static safety.

  • The annotation burden of adding Sendable to many types seems low to me. The rationale regarding mindfulness about public API contracts is entirely compelling. The migration path for Error seems sensible. I approve of the proposal’s approach to “rip the band-aid off” in one go rather than introducing an unsafe actor regime first, then trying to migrate to safety later.

A few of other things that still stand out to me, which do not necessarily need to hold up this proposal, but deserve future consideration:

  • Being unable to say if val is Sendable or val as? Sendable is a generality gap that is going to force a few scattered pieces of framework code into nasty corners. It deserves attention. (Thumbnail sketch: We have some heterogenous collection of assorted values, a cache, maybe, or lookup dynamically constructed from configuration, who knows. We need to share values in it across concurrency domains. We have some boxing mechanism that allows for safe sharing of non-async-sendable types, but it’s costly, and we want to optimize it away with a runtime type check when possible. Oops.)
  • What is @_marker, really? The proposal sketches out the loose meaning clearly enough, but it’s not quite clear why it needs to exist at all, or what it really means in practice. Is the import nothing more than the fact that Sendable is a compile-time-only type, and the compiler doesn’t emit any runtime support for it? Is it thus purely optimization, or is there a reason why a Sendable type can’t exist at runtime? Are there other types that have similar problems, and should @_marker be a public feature?
  • The spelling of @unchecked Sendable reads clearly enough, which is the main concern for this proposal, but it raises questions about annotations on protocol conformances. Is there some generality here we need to consider — in terms of user mental model, in terms of language capability? Or is this entirely a syntactic one-off, and developers should treat @unchecked Sendable as a single non-decomposable thought, much like @private(set)? What precedent does this create, if any? (Are there any per-conformance annotations in the language already?)
6 Likes

Yes, thank you, this was an oversight; it's covered in the announcement, but not in my brief summary of the announcement here. I've added a bullet briefly describing this change.

1 Like

Instead of having implicit conformance, when you pass a non-Sendable value to where one is expected (say to an actor), could Xcode offer a fix-it to add it for you?

1 Like

I'm mostly happy with the implicit conformance for non-public types. For public types, though, the potential harm is that authors may be vending API that they do not realize, which could constraint future changes in unexpected ways.

If I, as a library author, write a struct

public struct S {
  var x: Int
}

I may not intend for this type to be usable across concurrency domains, but the definition written here (if implicit conformances were permitted for public types) would preclude me from ever making this type concurrency-unsafe unless I was willing to have a potential source break. At the very least we'd need a way for library authors to spell "do not make this implicitly Sendable, it may become un-Sendable in the future."

IMO, this is similar to the reason we disallow the synthesis of a public memberwise initializer—it's not that exposing the memberwise initializer is potentially incorrect, it's that it potentially causes library authors to expose public API without even realizing it.

3 Likes

It’s not clear to me how this issue would apply to frozen types. The proposal explicitly excludes either public or frozen types, but if you’ve ruled out ever changing the stored members, then...

This is true, but I actually think we need such a spelling anyway to disable retroactive conformances via @unchecked Sendable regardless of how we decide the implicit conformance issue.

For instance, if we decide that we don’t want raw pointers to be sendable (for reasons I enumerate above and in line with Rust’s rationale for the same decision), then it would be rather useless if any third-party library could vend extension UnsafeMutableRawPointer: @unchecked Sendable { } for all unwitting users of that library. The same line of reasoning for other standard types we’ve already decided should not conform to Sendable.

(The brain-dead but I think sufficient solution here would be a delightful marker protocol named Unsendable. This trichotomy would allow an optimal expression of user intent: Can this type be sent across concurrency domains without manually implemented internal synchronization? (A) Yes, always. (B) No, never. (C) Let the compiler decide.)

3 Likes

Yeah, I can’t really think of an objection here off the top of my head. Maybe there’s a situation where a type would want to start using it’s existing members in a concurrency-unsafe way? Say, initializing an Int value from a pointer and then converting it back internally. That’s pretty contrived, though.

Eh, I don’t know. We don’t have a general facility for preventing retroactive conformances that a type author has determined can never be valid, so I don’t really see why we’d need it for Sendable.

I suppose there’s an argument that an improper Sendable conformance is more ‘fundamentally unsafe’ than, say, an improper Equatable conformance, but for virtually any semantic guarantee that a protocol makes you could easily write a program that exhibits unsafe behavior in the face of a conformance that violates the guarantee.

Regardless, I still think we’re better off forcing library authors to make API promises explicitly than hoping that they remember to opt-out of implicit API promises.

I can sort of understand structs being implicitly Sendable in a world where actors and structured concurrency were built into Swift 1, then it would probably make sense for vanilla structs to be completely capable of being sent across concurrency domains out of the box. You, the developer, think naturally of what structures you want to have, and it-just-works with concurrency.

I do wonder in that world if things like this would be necessary:

struct X<T> {
  var value: T
}
extension X: Sendable where T: Sendable { }

If structs just-work, why don’t generic structs? Are generics an advanced feature of Swift, or are they fundamental, or somewhere in the middle?

To get the implicit behaviour, I also have to add this Sendable concept:

struct X<T: Sendable> {  // implicitly conforms to Sendable
  var value: T
}

That feels weird in what I imagine an intro-to-this-imaginary-Swift-1 tutorial looks like. To introduce the concept of generics, I soon also have to introduce the concept of Sendable? But all my tutorial was trying to do was basic UI concepts and it send something to a background queue so suddenly I have to dive in much deeper on how Swift treats types in a concurrent world. Pretty sure getting things off the main queue is a regularly recommended best-practice by Apple, so I feel like this wouldn’t be an advanced tutorial.

A barely-thought-through alternative:

struct X<T: valuetype> {  // implicitly conforms to Sendable
  var value: T
}

struct Y<T> {  // does NOT implicitly conform to Sendable
  var value: T
}

Why? This valuetype concept is probably easier to explain than Sendable.

My point is to ask — how long into a pretty standard Swift tutorial would you have to introduce this concept of Sendable due to talking about generics or similar?

I agree that @concurrent is stronger, and I also think it is generally more understandable and meaningful: a @concurrent function can be executed concurrently, whether with itself or with respect to the place it was formed. That implies that it conforms to Sendable.

As much as I would want the Sendable/@sendablenaming to line up, I think@concurrent` remains the better name for the attribute.

I agree that this makes a lot of sense, and am happy to ban such extensions.

I agree that the latter is the clearest option, and the only one we should support.

I tend to agree with you. As I've noted before and has been mentioned here, I am not at all concerned about types that start out being Sendable and then add some reference-type-ish field that makes them not Sendable. We have tools that catch it, and one has almost surely also broken value semantics with such a change, which will be the bigger surprise.

This is a good point. The implicit Sendable makes it much more likely that we'll end up with a surprise with unsafe pointers.

It's an important optimization, because the vast majority of types will conform to Sendable one way or another. It also makes it possible to introduce Sendable conformances in a manner that can be backward-deployed.

I think this is a good feature to make public at some point in the future, because there are undoubtedly other semantic tags we might want later (e.g., the ValueSemantics protocol idea that came up in earlier pitches).

The language does not have per-conformance annotations yet, although the idea has come up for a couple of other reasons:

  • @available on conformances is a recent addition (Swift 5.4), and is currently inferred from the availability of the extension. We could narrow this to an annotation on the specific conformance that has availability.
  • @retroactive on a conformance could acknowledge that a conformance is in neither the original type's module nor the protocol's module. Retroactive conformances can cause of some confusion, so there's a possible future where folks want to know about them and mark them as such explicitly.
  • Actors pitch #4 has a "future direction" for isolated conformances.
  • Scoped conformances could put access control on the specific conformance.

The compiler can generate Fix-Its to help migrate code, and IDEs or other tools can help you apply those Fix-Its.

Doug

2 Likes

If anything this might be a strong argument to go down this road: having something that somewhat implicitly signals "hey you broke value semantics here" might be useful. It won't catch all cases, but it will catch many.

2 Likes

Sendable brings more questions than explanations to me: what are the sources and the destinations of the sendable values? I cannot find rationales behind why the original ValueSemantic is dropped (IMO this name is much better to understand and conform to than both Sendable and ConcurrentValue).

1 Like

Hmm, I don't know that I'm so compelled by this. My personal concern is less about the ability of library authors to catch the source break, and more about the ability to constrain themselves in unexpected or surprising ways. Yeah, I suppose it's better to realize you have a potential source break so you can do a major semver bump, or whatever, but that doesn't help the library author who simply doesn't want to break source by introducing a property. Allowing the implicit conformance would put the onus on the author to always think "hm, I might possibly want to make this concurrency-unsafe in the future, better opt-out."

I'm not as troubled by the break implied by loss of value semantics since IMO clients relying on the value semantics of a type which makes no such explicit guarantee are more clearly buggy than clients which rely on conformance to a protocol.

I agree that this might be independently useful, but not at the cost of, essentially, forcing library authors to guarantee value semantics for all value-semantics-composed types by default, with explicit opt-out required. We could have a tool that alerts authors to violations of value semantics without imparting implicit API promises on library authors.

I believe the reason is that there is no agreement on what, exactly, value semantics are, nor on how to map those semantics to the requirements for safely sending a value across concurrency domains.

Long story short, value semantics are not what we're after. It is some notion of "this is safe to send/share across actor/execution/thread contexts", which is why Send(able) works well. And "ValueSemantics" is just a small subset of it -- there exist plenty of non-value semantics types which are totally fine and expected to be sent between actors.

// I need to finish my review / writeup for this thread, will post soon... but it's shaping up very good :+1:

4 Likes

Hmmm, if the name split that way, I would worry that @concurrent implies that a closure will execute concurrently, not merely that it can.

IOW, on a naive reading,

let longRunningTask: @concurrent () -> Void

…wrongly seems to imply that a longRunningTask() call will not block.

I also grant, however that @sendable () -> Void doesn’t communicate much on the naive reading, and the @asyncSendable () -> Void that I proposed upthread is a bit unwieldy.

Maybe @concurrencySafe?

And I take it the concern is just the sheer size of all of those witness tables…?

That does make sense.

I do wonder if there isn’t some ingenious way to finagle runtime support for memberless protocols that requires neither witness tables nor ABI breakage…but I assume you all understand this many orders of magnitude better than me, and have already thought it through! Given that, I can let go my wish for is Sendable.

That’s an exciting one! Thanks for the context. It does make the conformance annotation syntax in this proposal look more comfortable.

1 Like