SE-0302 (second review): Sendable and sendable closures

Hello, Swift community.

The second review of SE-0302: Sendable and @sendable closures begins now and runs through March 8th, 2021.

This is a narrow re-review, and reviewers should focus on the changes made in the first revision, which are described in the announcement post. To briefly summarize them:

  • The @concurrent function attribute is now named @sendable.
  • The ConcurrentValue protocol is now named Sendable.
  • The UnsafeConcurrentValue protocol no longer exists, but Sendable conformances may be explicitly decorated with @unchecked, e.g.
    class MyClass: @unchecked Sendable {}
    
  • Non-public struct and enum types now implicitly conform to Sendable if their stored properties and case payloads all conform.

Reviewers may also find the previous review thread interesting.

Reviews are an important part of the Swift evolution process. All review feedback should be either on this forum thread or, if you would like to keep your feedback private, directly to the review manager. If you do email me directly, please put "SE-0302" somewhere in the subject line.

What goes into a review?

The goal of the review process is to improve the proposal under review through constructive criticism and, eventually, determine the direction of Swift. When writing your review, here are some questions you might want to answer in your review, keeping in mind that you are just reviewing the changes listed above:

  • What is your evaluation of the proposal?
  • Is the problem being addressed significant enough to warrant a change to Swift?
  • Does this proposal fit well with the feel and direction of Swift?
  • If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
  • How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

More information about the Swift evolution process is available at:

https://github.com/apple/swift-evolution/blob/master/process.md

As always, thank you for contributing to Swift.

John McCall
Review Manager

12 Likes

The semantics I would expect from @concurrent functions vs @sendable functions
are very similar, but not identical:

@concurrent focuses on the fact that (identical copies of) such functions can be called simultaneously from multiple concurrency domains (actors/executors/threads etc.).

@sendable focuses on the fact that (identical copies of) such functions can be transferred between concurrency domains, but I would not necessarily assume that they can be called when in a different concurrency domain without additional external locking or the guarantee that the function only exists in the current concurrency domain.

Thus I believe @concurrent to be stronger (and possibly strictly stronger) than @sendable.
However, I am uncertain what a correct response to this discrepancy would be.

3 Likes

Generally speaking, pretty nice improvement to original PR.

According to new Sendable @sendable semantics, it'd be better to rename UnsafeTransfer to UnsafeSender - a propertyWrapper of : @unchecked Sendable semantic wrapper.

@propertyWrapper
struct UnsafeSender<Wrapped> : @unchecked Sendable {
  var wrappedValue: Wrapped
  init(wrappedValue: Wrapped) {
    self.wrappedValue = wrappedValue
  }
}
actor MyAppActor {
  public func doStuff(@UnsafeSender dict: NSMutableDictionary) async
}

It would be very nice if the core team would define what @uncheck vs checked (example CheckedContinuation ) will mean in Swift. Similarly how it is defined that Unsafe in Swift relates only to memory safety.

Presumably the @unchecked Sendable type will be “checked” for ownership violations?

I’d like to suggest @forced Sendable as an alternative spelling. Reminds me of git force push.

Maybe even @coerced Sendable.

How will protocol inheritance be handled? Let's say I have:

protocol MyProtocol: Sendable {}

Is the following allowed?:

struct MyStruct: @unchecked MyProtocol {}

Or will explicit conformance to the Sendable protocol be required?:

struct MyStruct: MyProtocol, @unchecked Sendable {}

I think the latter is the clearest option.

Edit: I'm explicitly talking about the @unchecked option here. I'm assuming that without @unchecked, conforming to MyProtocol will just imply Sendable.

2 Likes

This reads pretty clearly to me: it's the "sendability" (or, in other words, the semantic validity of conformance to Sendable) that's unchecked. It clearly does not imply that anything else about Swift's rules is unchecked.

UnsafeTransfer is explicitly not a part of this proposal, just included "to illustrate aspects" of future directions, so we don't need to bikeshed the name.

2 Likes

@john_mccall, I think it may be important to add also the following that's mentioned in the document's revision history:

  • Add implicit conformance to Sendable for non-public, non-frozen struct and enum types.

I'd like to briefly focus the review on that point. Specifically, the proposal states:

A struct or enum can only be made to conform to Sendable within the same source file in which the type was defined.

For non-public, non-frozen structs and enums, the Sendable conformance is implicitly provided when conformance checking (described in the previous section) succeeds

I worry that these two rules will not compose well. Library authors who are not fully paying attention to how their value types work with concurrency may find that everything "works fine" only to neglect to vend a checked conformance [e.g., when they extract a type from a working app into a shared library], requiring users to write unchecked retroactive conformances.

I think that if a struct or enum is provably safe to send across concurrency domains, and we buy the justification for implicit conformance for this protocol as opposed to, say, Hashable, then it should apply to public and non-public, frozen and non-frozen structs and enums.

I should raise again the point about considering whether we ought to align with Rust in not conforming the unsafe pointer types to Sendable in the context of this implicit conformance. That would be because structs (and tuples!) that clearly are not safe to conform to Sendable would be implicitly conformed if one of their members is an unsafe pointer.

3 Likes

@unchecked is supposed to mean, “don’t worry, the compiler can’t check but I’ve made sure that it’s safe via internal means.”

A third-party retroactive conformance by construction has no control or sometimes even visibility into how the type is implemented and can’t actually make that guarantee, so whatever the documentation is of that decision, it’s bunk: this would only be a workaround for a missing first-party conformance.

The better solution is to make it easier for first-party authors not to forget that conformance in the first place. For this reason, I argue that implicit conformance should apply to all eligible value types or none, not only a subset of them based on visibility and resilience, which is likely to cause people to forget.

4 Likes

I still think the implicit conformance could turn out to be a mistake. It will lead to confusing non-local compiler errors. I don't think the annotation is as big a deal as it is being made out to be, at least for application developers, and I'm willing to tolerate some boilerplate in library code.

In practice, application developers will only need to provide a conformance for types they actually need to send across a concurrency boundary. In most well written applications this will be a small portion of the types. Drawing on an example from existing experience, most application code I see does not declare an Equatable conformance if the type does not actually need to be compared in the codebase. To say this another way, just because a valid conformance is possible does not mean it is necessary.

Further, I find the argument that we want to defer the need to learn about Sendable dubious. Swift developers will need to understand at least the basics of its concurrency model and this protocol is a very important brick in the foundation.

7 Likes

Since we've already severely limited Sendable's abilities by making it a "marker" protocol, the biggest action-at-a-distance hazard I see with implicit conformances to it is name lookup. If we also say that extension Sendable extensions are not allowed to declare members, that should contain that effect, so that modifying a type's layout does not cause members to flit in and out of existence everywhere else. At that point, I think most of the consequence Sendable conformance appearing or disappearing ought to be contained to first-order "does not conform" errors in call sites that require sendability.

1 Like

I'm referring to a user attempting to send a value across a concurrency boundary and receiving an error because no implicit Sendable conformance was provided. This could be due to a member of a member of a member, i.e. arbitrarily distant from the actual type the user attempted to send. @Chris_Lattner3 and I discussed this in one of the earlier threads.

That's not really any different that the errors users see with automatic Codable or Hashable conformance, so it doesn't seem to make anything worse. Those errors can identify which property of a property (ad infinitum) breaks the conformance, so it should work for this case as well. Improvements to the diagnostic would help in all cases as well.

1 Like

It is actually quite different because Equatable conformance has to be explicitly added so you get the error in the expected location.

I would think the diagnostic could point wherever we want, starting at the point at which you try to pass it between contexts, then to the type, then to the properties preventing the conformance. It's not really any different than trying to decode a type that has no Decodable conformance.

1 Like

I think it makes a lot of sense that a “marker” protocol can’t make new APIs appear on types, even via extensions. (Users could still, of course, create a protocol refining a marker protocol and have those types adopt that conformance.)

If this is agreed, then it simplifies our reasoning greatly regarding the possible risks/benefits of implicit conformance to Sendable:

What’s the possible harm of implicit conformance where the author hasn’t thought about it? If the rules are sound surrounding which value types can conform via the “checked conformance” route, the only drawback I can see is that future changes to the type could accidentally take away the conformance. In that case, an end user could always add it back with an “unchecked” conformance. This seems strictly better than the alternative of not conforming at all.

2 Likes

Finally chiming in here, trying to earn a bit of my (IMO entirely undeserved, but you all are very kind) authorship credit:

I like this form of the proposal. The high-level mental model is easy enough to grasp heuristically, and the fiddly details — while potentially surprising in context — do all follow naturally from that high-level model.

Speaking to the specific questions in this follow-up review:

  • Sendable is a better name than Concurrent. The latter might imply that multiple concurrency domains can simultaneously use a shared value of the type; the former makes it clear that it is transferring, not sharing, that is safe.

    I do worry a bit that Sendable is an over-broad name. Sending where, across what boundary? Between processes? Over the network?

    A yet-clearer name would be ActorSendable, though that wrongly limits the meanings to actors instead of all forms of concurrency. An even-yet-clearer-but-unacceptably-ugly name would be CrossConcurrencyDomainSendable (yikes). This rabbit hold seems like a bad one; I am willing to accept that, like Codable and @escaping, a potentially more general word takes on a very specific meaning with no further qualification because of how common that one specific meaning is. Given how pervasive the term would be, I’m sure a web search for “swift sendable” would turn up answers for confused new users. Sendable thus strikes me as acceptable.

    I’ll float one possible alternative: AsyncSendable. Here it is in context:

    struct Person : AsyncSendable {
      var name: String
      var age: Int
    }
    
    @propertyWrapper
    struct UnsafeTransfer<Wrapper> : @unchecked AsyncSendable {
      ...
    }
    
    actor MyContactList {
      func filteredElements(_ fn: @asyncSendable (ContactElement) -> Bool) async -> [ContactElement] { … }
    }
    

    Advantages:

    • It’s clear that the name refers to concurrency, not other kinds of sending.
    • It ties to the async keyword with which this proposal interacts.
    • It perhaps improves searchability.
    • It’s short.
    • It still loosely respects Rust’s term of art.

    Marginally better? Perhaps. As I wrote above, Sendable is acceptable; this is not something I have Strong Feelings™ about.

  • Being able to mark individual properties @unchecked instead of the entire type is a very compelling idea, and we should consider it sooner rather than later. Having one nuisancy property spoil the safety checking for an entire type is a recipe for future bugs: a large type gets marked @unchecked Sendable because of one dark corner, then another developer comes along later and adds features to another part of the type, not spotting the @unchecked at the top of the file and expecting Swift’s usual level of static safety.

  • The annotation burden of adding Sendable to many types seems low to me. The rationale regarding mindfulness about public API contracts is entirely compelling. The migration path for Error seems sensible. I approve of the proposal’s approach to “rip the band-aid off” in one go rather than introducing an unsafe actor regime first, then trying to migrate to safety later.

A few of other things that still stand out to me, which do not necessarily need to hold up this proposal, but deserve future consideration:

  • Being unable to say if val is Sendable or val as? Sendable is a generality gap that is going to force a few scattered pieces of framework code into nasty corners. It deserves attention. (Thumbnail sketch: We have some heterogenous collection of assorted values, a cache, maybe, or lookup dynamically constructed from configuration, who knows. We need to share values in it across concurrency domains. We have some boxing mechanism that allows for safe sharing of non-async-sendable types, but it’s costly, and we want to optimize it away with a runtime type check when possible. Oops.)
  • What is @_marker, really? The proposal sketches out the loose meaning clearly enough, but it’s not quite clear why it needs to exist at all, or what it really means in practice. Is the import nothing more than the fact that Sendable is a compile-time-only type, and the compiler doesn’t emit any runtime support for it? Is it thus purely optimization, or is there a reason why a Sendable type can’t exist at runtime? Are there other types that have similar problems, and should @_marker be a public feature?
  • The spelling of @unchecked Sendable reads clearly enough, which is the main concern for this proposal, but it raises questions about annotations on protocol conformances. Is there some generality here we need to consider — in terms of user mental model, in terms of language capability? Or is this entirely a syntactic one-off, and developers should treat @unchecked Sendable as a single non-decomposable thought, much like @private(set)? What precedent does this create, if any? (Are there any per-conformance annotations in the language already?)
6 Likes

Yes, thank you, this was an oversight; it's covered in the announcement, but not in my brief summary of the announcement here. I've added a bullet briefly describing this change.

1 Like

Instead of having implicit conformance, when you pass a non-Sendable value to where one is expected (say to an actor), could Xcode offer a fix-it to add it for you?

1 Like

I'm mostly happy with the implicit conformance for non-public types. For public types, though, the potential harm is that authors may be vending API that they do not realize, which could constraint future changes in unexpected ways.

If I, as a library author, write a struct

public struct S {
  var x: Int
}

I may not intend for this type to be usable across concurrency domains, but the definition written here (if implicit conformances were permitted for public types) would preclude me from ever making this type concurrency-unsafe unless I was willing to have a potential source break. At the very least we'd need a way for library authors to spell "do not make this implicitly Sendable, it may become un-Sendable in the future."

IMO, this is similar to the reason we disallow the synthesis of a public memberwise initializer—it's not that exposing the memberwise initializer is potentially incorrect, it's that it potentially causes library authors to expose public API without even realizing it.

3 Likes

It’s not clear to me how this issue would apply to frozen types. The proposal explicitly excludes either public or frozen types, but if you’ve ruled out ever changing the stored members, then...

This is true, but I actually think we need such a spelling anyway to disable retroactive conformances via @unchecked Sendable regardless of how we decide the implicit conformance issue.

For instance, if we decide that we don’t want raw pointers to be sendable (for reasons I enumerate above and in line with Rust’s rationale for the same decision), then it would be rather useless if any third-party library could vend extension UnsafeMutableRawPointer: @unchecked Sendable { } for all unwitting users of that library. The same line of reasoning for other standard types we’ve already decided should not conform to Sendable.

(The brain-dead but I think sufficient solution here would be a delightful marker protocol named Unsendable. This trichotomy would allow an optimal expression of user intent: Can this type be sent across concurrency domains without manually implemented internal synchronization? (A) Yes, always. (B) No, never. (C) Let the compiler decide.)

3 Likes