1

Hello, Swift community.

The review of SE-0302: ConcurrentValue and @concurrent closures begins now and runs through March 1st, 2021.

This review is part of the large concurrency feature, which we are trying to review in many small portions. While we've tried to make it independent of other concurrency proposals that have not yet been reviewed, it may have some dependencies that we've failed to eliminate. Please do your best to review it on its own merits, while still understanding its relationship to the larger feature; I know that's a lot.

Reviews are an important part of the Swift evolution process. All review feedback should be either on this forum thread or, if you would like to keep your feedback private, directly to the review manager. If you do email me directly, please put "SE-0302" somewhere in the subject line.

What goes into a review?

The goal of the review process is to improve the proposal under review through constructive criticism and, eventually, determine the direction of Swift. When writing your review, here are some questions you might want to answer in your review:

  • What is your evaluation of the proposal?
  • Is the problem being addressed significant enough to warrant a change to Swift?
  • Does this proposal fit well with the feel and direction of Swift?
  • If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
  • How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

More information about the Swift evolution process is available at:

https://github.com/apple/swift-evolution/blob/master/process.md

As always, thank you for contributing to Swift.

John McCall
Review Manager

17 Likes
2

Definitely +1 for this!
I'm happy to see safe concurrent value sharing incorporated in 'concurrency 1.0' instead of adding it later as was previously proposed.

1 Like
3

This proposal has come a long way and ended up in a great place! Thanks to the authors for all the hard work in iterative refinement. :pray:

A marker protocol cannot be used in a generic constraint for a conditional protocol conformance to a non-marker protocol.

I must have missed this before. It isn’t immediately obvious why this would require runtime support. I don’t have a concrete example of when this would be needed, but it does seem like it could turn out to be an unfortunate limitation for marker protocols someday. Is this a hard limit for implementation reasons or is it a limit that could maybe be lifted eventually?

4

By my reading this would be a hard limit because if we have:

protocol P {}
struct S<T> {}
extension S: P where T: ConcurrentValue {}

then querying is/as? P for various parameterizations of S<T> would dynamically depend on whether T: ConcurrentValue, which is unknowable after compile time.

4 Likes
5

Oh, that makes perfect sense. Thanks! It would be great to have this explanation added to the proposal for future readers.

6

after reading this proposal, definitely +1 for me

a context-less closure defaults to be non- @concurrent

// defaults to @escaping but not @concurrent
let fn = { (x: Int, y: Int) -> Int in x+y }

why not infer fn is a @concurrent function?

1 Like
7

I’m a little confused with default escaping here. What about SE-103

8

Should static mutable variables participate in concurrency? Should we enforce types with static mutable variables to be ConcurrentValueUnsafe? Seems to me like static stored variables don’t play well with the notion of concurrency.

9

Note that global and static let constants are concurrency-safe if they conform to ConcurrentValue, but variables aren't since you still need to ensure exclusive access (with a mutex, queue, actor...).

There's a "global actors" proposal floating around where you can annotate a global variable with a global actor to force serialization of its access through it. I assume property wrappers could be used to enforce serialization of access through other methods. Looks like this proposal has nothing to say about global and static variables however. I assume this is left for other proposals, but I think it's worth clarifying.

10

Hmm, this is a very interesting proposal.

My opinion is that it's pretty ugly, but it's necessary ugliness in order to safely support concurrency. What I mean by that is the business of adding this marker to almost every standard library type, as well as most types in the libraries you publish, and then likely having a file in your projects where you patch up a bunch of types from libraries whose authors didn't go around marking almost every type they expose (and hoping that they stay accurate, even across semver minor updates which add private data members).

But I understand why it's needed - we've never really defined value semantics as a language-level concept, so no code today is annotated to support safe concurrency, and lots of code will continue to be written without paying mind to concurrency so these holes will likely persist. Therefore we need a way to add those annotations/conformances, but retroactive conformances are bad, hence we limit them to compile-time and call them "markers". Still, it's a bit yucky.

  1. What about resilient types? By definition, they are updated independently of your application, and are free to evolve and add new data members. Any assumptions you make about a type on iOS 14 could become invalid and unsafe when your users update to iOS 15. IMO, we should ban any attempt to claim that a resilient type is a ConcurrentValue (even via UnsafeConcurrentValue). Since our only resilient types come from Apple's SDKs, this won't be a problem so long as Apple correctly annotates everything they won't evolve to be concurrency-unsafe.

  2. Except for the cases listed below, all struct, enum, and class types in the standard library conform to the ConcurrentValue protocol.
    ...

    • Lazy algorithm adapter types: the types returned by lazy algorithms (e.g., as the result of array.lazy.map { … }) never conform to ConcurrentValue . Many of these algorithms (like the lazy map ) take non- @concurrent closure values, and therefore cannot safely conform to ConcurrentValue .

    What about AnyCollection and friends (also AnyHashable)? Surely they also can't conform to ConcurrentValue - otherwise I could just erase any of the lazy algorithm types and pass them around freely.

    In fact, existentials in general are quite problematic, since they erase compile-time knowledge of a value's type and allow recovering that knowledge at runtime - but ConcurrentValue is a compile-time only thing. You couldn't, say, write something which hides concurrent processing as an implementation detail like this:

    protocol MyValueProtocol { /* ... */ }

func process(_ value: MyValueProtocol) {
  if let conc_value = value as? MyValueProtocol & ConcurrentValue {
    _processConcurrently(conc_value)
  } else {
    _processSerially(value)
  }
}

    The only way to use ConcurrentValue with existentials is for that constraint to bubble up, even if it has nothing to do with the semantics of the process function or the MyValueProtocol protocol.

    This is what the standard library has chosen to do for Error, and to be honest it is a bit disappointing that we're going to be restricting the kinds of types which can be thrown solely to support concurrency, even for code which has no interest in concurrency. Is there any way to narrow this? For example by only requiring that async functions throw errors which are concurrency-safe?

    • The change to keypath literals subscripts will break exotic keypaths that are indexed with non-standard types.

    I disagree with the idea that using non-standard-library types in a subscript is somehow "exotic". It may be that most values captured in keypaths can be marked as concurrency-safe, but again, it's still disappointing that we're potentially disallowing code that doesn't care about concurrency.

    While the captured values are erased from the keypath's type, I wonder if there is some way we could make a kind of ConcurrentKeyPath with a failable initializer that inspects a keypath's actual captured values to determine whether it is concurrency-safe. I suppose that would require ConcurrentValue to have some kind of runtime representation, though.

  4. Nested functions are also an important consideration, because they can also capture values just like a closure expression. We propose requiring the @concurrent attribute on nested function declarations:

    I think this should say "allowing" rather than "requiring", since non-concurrent nested functions will still be allowed (won't they?). Hopefully this sets a precedent which would allow us to add @noescape nested functions one day; sometimes the compiler fails to determine that a nested function does not escape, which can result in massive performance penalties.

2 Likes
12

Since it’s more of an implementation detail, perhaps it should not be separate protocols? An attribute might be more appropriate. For example:

struct MyNSPerson : @unchecked ConcurrentValue {
  private var name: NSMutableString
  var age: Int
}
4 Likes
13

We could do that, but you'd have to decide what to do about other closures, e.g. we couldn't infer this as concurrent:

var x = 42
let fn = { (a:Int) in x += a }

and you don't want the behavior of this to be ambiguous:

var x = 42
let fn = { (a:Int) -> Int in x + a } // capturing x by reference or copy?
x += 1
print(fn(0)) // 42 or 43?

It is simpler to just have a consistent rule that doesn't depend on the body of the closure.

SE-0103 describes the rule that applies to a variable of closure type, not closure expressions.

You're right, we definitely need to define a model for global variables as follow-on to this proposal. This is something that hasn't been nailed down yet, there are multiple possible different paths. The direction I'm in favor of is described in this writeup, but that is far from the only path.

Resilience is an important topic here - you're right that this needs to be clear. The proposal is consistent with how swift do this: adopting the [Unsafe]ConcurrentValue protocol marker in a public/open type is a promise that all future versions of that type will be concurrency safe. This is the same as adopting equatable or hashable publicly: you are promising that all future versions of the type conform to the requirements of equatable and hashable. ConcurrentValue works the same way.

You're right, this shouldn't be marked either. @Douglas_Gregor What do you think? Karl's points about existentials are also interesting.

Sure, but it is easy to adopt the ConcurrentValue protocol into those types, so the should be fine in this system, there is just a slight source compatibility roadbump in Swift 6 mode where they'll have to add the protocol conformance. If you are actually using an unsafe-for-concurrency value as an index, but you know it isn't being used in a concurrent way, you can use UnsafeConcurrentValue or an adaptor type of some sort.

Yes, you're right, I clarified the wording.

Yeah, this is a valid point. I'd love to hear alternate naming suggestions that would solve this concern.

This is an interesting idea. The main tradeoff I see here in that we're adding language syntax complexity that is otherwise unnecessary (we don't allow attributes there right now), but it is nice in that it solves the "MyNSPerson is a CV for clients" problem.

-Chris

3 Likes
16

As a general rule, I don’t think it’s reasonable for Swift to set a baseline that non-concurrent code will be completely unaffected by the concurrency feature. We do need to preserve source compatibility (at least in current language modes), and of course we want to avoid pushing unnecessary complexity onto programs. But concurrency is an almost ubiquitous reality these days, and when different language goals come into conflict, it is not abstractly unreasonable to resolve that conflict in favor of what benefits the concurrent environments that nearly all programs are written in.

Error is a good example of this. It would be possible to only restrict errors to be ConcurrentValues when they’re thrown between concurrency domains, but the consequences would be dire: it would create an enormous source of static complexity by forcing a large number of throwing functions to be explicit about whether they throw a concurrent or non-concurrent error, and there would be a widespread error of omission where functions only declare the latter, and thus force callers to filter out the theoretical possibility of a non-concurrent error. All this to allow errors to embed references to arbitrary mutable reference types, which usually isn’t a particularly good idea in the first place.

9 Likes
17

It looks to me like @_marker could itself be a marker protocol. Meta. I think this is nicer:

protocol ConcurrentValue: MarkerProtocol {}

Where MarkerProtocol is magically compiler-defined (kinda like protocol P: AnyObject)

18

We don’t want marker-ness to always be inherited (e.g. protocols should be able to imply ConcurrentValue without themselves becoming markers), so it’s at least a little unlike the normal protocol implication model.

2 Likes
19

That does raise the interesting question of whether it should be possible for a protocol to imply UnsafeConcurrentValue, though. This also seems related to Manolo’s question.

20

I was under the impression that, in terms of implementation work, parsing for this was already implemented (not sure whether in the main branch or in the S4TF only) for the previously discussed @memberwise conformance synthesis. But yes, it would add to the user-facing language.

21

I think this proposal has evolved very well through several iterations. It definitely addresses what was a missing component of the overall "concurrency version 1.0" landscape and is a significant part of the puzzle. Overall, it fits well with the feel and direction fo Swift. I have put in effort to follow along through several iterations and have done a careful reading of the current version.

I would like to comment on two related issues. I hate that they are matters of bikeshedding, but on reflection, I think it is perhaps germane both to the discussion about UnsafeConcurrentValue and to the review prompt about other languages with this feature:

First, while I appreciate that the Value in ConcurrentValue is a nod to the value semantics discussion we've had before, there's something a bit incongruent about an internally synchronized reference type being referred to as a Value, even if an Unsafe one, as we've really all along been using "value" to distinguish from reference semantics.

But drop the word Value from ConcurrentValue, and you'd get a curious issue where data types are being declared Concurrent even though they don't "run." Folks have been using the term "concurrency-safe" to describe what ConcurrentValue really stands for, and if "concurrent-able" were a word or at least rolled off the tongue well enough to be a word, then our problem would be solved.

Second, I think the term "@concurrent function" is a little unfortunate: it may well be clear to users who are steeped in Swift concurrency design, but I think to other users it could be confusable.

When we speak colloquially of "concurrent functions," I think of "concurrent forEach" or "concurrent map" and not their predicates. Even as the predicates are evaluated concurrently in the context of a concurrent forEach or map, it's really the parameter in the declaration of a concurrent forEach that's "concurrent" and not the type of that parameter, which is merely "concurrent-able" (cf. our discussion about property wrappers used on parameters).

If we can help it, it would be nice to have a clearer naming to observe the distinction so that we don't have "@concurrent functions" and "concurrent functions." Sadly, again, "concurrent-able" isn't a word, and it wouldn't make for one that rolls off the tongue.

So, can there be a solution?

In the development history of this proposal, the protocol in question has taken on several names (or been considered to be related to other protocols with those names), including ActorSendable and ValueSemantics. In reading the final proposal itself, I also see that the ConcurrentValue, UnsafeConcurrentValue, and @concurrent are explained as follows:

[A] key question is: "when and how do we allow data to be transferred across concurrency domains?"

The ConcurrentValue protocol models types that are allowed to be safely passed across concurrency domains by copying the value. This includes value-semantic types, references to immutable reference types, internally synchronized reference types, @concurrent closures,...

Any class may be declared to conform to UnsafeConcurrentValue.... This is appropriate for classes that use access control and internal synchronization to provide memory safety

A @concurrent function type is safe to transfer across concurrency domains (and thus, it implicitly conforms to the ConcurrentValue protocol).

(Emphases added.)

Therefore, I would want to see if we could consider the following names:

  • Values that are safe to transfer across concurrency domains should conform to Transferrable. (I wouldn't bother with trying to incorporate "across concurrency domains" into the name, just as we have a similarly laconic Codable.)
  • Classes that are safe to transfer across concurrency domains because they use access control and internal synchronization should conform to Synchronized (or, perhaps, UncheckedSynchronization).
  • Functions that are safe to transfer across concurrency domains should be @transferrable functions.

This ends up looking remarkably like Rust's names Send and Sync, and I take that as a good sign that we're describing similar concepts with similar words.

An interesting point pops up in comparing Rust with the proposed design for Swift:

In Rust, raw pointers are neither Send nor Sync.
Rationale, as I understand it: Because they provide no safety guarantees, they cannot guarantee that they are safe to transfer across concurrency domains.

In Swift, it is proposed here that Unsafe(Mutable)(Buffer)Pointer unconditionally conform to ConcurrentValue, not merely UnsafeConcurrentValue.
Rationale: Because they are completely unsafe, it is irrelevant whether it is safe or unsafe to transfer them across concurrency domains; we do not want any limits on a user in terms of transferring a pointer across concurrency domains unsafely.

I am not sure which choice is superior, but it would be interesting to hear someone who knows Rust well as to how that community feels about the design decision in that language.

12 Likes
22

Yes, @Karl is right, these shouldn't conform to ConcurrentValue. The implementation is correct (none of the Any* types is marked as ConcurrentValue), but I forgot to mention these in the writeup. We'll add them.

As for actual existential types, you can state ConcurrentValue as part of an existential type (e.g., Codable & ConcurrentValue), but you cannot have ConcurrentValue on the right-hand side of is or as?.

Doug

4 Likes
23

Would it make sense to add the restriction that a protocol cannot refine a marker protocol?

Otherwise the protocol would ‘inherit’ properties of marker protocols like the fact that the conformance can only be added in the same source file as the definition of the conforming type.

UPDATE: I just noticed that the conformance in the file where the type is defined only applies to ConcurrentValue and not to marker protocols in general. Is this restriction expressed as arguments to the @_marker attribute somehow, or is it hard coded as an attribute of ConcurrentValue?