SE-0433: Synchronous Mutual Exclusion Lock

I like very much load and store.

They do not look like a regular property, and misuses can be caught by the eye:

// Incorrect, and looks incorrect 👍
if mutex.load() > 5 {
    mutex.store(0)
}

They avoid the proliferation of withLock { $0 } and withLock { $0 = ... } quite nicely. Since they are present, they ward off the danger of users adding their own overly seductive property.

And I also like math operations (An increment() method that returns the new value when value is Numeric is handy, for example).

Thanks @Joe_Groff for exploring this area.

2 Likes

I'm not sure I agree. To the trained eye, those look incorrect, but IMO they look extremely sensible to the untrained. Which, hey, maybe the inexperienced shouldn't be fooling around with atomics [oops, meant to say mutexes] too much, but it's trivial to imagine a greener developer coming in and making a small change to good code and doing something exactly like that example.

withLock { … } with perhaps a convenience load only seems much Swiftier and safer to me. That said, I haven't worked with it much, and perhaps the burden from it is much more onerous than I'm envisioning.

4 Likes

My key experience was massaging a big code base towards Swift 6 and strict concurrency checks.

In many occasions (especially in tests), I was mutating a variable (a plain scalar, an Int, a String, or a brave sendable struct) from various isolation contexts. Typically: setup in main thread, write eventually, then check back in main thread - something that region-based isolation can't really deal with. Sometimes the compiler correctly spots an actual race condition that needs fixing. Sometimes the compiler emits a false positive warning, because it can not understand what is going on. In many cases, using a Mutex was the correct decision. It soothes compiler's concerns, and even when the Mutex is too much (because the unsafe code is practically safe), it's sometimes useless to be overly smart - efficiency is not always needed.

To me, Mutex is a type that 1. can solve a lot Swift Concurrency warnings, 2. should be easier to use than the dreadful Atomics, 3. already exists in the toolkit of nearly all developers anyway, with convenience methods that would make most readers of this thread blemish. The stdlib Mutex will be adored and abused when it eventually ships. Its ergonomics should be excellent, and aware of its future uses.

Also, as an author of a Swift wrapper around SQLite, I know pretty well that many people have difficulties thinking in terms of transactions. No matter how hard you try to push them in the direction of correctness, expect disappointment. This is not a reason to make the life of people who can think about transactions more difficult. The solution is patience and documentation, not depriving everybody of the good stuff.

2 Likes

Thank you, that provides some valuable context.

One thought: if the most common place you feel this friction is in tests, that seems like a prime place where it'd be easy to add some private convenience methods. I think most people have more tolerance for that sort of unconventional usage in tests than in non-test code.

To what extent do you think having a convenience load (but not store) would alleviate the pain you're feeling?

The more I think about just having a load method but not a store, the more I like that it fits with the principle of progressive disclosure. It's easy to imagine a scenario in which an inexperienced person is wanting to read a value, they see it's a Mutex<Int> rather than the Int they were expecting, but then they see that it has a load method, they try that, it works, and they go on with their day. Commonly the story ends here; but, if later on, they need to mutate the value and don't find a store, they're forced to look into mutexes deeper. Alongside discovering withLock, they've got a prime opportunity to learn why the API was designed the way it was and give some thought into how they should structure their code for transactional correctness. I had a similar learning experience when first encountering APIs like withUnsafe… and withCheckedContinuation, etc.

2 Likes

Weirdly, this is exactly what I like about the proposed API. The transaction is directly visible in the source as a block. No magic, no guessing. Yes, it's sometimes a little more code to write, but the for the reader it is immediately clear what's going on.

17 Likes

:thinking: I normally think just like you. In GRDB all database access are wrapped in a closure and a transaction – you have to use very verbose apis to opt out the safe default. Why am I so triggered by mutex.withLock { $0 } and mutex.withLock { $0 = ... }?

1 Like

As a data point, I made the same choice for MMIO. Users must explicitly call read(), write(), or modify() to interact with mapped registers. I really wanted to make it easy to operate on individual fields but exposing bit fields API is just too error prone, e.g.:

register.field0 = 0b101 // Implicit read - modify - write
register.field1 = 0b1 // Implicit read - modify - write

Instead users need to write:

register.modify { // read before closure
  $0.field0 = 0b101 // modify
  $0.field1 = 0b1 // modify
} // write at closure return

Which avoids some really gnarly issues you can hit when programming registers in C with volatile pointers and C-style bitfields.

How about if the “withLock” parts were gone?

mutex { $0 += 1 }

That can be made to work by adding a callAsFunction method to Mutex that takes a trailing closure.

The entire point of a mutual-exclusion lock is to perform operations while holding the exclusive lock, so it seems fitting to me that this natural operation should have a straightforward spelling at call sites.

The name of the variable (“mutex”) indicates which lock is being used, and the block in curly braces specifies what to do while holding it. I don’t think “withLock” adds anything here, so reducing the boilerplate makes sense.

8 Likes

Obtuse is fundamentally unsafe

Always try to make what users want to write be the best way to write something.

The challenge with making "safe" APIs that are also harder to use is that while a little push can work, if they're too much harder to use it's counter-productive: instead of hinting that the problem domain is more complex than beginners assumed, and provoking them into thinking about it more deeply, it can end up just pushing them to StackOverflow or ChatGPT where they'll just copy-paste something they don't understand either, which will probably be wrong in even more subtle and nefarious ways.

This is essentially a leaky abstraction problem.

If a solution can be found that's safe but also intuitive and sufficiently simple, that's much more likely to succeed.

With that in mind, I'm thinking Swift (the language and/or compiler) needs to be smarter, not the programmer. e.g. what if:

foo.value += 1

…were automatically translated to:

foo.withLock {
    $0 += 1
}

By which I mean more like @autoclosure parameters, not merely a _modify property accessor. It should permit arbitrarily many references to and uses of the value (subject to the normal rules that preclude ambiguous 'simultaneous' mutation), e.g.:

foo.value = abs(foo.value) + calculateDelta(from: foo.value)

I posit that folks who find this too magical may be biased by convention; that new programmers will learn this just fine and not find it at all "magical" (in the bad sense) that this works as they intended. If you've worked with beginner programmers you'll know that they are in fact already surprised that it could ever not work the way they intended.

There's plenty of leeway there for the compiler to restrict what can be done inside that [figurative] implicit closure, if necessary to prevent abuse or likely mistakes. e.g. it could prevent use of any other locks (in order to avoid acquisition ordering problems - or better yet, simply ensure that lock acquisition order is globally consistent).

Or it could prevent blocking or suspension points, requiring more explicit syntax for their use in order to better convey the author's intention through the code. e.g.:

foo.value = await next(after: foo.value)
// ❌ Suspension points are not allowed while directly modifying
//    a `Lock`'s value, as it's not clear that the author meant to
//    suspend while holding the lock.
//
//      foo.value = await next(after: foo.value)
//                  ^^^^^
//
// FixIt 1:  Perform the suspending operations separately
//           (if correct to do so)
//
//   let next = await next(after: foo.value)
//   foo.value = next
//
// FixIt 2:  Use an explicit critical section
//
//   await foo.withLock {
//      $0 = await next(after: $0)
//   }

I think it's reasonably intuitive - and at the very least reasonable to expect beginners to learn - that individual statements are generally "atomic" but multiple statements are not, unless you explicitly make them so.

A [well-written] context-specific compiler error (or warning) is going to be much more successful than just missing APIs (or API documentation, or long-forgotten Swift Forums threads, or buried in an otherwise well-meaning Evolution Proposal, etc).

1 Like

For clarity, most of the time I avoid using $0 (or $1, $2 etc..). Also always assigning the Mutex to let mutex = ... seems to me not readable. I have no problem with the closure and withLock that clearly encompasses the transaction. Something like:

let fifo = Mutex<[Int]>([])

func enqueue(item: Int) {
    fifo.withLock { array in
        array.append(item)
    }
}

Naming the closure parameter array or storage instead of $0 and naming the Mutex fifo instead of mutex seems to me readable.

1 Like

Of course the lock instance and the closure parameter can be given whatever names the programmer wants. I was not suggesting otherwise. You can substitute any metasyntactic variable names you like.

I was simply taking an existing example from the thread, and showing how it would look if withLock did not have to be there.

It would be even less intuitive to new users if your example were safe:

foo.value = abs(foo.value) + calculateDelta(from: foo.value)

but this wasn’t:

foo.value = abs(foo.value)
foo.value += calculateDelta(from: foo.value)

There is fundamental complexity when dealing with mutex-protected values, and I strongly believe that trying to hide that from the programmer instead of encouraging them via syntax to understand what’s going on will only lead to more problems.

11 Likes

This has been said about practically every aspect of programming languages. And programming in general. It is the nature of abstraction. We must be careful, as experienced programmers, to not become stubborn and set in our ways.

The pertinent point is that it is natural and assumed, by beginners, that simple statements work as they appear on face value. Obviously x += 1 should increment x by one - how on earth could it possibly not? And then they get long lectures about a bunch of stuff that really they shouldn't care about, explaining why a bunch of dubious design decisions have accumulated and conspired to make such a trivial operation not work correctly.

There's good reason that Python is a very popular language and is considered particularly beginner-friendly. It's better than most about doing what you mean.

And indeed one day people may be flabbergasted that a simple sequence of statements is not "atomic" as it obviously should be on face value. But baby steps.

3 Likes

And it has with statement (would be nice to have in Swift some day) in it to handle cases similar to this with mutex.

While it makes a lot of sense of language being "smarter" than a programmer, (a) hiding via implicit transformations (b) advanced and complex behaviour that has to be understanded properly is more harmful in long term. Unless there is an obviously better approach (which probably exist, yet not discovered), it is better to stick with more verbose code IMO.

Clearly, there would be users who doesn't bother themselves with detail and abuse SO/GPT solution without consideration (I have a slight believe that ChatGPT might warn, but that is only hope for the best given my experience). They would be able to do so even without such tool in stdlib, so the problem will remain here, but just a bit better hidden. But the issue better to arise sooner than later. Verbosity of API would make some of the people not using mutex at all, some to go and look up why there is such odd way to interact with that, and some will not bother and make an extension. That's the good intentions that here might be harmful, especially when you are confident about using some of the tools, since (a) you are forgetting that you can make a mistake, (b) there are a lot of developers who hasn't got a lot of experience with such primitives and not that confident on using them.

I think there is some merit in providing conveniences like the aforementioned load() if only to discourage developers writing a property accessor:

extension Mutex {
  var value: Value {
    get { withLock { $0 } }
    set { withLock { $0 = newValue } }
  }
}

FWIW because the type is annotated with @ _staticExclusiveOnly it's impossible to declare as a var:

‼️ Variable of type 'Mutex<Int>' must be declared with a 'let'
var foo = Mutex<Int>(5)

Therefore impossible to mutate via a property setter:

let foo = Mutex<Int>(5)

‼️ Left side of mutating operator isn't mutable: 'foo' is a 'let' constant
foo.value += 5

And this being a runtime error as the lock is not recursive:

foo.withLock { $0 = lock.value }
3 Likes

Unfortunately, I think it is actually possible. The toolchain I have locally only has Atomic, not Mutex, but I get the same error and can "fix" it by making the setter nonmutating.

3 Likes

I replaced my value property with load() and store(_:), and immediately found misuses of the convenience value property. I'm now fully convinced that value is much too much error-prone.

I was particularly guilty of:

// Won't compile as mutex.load().mutatingMethod() because
// the returned value is immutable 👍
mutex.value.mutatingMethod()
mutex.value.subcomponent = ...

Those indeed need withLock (and I'm completely happy with it).

In full honesty, load() allows racy mutations (class instances and nonmutating setters), and is thus not 100% foolproof:

// Compiles, and mutates outside of the critical section
mutex.load().frobnicate()

Thanks to all cautious members of the forum. I now rely on the wisdom of the LSG :-)

21 Likes

This proposal has been accepted. Thank you all for your thoughtful feedback.

6 Likes