To be perfectly honest, I don't think there's any design which is going to make this easy. Programmers working in concurrent systems need to learn to think transactionally, which is already a stretch, because there are inevitably going to be ways to compose transactions together non-transactionally. Swift can
stop you from writing await actor.foo = actor.foo + 1, and maybe it can advise you that await actor.setFoo(actor.foo + 1) still looks really questionable, but it can't actually force you to add a proper incrementFoo() method or whatever makes sense transactionally for your situation, and that's always going to be the biggest problem here.
Composing actor operations by preventing interleaving during async actor functions seems like it's just trading one problem for another, because now every async actor operation is a potentially unbounded serialization point, and if scalability is important, that can easily be just as wrong as the potential non-atomicity, and you have to chase all of those points down and rework the code. I've seen so many towers of awful workarounds based on recursive locks and careful unlock/relock regimens. Eventually you're back in the situation that Swift puts you in: calls to peer actor operations have to be treated as re-entrant because they might need to unblock actor progress — either they're written that way currently or they will be in the future.
I agree with Adrian's point that the more pressing issue is not having a way to maintain FIFO-ness except by basically switching to channels with AsyncSequence.