I see this viewpoint onto concurrency a lot, and it's the root cause of the frustrations, not the actual limitations.
These warnings, errors and limitations are not here to annoy you, and you don't put them into place to "make the compiler happy".
They're there to make you stop and think about your problem facing with concurrency, and resolve it in an appropriate way. They're there to "make the future you happy, who doesn't have to debug irreproducible races and weird states".
Whenever people bring up "just a counter" or "just a variable" examples, I quite honestly say don't use an actor for those (and that's coming from someone entire career with actors pretty much). It's fine to use a lock or atomics, they're there for a reason. Not everything has to be an actor, a lot of things are, but not everything - in Swift at least (and not in some hypothetical actors-only language). I see people getting frustrated because they assume everything must be actors now, but that's not true. After all, we introduced Mutex and low-level atomics, just now, and there may be more things to follow.
You do have to think about concurrency (you always should have but you might have gotten away with it), and sloppy programming won't fly anymore, but that's it, other synchronization mechanisms are viable as well -- especially for "just contains a bunch of values" things. It would not make sense at all to make a sophisticated concurrent datastructure an actor, there's a reason they're highly specialized for high performance using advanced locking or lock free programming techniques.
Yes, we should be pushing the envelope of the ergonomics story and making it easier, but the setters are a touchy subject as they would almost definitely result in breaking assumptions about atomicity and result in more mysterious hard to debug bugs rather than in those situations having to rethink things.
I really don't want to be dismissive here, but out of all the ergonomic improvements I'm personally just not sure we get the most gain, and least pain from the specific async setters topic. More specific real world examples would definitely help so we can think them through together. @Andropov's writeup was very interesting for example.