Ok, confirmed my hypothesis: the issue is due to Task.sleep rather unintuitively requiring a thread from the pool even when it's run on the main actor. The main thread itself is actually completely responsive, but the timer gets delayed.
I'll make sure there's a bug tracking improving Task.sleep to avoid this (I think there already is one that will cover this for other reasons, but if I'm mistaken about that I'll file one).
Thanks for the test case! What a curious bug.
[edit] oh and the reason the priority matters is because the thread count cap on the pool is one thread per CPU core per QoS bucket, so when it's .medium the main thread is able to get a pool thread anyway.