Working through a related problem recently, I found that the way to overcome the testing-related problems with startup of an AsyncSequence was to ensure that the code within startObserving was called in the init method of the owner, rather than after initialization, so in your case as part of the init method of your Observer actor.
While the rest of the long thread would be a red herring, this specific post lays out the code to do that, and the test results: Reliably testing code that adopts Swift Concurrency? - #73 by babbage (Later: I see now you actually participated in the early part of that thread, but the linked post was a late follow-up.)
If you still need to be able to turn on and off whether or not the Observer actually acts on observations, you could still have a bool like shouldObserve, toggle that on and off again, and then guard shouldObserve else { return } as the first line of for try await element in sequence... However, the order of toggling that vs. receiving actual events it is observing would likely be in a non-deterministic order given it is an Actor. So more likely it would be better to instantiate an Actor whenever you want to be observing, start observing in its init method, and then discard the actor instance when you no longer want to be observing.
Would be interested to hear if this slightly different approach to your underlying problem is a solution for you or not. 