Unfortunately not, no. At least not without writing a new subcommand to call instead of swift test or forking Swift and changing the way swift test works
Make sure Homebrew hasn't screwed up the Swift installation by running sudo xcode-select -s /Applications/Xcode.app (or wherever you have Xcode installed). Homebrew has a habit of overriding the command line tools with whatever it needs which can cause issues like this
I have to say that in about 10 years of using MacPorts first and for about 6-7 of those years of using Homebrew (created a portable config / setup that goes on every of the team’s machines including the CI build agents which run macOS, now bare metal IaaS on Flow.Swiss) I have never seen it mess with the command line tools path which does not mean it cannot happen, maybe we avoided it. I can add it to the list of things to be on the lookout for, but not something I can complain about Homebrew. It definitely helped us automate our setup more.
Why do you hate homebrew (I know hate is a strong word :))? I have much more of an issue with either the lack of access to some command line tools or having an older version than needed because of licensing constraints on macOS… for a lot of things this helps with an environment that is closer to upstream and easier to get help with as a result.