Not all of it's relevant to your question, but you might be interested in Survey: How do you build Swift? where a lot of contributors described their build setups.
Personally, I don't build in Xcode often, but I still use it for the bulk of my editing. I more or less use the setup from the first post in the thread I linked: A debug Xcode build which provides a project I can use for editing and the occasional debug session, and a release-with-asserts ninja build which I use for incremental work, running lit tests, etc. This allows me to use Xcode's autocomplete, jump to definition, etc. while still building with ninja which I've found is generally a bit faster and more reliable. I've also used VSCode a little and have found that it works pretty well too.