I thought this sounded familiar...
I think it's been clear for a while that the way we've implemented macros has some advantages, but scaling isn't one of them.
It works, but if you have, say, 20 packages in your dependency graph, all using macros for various cool features, written against whatever version of SwiftSyntax was latest for them at the time, you're going to end up downloading and building a lot of copies of that library. Every design has strengths and weaknesses, and this is ours.
The simplest short-term solution I can think of would be to cache SwiftSyntax build artefacts, so that "clean builds" aren't quite totally clean builds. They would be more similar to the kind of clean build you get when using binary dependencies -- your code would be built from scratch, but some parts (the various copies of SwiftSyntax) would be precompiled.