You shouldn’t have to do anything at all. After installing Xcode via the App Store and opening it once, $ swift build should just work.
The other fancy stuff mentioned on Swift.org - Download Swift is only if you are wanting to use an older version of Swift or try it out in its development state or something.
How does
swift buildknow to look in/Library/Developer/Toolchainsin the first place?
Because in looks relative to itself by default. The source is here. The bin directory discovery is here, so maybe you can help it in the right direction with SWIFTPM_CUSTOM_BINDIR?
But I repeat that none of this should be necessary in the first place. It should just work. Hopefully someone more knowledgeable than myself can figure out why it didn’t in your case.