On macOS 13.5, with Xcode 14.3.1, trying to build a Swift toolchain (to then use within Xcode), based on the the instructions in the readme: GitHub - apple/swift: The Swift Programming Language I hit a problem:
ld: warning: ignoring file /opt/homebrew/lib/libzstd.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
Undefined symbols for architecture x86_64:
"_ZSTD_compress", referenced from:
llvm::compression::zstd::compress(llvm::ArrayRef<unsigned char>, llvm::SmallVectorImpl<unsigned char>&, int) in libLLVMSupport.a(Compression.cpp.o)
"_ZSTD_compressBound", referenced from:
llvm::compression::zstd::compress(llvm::ArrayRef<unsigned char>, llvm::SmallVectorImpl<unsigned char>&, int) in libLLVMSupport.a(Compression.cpp.o)
"_ZSTD_isError", referenced from:
llvm::compression::zstd::compress(llvm::ArrayRef<unsigned char>, llvm::SmallVectorImpl<unsigned char>&, int) in libLLVMSupport.a(Compression.cpp.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[2137/4537][ 47%][493.962s] Building C...les/clang-tblgen.dir/NeonEmitter.cpp.o
ninja: build stopped: subcommand failed.
This is because my libzstd in homebrew is thin (ARM64), but the toolchain build is building a fat toolchain, and needs an x86_64 version of this library. I've checked homebrew, but this formula doesn't have a "universal" variant.
How are other people avoiding this? Can I build a "thin" toolchain? Is there an easy way to get a fat libzstd? Should I just not be building a toolchain at all?
Yeah, the README really positions build-toolchain as an alternative to the other build styles that can then be installed into Xcode, so I thought that was what I needed.
In fact, although following the instructions you linked has worked, I now don't know how to use this compiler within Xcode. All the docs I've found in the Swift repo either say "use build-toolchain to use your Swift within Xcode" or "use your new compiler on the command-line"...
Correct, installing a newly built toolchain for usage in Xcode is not a common use case. .pkg files that are distributed on Swift.org - Download Swift are not built locally, but on CI and are code-signed. When building locally you're much more likely to use it from command-line, run lit tests, and integration tests. After proving that your toolchain works with those test runs, you can submit a PR and build a new toolchain on CI with that PR.
Nevertheless, if you still want to get it recognized by Xcode, the file system layout of a freshly built toolchain with build-script is quite close to what you get after installing one from a .pkg file, so maybe you can get away with manually creating a directory for it in ~/Library/Developer/Toolchains/ and copying there build artifacts you get from your build-script run. I haven't tried this method though, but this is where I'd start looking if there was such a need.
You can also pass --installable-package=$(pwd)/installable-package.tar.gz --install-destdir=$(pwd)/installable-package to build-script, which will create a tarball for you to unpack into your freshly created ~/Library/Developer/Toolchains/<new_toolchain>.xctoolchain directory. Make sure that the new toolchain directory layout matches that are already installed and proven to work.
I was unable to find sufficient similarity between the output structure of the normal build and what's expected of a toolchain, nor to make --installable-package work at all, so I reverted to a worse solution:
I built libzstd myself from source, fat (it supports this natively, so long as you don't want lz4 or xz support, neither of which seem to have native fat support nor universal options in homebrew)
I installed it to /usr/local
I symlinked it into /opt/homebrew/lib
At this point the build-toolchain script did work (consuming 184GB of disk space, which, I, wow), and I was able to install the toolchain in such a way that Xcode recognizes it.
I was able to build our app for "any iOS device (arm64)" using this toolchain, up to the point of linking an executable, at which point it failed: File not found: /Library/Developer/Toolchains/swift-LOCAL-2023-09-04-a.xctoolchain/usr/lib/clang/13.0.0/lib/darwin/libclang_rt.profile_ios.a. I copied this file from Xcode 14.3.1 (which oddly uses a later clang version, which might later be an issue?) but got me past this problem temporarily.
Unfortunately, this "main branch" Swift crashes when compiling some of our code, so I still don't really have a working toolchain... another problem for another day.
Can you elaborate on this? Did this produce an error message? If so, which exact invocation did you use to get to it? This should be filed as a bug so that we can improve the build process.
Sorry, I didn't pay much attention to it :/ I added your --installable-package=$(pwd)/installable-package.tar.gz --install-destdir=$(pwd)/installable-package to the end of my previously-working build-script invocation, and it failed trying to interact with a file inside $(pwd)/installable-package that it hadn't actually created.
Wrapping up this thread: After trying to build a few tags and failing either to build, or to produce a toolchain that could actually compile our code (eg. due to debug assertions), and given the extreme disk space requirements (I have a 500GB SSD, and I ran out of disk space for other things with over 150GB of Swift-related stuff hanging around. Someone on a 256GB disk doesn't stand a chance), I've given up on being able to contribute to Swift
Would you be able to share the exact error messages that you stumbled upon?
Did you build with --debug? This is not a configuration I'd recommend, especially for first-time contributors precisely for the reason you've listed. Most of the time you don't need debug builds, especially enabled in the whole LLVM and Swift toolchain stack all at once. You should only be enabling debugging information in a single component that you certainly know you will be attaching a debugger to, and build-script has plenty of options for fairly granular control of this.
I came across this thread as I had the exact same problem as the OP trying to build a toolchain today. The solution was to download the libzstd.dylib source and build for intel and use lipo to merge it into the /opt/homebrew/lib/libzstd.dylib (the easiest way to build for intel for me was to run make with arch -x86_64.) There were a couple of other minor concurrency errors to fix checking out tag swift-DEVELOPMENT-SNAPSHOT-2025-02-06-a using Xcode 16.2. This was the last tag I found able to generate an LLVM.xcodeproj.
@Max_Desiatov is right though that build-toolchain is about the last thing you want to be using if you can avoid it. It's better to iterate using the ninja builds and simple scripts or tests then roll a toolchain for testing on a larger codebase inside Xcode. If you're only working on the compiler one way I've found to work is to download a pre-built toolchain of the same tag from swift.org instead and copy the swift-frontend binary built by ninja into it as you iterate. I imagine if you knew the right .dylibs and .swiftmodules to update you could almost patch the standard library this way as well though your milage would definitely vary. In the end if you can get it to work and have a spare few hours and 150GB free disk space it's easier to roll a new toolchain.
One thing that could change is that bizarrely the utils/build-toolchain script features very prominently on the very top level README of the Swift repo. It should perhaps be buried a few levels of README/GETTING STARTED.md down. While it's difficult to have it run through to the end as it exercises every part of the swift repo it's the easiest way to test inside Xcode if you don't have CI access so it hasn't outlived it's usefulness yet and please don't delete it! It's just a shorthand wrapper for the utils/build_script anyway so it isn't inherently more difficult to support. Another thing that might be useful would be if it only built for the local architecture.
I agree with the build-toolchain part, but not the rest of what you're saying. build-script is how the toolchain is built and tested in CI. It generates ninja build manifests after all that one can use for local incremental builds. If the contributor's goal is to build an installable toolchain, the easiest way is to replicate the CI environment including its build-script invocation with the used preset.
Specifically on topic, to avoid libzstd errors I recommend either creating a new build-script preset or modifying the existing ones to remove infer-cross-compile-hosts-on-darwin setting. Not only this will cut your build time in half due to skipping cross-compilation from Intel to Apple Silicon or vice versa, it won't require you to have libzstd installed for both platforms as a consequence.
Taking this further I've opened a PR to add a --single-arch option to the build_toolchain script and tested it (it's now possible to turn around a toolchain in about half an hour). Altering the presets didn't seem easy as the option to remove was at the top level and this was a way to localise the risk of the change while giving people a chance of solving the problem in the OP if they look through the script.
