Cannot build SwiftPM using build-script

Development Package Manager
1

Hi all,
I feel like this must be a fairly basic mistake!
I've tried to build the compiler and spm using utils/build-script in the swift distribution and it's failing...

utils/build-script -R -p

... cmark, llvm, swift and the standard library build fine for about an hour then ...

--- Cleaning swiftpm ---
+ /Users/carl/Documents/Code/swift/swiftpm/Utilities/bootstrap clean --build-dir /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/swiftpm-macosx-arm64
--- bootstrap: note: Cleaning
--- Building swiftpm ---
+ /Users/carl/Documents/Code/swift/swiftpm/Utilities/bootstrap build --release --swiftc-path /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/toolchain-macosx-arm64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swiftc --clang-path /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/toolchain-macosx-arm64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang --cmake-path /Users/carl/CMake.app/Contents/bin/cmake --ninja-path /Users/carl/homebrew/bin/ninja --build-dir /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/swiftpm-macosx-arm64 --llbuild-build-dir /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/llbuild-macosx-arm64 --reconfigure
--- bootstrap: error: unable to find swiftc at /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/toolchain-macosx-arm64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swiftc
ERROR: command `['/Users/carl/Documents/Code/swift/swiftpm/Utilities/bootstrap', 'build', '--release', '--swiftc-path', '/Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/toolchain-macosx-arm64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swiftc', '--clang-path', '/Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/toolchain-macosx-arm64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang', '--cmake-path', '/Users/carl/CMake.app/Contents/bin/cmake', '--ninja-path', '/Users/carl/homebrew/bin/ninja', '--build-dir', '/Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/swiftpm-macosx-arm64', '--llbuild-build-dir', '/Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/llbuild-macosx-arm64', '--reconfigure']` terminated with a non-zero exit status 1, aborting

Looking in /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert I see all the directories I would usually see for the compiler builds I've done over the years...

ls -l /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert
total 0
drwxr-xr-x  19 carl  staff   608 30 Jul 09:28 cmark-macosx-arm64
drwxr-xr-x   4 carl  staff   128 26 Jul 14:23 earlyswiftdriver-macosx-arm64
drwxr-xr-x  29 carl  staff   928 26 Jul 14:34 llvm-macosx-arm64
drwxr-xr-x  34 carl  staff  1088 26 Jul 15:00 swift-macosx-arm64

I've never seen a toolchain-macosx-arm64 directory there. Is this a regression in building SPM as part of the standard swift compiler build?

2

SwiftPM cannot be built with just the compiler alone, as it also depends on other corelibs like libdispatch and llbuild being built first. Further, to make things easier, all those components are assembled into a toolchain in the directory you asked for the final toolchain to be installed before building SwiftPM, which is something like build/Ninja-ReleaseAssert/toolchain-macosx-arm64/ alongside the other builds if you didn't specify an installation directory.

To build SwiftPM with build-script, you're better off using one of the build presets. Alternately, adding these flags should make it work: --libdispatch --xctest -b --install-llvm --install-swift --install-libdispatch --install-xctest --install-llbuild.

3

Ah OK. Makes sense. Is there anything that documents or lists the build presets and which ones to use? The utils/build-presets.ini file is gigantic and hard to comprehend.

Even just knowing what people "usually" use would be help

4

I tried the above settings but it still didn't seem to work...

utils/build-script --clean -R -p --libdispatch --xctest -b --install-llvm --install-swift --install-libdispatch --install-xctest --install-llbuild
[utils/build-script] NOTE: Using toolchain default
+ /usr/libexec/PlistBuddy -c 'Print :SupportedTargets:macosx:Archs' /Users/carl/Xcode-15.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.4.sdk/SDKSettings.plist
Starting clean build now.
+ rm -rf /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert
+ mkdir -p /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert
--- Building earlyswiftdriver ---
+ /Users/carl/Documents/Code/swift/swift-driver/Utilities/build-script-helper.py build --package-path /Users/carl/Documents/Code/swift/swift-driver --build-path /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/earlyswiftdriver-macosx-arm64 --configuration release --toolchain /Users/carl/Xcode-15.3.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr --ninja-bin /Users/carl/homebrew/bin/ninja --cmake-bin /Users/carl/CMake.app/Contents/bin/cmake --local_compiler_build
Building Swift Driver dependency: llbuild
CMake Warning (dev) at CMakeLists.txt:91 (find_package):
  Policy CMP0148 is not set: The FindPythonInterp and FindPythonLibs modules
  are removed.  Run "cmake --help-policy CMP0148" for policy details.  Use
  the cmake_policy command to set the policy and suppress this warning.

This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_C_COMPILER


Building Swift Driver dependency: TSC
Building Swift Driver dependency: Argument Parser
Building Swift Driver dependency: Yams
Building Swift Driver for target: arm64-apple-macosx10.15
CMake Error in cmake/modules/CMakeLists.txt:
  export called with target "SwiftDriver" which requires target "CSwiftScan"
  that is not in any export set.


CMake Generate step failed.  Build files cannot be regenerated correctly.
Traceback (most recent call last):
  File "/Users/carl/Documents/Code/swift/swift-driver/Utilities/build-script-helper.py", line 690, in <module>
    main()
  File "/Users/carl/Documents/Code/swift/swift-driver/Utilities/build-script-helper.py", line 687, in main
    handle_invocation(args)
  File "/Users/carl/Documents/Code/swift/swift-driver/Utilities/build-script-helper.py", line 188, in handle_invocation
    build_using_cmake(args, toolchain_bin, args.build_path, targets)
  File "/Users/carl/Documents/Code/swift/swift-driver/Utilities/build-script-helper.py", line 452, in build_using_cmake
    build_swift_driver_using_cmake(args, target, swiftc_exec, driver_dir,
  File "/Users/carl/Documents/Code/swift/swift-driver/Utilities/build-script-helper.py", line 535, in build_swift_driver_using_cmake
    cmake_build(args, swiftc_exec, driver_cmake_flags, driver_swift_flags,
  File "/Users/carl/Documents/Code/swift/swift-driver/Utilities/build-script-helper.py", line 553, in cmake_build
    subprocess.check_output(cmd, cwd=build_dir)
  File "/Users/carl/Xcode-15.3.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/Users/carl/Xcode-15.3.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/carl/CMake.app/Contents/bin/cmake', '-G', 'Ninja', '-DCMAKE_MAKE_PROGRAM=/Users/carl/homebrew/bin/ninja', '-DCMAKE_BUILD_TYPE:=Release', '-DCMAKE_Swift_FLAGS=-module-cache-path "/Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/earlyswiftdriver-macosx-arm64/module-cache" -target arm64-apple-macosx10.15 -sdk /Users/carl/Xcode-15.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.4.sdk', '-DCMAKE_Swift_COMPILER:=/Users/carl/Xcode-15.3.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swiftc', '-DCMAKE_OSX_DEPLOYMENT_TARGET=10.15', '-DCMAKE_OSX_ARCHITECTURES=arm64', '-DLLBuild_DIR=/Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/earlyswiftdriver-macosx-arm64/release/dependencies/llbuild/cmake/modules', '-DTSC_DIR=/Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/earlyswiftdriver-macosx-arm64/release/dependencies/swift-tools-support-core/cmake/modules', '-DYams_DIR=/Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/earlyswiftdriver-macosx-arm64/release/dependencies/yams/cmake/modules', '-DArgumentParser_DIR=/Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/earlyswiftdriver-macosx-arm64/release/dependencies/swift-argument-parser/cmake/modules', '/Users/carl/Documents/Code/swift/swift-driver']' returned non-zero exit status 1.
ERROR: command `['/Users/carl/Documents/Code/swift/swift-driver/Utilities/build-script-helper.py', 'build', '--package-path', '/Users/carl/Documents/Code/swift/swift-driver', '--build-path', '/Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/earlyswiftdriver-macosx-arm64', '--configuration', 'release', '--toolchain', '/Users/carl/Xcode-15.3.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr', '--ninja-bin', '/Users/carl/homebrew/bin/ninja', '--cmake-bin', '/Users/carl/CMake.app/Contents/bin/cmake', '--local_compiler_build']` terminated with a non-zero exit status 1, aborting
5

People should "usually" use the same preset as used on a CI job corresponding to their host platform. The preset invocation should be present close to the top of CI job's build log, below the update-checkout invocation.

6

Cool.

I tried...

utils/build-script --preset=buildbot_osx_package        
[utils/build-script] ERROR: no value found for install_destdir in "%(install_destdir)s"

So that's not getting me a lot further. Has no one done what I'm trying to do? Im a bit confused why it isn't easier!?

7

build-script is very much a power tool: if you stray from the presets, you really have to know what you're doing to hold it right. :wink:

If you take a look at the log, it says the earlyswiftdriver build is failing, which you said worked fine before. None of the flags I listed affect that, so either you changed something else or it's a spurious failure.

The build-script help text has some info: run ./swift/utils/build-script --help and read the doc on Preset modes.

8

Thanks. The --help from build-script was the first place I checked!

It talks about build script presets but doesn't really list them. I'm happy not to stray from the presets. In the above build (which was as recommended by Max?) I just tried to use what I thought the OSX presets would be to build SPM. Honestly I just want to do the simplest build with latest source code, I haven't tweaked or done anything unusual on this branch. It should be completely vanilla. I just checked out swift, did update-checkout as the instructions say then tried running the build script.

How do most developers build SPM? Out of frustration, I tried doing swift build -c release in the swiftpm folder, which works, but the SPM built there is using swift from my installed Xcode, which is something like swift 5.10 or something, so then it won't work with the swift I just compiled from the toolchain, so it's useless for me (I'm trying to build a swift 6 toolchain and SPM just using the latest code).

Feeling a bit stuck in a blind alley still.

9

What exactly makes you think so? SwiftPM can be built with any compatible toolchain, which as of this moment is Swift 5.9 or later. And yes, I would say most developers should just use swift build for their local use (maybe -c release is preferred) or development (then the default debug mode is better).

10

Perhaps my comment above is too terse.

Out of frustration, I build swift from source in a source code directory somewhere using the build script in its simplest form, something like utils/build-script -R and get a built compiler, swift-frontend, swift-driver, llvm, libraries, a standard library, etc. and it works as a compiler. Because this is using the latest trunk, it's a swift 6.0 compiler.

But note this is not including swiftpm because I haven't managed to get utils/build-script to build it yet.

Now if I build swiftpm by running swift build inside the swiftpm directory, it builds swiftpm in a subdirectory there using the command line tools on my Mac from Xcode (/usr/bin/swift etc.) which are from Xcode 15.3.

Note: these are not the swift compiler I built above. It does not have swiftpm so it cannot build swiftpm using swiftpm. A classic bootstrapping problem.

The built swiftpm is in .build/arm64../release or something like that. It contains the compiled tools you'd expect to see swift-package, swift-build, swift-test, etc. plus a lot of build intermediates and the important module files like PackageDependency.swiftmodule and so on.

I can copy those into the same folder as my compiled swift-frontend and swift-driver binaries from the main compiler build and then invoke them as you'd expect. Something like <path to my swift and swiftpm binaries folder>/swift package update inside a test project will attempt to start the package manager.

This then fails because PackageDependency.swiftmodule and other modules have been built by a different version of swift, so swiftpm cannot use them, because it's trying to build the manifest with a different version of swift (6.0) from the version of swift (5.10) that built the swiftpm binaries and the modules it needs.

So it seems like it needs to be built a bit more coherently. That's why building swiftpm using swiftpm isn't working at this point.

11

It is built pretty much the same way that build-script builds it. Most probably you haven't copied required PackageDescription and other libraries into appropriate directories together with SwiftPM executables, which is what build-script presets also do for you. The version of the toolchain you compile it with is irrelevant.

If you want to build an end-to-end installable toolchain package, I recommend replicating invocations from CI jobs. Follow the build logs for toolchain jobs invoked by @swift-ci build toolchain. Here's a macOS invocation for some old PR I have in shell history (there's no guarantee this exact invocation will work for you if preset parameters changed since then)

./utils/build-script --preset=buildbot_osx_package,no_test \
  --cmake-c-launcher "$(which sccache)" \
  --cmake-cxx-launcher "$(which sccache)" \
  "install_destdir=$(pwd)/../swift-nightly-install" \
  "installable_package=$(pwd)/../swift-PR-70629-993-osx.tar.gz" \
  "install_toolchain_dir=${HOME}/Library/Developer/Toolchains/swift-PR-70629-993.xctoolchain" \
  "install_symroot=$(pwd)/../swift-nightly-symroot" \
  "symbols_package=$(pwd)/../swift-PR-70629-993-osx-symbols.tar.gz" \
  "darwin_toolchain_bundle_identifier=org.swift.pr.70629.993" \
  "darwin_toolchain_display_name=Swift PR - 70629 (993)" \
  "darwin_toolchain_display_name_short=Swift PR - 70629 (993)" \
  "darwin_toolchain_xctoolchain_name=swift-PR-70629-993" \
  "darwin_toolchain_version=70629.993.0" \
  "darwin_toolchain_alias=PR70629"

It'll take ages to complete though, since it builds everything twice for each architecture to produce universal binaries for macOS. For that reason sometimes it's faster to iterate on toolchain packages on Linux.

As pointed out before, it's order of magnitudes faster to just build SwiftPM with swift build and use that directly if you're interested in SwiftPM only.

12

Worth a try next time I’m back at my desk.

I copied o er the swift modules but they are built with swift 5.10 so they can’t be linked by. Swiftpm when building the manifest.

I’m assuming the build script presets must do something like build swiftpm and modules like PackageDescription using cmake, ninja and the just built swift compiler. Much like the build script builds the standard libraries.

That’s probably how it avoids this chicken and egg problem.

The built swiftpm and modules work fine so long as I use them with the swift 5.10 from Xcode.

13

That's only true for Windows as a host platform. On other platforms SwiftPM is built with SwiftPM.

14

No, he's right to an extent. As part of the build-script build, a bootstrap version of SwiftPM is built using CMake, then that bootstrap SwiftPM is used to build the final SwiftPM and PackageDescription modules.

@carlos42421, can you try rerunning the long command, that you said failed yesterday when building earlyswiftdriver, a couple more times? See if that CSwiftScan error is reproducible or spurious.

15

That's an implementation detail and irrelevant to the issue at hand. With Swift toolchains builds requiring an installation of Swift on the host, we can assume that SwiftPM is available too. Thus swift-bootstrap product is no longer necessary and may be removed in the future.

Whether someone is building SwiftPM with swift build or build-script with presets, they shouldn't notice any change as a result. For all intents in purposes it's best to assume that SwiftPM is always built with SwiftPM (except Windows), which is true even if swift-bootstrap is still involved and built with CMake at some point.

16

The difference is what he pointed out: swift build will build the trunk SwiftPM and PackageDescription modules with the Swift 5.10 compiler by default, then he can't use those 5.10-versioned PackageDescription modules with the trunk 6.1 compiler he just built. build-script takes care of all this for you.

17

Yeah, that's exactly what I was experiencing @Finagolfin.

I tried just updating all repositories to the latest, doing a clean and trying again. It seemed to get further.

This is the command I used:
utils/build-script --clean -R -p --libdispatch --xctest -b --install-llvm --install-swift --install-libdispatch --install-xctest --install-llbuild

This built llvm, then "installed" it so it's definitely getting further...

--- Installing llvm ---
+ env DESTDIR=/Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/toolchain-macosx-arm64 /Users/carl/CMake.app/Contents/bin/cmake --build /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/llvm-macosx-arm64 -- install-llvm-ar install-llvm-cov install-llvm-profdata install-IndexStore install-clang install-clang-resource-headers install-compiler-rt install-clangd install-LTO install-lld install-dsymutil
[0/14][  0%][1.411s] cd /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/llvm-macosx-arm64/tools/llvm-ar && /Users/carl/CMake.app/Cont...in/cmake -DCMAKE_INSTALL_COMPONENT="llvm-ar" -P /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/llvm-macosx-arm64/cmake_install.cmak
-- Install configuration: "Release"
-- Installing: /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/toolchain-macosx-arm64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/llvm-ar
[1/14][  7%][4.651s] cd /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/llvm-macosx-arm64/tools/lto && /Users/carl/CMake.app/Contents/bin/cmake -DCMAKE_INSTALL_COMPONENT="LTO" -P /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/llvm-macosx-arm64/cmake_install.cmake
-- Install configuration: "Release"
-- Installing: /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/toolchain-macosx-arm64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib
-- Installing: /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/toolchain-macosx-arm64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/llvm-c/lto.h
[2/14][ 14%][4.831s] cd /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/llvm-macosx-arm64/tools/llvm-profdata && /Users/carl/CMake.ap...ke -DCMAKE_INSTALL_COMPONENT="llvm-profdata" -P /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/llvm-macosx-arm64/cmake_install.cmak
-- Install configuration: "Release"
-- Installing: /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/toolchain-macosx-arm64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/llvm-profdata
[3/14][ 21%][4.947s] cd /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/llvm-macosx-arm64/tools/clang/lib/Headers && /Users/carl/CMak...E_INSTALL_COMPONENT="clang-resource-headers" -P /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/llvm-macosx-arm64/cmake_install.cmak
-- Install configuration: "Release"
-- Installing: /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/toolchain-macosx-arm64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/builtins.h
-- Installing: /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/toolchain-macosx-arm64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/float.h
-- Installing: /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/toolchain-macosx-arm64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/inttypes.h
-- Installing: /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/toolchain-macosx-arm64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/iso646.h
-- Installing: /Users/carl/Documents/Code/swift/build/Ninja-ReleaseAssert/toolchain-macosx-arm64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/limits.h

And I can see files in there now...

ls toolchain-macosx-arm64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/
bin	include	lib	local

So that's more likely to work.

The build script then seemed to build the compiler and standard library etc.

Then it broke trying to build lib dispatch, which it said (I suppose fairly reasonably) isn't supposed to be built for Darwin....

[2588/2588][100%][967.813s] Running utility command for swift-benchmark-macosx-arm64
Error: build-script does not support building libdispatch on macOS?!
Usage: build-script-impl [--help|-h] [ --SETTING=VALUE | --SETTING VALUE | --SETTING ]*

I'll try again but without lib dispatch and see if that gets further...

EDIT: using utils/build-script -R -p -b --install-llvm --install-swift --install-llbuild I managed to get a clean build of swift, llvm, llbuild and spm. I have yet to try it in my toolchain but it looks good. Thank you everyone!

Carl