How to build SwiftPM using a locally built Swift toolchain

Development Package Manager
, ,
1

Hi all,
I am trying to debug an issue in one of the Swift toolchain snapshots (swift-DEVELOPMENT-SNAPSHOT-2023-05-09-a specifically).

Hardware and OS

Hardware - Apple M2 Max
OS - Ventura 13.4

Summary about the issue

The issue happens while building a Swift project in release mode using the package manager.

Local setup

1. Building Swift from source

a. First, I set up my local Swift development environment using the instructions in the Swift repo on Github.

b. In addition to the steps mentioned on Github, I checked out the swift-DEVELOPMENT-SNAPSHOT-2023-05-09-a tagged versions of all the repos using the following command. I did this because I want to debug issues in a specific toolchain snapshot.

utils/update-checkout --tag swift-DEVELOPMENT-SNAPSHOT-2023-05-09-a

c. To build the project I used the build command mentioned on Github, verbatim.

utils/build-script --skip-build-benchmarks \
  --skip-ios --skip-watchos --skip-tvos \
  --swift-darwin-supported-archs "$(uname -m)" \
  --sccache --release-debuginfo --swift-disable-dead-stripping

d. This step completed successfully and I had swift, swiftc and other binaries in my build folder at <WORKSPACE_ROOT>/build/Ninja-RelWithDebInfoAssert/swift-macosx-arm64/bin/.

2. Building Swift PM from source

a. After having built Swift from source I attempted to built Swift PM from source using the following command.

utils/build-script --skip-build-benchmarks \
  --swift-darwin-supported-archs "$(uname -m)" --release-debuginfo \
  --swiftpm --install-swiftpm \
  // This is the path to the locally built Swift executables.
  // I'm using the locally built executables because that's what I'm guessing 
  // the Swift PM must have been built with, in the toolchain snapshot.
  --native-swift-tools-path <WORKSPACE_ROOT>/build/Ninja-RelWithDebInfoAssert/swift-macosx-arm64/bin/

b. The build command finally fails when trying to compile the actual Swift PM sources, with the following error.

/Users/kshitij/workspace/swift-project/swiftpm/Sources/PackageCollectionsSigning/CertificatePolicy.swift:103:93: error: cannot convert value of type 'PolicySet' (aka 'AnyPolicy') to expected argument type '() throws -> Policy'
            var verifier = Verifier(rootCertificates: CertificateStore(trustStore), policy: policySet)
                                                                                            ^
/Users/kshitij/workspace/swift-project/swiftpm/Sources/PackageCollectionsSigning/CertificatePolicy.swift:103:28: error: generic parameter 'Policy' could not be inferred
            var verifier = Verifier(rootCertificates: CertificateStore(trustStore), policy: policySet)
                           ^
/Users/kshitij/workspace/swift-project/swiftpm/Sources/PackageCollectionsSigning/CertificatePolicy.swift:103:28: note: explicitly specify the generic arguments to fix this issue
            var verifier = Verifier(rootCertificates: CertificateStore(trustStore), policy: policySet)
                           ^
                                   <<#Policy: VerifierPolicy#>>

Questions

  1. Is this the right way to build and install Swift PM using a locally built Swift toolchain?
  2. Is it possible that the Swift PM in the swift-DEVELOPMENT-SNAPSHOT-2023-05-09-a toolchain snapshot never actually built using the Swift in the same toolchain snapshot? How can I even check the build status of a given toolchain snapshot? Or can I assume that all builds/tests for a given toolchain snapshot succeeded?
  3. Same question as the one asked here. Is it possible to tell your system Swift PM (one that comes with Xcode, let's say) to use the executables and libraries from the Swift you built locally?

I would truly appreciate your help :slight_smile:.

2

The build error looks like an old swift-certificates is being used. Personally, I don't build with build-script locally, so not sure how that happens or how to fix it. I just build using swift build in a SwiftPM checkout and then use shell aliases to invoke the various tools that were built from source.

For 3, there are various approaches. One would be to adjust PATH to find the tools you built before the ones installed in the system. Another one can be building a full toolchain and using xcrun --toolchain to invoke tools from it (or Xcode's toolchain menu). Note that Xcode links its own version of libSwiftPM for its integration with packages and it is currently not possible to replace that with a copy you build yourself.

3

Could you elaborate this a bit?

4

It's really just something like this

alias sb=$SWIFTPM_DIR/.build/debug/swift-build

where SWIFTPM_DIR is the directory where my working copy of SwiftPM is checked out. This way I can invoke the "regular" SwiftPM using swift build and my copy with sb.

5

So I was actually able to build my Swift PM using utils/build-script and now have the various Swift PM binaries - swift-build, swift-package built into the <WORKSPACE_ROOT>/build/swiftpm-macos-arm64/release/ directory.

Now,

  1. When running utils/build-script I also specified the --install-swiftpm option. I was thinking that this would copy the generater PM binaries (and their prerequisite libraries) into the directory I specified using --install-prefix but that is not what it seems to be doing. Would you know how I would get the build-script to copy the PM binaries to the same directory where my Swift binaries live?

  2. How does the Swift PM decide what swiftc it should use? One way is if I specify the SWIFT_EXEC arg. But if I don't what is the default?