Testing installation of toolchain

Development Source Tooling
1

Is any robust means available to test a system installation of the Swift toolchain, for correctness, on a system operating Linux?

To be clear, I am not seeking simply to test the correctness of the files generated from building Swift, or to test a library or application developed with Swift.

I am also not seeking to test a build of Swift currently located within its build tree.

Based on an assumption that the installed files already have been tested as such, I need a means to ensure that all of the components and their dependencies have been installed to the system such that proper operation may be reasonably expected when invoked. Such a test process is useful for testing automated system installation processes.

An example of a process that satisfies the requirement would be running a test suite that attempts to invoke Swift tools already installed on the system, and that does not itself contain or seek to obtain Swift tools. Such a test suite should attempt to invoke and to stress as many as possible of the various components and functionalities in the toolchain.

Simply invoking executable tools to check for non-error return status, or building and running a basic "Hello world" application, would not be adequately robust means of testing.

2

It depends what you mean by "already installed on the system": if you mean that there is some standard system directory and procedure that the root user is expected to install Swift to for all users, there is no standard system installation guideline like that.

If you simply mean downloading the full toolchain from swift.org or elsewhere and unpacking it locally, then running tests to make sure it is reasonably complete and not missing anything important, that is what the integration tests are for, which are run on every toolchain build.

Further, there is the source compatibility suite, which I think is run on every release toolchain.

3

I am coming to understand that there is no standard guideline. I am hopeful to achieve a nonstandard process that can be confirmed as valid through proper testing.

The compatibility test appears to target active projects written in Swift, with the purpose of detecting incompatibilities across releases of the language tools.

The integration tests apparently expect to find components not included in the basic toolchain distribution, such as a utility called FileCheck.

As such, neither suggestion, unfortunately, appears to be of much use for the objective.

4

Why would the integration tests have to only use tools from the Swift toolchain to test that it is working? FileCheck is available in most linux distros, not hard to install.

5

Additional dependencies are not necessarily problematic, but the integration-test project seems to expect that all the LLVM-related binaries share a common location, and a further consideration is potential incompatibility between binaries generated from the LLVM main project versus the version forked for Swift.

6

No, there is no such expectation, if you're talking about the few that it uses to run the integration tests themselves. It merely reuses the freshly built LLVM binaries from the toolchain build, but doesn't have to.

The 3-4 LLVM scripts and executables that the integration tests look for to check the toolchain likely have no difference between stock LLVM and the fresh Swift toolchain build:

[2026-03-18T08:23:55.380Z] + python3 /home/build-user/llvm-project/llvm/utils/lit/lit.py . -v --time-tests --param package-path=/home/build-user/build/buildbot_linux/none-swift_package_sandbox_linux-x86_64 --param test-exec-root=/home/build-user/build/buildbot_linux/none-swift_package_sandbox_linux-x86_64/tests --param llvm-bin-dir=/home/build-user/build/buildbot_linux/llvm-linux-x86_64/bin --timeout=600
[2026-03-18T08:23:55.380Z] lit.py: /home/build-user/swift-integration-tests/lit.cfg:139: note: testing package: '/home/build-user/build/buildbot_linux/none-swift_package_sandbox_linux-x86_64'
[2026-03-18T08:23:55.380Z] lit.py: /home/build-user/swift-integration-tests/lit.cfg:152: note: testing using 'FileCheck': '/home/build-user/build/buildbot_linux/llvm-linux-x86_64/bin/FileCheck'
[2026-03-18T08:23:55.380Z] lit.py: /home/build-user/swift-integration-tests/lit.cfg:153: note: testing using 'readelf': '/home/build-user/build/buildbot_linux/llvm-linux-x86_64/bin/llvm-readelf'
[2026-03-18T08:23:55.380Z] lit.py: /home/build-user/swift-integration-tests/lit.cfg:154: note: testing using 'readobj': '/home/build-user/build/buildbot_linux/llvm-linux-x86_64/bin/llvm-readobj'
7

The test depends on two parameters, package-path and llvm-bin-dir. It is not explained how LLVM binaries would be taken from elsewhere, such as the system path.

My test results are as follows:

$ lit -sv --param package-path=/usr/local/lib/swift --param llvm-bin-dir=/usr/local/lib/swift/bin .
lit: /home/user/swift-integration-tests/lit.cfg:120: note: 'pexpect' module unavailable, skipping related tests
lit: /home/user/swift-integration-tests/lit.cfg:139: note: testing package: '/usr/local/lib/swift'
lit: /home/user/swift-integration-tests/lit.cfg:152: note: testing using 'FileCheck': '/usr/local/lib/swift/bin/FileCheck'
lit: /home/user/swift-integration-tests/lit.cfg:153: note: testing using 'readelf': '/usr/local/lib/swift/bin/llvm-readelf'
lit: /home/user/swift-integration-tests/lit.cfg:154: note: testing using 'readobj': '/usr/local/lib/swift/bin/llvm-readobj'
lit: /home/user/swift-integration-tests/lit.cfg:180: note: Failed to find swift benchmarks, skipping related tests.
lit: /home/user/swift-integration-tests/lit.cfg:187: note: testing using 'swift': '/usr/local/lib/swift/usr/bin/swift'
lit: /home/user/swift-integration-tests/lit.cfg:192: note: testing using 'swiftc': '/usr/local/lib/swift/usr/bin/swiftc'
lit: /home/user/swift-integration-tests/lit.cfg:201: note: testing using 'repl_swift': /usr/local/lib/swift/usr/bin/repl_swift
lit: /home/user/swift-integration-tests/lit.cfg:210: note: testing using 'docc': /usr/local/lib/swift/usr/bin/docc
lit: /home/user/swift-integration-tests/lit.cfg:215: note: testing using 'wasmkit': /usr/local/lib/swift/usr/bin/wasmkit
lit: /home/user/swift-integration-tests/lit.cfg:219: fatal: swift does not exist!

How would the problem resolve, if, for example, FileCheck were simply somewhere in the system path?

8

Simple, read the basic doc, which I helped write.

In case it isn't clear, the four llvm-bin-dir binaries can come from anywhere and should work, while the package-path is the Swift toolchain that is checked for the Swift compiler and Swift-forked LLVM binaries, by some simple integration tests.

9

I have done my best to follow the directions, but in all of my trials, the results are essentially the same, as the one I showed previously.

My most recent trial appears as follows:

$ python3 /usr/lib/llvm-20/build/utils/lit/lit.py -sv --param package-path=/usr/local/lib/swift --param llvm-bin-dir=./usr/lib/llvm-20/bin/ .
lit.py: /home/user/swift-integration-tests/lit.cfg:120: note: 'pexpect' module unavailable, skipping related tests
lit.py: /home/user/swift-integration-tests/lit.cfg:139: note: testing package: '/usr/local/lib/swift'
lit.py: /home/user/swift-integration-tests/lit.cfg:152: note: testing using 'FileCheck': './usr/lib/llvm-20/bin/FileCheck'
lit.py: /home/user/swift-integration-tests/lit.cfg:153: note: testing using 'readelf': './usr/lib/llvm-20/bin/llvm-readelf'
lit.py: /home/user/swift-integration-tests/lit.cfg:154: note: testing using 'readobj': './usr/lib/llvm-20/bin/llvm-readobj'
lit.py: /home/user/swift-integration-tests/lit.cfg:180: note: Failed to find swift benchmarks, skipping related tests.
lit.py: /home/user/swift-integration-tests/lit.cfg:187: note: testing using 'swift': '/usr/local/lib/swift/usr/bin/swift'
lit.py: /home/user/swift-integration-tests/lit.cfg:192: note: testing using 'swiftc': '/usr/local/lib/swift/usr/bin/swiftc'
lit.py: /home/user/swift-integration-tests/lit.cfg:201: note: testing using 'repl_swift': /usr/local/lib/swift/usr/bin/repl_swift
lit.py: /home/user/swift-integration-tests/lit.cfg:210: note: testing using 'docc': /usr/local/lib/swift/usr/bin/docc
lit.py: /home/user/swift-integration-tests/lit.cfg:215: note: testing using 'wasmkit': /usr/local/lib/swift/usr/bin/wasmkit
lit.py: /home/user/swift-integration-tests/lit.cfg:219: fatal: swift does not exist!

For this case, I installed LLVM tools for my distribution. My distribution is based on Ubuntu 24.04, and the package I selected is llvm-20-tools.

Several matters seem relevant.

First, LLVM tools are packaged with a major semantic version in the package name, in contrast to the more typical case of a package being named for the project in general, with the package manager handling upgrades based on metadata that distinguishes the different versions of the same package.

Second, the files provided by the package are placed in separate paths depending on the version.

Combined, the two observations suggest that backward compatibility is not provided among the various versions, and that therefore, the one selected must be exactly equal to a particular compatible version, for a particular use, or must occur within an acceptable range of particular versions.

I have no guidance, however, for which version is appropriate, and the matter is further complicated by older versions of the tools not being packaged for newer versions of the distribution. For example, based on my best understanding, the recommendation you offered, llvm-12-tools is available for Ubuntu 20.04 (jammy), but not 22.04 (noble), the prior obsolete by being four years old, against the two-year cycle used in Ubuntu for LTS.

Regardless, the output from the trial leaves me with little sense of the cause of the failure. I appreciate assistance understanding the various issues, and finding a path to proceed.

10

@Finagolfin:

I have read the document you wrote, and tried to apply it, to the my best of understanding, through a variety of trials, described in my previous posts.

Unfortunately, I am finding neither the documentation or the error messages adequately clear, for me to interpret meaningfully, based on my current level of experience with the tools.

Would you please try to address whether my usage is correct, and help me find a resolution to the reported errors?

Thank you.

11

Hey, got busy with the recent release and other stuff, but I believe that error is telling you what the problem is. Specifically, there is no Swift compiler at note: testing using 'swift': '/usr/local/lib/swift/usr/bin/swift'.

That lit command expects the three LLVM tools to be at the exact llvm-bin-dir provided, while the Swift package-path is the root of your toolchain, ie the Swift compiler should be at package-path/usr/bin/. The version of the LLVM tools won't matter.

On Fedora 43, I simply installed the llvm package by invoking sudo dnf install llvm, then ran ./litTest -sv --param package-path=/path/to/swift/toolchain --param llvm-bin-dir=/usr/bin . in the integration test directory to successfully test a Swift toolchain.

1 Like
13

I have launched the test suite with the command

python3 /usr/lib/llvm-20/build/utils/lit/lit.py -sv --no-progress-bar --param package-path=/home/user/swift/ --param llvm-bin-dir=/usr/lib/llvm-20/bin .

Ten tests are reported as unsupported, and one as failed, the test for WASM.

******************** TEST 'swift-package-tests :: wasm/swiftpm.md' FAILED ********************
Exit Code: 2

Command Output (stdout):
--
# RUN: at line 12
rm -rf /tmp/swift-integration-tests/wasm/Output/swiftpm.md.tmp.dir
# executed command: rm -rf /tmp/swift-integration-tests/wasm/Output/swiftpm.md.tmp.dir
# RUN: at line 13
mkdir -p /tmp/swift-integration-tests/wasm/Output/swiftpm.md.tmp.dir/swift-sdks
# executed command: mkdir -p /tmp/swift-integration-tests/wasm/Output/swiftpm.md.tmp.dir/swift-sdks
# RUN: at line 19
/home/user/swift/usr/bin/swift sdk list --swift-sdks-path /tmp/swift-integration-tests/wasm/Output/swiftpm.md.tmp.dir/swift-sdks | /usr/lib/llvm-20/bin/FileCheck --check-prefix CHECK-SDK-LIST-BEFORE /home/user/swift-integration-tests/wasm/swiftpm.md
# executed command: /home/user/swift/usr/bin/swift sdk list --swift-sdks-path /tmp/swift-integration-tests/wasm/Output/swiftpm.md.tmp.dir/swift-sdks
# executed command: /usr/lib/llvm-20/bin/FileCheck --check-prefix CHECK-SDK-LIST-BEFORE /home/user/swift-integration-tests/wasm/swiftpm.md
# RUN: at line 21
find "%{swift-sdk-generator_srcdir}/Bundles" -mindepth 1 -maxdepth 1 | xargs /home/user/swift/usr/bin/swift sdk install --swift-sdks-path /tmp/swift-integration-tests/wasm/Output/swiftpm.md.tmp.dir/swift-sdks | /usr/lib/llvm-20/bin/FileCheck --check-prefix CHECK-SDK-INSTALL /home/user/swift-integration-tests/wasm/swiftpm.md
# executed command: find '%{swift-sdk-generator_srcdir}/Bundles' -mindepth 1 -maxdepth 1
# .---command stderr------------
# | find: '%{swift-sdk-generator_srcdir}/Bundles': No such file or directory
# `-----------------------------
# error: command failed with exit status: 1
# executed command: xargs /home/user/swift/usr/bin/swift sdk install --swift-sdks-path /tmp/swift-integration-tests/wasm/Output/swiftpm.md.tmp.dir/swift-sdks
# .---command stderr------------
# | Error: Missing expected argument '<bundle-path-or-url>'
# | Help:  <bundle-path-or-url>  A local filesystem path or a URL of a Swift SDK bundle to install.
# | Usage: swift sdk install <bundle-path-or-url> [--checksum <checksum>] [--color-diagnostics] [--no-color-diagnostics]
# |   See 'sdk install -help' for more information.
# `-----------------------------
# error: command failed with exit status: 123
# executed command: /usr/lib/llvm-20/bin/FileCheck --check-prefix CHECK-SDK-INSTALL /home/user/swift-integration-tests/wasm/swiftpm.md
# .---command stderr------------
# | FileCheck error: '<stdin>' is empty.
# | FileCheck command line:  /usr/lib/llvm-20/bin/FileCheck --check-prefix CHECK-SDK-INSTALL /home/user/swift-integration-tests/wasm/swiftpm.md
# `-----------------------------
# error: command failed with exit status: 2

After some reading, I find that WASM may be supported only through an alternative development package. Should I be concerned about the one particular failure, or the ten tests reported as unsupported?

14

Any time you're unsure about some tests, you can check the CI for a passing run and see what it shows. Every toolchain build runs these integration tests at the end, so picking a recent run of the Fedora toolchain CI shows the following:

Total Discovered Tests: 41
    Unsupported: 10 (24.39%)
    Passed     : 31 (75.61%)

So it looks like that's to be expected, and I saw the one wasm failure myself, which is probably because the CI expects the swift-sdk-generator source to be checked out for its testing, but that won't be there for an arbitrary toolchain.

15

Would you please clarify "checked out for its testing".

I want to avoid installing new components to the system, instead simply testing the basic components after being installed. Can the test suite simply be directed to the generator source at an arbitrary location?

16

That test says it is only meant to be run on CI, where the Wasm SDKs are built separately. If you want to run it too, try reading it and modifying its source for your use.

17

Thank you for the explanation.

I feel it is unfortunate that the design of the test suite is such that any testing on an actual target system must result in failure.

For an informed individual performing manual testing, such a design may not be particularly troublesome.

For testing an installation on a target system, however, which is different from a CI test of the correctness of the software itself, an automated process will read the exit status indicating failure, without any distinction between a failure caused by the test being intended for a different environment, versus caused by the installation functioning incorrectly.