Testing installation of toolchain

Development Source Tooling
1

Is any robust means available to test a system installation of the Swift toolchain, for correctness, on a system operating Linux?

To be clear, I am not seeking simply to test the correctness of the files generated from building Swift, or to test a library or application developed with Swift.

I am also not seeking to test a build of Swift currently located within its build tree.

Based on an assumption that the installed files already have been tested as such, I need a means to ensure that all of the components and their dependencies have been installed to the system such that proper operation may be reasonably expected when invoked. Such a test process is useful for testing automated system installation processes.

An example of a process that satisfies the requirement would be running a test suite that attempts to invoke Swift tools already installed on the system, and that does not itself contain or seek to obtain Swift tools. Such a test suite should attempt to invoke and to stress as many as possible of the various components and functionalities in the toolchain.

Simply invoking executable tools to check for non-error return status, or building and running a basic "Hello world" application, would not be adequately robust means of testing.

2

It depends what you mean by "already installed on the system": if you mean that there is some standard system directory and procedure that the root user is expected to install Swift to for all users, there is no standard system installation guideline like that.

If you simply mean downloading the full toolchain from swift.org or elsewhere and unpacking it locally, then running tests to make sure it is reasonably complete and not missing anything important, that is what the integration tests are for, which are run on every toolchain build.

Further, there is the source compatibility suite, which I think is run on every release toolchain.

3

I am coming to understand that there is no standard guideline. I am hopeful to achieve a nonstandard process that can be confirmed as valid through proper testing.

The compatibility test appears to target active projects written in Swift, with the purpose of detecting incompatibilities across releases of the language tools.

The integration tests apparently expect to find components not included in the basic toolchain distribution, such as a utility called FileCheck.

As such, neither suggestion, unfortunately, appears to be of much use for the objective.

4

Why would the integration tests have to only use tools from the Swift toolchain to test that it is working? FileCheck is available in most linux distros, not hard to install.

5

Additional dependencies are not necessarily problematic, but the integration-test project seems to expect that all the LLVM-related binaries share a common location, and a further consideration is potential incompatibility between binaries generated from the LLVM main project versus the version forked for Swift.

6

No, there is no such expectation, if you're talking about the few that it uses to run the integration tests themselves. It merely reuses the freshly built LLVM binaries from the toolchain build, but doesn't have to.

The 3-4 LLVM scripts and executables that the integration tests look for to check the toolchain likely have no difference between stock LLVM and the fresh Swift toolchain build:

[2026-03-18T08:23:55.380Z] + python3 /home/build-user/llvm-project/llvm/utils/lit/lit.py . -v --time-tests --param package-path=/home/build-user/build/buildbot_linux/none-swift_package_sandbox_linux-x86_64 --param test-exec-root=/home/build-user/build/buildbot_linux/none-swift_package_sandbox_linux-x86_64/tests --param llvm-bin-dir=/home/build-user/build/buildbot_linux/llvm-linux-x86_64/bin --timeout=600
[2026-03-18T08:23:55.380Z] lit.py: /home/build-user/swift-integration-tests/lit.cfg:139: note: testing package: '/home/build-user/build/buildbot_linux/none-swift_package_sandbox_linux-x86_64'
[2026-03-18T08:23:55.380Z] lit.py: /home/build-user/swift-integration-tests/lit.cfg:152: note: testing using 'FileCheck': '/home/build-user/build/buildbot_linux/llvm-linux-x86_64/bin/FileCheck'
[2026-03-18T08:23:55.380Z] lit.py: /home/build-user/swift-integration-tests/lit.cfg:153: note: testing using 'readelf': '/home/build-user/build/buildbot_linux/llvm-linux-x86_64/bin/llvm-readelf'
[2026-03-18T08:23:55.380Z] lit.py: /home/build-user/swift-integration-tests/lit.cfg:154: note: testing using 'readobj': '/home/build-user/build/buildbot_linux/llvm-linux-x86_64/bin/llvm-readobj'
7

The test depends on two parameters, package-path and llvm-bin-dir. It is not explained how LLVM binaries would be taken from elsewhere, such as the system path.

My test results are as follows:

$ lit -sv --param package-path=/usr/local/lib/swift --param llvm-bin-dir=/usr/local/lib/swift/bin .
lit: /home/user/swift-integration-tests/lit.cfg:120: note: 'pexpect' module unavailable, skipping related tests
lit: /home/user/swift-integration-tests/lit.cfg:139: note: testing package: '/usr/local/lib/swift'
lit: /home/user/swift-integration-tests/lit.cfg:152: note: testing using 'FileCheck': '/usr/local/lib/swift/bin/FileCheck'
lit: /home/user/swift-integration-tests/lit.cfg:153: note: testing using 'readelf': '/usr/local/lib/swift/bin/llvm-readelf'
lit: /home/user/swift-integration-tests/lit.cfg:154: note: testing using 'readobj': '/usr/local/lib/swift/bin/llvm-readobj'
lit: /home/user/swift-integration-tests/lit.cfg:180: note: Failed to find swift benchmarks, skipping related tests.
lit: /home/user/swift-integration-tests/lit.cfg:187: note: testing using 'swift': '/usr/local/lib/swift/usr/bin/swift'
lit: /home/user/swift-integration-tests/lit.cfg:192: note: testing using 'swiftc': '/usr/local/lib/swift/usr/bin/swiftc'
lit: /home/user/swift-integration-tests/lit.cfg:201: note: testing using 'repl_swift': /usr/local/lib/swift/usr/bin/repl_swift
lit: /home/user/swift-integration-tests/lit.cfg:210: note: testing using 'docc': /usr/local/lib/swift/usr/bin/docc
lit: /home/user/swift-integration-tests/lit.cfg:215: note: testing using 'wasmkit': /usr/local/lib/swift/usr/bin/wasmkit
lit: /home/user/swift-integration-tests/lit.cfg:219: fatal: swift does not exist!

How would the problem resolve, if, for example, FileCheck were simply somewhere in the system path?

8

Simple, read the basic doc, which I helped write.

In case it isn't clear, the four llvm-bin-dir binaries can come from anywhere and should work, while the package-path is the Swift toolchain that is checked for the Swift compiler and Swift-forked LLVM binaries, by some simple integration tests.

9

I have done my best to follow the directions, but in all of my trials, the results are essentially the same, as the one I showed previously.

My most recent trial appears as follows:

$ python3 /usr/lib/llvm-20/build/utils/lit/lit.py -sv --param package-path=/usr/local/lib/swift --param llvm-bin-dir=./usr/lib/llvm-20/bin/ .
lit.py: /home/user/swift-integration-tests/lit.cfg:120: note: 'pexpect' module unavailable, skipping related tests
lit.py: /home/user/swift-integration-tests/lit.cfg:139: note: testing package: '/usr/local/lib/swift'
lit.py: /home/user/swift-integration-tests/lit.cfg:152: note: testing using 'FileCheck': './usr/lib/llvm-20/bin/FileCheck'
lit.py: /home/user/swift-integration-tests/lit.cfg:153: note: testing using 'readelf': './usr/lib/llvm-20/bin/llvm-readelf'
lit.py: /home/user/swift-integration-tests/lit.cfg:154: note: testing using 'readobj': './usr/lib/llvm-20/bin/llvm-readobj'
lit.py: /home/user/swift-integration-tests/lit.cfg:180: note: Failed to find swift benchmarks, skipping related tests.
lit.py: /home/user/swift-integration-tests/lit.cfg:187: note: testing using 'swift': '/usr/local/lib/swift/usr/bin/swift'
lit.py: /home/user/swift-integration-tests/lit.cfg:192: note: testing using 'swiftc': '/usr/local/lib/swift/usr/bin/swiftc'
lit.py: /home/user/swift-integration-tests/lit.cfg:201: note: testing using 'repl_swift': /usr/local/lib/swift/usr/bin/repl_swift
lit.py: /home/user/swift-integration-tests/lit.cfg:210: note: testing using 'docc': /usr/local/lib/swift/usr/bin/docc
lit.py: /home/user/swift-integration-tests/lit.cfg:215: note: testing using 'wasmkit': /usr/local/lib/swift/usr/bin/wasmkit
lit.py: /home/user/swift-integration-tests/lit.cfg:219: fatal: swift does not exist!

For this case, I installed LLVM tools for my distribution. My distribution is based on Ubuntu 24.04, and the package I selected is llvm-20-tools.

Several matters seem relevant.

First, LLVM tools are packaged with a major semantic version in the package name, in contrast to the more typical case of a package being named for the project in general, with the package manager handling upgrades based on metadata that distinguishes the different versions of the same package.

Second, the files provided by the package are placed in separate paths depending on the version.

Combined, the two observations suggest that backward compatibility is not provided among the various versions, and that therefore, the one selected must be exactly equal to a particular compatible version, for a particular use, or must occur within an acceptable range of particular versions.

I have no guidance, however, for which version is appropriate, and the matter is further complicated by older versions of the tools not being packaged for newer versions of the distribution. For example, based on my best understanding, the recommendation you offered, llvm-12-tools is available for Ubuntu 20.04 (jammy), but not 22.04 (noble), the prior obsolete by being four years old, against the two-year cycle used in Ubuntu for LTS.

Regardless, the output from the trial leaves me with little sense of the cause of the failure. I appreciate assistance understanding the various issues, and finding a path to proceed.

10

@Finagolfin:

I have read the document you wrote, and tried to apply it, to the my best of understanding, through a variety of trials, described in my previous posts.

Unfortunately, I am finding neither the documentation or the error messages adequately clear, for me to interpret meaningfully, based on my current level of experience with the tools.

Would you please try to address whether my usage is correct, and help me find a resolution to the reported errors?

Thank you.

11

Hey, got busy with the recent release and other stuff, but I believe that error is telling you what the problem is. Specifically, there is no Swift compiler at note: testing using 'swift': '/usr/local/lib/swift/usr/bin/swift'.

That lit command expects the three LLVM tools to be at the exact llvm-bin-dir provided, while the Swift package-path is the root of your toolchain, ie the Swift compiler should be at package-path/usr/bin/. The version of the LLVM tools won't matter.

On Fedora 43, I simply installed the llvm package by invoking sudo dnf install llvm, then ran ./litTest -sv --param package-path=/path/to/swift/toolchain --param llvm-bin-dir=/usr/bin . in the integration test directory to successfully test a Swift toolchain.

1 Like