Source compat suite anyone?

The search function yields lots of results for “source compat suite” – but those are all CI Notifications ;-)
Is there any detailed information on how to use the scripts?

[Like choosing which platforms to inculde or exclude (or just not fail for the majority of projects ;-)]

Good question (although possibly the wrong topic; maybe something under Development since it’s not relevant to most developers). @Rostepher, who’s the best person to answer this?

1 Like

I guess this is a hard nut to crack… to add some details:
The CI logs contain the right call to trigger the tests — but it needs some sandbox configuration files, which are not available (and there is not documentation for .sb files either :slightly_frowning_face:)

@Tino have you looked at https://github.com/apple/swift-source-compat-suite#building-projects? This might be able to give you some answers about how to build the source compatibility suite.

To build all projects against a specified Swift compiler locally, use the runner.py utility as shown below.

./runner.py --swift-branch master --projects projects.json --include-actions 'action.startswith("Build")' --swiftc path/to/swiftc
Use the --include-repos flag to build a specific project.

./runner.py --swift-branch master --projects projects.json --include-actions 'action.startswith("Build")' --include-repos 'path == "Alamofire"' --swiftc path/to/swiftc
By default, build output is redirected to per-action .log files in the current working directory. To change this behavior to output build results to standard out, use the --verbose flag.

I did - but that results in failures for most of the builds (afaics, non-macos targets never work).
The “right” call is

./runner.py --swiftc /Users/buildnode/jenkins/workspace-private/swift-master-source-compat-suite/build/compat_macos/install/toolchain/usr/bin/swiftc --projects projects.json --include-actions 'action.startswith("Build")' --swift-branch master --sandbox-profile-xcodebuild ../../../workspace-private/swift-source-compat-suite-sandbox/sandbox_xcodebuild.sb --sandbox-profile-package ../../../workspace-private/swift-source-compat-suite-sandbox/sandbox_package.sb

but sandbox_xcodebuild.sb doesn’t seem to exist on public servers :-(

sandbox profile is not open sourced, however you should be able to run it without the sandbox flag

For example:

./runner.py --swiftc <swiftc_path> --projects projects.json --include-actions 'action.startswith("Build")' --swift-branch master
1 Like

The script ran on the first try – but only a handful of targets passed the tests.
As many macos builds have been successful, I looked into the build of the toolchain and noticed that you have to add some flags to build for multiple platforms.
So, I rebuilt swiftc, but as that didn’t help, so I started looking for documentation (actually, I’d even prefer to just build for macOS anyways, because I just want to check if the code itself is compatible with a modified toolchain…)

But as you wrote that runner.py doesn’t needs to be started in the exact same way as Jenkins does, I just triggered it with the regular swiftc – and that did produce only a single failure so far, so I’m sure it’s still an issue with the build of the toolchain…

I started looking into the compat suite sources to check wether there is an easy way to simply skip ios and the other ARM-platforms, but afaics, it might be easier for me write a small script that parses and filters projects.json on its own.

Thanks for your help!