PR Test presets for LLBuild

Development LLBuild
1

Hi all,

We recently hit an issue where an llbuild change broke Swift CI, because our PR testing doesn't test everything downstream correctly. To fix this, and better match how we are testing the rest of our toolchain projects, I want to add testing of sourcekit-lsp to the llbuild PR tests. In looking into that, I discovered our existing PR tests for llbuild have some other deficiencies, and I'd like some feedback on how they should be configured.

I'd be happy to setup the presets for these jobs if you can help me answer the above questions.

1 Like
2

Edited to clarify: the Linux smoke bot does run the llbuild tests. It was another case where the jenkins page is giving an incorrect description of what the job runs. The only bot that doesn't run the llbuild tests is the non-smoke macOS bot.

3

I haven't read/thought about all of this stuff, but I want to give a big +1 to adding presets. All bots should use presets for reproducibility. IMO if we ever do not use a preset it is a bug.

4

I have verified the information with the Swift CI bots.

  • @swift-ci please smoke test Linux
    • https://ci.swift.org/view/Pull%20Request/job/swift-llbuild-PR-Linux-smoke-test/
    • Currently uses -R -T --llbuild --swiftpm --xctest --foundation -- --reconfigure --skip-test-swift --skip-test-libdispatch --skip-test-cmark --skip-test-foundation --skip-test-xctest
    • This should have a preset so that we can change it without changing the Jenkins config.
    • The job claims to use skip-test-llbuild , but in reality it does run the tests.

Yes, we should create a preset for this job.

Current config on the bot:

"${WORKSPACE}/swift/utils/build-script" -R -T --llbuild --swiftpm --xctest --foundation -- --reconfigure --skip-test-swift --skip-test-libdispatch --skip-test-cmark --skip-test-foundation --skip-test-xctest
  • @swift-ci please smoke test macOS
    • https://ci.swift.org/view/Pull%20Request/job/swift-llbuild-PR-osx-smoke-test/
    • Currently uses --test --validation-test --lit-args=-v --compiler-vendor=apple --reconfigure --verbose-build --build-ninja --build-swift-stdlib-unittest-extra --build-subdir=buildbot_incremental --release --assertions --llbuild --swiftpm --skip-test-swift --skip-test-cmark --cmake-c-launcher which sccache --cmake-cxx-launcher which sccache
    • This should have a preset , but otherwise looks OK to me.

Yes, we should also create a preset for this job.

Current config on the bot:

"${WORKSPACE}/swift/utils/build-script" --test --validation-test --lit-args=-v --compiler-vendor=apple --reconfigure --verbose-build --build-ninja --build-swift-stdlib-unittest-extra --build-subdir=buildbot_incremental --release --assertions --llbuild --swiftpm --skip-test-swift --skip-test-cmark

cd ${WORKSPACE}/llbuild
xcodebuild -scheme llbuild SWIFT_EXEC="$WORKSPACE/buildbot_incremental/swift-macosx-x86_64/bin/swiftc" -UseNewBuildSystem=0 test

Current config on the bot:

"${WORKSPACE}/swift/utils/build-script" --preset="buildbot_linux_1604" install_destdir="${SWIFT_INSTALL_DIR}" installable_package="${SWIFT_INSTALLABLE_PACKAGE}"

Current config on the bot:

"${WORKSPACE}/swift/utils/build-script" "--preset=buildbot_swiftpm_macos_platform,tools=RA,stdlib=RA"
5

Did you change this recently? The logs show 1404, not 1604.

6

I've put up a PR to cover the smoke testing changes to test sourcekit-lsp and add presets for llbuild Add presets for llbuild smoke tests and add testing of sourcekit-lsp by benlangmuir · Pull Request #27103 · apple/swift · GitHub.

7

Smoke test changes look good. I have no problem with, and would indeed prefer, that we move the full linux tests to 16.04 to match Swift itself. And for the full macOS tests, it seems a bug to me that it has skip-test-llbuild.

8

Thanks! I merged the new presets.

Here's what I think the bot changes should be (@mishal_shah):

2 Likes
9

The bots are now using the new presets. Thanks all!