I'm looking into changing some build ordering in build-script and wanted to check if there were any problems anyone might have that I'm overlooking.
Currently the build-script builds, installs, and tests a small set of core projects before doing anything for the wider set of projects. Some projects, like swiftpm and swift-driver rely on the toolchain being fully built and installed before they can start building. Because of the current ordering, Swift will be tested before swift-driver gets a chance to build.
The direct motivation for this change is that we would like to make it possible to run the Swift test suite against swift-driver, which would require running tests later, after all the components have been built.
The current ordering in build-script looks like this:
Build all core build products
Test all core build products
Install all core build products
For each other build products:
Build
Test
Install
And I would like to make it something more like this:
For each build product:
Build
Install
For each build product:
Test
This is hopefully a more flexible ordering for this purpose.
Possible downsides I can think of right now are minor:
Some builds with tests might fail slightly later, but others will fail faster, so this is just a tradeoff.
It installs products before testing them, but since installs go to the toolchain in the build directory I'm not sure if this is a problem to anyone.
One problem I can see with this is that the compiler is first "tested" on swiftpm and swift-driver, instead of against its own test suite. In CI, if I get a failure involving these projects, narrowing them down can be much more work (especially with swiftpm) compared to narrowing down a failure in the test suite. So one needs to be mentally aware that even when one sees a failure in CI when building swiftpm or swift-driver, one should try running the compiler's own test suite locally before trying to reproduce the issue with swiftpm or swift-driver.
This is already a problem with the stdlib to some extent, but it is generally it's not a big one (at least for me) since I typically build the stdlib once before submitting a PR.
IMO, it would be better if we could move the driver tests so that those run later (maybe they are run as part of swift-driver's test suite and not as part of the compiler's test suite), instead of moving everything to build first before running tests.
Could you elaborate on when tests might fail earlier under the new scheme?
Just builds with tests, things will fail faster in the case that there's a compilation failure in any of the products you're building, and somewhat slower in the case that you're building more products but have a failure in the Swift tests rather than any other tests.
Now that you point it out, having the compiler be used for things before running the test suite does seem like it could make things more difficult.