Our test suite takes a long time to run, e.g. up to 20 minutes on CI. In order to iterate faster, we would love to split up tests and run them in parallel, but the "obvious" approaches to it don't seem to work. This is particularly surprising for non-solution 2 (using swift test --filter), which seems like a bug or shortcoming to me. We did find a workaround, but it's a bit awkward and hard to extend.
Non-solution 1
The first idea is to use swift test --parallel. While this could potentially speed up tests on a machine with multiple cores, at least Gitlab only seems to run a single core per job, so in this case there is no speedup by choosing this option. While there is an option for Gitlab to run multiple instances of the same job in parallel, it is my understanding that this would require swift test --parallel to be able to receive an argument indicating the number of the job (similar to the example in the Gitlab documentation). I don't think such a capability exists?
Non-solution 2
Alternatively, we might just manually create different jobs that test subsets of the test cases independently; this is fine since we know e.g. which test sets take a long time, and it's also more deterministic than the previous version, even though it requires a bit more maintenance. While XCTest unfortunately doesn't seem to support advanced tagging and filtering like some other tools, for our use cases the basic regex-based filter would essentially suffice. Unfortunately, we discovered that splitting tests up by the use of swift test --filter makes the whole CI build significantly slower than not splitting tests up. Since swift test --filter also produces output in quite a different way than normal swift test (e.g. it doesn't print a summary of all the tests at the end), my working hypothesis is that using the --filter option somehow causes the test framework to re-run some app setup code for every test which incurs quite a performance overhead in our use case (due to costly initialisation logic in our app) as opposed to having this setup code run only once. I wonder if this is known behaviour or some kind of bug?
Workaround
The only working solution that we've found so far is a bit messy; it consists in basically just deleting test files on CI before running the tests (for this to work, --enable-test-discovery needs to be used in lieu of LinuxMain.swift). I don't think it's a very neat solution, but it seems to work and speeds our builds up significantly.
Based on experience, when a thread gets no replies within a day or two, the reason is that no one (who is reading that section of the forms) knows the answer, and everyone is hoping someone else who knows more will provide the answer.
No one is conspiring to that end, it just happens.
At that point, I can think of only two things left to try:
Look at the source, see who the main contributors to the relevant sections of code are and try to ping them. This can be difficult if their user names differ between here an GitHub.
File a bug, even if you aren’t sure if it is a bug or by design. Then at least it will appear in someone’s (extremely long) to‐do list. Bug reports tend to eventually get replies, albeit much more slowly than on these forums.
You can also pat yourself on the back for finding a question really worth asking. When every reader has a quick answer, the question is probably a newbie one that has been answered a hundred times across the web. But when no reader can help, then it is likely a still unsolved puzzle. And the threads dedicated to those are the ones truly worthy of our time, thought and creativity. I wish the forum software advertised these threads instead of the popular but wasteful ones.
swift test does have a job option but that controls the number of tests to run in parallel when executing the entire test suite. The GitLab feature seems something specialized where the jobs are split into N jobs and only a single split job runs in an invocation. This is not something that is supported by SwiftPM.
Non-solution 2
Your hypothesis is right and --filter spawns a new subprocess for individual test cases. This causes the setup code run for every single test case which is costly in your case. We should work towards improving the parallelism heuristics in the swift test tool. Can you file a bug for this?
Here are some workarounds that might work better for your use case:
Workaround 1:
You can move your costly tests into a local package in the same repository and then run those in a separate job. Assuming you're using Xcode to develop on desktop, you can create an empty workspace and add both of these packages so you can run tests of either package locally without having to switch between them.
Workaround 2:
This is a bit "hacky" but should be an OK workaround: Conditionally add/remove test targets from the package manifest. Here's an example from SwiftPM's own manifest. This also requires automatic test discovery.
Workaround 3:
One other option is driving the test execution process yourself. This is a bit risky/advanced as test execution is platform dependent and the underlying implementation detail can evolve overtime. With that disclaimer, here's how you can execute the tests yourself on Linux:
This looks like it is solved by this PR on the master branch.
In the meantime, I am using Workaround 2 with a Makefile to conditionally remove tests with an environment variable. Can't wait for this to land so I can simplify that all to a "--filter".