Running all tests optimization tips

Hi,

For some time now I'm seeing that test suit is taking more time that it used to run on local machine. Normally I just run all tests before a commit and push, or to see if on change diagnostics all covered all tests. Not really sure if its related to hardware in specific(probably is) but also test suit is incrementally growing so its expected to take longer.

For example the last run took 2701.21s which is ~45min running on -- Testing: 6533 tests, 4 workers -- with llvm-project/llvm/utils/lit/lit.py -sv --param swift_site_config=build/Ninja-ReleaseAssert/swift-macosx-x86_64/test-macosx-x86_64/lit.site.cfg swift/test/

So the questions here are more:

  • If there are some flags that I can pass to it to speed up this process?
  • There is a way to force more worker threads?
  • There are some tips and tricks from your development workflow to avoid loosing too much time on running this?

Any advice is appreciated!
Thanks in advance =]

I very much recommend running only a subset of the test suite for day to day development. CI runs can catch the little mistakes 9 times out of 10. For example, when I work on the Clang Importer, I try to only run the tests under tests/ClangImporter. You can tell lit to do this by mentioning the subdirectory you want to test or by using --filter. I don’t ever remember the structure of the tests suite well enough to use filter correctly, but it’s there if you need it. You can also utter the names of specific tests instead of running the whole suite under a directory. Very helpful for iterating on a single test.

4 Likes

Apart from --filter that Robert suggested, you can try using --max-test-timeout 1 (in units of seconds). This can be something in-between --filter and running the smaller test suite.

also test suit is incrementally growing so its expected to take longer.

FWIW, I don't see this as something that needs to be true. One would hope that the compiler is also incrementally growing faster. :smiley:

1 Like

Thanks for the answers Robert, Varun :)

True :laughing:

Terms of Service

Privacy Policy

Cookie Policy