GitHub Draft PR: [test] Improve testing of Swift features by drodriguez · Pull Request #76740 · swiftlang/swift · GitHub
This is quite a big change, and it requires modifications in the flow of the compiler developers that work in experimental features, hopefully for the better and hopefully not intrusively. I tried to make the new flow as good as I could while still covering the cases that normally hurt us more as downstream maintainers.
The testing infrastructure of Swift.org is limited and humans are not flawless, so from time to time, some testing done in experimental Swift features end up breaking the test suite in the less tested configurations of the compiler. Most of these problems are easy to solve, but until know there was no help in doing so.
I would like to open the floor to anyone that is interested in reading through the proposal (copied below from the GitHub PR), specially the day to day developers of the compiler. Would this improve your work? Would this be too intrusive? Can it be done differently? Can it be improved?
Please add your comments here or comment directly in the PR if the code context is important.
Thanks.
Features in Swift can be experimental, upcoming or base features. During their lifetime features evolve from one category to other categories. Usually experimental features are only available in "asserts" compilers, but they can also be experimental features in "non asserts" compilers. Upcoming and base features are available in both "asserts" and "non asserts" compilers.
When coding a new feature, the feature normally will start as an experimental feature. In the past this has required to mark the tests that use that feature with REQUIRES: asserts to avoid the test failing when testing in a "non asserts" compilers. This requisite is not always follow, and we forget to add those REQUIRES: asserts from time to time. This causes breakages for people that test the "non asserts" compilers. For some experimental features that are available in production compilers the REQUIRES: asserts is not even needed, which can make adding those lines work against one intentions. When the feature graduates from experimental to upcoming or base, we sometimes forget to update the the related tests to remove those REQUIRES: asserts, so a "non asserts" compiler will not actually execute those tests, and bugs can be introduced by mistake in those compilers.
The changes in this PR introduce a system that aims to simplify testing those experimental features and avoid some of the problems noted above.
The first change is take the canonical Features.def and transform its contents in something that LLVM Lit can create available_features for. This is done abusing the Clang preprocessor to transform the .def file into a Python file that can be loaded by lit.site.cfg during testing. This is done for each build and will pick up changes in the Features.def as they happen, so it will always be up-to-date. Additionally it understand when features as available depending on "asserts" or "non asserts" compilers, and will not incorrectly require an "asserts" compiler for non-production features, or let experimental features be tested in a "non asserts" compiler.
The second part of the change is keeping the tests up-to-date with the features they are testing, so each test that uses a feature is marked as such. This is done with a test itself, which greps through the existing tests, checks for the usage of -enable-experimental-feature or -enable-upcoming-feature and warns the user about the missing REQUIRES: lines (failing the test suite, so nobody can submit a test that skips the requirements).
Finally, the last change is modifying a huge number of tests to follow the new rules. All the tests that currently use -enable-experimental-feature or -enable-upcoming-feature have been annotated with REQUIRES: lines, and the (now unnecessary) REQUIRES: asserts have been removed.