having now written a handful of FileCheck tests for SIL, i was wondering if anyone had any tips for effective patterns/guidelines for such tests. i somewhat frequently find myself doing what feels like just copy/pasting compiler output and CHECK:ing that it's there, which usually gives me pause. do you have any recommendations for what sorts of things you generally find valuable in FileCheck tests? what bits of output should be typically be ignored vs captured and used for expectations? what would you point to as examples of 'good' FileCheck tests of SIL code?
some specific questions that have occurred to me:
when do you use CHECK-LABEL? matching on the name of a SIL function makes sense to me, but should that always be paired with a matching check at the end (this doesn't seem super common anecdotally)?
is just copy-pasting full output directly into a test a potential maintenance burden? i assume it could be, but it's not totally clear to me which aspects of the output are at more or less risk of being broken if someone changes unrelated logic that affects codegen, etc
when do you use CHECK-NEXT vs CHECK?
when do you typically ignore subpatterns (like with {{.*}}) vs capture them?
It is good practice to make it so that tests only check for conditions relevant to the test. Most of the specifics of the raw compiler output for a SIL dump is not important for many tests, so even if you start by copying and pasting it directly into CHECK: lines, it's worth going over it to remove irrelevant instructions, comments, type annotations, etc. You can use [[FOO:%.*]] captures to avoid hard-coding specific instruction numbers. CHECK-NEXT: is useful for robustness when sequences of instructions must occur one after the other.
CHECK-LABEL: works well most often for matching declaration introducers. FileCheck works hierarchically with labels, matching all of the -LABEL patterns first and using them to segment the output, so that the regular check lines in between each label in the test file are scoped to only match lines between the label matches in the output. This gives better recovery, where a failed check line expected in one function is less likely to interfere with check lines that are expected in other function bodies.
To echo Joe, stripping out irrelevant items is a really good idea, because it will save someone else (maybe future you) from having to update a pile of tests that failed because they're expecting over-specific output but that aren't actually broken. Not doing it is definitely a maintenance burden, but that fact only really becomes clear when someone makes a change that triggers a pile of unnecessary test failures.
Stripping things out also, as a side-effect, makes your test cleaner and easier to read, since people can see what you actually care about and are testing.
(I'd say this applies to any use of FileCheck, not just for SIL, FWIW.)
In the same spirit, I would only use CHECK-NEXT if I'm testing something about the next line specifically. If you're tempted to use CHECK-NEXT, I think it's worth contemplating whether it would be reasonable for something to be inserted between the two lines, and if doing so should fail your test. If you're only doing it because you're worried that a CHECK would pick up something similar from much later in the output, you could use CHECK-LABEL to segment things first.
On ignoring sub patterns, likewise. Is the exact sub pattern relevant to the test? Is it relevant that it must be the same on other lines? If either are true, capture it and use the capture. If not, don't.