Hey folks! I’m noodling on my very first improvement to Swift Testing. The basic premise is that it would be sweet to be able to filter/skip tests & suites based on tags that you’ve applied to those tests & suites. I take one approach to it in my proposal below, though there are definitely other valid ways to do this. I’d love to know what y’all think!
Most recent version of the proposal: nnnn-tag-based-test-execution-filtering.md
This feature has been requested multiple times (example1, example2). So thank you for kicking of an actual proposal.
Note: it is possible to filter on tags when using the VSCode extension or Xcode. However, it indeed does not work for the swift test cli. Which is where your proposal comes in.
I'll make sure we add this to the agenda for the next Swift Testing workgroup meeting.
I accept the rationale of adding the special prefix syntax to the existing arguments, but also dropping support for regular expressions is a little too much IMO.
Yeah that's a fair criticism. I think that in practice it won't be an issue in most situations because the space of tag names in a project is likely way smaller than the space of test/suite names, so enumerating them all is less of a burden.
That said, in principle I don't have any issues with your criticism.
The regular expression can be applied to the string after the "tag:" prefix, yes? Then I would say it's simple enough to just do so.
+1 to the overall proposal, and thank you @goose for putting this together!
One bit of feedback: Since you're proposing the introduction of a tag: prefix specifically to match against tags, it may make sense to also (retroactively) introduce a prefix representing the existing matching behavior, which is based on test identifier.
For example, you could introduce a prefix named something like test:, id: or testID:, which has equivalent semantics to the current --filter/--skip flags. And to maintain compatibility, an unprefixed argument would implicitly be treated as if it had that prefix. Then, if a user wanted to be explicit about the matching criteria, they could explicitly provide that prefix. That could also provide a pathway (as a future direction) to specify whether or not the argument should be a regex/pattern match (i.e. the current behavior) or whether it should be a literal/exact match, regardless of prefix.
Just want to chime in that I really like this feature.
I tend to use tags to organize tests in types: e2e, unit, integration. In particular, a simple way to exclude/filter integration tests (for me: tests that depend on live other systems) would help me a lot as I tend to run those in a separate step in CI anyway. Off course, I could collect them in their own suite or prefix the test with something, those are crude workarounds. And error prone: stringly typed. It feels like this is exactly what tags are meant for.
I also like the tag:foo syntax. It feels like that's something I see used in more places, though I don't have a concrete example. 
Thanks for this pitch. It's something I could see myself using.
I support the this proposal and is definitely something I would like to see. Similar to others on this thread, I am categorizing my tests based on some criteria (e.g.: unit tests, integration, end to end), and this feature would allow me to better collect coverage data based on the "unit tests" - as there is no ideal way to achieve this today.
I like the syntax of tag:<label> as well as @smontgomery idea of adding a test identifier option too as this can lead the path the future options (should there be a need).
I don't have a strong opinion on whether the string after the tag: should be a regular expression other than the current behaviour is a bit confusing at first as it's a case sensitive regular expression. And it would be great to have consistency to avoid confusion and ambiguity.
Can we explicitly augment the proposal to describe the behaviour when an "unexpected" (for a lack of a better work) argument?
e.g.: what will --filter tag:, --filter tag\: match? should this return an error (or a specific return code) since there were no matches?
It's pretty standard syntax for search engine refinements. For example, GitHub code search lets you write label:xyz for the equivalent filtering operation.
Not to speak for @goose but I think with the latest discussion, we'd expect tag\: not to match anything. It would be reasonable to have Swift Testing and/or Swift Package Manager reject unrecognized prefixes with a diagnostic.
Thanks for all the discussion so far! Just to summarize where things have landed thus far:
id: prefix for explicitly matching tests & suites by name. This is functionally the only way --filter and --skip work today. As @smontgomery suggested: "an unprefixed argument would implicitly be treated as if it had [the id:] prefix."There are two main ways the argument to --skip and --filter could be "unexpected". Either the prefix isn't recognized, or the regular expression is invalid. Invalid regular expressions are beyond the scope of this change, I think, so we'll just preserve the existing behavior (i.e. the CLI exits with a non-zero status code and an error message).
For the scenario where the prefix is unrecognized but it has the overall shape of a prefix (like tag\: or someNonExistentPrefix:, I'm thinking that we would alert the user with a diagnostic saying the prefix isn't valid. We could also include a brief line in the diagnostic saying something like:
If the `someNonExistentPrefix:` prefix is instead a test ID you are trying to match on, use `id:someNonExistentPrefix:`
What do we think about this overall path forward?
This change will likely require some elbow grease in SwiftPM to recognize the id: prefix (and others!) and strip it off or ignore it when filtering XCTest. Not hard, but something to keep in mind.
Here are some of my thoughts. prefix and value are as follows --filter <prefix><value>
If the prefix is valid, but has value is nil -> actionable diagnostic error
if the prefix is invalid -> actionable diagnostic error
if prefix is valid, and value is not nil -> no diagnostic error
Based on the pitch, a valid prefix is tag: or id: (and equivalent)
With some apologies for the bikeshedding, but not enough to stop me:
I think the --filter CLI option is not as descriptive as it could be. I think that --skip is much clearer as to what will happen, but --filter could be either "only run tests matching this tag", or "skip running tests matching this tag". I think that something like --only, --focus, or --include would be better.
Of these, I like --include the best: it mirrors what --skip implies the best. That said, for those familiar with more C-like build systems, --include can conceptually conflict with the idea of including libraries.
--only would also work, but I feel isn't as discoverable or meaningful as --include. It also implies that you can only use 1 --only, instead of many.
--focus is my second favorite of these suggestions. If you're familiar with tools like Quick or rspec, you're familiar with the idea of focusing tests. But if you're not familiar with those tools, then --focus isn't as obvious.
Other testing framework make a distinction between executing test cases vs tags.
Robot Framework supports --test <testPattern>, --include <tagPattern> and --exclude <tagPattern>. [1]
pytest, a popular testing framework in python, allows for specifying custom marks, and then executing specific marks (via python -m <marks> , specific tests by ID pytest <test-identifier>, or selecting tests based on their name via `-k [2].
Based on this small sample, each testing frameworks have different arguments to select tests and tags for execution in a test run. I'm not sure if Swift should do the same.
But based on the test names and @younata post, I would be find if we update the current --filter to --include. If we decided to proceed with this, we need to deprecate the current option with an actionable message informing the user the new option to use.
While we are at it, we should also consider deprecating --skip in favour of --exclude, though I don't have a strong opinion on this argument yet, as I haven't had the need yet to excludes tests.
In terms of naming, I’d like a convention with clear opposites:
include, we should also have exclude;filterIn / filterOutskip, only would be the logical pair.I agree that filter / skip are not the most logical opposites.
Of the ones outlined, I like only / skip best.
--filter is already part of the swift test interface, and I don't think we want to go renaming it?
it would be a high cost. But it's not off the table for me. curious how the community views this.
FWIW I would vote for:
--filter +something-to-include
--filter -something-to-exclude
I think the --filter/--skip flag names are outside of the scope of this pitch though, no?
I'm all for making these flags more intuitive but changing the names shouldn't be necessary to land tag-based test execution filtering.