Rethinking symbolic vs. string literal tags

Related Projects swift-testing
,
1

"Symbolic" tags are instances of Tag referred to by Swift symbol name. For instance, swift-testing includes the symbolic tags .red, .orange, .yellow, etc. out of the box. Symbolic tags are useful because they are harder to misspell and can participate in IDEs' autocompletion features.

Today, every tag wraps a string regardless of how it is declared:

extension Tag {
  static var critical: Self { "unimportant" }
}

The above tag has two "names", in essence, and can be referred to as .critical or as "unimportant" when added to a test. The two names are equivalent.

A tag needs some form of identity, which is why all tags wrap string. If a tag were declared with just Tag() with no associated metadata, then any two tags would compare equal, and there would be no way to distinguish them. If some statically available information (such as file ID and line number) were used to unique tags, then changes elsewhere in a source file could cause tags' identities to change unexpectedly.

Problem

String values are important for Swift Package Manager because when working at the command line, all you have to help you out are (about) 102 keys and the --help option. If tags map to and from strings, then you can write something like swift test --filter-by-tag '.critical' and have it "just work." But because a tag may have two distinct names, we need to make decisions like:

  1. Should a test tagged "unimportant" be run if a developer asks to run tests with the .critical tag?
  2. If a .critical tag is defined in two different contexts (say, two fileprivate declarations in different files), should they be treated as equivalent even if they have different raw string values?
  3. Should let critical: Tag = ... at file scope be treated as equivalent to .critical (a static member of Tag)?
  4. Are any of these possible equivalences transitive? If .critical and critical are the same, is critical the same as "unimportant"?
  5. What happens if somebody declares a tag with a string value that's only known at runtime (e.g. taken from an environment variable)?
  6. If a tag is declared symbolically in another module (e.g. let experimental: Tag = "Swift Experimental Feature") how do we find out its string representation and consistently recognize it as equivalent if we don't have access to the source code of the other module?

We don't have good answers to these questions, and we've found that any solution we try to design ends up unsound or, at best, inexplicable to the end user.

Proposed solution

The solution I'm proposing refactors Tag pretty significantly. Gone is RawRepresentable conformance. In its place are two disjoint families of tags: tags declared as static members of Tag, and tags declared as compile-time constant string literals.

Static members of Tag

Tags declared as static members use a new @Tag macro (name subject to review) to indicate at compile-time that they are tags that may be applied to tests:

extension Tag {
  @Tag static var critical: Tag
}

@Test(.tags(.critical)) func f() { ... }

We may want to change the exact shape of this macro in the future, especially with recently landed changes to swift-syntax to allow gathering the lexical context of a macro. We may in the future change the above to something like:

extension Tag {
  static let critical = #tag
}

@Test(.tags(.critical)) func f() { ... }

But that's a future direction we are unable to implement at this time, so best not to dwell on it.

Note that tags declared this way do not have raw string values associated with them. The ambiguity of .critical versus "unimportant" above is eliminated. We also disallow applying this macro to symbols that are not members of Tag, so a file-scoped critical tag is not possible, and that ambiguity is eliminated too.

String literals

Tags declared as string literals behave more or less the same as before:

@Test(.tags("unimportant")) func f() { ... }

Today, Tag conforms to ExpressibleByStringLiteral, allowing one to use a string literal in place of an instance of Tag. That's a problem if we make this change because it becomes possible to misspecify a symbolic tag in a way we can't catch with the @Tag macro:

extension Tag {
  static var critical: Self { "unimportant" }
  // ❌ BAD: looks like a symbolic tag at the call site, but is defined as a
  // string literal tag here.
}

So we will need to also remove ExpressibleByStringLiteral conformance. To allow developers to specify tags as string literals, we can overload the .tags() trait to take either predeclared instances of Tag or actual string literals (instances of String) which swift-testing can translate internally. We lose the ability to mix the two in a single call to .tags(), but making two calls side-by-side is trivial.

Expressing tags at the command line

We define a trivial transformation function to/from tag and string: if the first character of a string is a period, then the string is interpreted as the name of a static member tag; otherwise, it is interpreted as the value of a string literal tag. A leading slash is dropped, but indicates that the tag is a string literal tag, so ".oddChoice" can be "escaped" and correctly interpreted in the unlikely event that a developer wishes to specify it. This format is easy to represent at the command line and easy to understand/explain:

# Run tests with the symbolic tag .critical or the string tag "unimportant"
swift test --filter-by-tags .critical unimportant

Macro knowledge of tags

One unfortunate caveat here is that the compiler, by way of the testing library's macro target, needs to have knowledge of tags. We'd really prefer it if the macro target had as little knowledge of individual traits as possible, but given the integration we'd like to build with tools like Swift Package Manager, it seems unavoidable that other parts of the swift-testing stack also need to understand them.

Macro targets are limited to examining syntax and do not have type information. In the future, we could explore changes to the language and compiler to allow better compile-time context for the testing library's macros and to improve the overall experience for developers when using tags.

Trying it out

These changes are on a branch of swift-testing, jgrynspan/rethink-tags. You can try them out today. A PR is open to merge this branch. Normally, we would want to merge the PR with API changes guarded by @_spi(Experimental), but it is not possible to do so in this case.

2 Likes