[Pitch] Data-dependent test serialization

Swift Testing has a test trait you can apply to a suite or parameterized test called .serialized:

@Suite(.serialized) struct TerminalTests {
  @Test func `Xterm color emulation`() {
    Environment.set("TERM", to: "xterm-256")
    #expect(Terminal.isColorEnabled)
  }

  @Test func `VT-100 emulation`() {
    Environment.set("TERM", to: "vt100")
    #expect(!Terminal.isColorEnabled)
  }
}

In this example, because the suite has been annotated .serialized, the two tests function run one after the other. However, they can still run in parallel with other unrelated tests, including tests in other suites that are also marked .serialized:

@Suite(.serialized) struct TerminalTests {
  // ...
}

@Suite(.serialized) struct CashRegisterTests {
  @Test func `Add money to the register`() {
    // ...
  }

  @Test func `Remove money to the register`() {
    // ...
  }
}

The intent here is to allow test authors to use .serialized while still maximizing parallelization.

I propose adding an overload of .serialized that lets test authors specify a data dependency. If a test is serialized for a given data dependency, then all other tests with the same dependency are also serialized with respect to that test.

We've received feedback that test authors expect the existing .serialized trait to serialize all tests it is applied to (regardless of which suites they are in). So I also propose changing its behaviour to do exactly that.

For the full proposal, click here.

Trying it out

To try this feature out, add a dependency to the main branch of swift-testing to your project:

...
dependencies: [
  ...
  .package(url: "https://github.com/swiftlang/swift-testing.git", branch: "main"),
]

Then, add a target dependency to your test target:

.testTarget(
  ...
  dependencies: [
    ...
    .product(name: "Testing", package: "swift-testing"),
  ]
)

Finally, import Swift Testing using @_spi(Experimental) import Testing.

13 Likes

Great to see this! I’ve definitely been bitten by the “not entirely serial” execution semantics previously. Thai is a nice way to spell it and I also think the behavior change to match the * is a great move. Tanks a lot, looking forward to this

4 Likes

I think the * is just too cute — is there a reason not to use serialized(for: .everything) or similar instead? Which also doesn't introduce another ad-hoc overload of * into the global namespace?

Previous iterations of this pitch had special ways to refer to the environment, and maybe filesystem? I'm concerned that tests may simply not be consistent in how they refer to these things. With a Foundation import, they could plausibly use \ProcessInfo.environment (not a static keypath, though I presume it would still work) and \FileManager.default. But that would only be a convention.

5 Likes

I really like the way keypaths are being used to express the dependencies. It's a lot of power with pretty much zero boilerplate and lets authors use the names of their actual "data", vs. the common pattern of defining static members of a type acting as a namespace.

I think the examples could use a bit of cleaning up though; many of them just say for: A, for: B and so forth, which certainly aren't keypaths. It sort of buries a small but important detail, which is that because the serialized function is defined as such

static func serialized<R, V>(for keyPath: KeyPath<R, V>) -> Self

where the root type is also an unconstrained generic, the keypath must be root-qualified. It's obvious once you know what's going on—something like .serialized(\.myField) would be meaningless since there's nothing to infer the receiver from, but it is a slightly different pattern than I think most folks are used to, where keypaths are used in a context where the root is easily inferred.

And Keith above managed to write almost the exact same words as I was working on my reply, so I think that points to something:

I'm not fond of the specific definition of a custom * operator to represent the unbounded parameter. It feels too cute for its own good. It's technically following the same pattern as lone ... in a subscript to represent an unbounded-on-both-ends range, and I can see a conceptual connection here to the use of * in availability annotations, but I worry that people will look at this and say "well swift-testing does it, so now I'm going to use * to represent "all things" in my API" and every overload for * needed to define that will be a death-by-a-thousand-cuts to type checker performance for multiplication operations.

Code completions aren't going to know what to do with that either, and most users writing Swift aren't going to think of putting * there just because they know availability annotations can do it. I think it's much more likely that a user would write .serialized(for: , then maybe a dot to see what completes in that context. I think you'd have a more discoverable API if you just defined something like a single element enum or a struct with a single static member named everything, since you have to define a custom overload anyway.

2 Likes

I’m excited for this, +1.

I just don't think that the existing .serialized trait ought to be deprecated. The existing documentation describing intra-Suite serialization is clear - we just wanted inter-Suite serialization! With even more documentation and examples, it's pretty clear what the default behavior is.

Keeping .serialized as wrapper for .serialized(for: *) would be a more preferable API design imo.

2 Likes

I agree. This is also a good place to introduce a dependency named .suite to restore the old behavior.

Big plus to this. The current approach of wrapping everything in an outer serialised suite is a bit painful so would be great to have a real solution to this

1 Like

Thanks for the feedback over the weekend, folks! I'll try to address things in order here.

The open question during development became "what is the namespace for .everything?" i.e. what Swift type does it live in? Well, Dependency, sure, but then why is it an instance of Dependency but I can't create my own instances of that type? And this led to Dependency being the namespace for all dependencies, but then see "Using a new tag-like type and macro to describe data dependencies" in alternatives considered for why this was a problem.

(But yes, I am very much open to using something other than * here. Happy to bikeshed.)

There is no type information available to the compiler when @Test is expanded that could be used to create a partially-qualified key path. It would make sense to me if it defaulted to the current type (i.e. the current suite) but this is not expressible in the language today. The team is open to exploring alternative namespaces for dependencies, but please do review alternatives considered for some more background here.

(See above.) For what it's worth, we could directly use the existing UnboundedRange_ type from the stdlib, but a dependency is not a range so we didn't.

The proposal marks the existing trait as to-be-deprecated with formal deprecation left to a future Swift release. This affects documentation and presentation in IDEs like Xcode, but does not normally cause the compiler to emit diagnostics. As a general rule, two interfaces that do the same thing are a code smell, but leaving in the unqualified .serialized will affect discoverability and will be confusing to test authors next to .serialized(for:).

That's the plan right now!

A named dependency whose effects depend on where it's used would likely be confusing in its own right when other dependencies explicitly act globally. A workaround is already offered here: \SuiteType.self can be used for this sort of functionality.

To be clear, my comment about this wasn't suggesting that you need to change anything about the design. I was just stating that the fact that the keypaths must always be root-qualified is subtle for end users who are probably used to inference and I think it would improve the proposal text to show this using more concrete examples instead of just the A and B placeholders that are currently there, which aren't keypaths at all.

2 Likes

Understood. I'll noodle on that section.

This feels like a valuable addition, and I appreciate the API choice to name key paths directly over e.g. some namespace type filled with dependency "key" declarations or similar.

Re: stating multiple dependencies, from the proposal:

@Test(.serialized(for: A), .serialized(for: B)) func ab() {}

Is it worth considering (or rejecting) a convenience version of serialized for stating multiple dependencies?

@Test(.serialized(for: A, B)) func ab() {}

We could do that, however you'll note that at least for the initial proposal, a test with multiple dependencies is treated as having an unbounded dependency, so this would be alternate syntax for .serialized(for: *) (ignore potential respelling). But it would come with a hidden cost if we later treat multiple dependencies differently from an unbounded dependency, because a test that errantly relies on A, B → * could start failing unexpectedly. (This cost exists for serialized(for: A), .serialized(for: B) too, but my gut tells me that's less likely to be overlooked.)