1

We propose promoting the existing .serialized trait to public API. This trait enables developers to designate tests or test suites to run serially, ensuring sequential execution where necessary.

Read the full proposal here.

3 Likes
2

If my vote makes any difference, I vote 100% for this to be added. It's incredibly useful to have, especially when running tests in parallel becomes the default.

3

This is essential near-term. +many!

Is the longer-term discussed somewhere?

(1) Wording could be clearer to elicit more reviews

to designate a parameterized test or test suite to run sub-tests serially , ensuring sequential execution

?: Sub-tests? I'm not sure how best to talk about member/child tests of a container/parent.

?: I would avoid implying sequential or ordered unless the order really is defined and enforced (even if the implementation is definitive now).

(2) Non-parameterized tests

It would be nice if it were impossible to put a .serialized trait on a non-parameterized test. Would it make sense to use a different term for the two targets? That might also help if suite-level and parameterized-test-level features diverge in the future. (Currently trait protocols are per-Suite-level and per-Test, making this suggestion impossible?)

(3) Future-proofing

Would it make sense to anticipate some future semantics, and consider usage in that light?

E.g., .serialized(with {group}: "name")?

(The proposal didn't refer to related features or discussions, so forgive me if you/others already have ideas in mind.)

Suites are tests organized for their own reasons. Developers might want to preserve parallelism for some tests in the suite over refactoring the suite to only include serialized tests, or might want to designate multiple serial member subsets.

Similarly, parameterized tests might partition on one of the inputs being combined (e.g., a mock persistence context), such that tests with the same element should be run serially, but tests with different instances of the element could run in parallel.

Also, it may help to name these groups, to document and find and understand the members.

The developer's mental model might be something like token/baton-passing concurrency: only one test with the baton can run at a time, for those tests needing the baton.

I realize this might not be in the cards for now, but can we ensure this proposal remain (source-) consistent with that model (or some other more flexible model)?

Could the proposed .serialized be a shorthand for .serialized(with:) where the default argument means "everything in this container"?

When implemented, using a StaticString could permit compile-time errors from mixing batons (e.g., across suites and tests). Developers could have a common static definition, and find all tests in the group as those using that symbol.

Also, ordering behavior could be specified as inter-baton/group orderings: first all the data-loading tests, then the analysis tests, then some mutation tests...

4

Tests in a suite are ordered by source location (top of the file to the bottom), while test cases are ordered the same as the collection they are generated from.

I am looking at adding a compile-time diagnostic for this misuse. Edit: I already did and completely forgot. :slight_smile:

It doesn't make sense to design out the syntax for such a feature since it isn't supported at this time, but if we add something like that in the future, it would be straightforward to also add an overload of .serialized that takes arguments.

1 Like
5

Thank you @Dennis!

Although we want to encourage parallelized testing where possible, the core testing team recognizes the need for serialized testing and the need to annotate tests that cannot run in parallel with other tests.

Feedback in this thread has been light but positive, and the proposal is accepted. However, there remain some open questions about future enhancements to the proposed API, so the team requests that Dennis amend the Future Directions section of the document before we merge #535 into the repository.

1 Like