How do you mark a test as "pending"?

Development Swift Testing
1

Suppose I'm writing some new feature—or I have identified an existing bug– and I know it is going to fail, but the idea is sometime in the very near future it will succeed. I would still like to run the test, but I don't want it to "fail" the suite. I'd like it to fail with some intermediate state, like pending. Is this a feature?

I've tried disabling the test, but this prevents you from running the test at all. Even if I ask for that test specifically to run, Swift Testing seems to prefer running zero tests rather than running the disabled test, and I don't think this makes any sense. I'd like to suggest that .disabled() should exclude the test from being automatically run by the parent suite, not in all cases whatsoever.

2 Likes
2

If you have a test that is currently failing, but will succeed in the future, use withKnownIssue():

@Test func `widget is flummoxed`() {
  withKnownIssue("Need to implement widgets") {
    var widget = Widget()
    widget.color = .green
    #expect(widget.isFlummoxed)
  }
}

I'm not sure what behaviour you're seeing with .disabled() that doesn't match your expectations. A test with the .disabled() trait won't run, but disabling it should not affect unrelated tests.

2 Likes
3

I don't think withKnownIssue is applicable here because this is a feature of the whole test, not any part of the test body. The idea is the test body, written as-is, ought to pass.

For example, write a test around a new feature, so that it fails the suite if you try to run it. Add the "disabled" trait to the test. If click on the arrow next to the test in Xcode, or if I select the test on swift test --filter, I get:

Suite "Table" passed after 0.001 seconds.
Test run with 1 test passed after 0.001 seconds.

This suggests that the test was run and passed, but this did not happen. The test was skipped, zero tests were run. If I specifically name the test and see it was run, I would expect the test to have been run, regardless of the trait.

1 Like
4

I think this request makes sense, and this isn’t the first time this idea has come up.

In terms of implementation, what I think we should do is change test selection behavior such that if a disabled test (EDIT: function) is specifically and explicitly selected — not just implicitly selected by having selected one of its containing suites — then that test ought to be run despite having a .disabled trait. And that would have the effect of allowing users to begin writing a test they know is going to fail initially, click directly on that test function in an integrated IDE or specify its name via swift test CLI, and have it run in order to iterate on it, but still maintain it as disabled.

Can you file a GitHub issue about this?

2 Likes
5

We would need to think carefully about how or whether this policy would be applied to suites which are themselves disabled, given how ConditionTrait is recursive and inherited by sub-tests. I’m curious to hear people’s thoughts on that aspect.

1 Like
6

I would naïvely expect that it wouldn't override individual tests' traits within the suite since they may be conditionalized for some unrelated reason. But I could be convinced otherwise.

2 Likes