[Pitch] Conditionally saving attachments (AKA attachment lifetimes)

Evolution Pitches
,
1

In ST-0009, we introduced attachments to Swift Testing in Swift 6.2. This feature allows you to create a file containing data relevant to a test.

In XCTest on Apple platforms, you can specify the lifetime of an attachment by setting the lifetime property of an XCTAttachment object. This property lets you avoid serializing and saving an attachment if its data won't be necessary (e.g. "if the test passes, don't bother saving this attachment.")

ST-0009 listed attachment lifetimes as a future direction for the feature. Well, the future is now! So I'd like to propose we introduce new API to Swift Testing to allow test authors to control whether or not a test's attachments should be saved or discarded.

Read the full proposal here.

Trying it out

To try this feature out, add a dependency to the main branch of swift-testing to your project:

...
dependencies: [
  ...
  .package(url: "https://github.com/swiftlang/swift-testing.git", branch: "main"),
]

Then, add a target dependency to your test target:

.testTarget(
  ...
  dependencies: [
    ...
    .product(name: "Testing", package: "swift-testing"),
  ]
)

Finally, import Swift Testing using @_spi(Experimental) import Testing.

2 Likes
2

The only piece that I can consider a gap is the ability to say never or always, to allow overwriting the platform-specific default behavior, and in the cases where you… never or always want the attachments regardless of test outcome

1 Like
3

always is the default, however on Apple platforms you have a test plan that may have configured the default differently. @smontgomery had some thoughts about this question he wanted to share.

I am hesitant to include something like always in this pitch, at least initially, because it has no real effect on non-Apple platforms (where there's no test plan to override the default.)

4

Ah, yes good point.

My only counter-argument is that sometimes developers want to be explicit with their intentionality regardless of the default behaviors of where the test executes, especially when they may have cross-platform support stories for running their test suites (on Apple & Windows for example)

It might be fine as a point for future-direction, just will want to call it out me thinks as such

1 Like
5

Firstly, thanks @grynspan for working on this. +1 on the overall concept. Having the ability to control when attachments are or are not preserved is important and useful, and I agree with using a trait to control it.

I think this proposal would benefit from two more things:

  1. A mechanism to configure the default retention policy: either "Keep always" (the current behavior) or "Discard on success" (new). This could be specified via a swift test CLI argument, environment variable, or some combination thereof.
  2. An additional factory function on the proposed AttachmentSavingTrait type indicating "keep always". This would primarily be of use when the default (from 1) has been set to "Discard on success", by allowing an individual test or suite to override that default policy.

One big reason I think this is important is to avoid excessive storage usage in CI. If a codebase has many tests which record attachments that are meant to help troubleshoot problems, but those tests are passing, the attachments will often be superfluous and waste both space and potentially incur greater transfer cost for system maintainers. But certain attachments are important regardless of the outcome of a test, so there needs to be a way to reliably preserve those regardless of the default retention policy.

1 Like
6

:index_pointing_up: this.

I agree that having a way of overriding environment/test plan/suite defaults is useful. Especially if you run tests in various IDEs and platforms. Having the option to explicitly state in the code how you want to handle saving/retaining attachments makes it more predictable and that to me sounds like a good idea.

1 Like
7

@smontgomery and I have discussed this a bit off-forum and an implementation of this feature would need to include changes to test hosting tools like Swift Package Manager and the VS Code plugin. I do see the value in it!

I think it would make sense to talk to the SwiftPM/VSCP stakeholders about adding a command-line option for this functionality (--default-attachments-behavior discard or something like that, API naming is hard) and then proceed from there in a follow-up proposal.

Pinging @plemarquand for his thoughts on changes to those components.

8

I’m in favour of having a flag for swift test to set the default attachment lifetime. Attachment location is already controlled by a flag, so it makes sense for attachment behaviour to be controlled the same way. It’s also easy to integrate in to tools, as VS Code already has the concept of additionalTestArguments that can be forwarded to each invocation, or you can configure the flag on your VS Code launch config.

It does feel like it could be outlined in a follow up proposal though, as its not integral to landing attachment lifetimes as a concept.

2 Likes