How to test package traits?

Evolution Discussion
,
1

Playing around with the new package traits feature, and it's almost great, but I'm running into one problem: I can't find a way to unit test non-default traits. Ideally, I'd like to do something like:

// swift-tools-version:6.1

import PackageDescription

let package = Package(
    name: "MyPackage",
    products: [
        .library(
            name: "MyPackage",
            targets: ["MyTarget"]
        ),
    ],
    traits: [
        "SomeTrait"
    ],
    targets: [
        .target(
            name: "MyTarget"
        ),
        .testTarget(
            name: "NoTraitTests",
            dependencies: ["MyTarget"],
            traits: []
        ),
        .testTarget(
            name: "WithSomeTraitTests",
            dependencies: ["MyTarget"],
            traits: ["SomeTrait"]
        )
    ]
)

What I'd like for this to do would be to compile the project twice; once with the trait enabled and once without, and run the unit tests in both configurations, to make sure both configurations work properly.

I do know about swift test --traits SomeTrait, but this seems to be all-in one way or another; I don't see a way to test both with one command. Furthermore, this option doesn't seem to be available at all when editing a package in Xcode's UI; I can't find any way to add arguments to the swift test command it uses under the hood, and somewhat strangely, whether it enables the trait or not appears to be somewhat random. It's usually off, but sometimes it just compiles with the trait enabled, and I'm not sure why.

Are there any tricks that I'm missing?

1 Like
2

Thanks for the feedback.

This was something that we called out as a future direction. I think it would be great to have a swift build/test --all-trait-combinations that uses all possible permutations. Right now you are correct that you have to manually spell out the different swift build/test invocations.

Since Xcode is out of scope of the Swift project and an Apple provided tool, do you mind filing a feedback to such a feature?

3

@FranzBusch I don't think --all-trait-combinations is a good idea. If you have 3 traits which are exclusive with each other then you need to be able to express that in the Package file. But that's not the main problem. The main problem is that if you have 10+ traits, each controlling a tiny unrelated portion of the behavior, do you really want to test 1000+ combinations? Probably not.

I think the best solution would be to be able to define multiple sets of traits for each test target and if you don't provide any flag that affects traits then the tests would be run multiple times, once with each set of traits. If you provide a flag that affects traits then all the tests will be run with just that set of traits. So if you have the 3-way exclusive traits and 10 other traits controlling unrelated parts, you could have just 3 set of traits and it would cover all the testing needs.

For the test targets that don't specify trait sets, they would be run with the default set of traits (or the traits provided by the command line flags).

With only one set of traits per test target (like in the OP), running the tests with a specific set of traits doesn't really have a good strategy. Either tests will fail or tests will be skipped. (Assuming multiple traits that are controlling unrelated parts of the code and each have a test target for the turned on and one for the turned off state, with all other trait turned off)

1 Like
4

Also if we have traits that don't change existing behavior, only add new APIs (like importing a 3rd party framework and providing convenience inits/protocol conformances/etc.), then I can see an "only run this test target if this trait is turned on" parameter (in addition to providing one or multiple sets of traits to use when running tests without specifying traits), so that tests about those additional APIs can live in their separate test target without needing #if trait around everything.

edit: I don't know if it was clear based on my comments but I think the main goal should be to have swift test test everything that the author of the package wants to test (minus multiple OSs), so that if someone wants to make a small change, they don't need to look for the correct test command in documentation/scripts/etc.