Test discovery on Linux

Development Package Manager
1

I've been slowly working on test discovery for Linux and I think the implementation is in a reasonable state now. I've tested it on some OSS Swift packages but it would be great if others can try it out and report any issues they run into. It can be enabled by passing --enable-test-discovery flag to the swift test invocation.

The toolchain is available here.

Known issues:

  • Running tests in release mode requires passing --enable-index-store flag.
21 Likes
2

Would would be really useful is being able to do a diff between user defined linuxmain files and what is auto generated and in this way I have more confidence that I can just delete all the manual files.

3

You can diff the output of swift test --list-tests and swift test --enable-test-discovery --list-tests.

You can also see the generated test list in .build/x86_64-unknown-linux/debug/testlist.derived if you want to do a manual inspection.

6 Likes
4

This is really amazing Ankit, thanks a lot for publishing the toolchain too.

I just tried on some projects I have laying around and found no issues in discovering tests in any of them, it correctly discovered all of our tests :heart:

I have discovered one difference though; which we actually do not rely on, but perhaps someone could / does, so might as well mention the case right away, even if we'd e.g. ship a first version without supporting that (which would already be tremendously useful, as I don't know if people actually do this base class "trick"):

Structure:

  • class CommonTests: XCTestCase { func test_common() {} }
  • final class SpecificOneTests: CommonTests {}

On mac this discovers the SpecificOneTests.test_common as well as the one in the base class, the current --enable-test-discovery does not discover the one in the SpecificOneTests.

Having that said, I'm not sure how critical this is, since I've not seen it used, but I could be totally mistaken and people could expect it to work. Even the existing PRed discovery would already be amazing and I'd love to be able to use it in normal swift builds :slight_smile:

2 Likes
5

I created a docker image on Docker Hub as norionomura/swift:pr-25685 that supports --enable-test-discovery.

5 Likes
6

Ah, yeah. This is a little complex to implement in the current approach but it should be technically possible. I don't think this is a common enough scenario though and it is easy to workaround (by just explicitly writing a new test method in the subclass and then calling the method in the super class).

1 Like
7

Yeah, agreed that it's not a show stopper; wanted to make sure we're aware of it (might want to document it).

All in all though this is awesome, can't wait :blush:

1 Like
8

thank you for putting this together @Aciid, this is fantastic for server developers

2 Likes
9

I've just been trying out the new --enable-test-discovery flag on the latest Swift 5.1 docker image and found it's great for tests, so thanks for the great work :+1:t2:

Unfortunately I've found that just running swift build still requires the LinuxMain.swift file to exist, even though it's not needed any more. I'm not sure why just doing a plain build without the tests requires the LinuxMain.swift file. Is there a reason for this and are there any plans to change it in future?

2 Likes
10

Just hit this issue when building an app inside Docker that's had the LinuxMain and all the allTests stripped out as it's no longer needed. A completely empty LinuxMain.swift fixes the issue but as mentioned it seems weird to require it for the build to pass (I'm assuming it's because the build command also attempts to build the Test target as well). It also isn't required when building on macOS. Another fix is to build it with the --enable-test-discovery flag as well.

11

can you check if this is an issue with the trunk snapshot?

12

@Aciid There is an issue when using multiple test targets (for instance AppTests and IntegrationTests):

/package/.build/x86_64-unknown-linux/debug/testlist.derived/IntegrationTests.swift:11:16: error: invalid redeclaration of '__allTests__[REDACTED]Tests'

This happens for each test class.

It seems each target adds the declarations for all targets' tests.

13

It’s working flawlessly for me even with multiple test targets containing multiple test cases inheriting from custom subclasses.

What version/toolchain of Swift are you using?

14

@SDGGiesbrecht @Aciid I misdiagnosed the problem. It has nothing to do with multiple test targets. The problem occurs with extensions of a test class over multiple files:

ExampleTests.swift

import XCTest
final class ExampleTests: XCTestCase {
    func testExample() {}
}

ExampleTests2.swift

extension ExampleTests {
    func testExample2() {}
}

Gives

[9/12] Compiling ExamplePackageTests ExampleTests.swift
/package/.build/x86_64-unknown-linux/debug/testlist.derived/ExampleTests.swift:11:16: error: invalid redeclaration of '__allTests__ExampleTests'
    static let __allTests__ExampleTests = [
               ^
/package/.build/x86_64-unknown-linux/debug/testlist.derived/ExampleTests.swift:5:16: note: '__allTests__ExampleTests' previously declared here
    static let __allTests__ExampleTests = [
               ^

This is with the latest Swift 5.1 docker image.

15

Ah, okay.

Then it looks like bug. Report it here: bugs.swift.org

(And if you are looking for a workaround in the meantime, then it should be relatively trivial to temporarily change each instance of extension ExampleTests to final class ExampleTestsExtension[1, 2, 3, ...]: XCTestCase. It would at least be less work than maintaining test manifests.)

16

I used this trick in a project where the same tests had to run against a local network mock and against a real network. The actual tests were in the base class, and the subclasses were responsible for configuring the differing environments. In Xcode, I had the tests in the base class marked as do-not-run.

17

I finally got around to creating a bug report: [SR-11951] Test discovery on Linux breaks with test class extensions in multiple files · Issue #4635 · apple/swift-package-manager · GitHub
That workaround should work. And if you want to share the setup and teardown methods, a subclass approach is another option.

1 Like
18

I've tried to adapt --enable-test-discovery for SwiftNIO but faced with some problems since there are tests which marked as deprecated and SPM on Linux doesn't like it (with -Xswiftc -warnings-as-errors flag). @johannesweiss filed [SR-12008] SwiftPM: On Linux, enable-test-discovery stumbles over deprecated tests · Issue #4628 · apple/swift-package-manager · GitHub. Seems like LinuxMain generates at [WIP] Implement test discovery on linux by aciidb0mb3r · Pull Request #2174 · apple/swift-package-manager · GitHub. Can we apply NIO's hack there? Or maybe there is a better option for it? I'm trying to prepare a patch (at least I think so-)). Thanks for any pointers :slight_smile:

1 Like
19
20

Is testing in release mode working for anyone else? I'm getting the following error:

$ swift test -c release --enable-test-discovery --enable-index-store
/src/.build/x86_64-unknown-linux/release/testlist.derived/InkTests.swift:2:18: error: module 'InkTests' was not compiled for testing
@testable import InkTests
                 ^

This happens with my project, as well as other projects I cloned from GitHub that use swift test ‑‑enable‑test‑discovery.

Should @testable only be added to the generated test list when testing with -c debug?