Swift Testing: test passes only when setting breakpoint

Using Swift
1

I’m trying to unit test version control commands with Swift Testing using Xcode 16.4 on macOS 15.6. For my first test I create a repo and test that the repo was created. I have the following unit test code:

class JujutsuCommandTests {
    var repo: Repo?

    init() {
        let testReposFolder = URL.documentsDirectory.appendingPathComponent(
            "Test Repos",
            conformingTo: .folder
        )
        let unitTestingFolder = testReposFolder.appendingPathComponent(
            "Unit Testing",
            conformingTo: .folder
        )
        repo = createRepo(location: unitTestingFolder)
    }

    deinit {
        if repo != nil {
            do {
                try FileManager.default.removeItem(at: repo!.folder.url)
            } catch {
                print(error)
            }
        }
    }

    @Test func createRepoCommandCreatesValidRepo() async throws {

        let validRepo = hasJujutsuRepo(
            location: repo?.folder.url ?? URL(filePath: "/")
        )

        #expect(
            validRepo,
            Comment(
                stringLiteral:
                    "Repo location: \(String(describing: repo?.folder.url))"
            )
        )
    }

}

Here’s the code for the hasJujutstuRepo function.

func hasJujutsuRepo(location: URL) -> Bool {
    let repoLocation = location.appendingPathComponent(
        ".jj",
        conformingTo: .folder
    )

    do {
        let resourceValues = try repoLocation.resourceValues(forKeys: [
            .isDirectoryKey
        ])
        if let isFolder = resourceValues.isDirectory {
            if isFolder {
                return true
            }
        }
    } catch {
        return false
    }
    return false
}

When I run the test, Xcode’s console prints that the repo was created, but the test fails. The comment shows the correct location for the repo created in the test. If I set a breakpoint inside the hasJujutsuRepo function, the test passes.

I tried removing the async from the test, but the test still fails unless I set the breakpoint.

What do I have to do to get the test to pass without setting a breakpoint?

1 Like
2

What’s in the createRepo(location: )?

3

This sounds like a timing issue and probably depends on how the repo/folder is actually created. What does the createRepo method do? Is it possible that it creates an instance, but the actual folder creation is delayed? If it invokes an external command (something like git init it could be that there’s some additional delay between that returning and the system actually creating the folder.

The reason you then see the test pass with a breakpoint would be that you’re literally waiting, i.e. giving it more time to “catch up”. The correct way to address this would then not to somehow make the test pass, but fix the production code to not prematurely return (or provide some other API to inform a caller that the operation has completed).

5 Likes
4

Here’s the code for the createRepo function.

func createRepo(location: URL) -> Repo? {
    let task = Process()
    let bundle = Bundle.main
    let resourceFolder = bundle.resourceURL
    task.executableURL = resourceFolder?.appendingPathComponent("jj")
    task.currentDirectoryURL = location.deletingLastPathComponent()
    task.arguments = ["git", "init", "--colocate", location.lastPathComponent]

    do {
        try task.run()
        return Repo(folder: Folder(url: location))
    } catch {
        print(error)
        return nil
    }
}

struct Repo: Hashable, Codable {
    var folder: Folder

    func hash(into hasher: inout Hasher) {
        hasher.combine(folder)
    }
}

struct Folder: Hashable, Codable {
    var url: URL
    var displayName: String {
        url.path.components(separatedBy: "/").filter { !$0.isEmpty }.last ?? ""
    }
}

The code uses the Process class to run the jj git init command to create the repo. The jj binary is bundled with the app.

5

Maybe let's start here… there's no "one right" opinion here but an ideal unit test should be cheap, fast, and free of side effects outside of the specific side effects you want to test. If a "unit test" makes production calls to a database, network, or filesystem you are IMO trending in the direction of "integration" test. Which isn't necessarily a "bad" thing… integration tests have their place.

My POV is your dependency on Process — and also maybe Bundle — should be injected with a test double implementation. You don't really need to test that Process works correctly… you need to test that you are using the APIs on Process correctly.

6

Unfortunately that means you need to actually use Process, unless you somehow know all of its behaviors and can encode them in the double. What you're actually testing is the input and output of the Process integration.

But ultimately yes, Process should be abstracted over in some way so you don't actually call into git during tests. You can then write separate tests for the actual Process integration to ensure you're calling it correctly, and you can tailor those tests to eliminate breaking patterns.

7

Process.run is asynchronous. Either pass in a terminationHandler or use waitUntilExit.

8

I'm not sure I agree this needs any knowledge of Process implementation details. We "spy" on the API calls and parameters passed and "stub" simple return values back from those calls.

When I said "test double" I did not mean a box or layer of indirection. I do not mean for these "test double" APIs to forward through to a legit Process implementation. I mean for these test double APIs to replace a legit Process implementation.

9

You probably also want to handle error return values. Process.run only throws for errors like executableURL doesn't point to an executable file. If the process returns a non-zero exit code, that will be signaled in the terminationStatus of the Process after it terminated.

10

Thank you for the tip on using waitUntilExit. Adding that call after calling task.run made the test pass with no breakpoint set.

Thank you to the people who suggested abstracting the call to Process so I’m not actually making the version control calls during the tests. I will do that and avoid future problems unit testing version control commands.

11

I would argue that these kind of tests have 0 value. You are testing that you successfully copy pasted your implementation detail into your tests.
If you are testing ~0 work/execution time logic, there is no reason to over-complicate tests with making them depend on the implementation details.
If you are testing non-zero work/execution time logic and you want to be able to run those tests in ~0 time then you want to have a way to switch between running the tests with and without mocks.