How do I set up a performance test suite?

Using Swift
, , ,
1

Excuse me in advance if Xcode questions are off topic because they're not strictly related to the open source parts of the Swift ecosystem.

My goal

I have a macOS application with a performance problem. I want to be a good boy, and be data-driven in my attempt to remedy that, by using performance tests to benchmark my progress.

In order for the benchmark to be worth anything, both the calling code (the unit test bundle) and the target (the Mac app) should be compiled in release mode, with optimizations on, right?

Well running regular unit tests in debug mode is quite handy (symbols for debugging, faster compiles, etc.). So it looks like I should make a new unit test bundle just for performance tests, is that right (question 1)?

Enabling optimizations

Assuming my premise in question 1 is correct, I added a new unit test bundle target to my project. I employed Ben Rimmington's trick to use an assertion to ensure optimizations are on:

override func setUp() throws {
    assert(false, "The performance tests aren't being run with optimizations on!")
}

I ran the XCTestCase with the diamond button in the left gutter, and sure enough, the assertion traps. I went into the Build Settings for the target, and set Swift Compiler - Code Generation > Optimization Level > Debug> to Optimize for Speed [-O]. Reran the tests, and that assertion no longer traps (it's being optimized out, woohoo!)

Now I added the same assertion to my app's module, in a public global method called verifyCompilerOptimizationsAreOn. I ran this from my performance testcase, and it traps.

Question 2: How do I tell Xcode that the FooAppPerformanceTests module should depend on a release build of FooApp?

App launches during unit tests

Also, I've noticed that running unit tests causes the target mac app to launch and exit briefly.

  1. Why is that? I would have expected that only UI tests would need to run the app for real, and that unit tests would just run the relevant code directly, in isolation from a running app.
  2. I can disable that behavior by setting Host Application to None in the general section of the target's settings, but then I get linker errors telling my that the symbols under test are no longer found.

Question 3: Is there a way to turn that behavior off, while still being able to run the test targets? The flash of the app is pretty annoying, though admittedly it's a nuisance I can live with it if I must.

2

You're right in saying that the "host application" is your issue here: when that setting is on the app does run and the test is ran inside of it. If you run the tests as standalone you need the code that you're testing as well in the target, as there's no app to piggyback on, so check if your code-to-be-tested is compiled in there

3

Hey Davide, thanks for take the time to respond :slight_smile:

To make sure I'm following: are you suggesting that I check off the test target in the "target membership" part of the sidebar, for every file under test?

Is there a way to depend on the app target as if it were a library, so that I can depend on the symbols in the binary, without manually pulling in all the source files to build in the test target? Of course I guess I could extract the code into a library, and pull that into both the app and the test target.

4

Yes, I intended to move all the code to the test target as you mention. And a possible workaround I was going to propose is to use a library for the code as you guessed :D