I'm trying to write a benchmark to quantify the performance of a slow NSOutlineView
that I'm trying to optimize. I think the optimizer is outsmarting me, and trying to erase my benchmark (since the sorted data isn't used).
How can I confirm that my benchmark really is testing the sort performance, without slowing my test down?
This is what I have so far:
func testSortByColumnPerformance() {
class MockOutlineView: NSOutlineView {
init(sortDescriptors: [ColumnIDs]) {
super.init(frame: .zero)
self.sortDescriptors = sortDescriptors.map { $0.sortDescriptor }
}
required init?(coder: NSCoder) {
fatalError("init(coder:) has not been implemented")
}
}
let mocks = [
MockOutlineView(sortDescriptors: [.a]),
MockOutlineView(sortDescriptors: [.a, .b]),
MockOutlineView(sortDescriptors: [.a, .b, .c]),
MockOutlineView(sortDescriptors: [.a, .b, .c, .d]),
MockOutlineView(sortDescriptors: [.a, .b, .c, .d, .e]),
]
let options = XCTMeasureOptions()
options.invocationOptions = [.manuallyStart, .manuallyStop]
options.iterationCount = 100
self.measure(options: options) {
let dataSource = configureMyDataSource()
self.startMeasuring()
dataSource.outlineView(mocks[0], sortDescriptorsDidChange: [])
dataSource.outlineView(mocks[1], sortDescriptorsDidChange: mocks[0].sortDescriptors)
dataSource.outlineView(mocks[2], sortDescriptorsDidChange: mocks[1].sortDescriptors)
dataSource.outlineView(mocks[3], sortDescriptorsDidChange: mocks[2].sortDescriptors)
dataSource.outlineView(mocks[4], sortDescriptorsDidChange: mocks[3].sortDescriptors)
self.stopMeasuring()
blackHole(dataSource.data)
}
}
Other context:
- I'm building both the performance test bundle target and app target in release mode (confirmed with
assert(false)
statements that don't crash. - I copied the
blackHole
andidentity
functions from here, and put them in my app target.// Just consume the argument. // It's important that this function is in another module than the tests // which are using it. @inline(never) public func blackHole<T>(_ x: T) { } // Return the passed argument without letting the optimizer know that. @inline(never) public func identity<T>(_ x: T) -> T { return x }
P.S. Would it be possible to bake these functions right into the standard library? I know the standard library is privileged by the optimizer in some ways, but it is it possible to exclude a function like this, to make sure the black hole really is black?