Prevent optimizer from culling my benchmark

AlexanderM · May 16, 2020, 6:01pm

I'm trying to write a benchmark to quantify the performance of a slow NSOutlineView that I'm trying to optimize. I think the optimizer is outsmarting me, and trying to erase my benchmark (since the sorted data isn't used).

How can I confirm that my benchmark really is testing the sort performance, without slowing my test down?

This is what I have so far:

func testSortByColumnPerformance() {
    class MockOutlineView: NSOutlineView {
        init(sortDescriptors: [ColumnIDs]) {
            super.init(frame: .zero)
            self.sortDescriptors = sortDescriptors.map { $0.sortDescriptor }
        }
        
        required init?(coder: NSCoder) {
            fatalError("init(coder:) has not been implemented")
        }
    }
    
    let mocks = [
        MockOutlineView(sortDescriptors: [.a]),
        MockOutlineView(sortDescriptors: [.a, .b]),
        MockOutlineView(sortDescriptors: [.a, .b, .c]),
        MockOutlineView(sortDescriptors: [.a, .b, .c, .d]),
        MockOutlineView(sortDescriptors: [.a, .b, .c, .d, .e]),
    ]
    
    let options = XCTMeasureOptions()
    options.invocationOptions = [.manuallyStart, .manuallyStop]
    options.iterationCount = 100
    
    self.measure(options: options) {
        let dataSource = configureMyDataSource()
                            
        self.startMeasuring()
        dataSource.outlineView(mocks[0], sortDescriptorsDidChange: [])
        dataSource.outlineView(mocks[1], sortDescriptorsDidChange: mocks[0].sortDescriptors)
        dataSource.outlineView(mocks[2], sortDescriptorsDidChange: mocks[1].sortDescriptors)
        dataSource.outlineView(mocks[3], sortDescriptorsDidChange: mocks[2].sortDescriptors)
        dataSource.outlineView(mocks[4], sortDescriptorsDidChange: mocks[3].sortDescriptors)
        self.stopMeasuring()
                            
        blackHole(dataSource.data)
    }
}

Other context:

I'm building both the performance test bundle target and app target in release mode (confirmed with assert(false) statements that don't crash.

I copied the blackHole and identity functions from here, and put them in my app target.

// Just consume the argument.
// It's important that this function is in another module than the tests
// which are using it.
@inline(never)
public func blackHole<T>(_ x: T) {
}

// Return the passed argument without letting the optimizer know that.
@inline(never)
public func identity<T>(_ x: T) -> T {
  return x
}

P.S. Would it be possible to bake these functions right into the standard library? I know the standard library is privileged by the optimizer in some ways, but it is it possible to exclude a function like this, to make sure the black hole really is black?