XCTestCase.measure() is very slow due to symbolication. How to make it faster?

Using Swift
1

For example this trivial testPerformance takes over 2 minutes at 100% CPU:

class MyTests: XCTestCase {
    
    func testPerformance() {
        measure {
            let _  = 1 + 1
        }
    }
}

The problem seems to be that constructing a XCSymbolicationRecord when reporting the test result is very slow in my setup. I'm not sure why this is ... generally working with my project in Xcode is fast. For example I can step thought project code in debugger without any big delays.

My project does include a big Rust library... maybe that's causing the problem? Is there anything I can do to make performance tests usable for my project?

Here's the stack whenever when I pause running the tests:

Thread 1 Queue : com.apple.main-thread (serial)
#0	0x00007fff68ae86a2 in __open ()
#1	0x0000000102ca2a34 in CSCppFileMemory::CSCppFileMemory(char const*, TRange<Pointer64>, unsigned int) ()
#2	0x0000000102ca01cb in new_partial_file_memory(char const*, unsigned long long, unsigned int, unsigned long long (CSCppFileMemory*) block_pointer) ()
#3	0x0000000102ca14eb in iterate_symbol_owners_from_archive(char const*, unsigned long long, unsigned int, CSCppArchitecture const&, char const*, bool, void (CSCppSymbolOwner*) block_pointer) ()
#4	0x0000000102ca04ef in iterate_symbol_owners_from_memory(CSCppMemory*, char const*, unsigned int, unsigned long long, unsigned int, CSCppArchitecture const&, char const*, bool, void (CSCppSymbolOwner*) block_pointer) ()
#5	0x0000000102ca0d46 in iterate_symbol_owners_from_path(char const*, bool, unsigned int, CSCppArchitecture const&, bool, void (CSCppSymbolOwner*) block_pointer) ()
#6	0x0000000102c907bf in CSSymbolicatorCreateWithPathArchitectureFlagsAndNotification ()
#7	0x0000000102c89133 in void extract_debug_maps_from_header<SizeAndEndianness<Pointer64, LittleEndian> >(TExtendedMachOHeader<SizeAndEndianness<Pointer64, LittleEndian> >&, TRawSymbolOwnerData<SizeAndEndianness<Pointer64, LittleEndian>::SIZE>&, TNList<SizeAndEndianness<Pointer64, LittleEndian> > const*, unsigned int, TRange<SizeAndEndianness<Pointer64, LittleEndian>::SIZE>) ()
#8	0x0000000102cc3be0 in TRawSymbolOwnerData<SizeAndEndianness<Pointer64, LittleEndian>::SIZE>* create_traw_symbol_owner_data_arch_specific<SizeAndEndianness<Pointer64, LittleEndian> >(CSCppSymbolOwner*, CSCppDsymData*) ()
#9	0x0000000102c7e69a in CSCppSymbolOwnerData* create_symbol_owner_data_arch_specific<SizeAndEndianness<Pointer64, LittleEndian> >(CSCppSymbolOwner*, CSCppDsymData*) ()
#10	0x0000000102c7e16d in create_symbol_owner_data2(CSCppSymbolOwner*, CSCppDsymData*) ()
#11	0x0000000102c7dd2c in CSCppSymbolOwnerCache::create_symbol_owner_data(CSCppSymbolOwner*, CSCppDsymData*) ()
#12	0x0000000102c7d6af in CSCppSymbolOwnerCache::data_for_symbol_owner(CSCppSymbolOwner*) ()
#13	0x0000000102c7d656 in CSCppSymbolOwner::data() ()
#14	0x0000000102c83ff3 in CSSymbolicatorGetSymbolWithAddressAtTime ()
#15	0x0000000102a769e1 in +[XCSymbolicationRecord symbolicationRecordForAddress:] ()
#16	0x0000000102a2798b in -[XCTestCase _symbolicationRecordForTestCodeInAddressStack:] ()
#17	0x0000000102a293ac in -[XCTestCase reportMetric:reportFailures:] ()
#18	0x0000000102a2b9db in -[XCTestCase measureMetrics:automaticallyStartMeasuring:forBlock:] ()
#19	0x0000000102a25a9b in -[XCTestCase measureBlock:] ()
#20	0x0000000105002c77 in BikeServiceTests.testPerformance() at /Users/jessegrosjean/Documents/github/Bike/clients/bike.swift/Common Tests/BikeServiceTests.swift:29
...
2

FWIW I filed this back in May as FB7698032 CoreSymbolification extremely slow with large archive file, and it's still an issue with Xcode 12.2. I haven't tried Xcode 12.3 yet. I'm seeing millions of open/mmap/close on a single .a file.

3

Is this issue never going to get fixed? There is no way to accurately use XCTTest's performance measurements. It's been like this for years now.

4

It might not be exactly what you ask for, but you might try out GitHub - ordo-one/package-benchmark: Swift benchmark runner as a SwiftPM command plug-in with CI support. as an alternative and see if it can provide you with some utility (shameless plug) - it won't give you backtrace info though, just running time (and some more metrics).