How do you run performance tests on Swift packages in Xcode?

Using Swift
, ,
1

I've been trying to run a performance test on a Swift package in Xcode for about an hour now, but I can't seem to get the "No baseline average for Time" message to go away. I've updated the baseline value through Xcode's UI multiple times and it just doesn't seem to stick. Is this issue unique to Swift packages or just a more general bug in Xcode?

Thanks!

3 Likes
2

The strange thing is that I can confirm setting a baseline average in Xcode adds the following two files to the Swift package:

.swiftpm/xcode/xcshareddata/xcbaselines/{Package Name}.xcbaseline/{UUID}.plist
.swiftpm/xcode/xcshareddata/xcbaselines/{Package Name}.xcbaseline/Info.plist

The former looks something like the following:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>classNames</key>
        <dict>
                <key>PackageTests</key>
                <dict>
                        <key>testPerformance()</key>
                        <dict>
                                <key>com.apple.XCTPerformanceMetric_WallClockTime</key>
                                <dict>
                                        <key>baselineAverage</key>
                                        <real>0.00903</real>
                                        <key>baselineIntegrationDisplayName</key>
                                        <string>Local Baseline</string>
                                </dict>
                        </dict>
                </dict>
        </dict>
</dict>
</plist>

And here's Info.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>runDestinationsByUUID</key>
        <dict>
                <key>D36CD09E-8356-41EE-9A77-1F6891B15F09</key>
                <dict>
                        <key>localComputer</key>
                        <dict>
                                <key>busSpeedInMHz</key>
                                <integer>…</integer>
                                <key>cpuCount</key>
                                <integer>…</integer>
                                <key>cpuKind</key>
                                <string>…</string>
                                <key>cpuSpeedInMHz</key>
                                <integer>…</integer>
                                <key>logicalCPUCoresPerPackage</key>
                                <integer>…</integer>
                                <key>modelCode</key>
                                <string>…</string>
                                <key>physicalCPUCoresPerPackage</key>
                                <integer>…</integer>
                                <key>platformIdentifier</key>
                                <string>com.apple.platform.macosx</string>
                        </dict>
                        <key>targetArchitecture</key>
                        <string>x86_64</string>
                </dict>
        </dict>
</dict>
</plist>

Even though Xcode automatically created these files for me, it doesn't appear to realize that it did:

No baseline average for Time

Is this a known issue with how Xcode handles performance tests for Swift packages or am I just misunderstanding how they're supposed to work?

1 Like
3

I still can't seem to get this to work. Just out of curiosity, how does everyone else run performance tests on Swift packages?

4

FWIW I'm having the same issue on Xcode 12.3

1 Like
5

Thanks for confirming—I'm glad to know I'm not the only one! (For the record, I last tested this on Xcode 12.4.)

1 Like
6

Same behaviour with Xcode 13 beta 2.

1 Like
7

And with Xcode 13.4.

I tried both with SPM and generating an Xcode project - same issue with both, seems amnesia has hit.

8

My suspicion is that if Xcode is opening a package rather than a project file, it doesn't have anywhere to store the results. Hence the amnesia.

When I was wanting them retained, I'd use the command line to create an Xcode project file and use that. However that approach is now being phased out.

9

But Xcode is storing other stuff in .swiftpm, not sure it's the best place, but performance baselines could be there:

hassila@ice ~/G/test-release-package (main)> ls .swiftpm/xcode
package.xcworkspace/ xcuserdata/
10

In fact, it is already being stored there. I just created a dummy package with a couple of performance tests, and in a subfolder of .swiftpm/xcode/xcshareddata/xcbaselines is a file (with random generated name) containing the following:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>classNames</key>
	<dict>
		<key>MyLibraryTests</key>
		<dict>
			<key>testCalc1()</key>
			<dict>
				<key>com.apple.XCTPerformanceMetric_WallClockTime</key>
				<dict>
					<key>baselineAverage</key>
					<real>0.032604</real>
					<key>baselineIntegrationDisplayName</key>
					<string>Local Baseline</string>
				</dict>
			</dict>
			<key>testCalc2()</key>
			<dict>
				<key>com.apple.XCTPerformanceMetric_WallClockTime</key>
				<dict>
					<key>baselineAverage</key>
					<real>0.033293</real>
					<key>baselineIntegrationDisplayName</key>
					<string>Local Baseline</string>
				</dict>
			</dict>
		</dict>
	</dict>
</dict>
</plist>

The problem is even though Xcode was writing the data there, it doesn't seem to think of looking there for previous results. :face_with_monocle:

1 Like
11

Unfortunately, the issue can be still observed in Xcode Version 14.0 beta 4.

2 Likes
12

Here is another suggestion:

1 Like