I've captured another run, with a longer file, from the 6/14 toolchain and the trace makes much more sense. I've uploaded it here. This is direct from swift-frontend.
1 Like
Looks like majority of time is spect trying to lookup members. Could you please capture a trace for the same code using Xcode 12.5 toolchain?
I've updated to Xcode 12.5.1 with the 5.4.2 compiler. Those debug symbols aren't available yet, are they?
I'm not sure.
I installed the symbolicated 5.4.1 toolchain but I can't run swift-frontend out of it due to an error: <unknown>:0: error: unable to load standard library for target 'x86_64-apple-macosx11.0'.
It's okay, no worries! It looks like the most expensive part of the lookup is access checking, you can try running the benchmark with -disable-access-control -disable-availability-checking to see how much would that improve the results.
Xcode 12.5.1:
Benchmark #1: xcrun swiftc -Xfrontend -disable-access-control -Xfrontend -disable-availability-checking -disallow-use-new-driver -typecheck a.swift
Time (mean ± σ): 59.6 ms ± 1.0 ms [User: 41.4 ms, System: 15.6 ms]
Range (min … max): 58.0 ms … 62.5 ms 48 runs
Benchmark #1: xcrun swiftc -Xfrontend -disable-access-control -Xfrontend -disable-availability-checking -disallow-use-new-driver -typecheck b.swift
Time (mean ± σ): 102.4 ms ± 1.4 ms [User: 83.1 ms, System: 16.5 ms]
Range (min … max): 100.3 ms … 105.6 ms 28 runs
Benchmark #1: xcrun swiftc -Xfrontend -disable-access-control -Xfrontend -disable-availability-checking -disallow-use-new-driver -typecheck c.swift
Time (mean ± σ): 443.9 ms ± 5.3 ms [User: 412.2 ms, System: 29.1 ms]
Range (min … max): 436.8 ms … 453.9 ms 10 runs
Benchmark #1: xcrun swiftc -Xfrontend -disable-access-control -Xfrontend -disable-availability-checking -disallow-use-new-driver -typecheck d.swift
Time (mean ± σ): 94.8 ms ± 1.5 ms [User: 74.9 ms, System: 17.4 ms]
Range (min … max): 92.1 ms … 97.7 ms 30 runs
Xcode 13b1:
Benchmark #1: xcrun swiftc -Xfrontend -disable-access-control -Xfrontend -disable-availability-checking -disallow-use-new-driver -typecheck a.swift
Time (mean ± σ): 64.1 ms ± 0.9 ms [User: 45.3 ms, System: 15.4 ms]
Range (min … max): 62.3 ms … 65.9 ms 44 runs
Benchmark #1: xcrun swiftc -Xfrontend -disable-access-control -Xfrontend -disable-availability-checking -disallow-use-new-driver -typecheck b.swift
Time (mean ± σ): 108.4 ms ± 1.5 ms [User: 88.5 ms, System: 16.8 ms]
Range (min … max): 105.8 ms … 112.5 ms 27 runs
Benchmark #1: xcrun swiftc -Xfrontend -disable-access-control -Xfrontend -disable-availability-checking -disallow-use-new-driver -typecheck c.swift
Time (mean ± σ): 532.7 ms ± 4.7 ms [User: 498.2 ms, System: 31.6 ms]
Range (min … max): 522.3 ms … 539.2 ms 10 runs
Benchmark #1: xcrun swiftc -Xfrontend -disable-access-control -Xfrontend -disable-availability-checking -disallow-use-new-driver -typecheck d.swift
Time (mean ± σ): 99.2 ms ± 1.8 ms [User: 78.6 ms, System: 18.2 ms]
Range (min … max): 96.9 ms … 106.2 ms 29 runs
6/14 5.5 Toolchain:
Benchmark #1: /Library/Developer/Toolchains/swift-5.5-DEVELOPMENT-SNAPSHOT-2021-06-14-a.xctoolchain/usr/bin/swiftc -Xfrontend -disable-access-control -Xfrontend -disable-availability-checking -disallow-use-new-driver -typecheck a.swift
Time (mean ± σ): 79.0 ms ± 1.1 ms [User: 57.1 ms, System: 16.3 ms]
Range (min … max): 76.7 ms … 81.2 ms 37 runs
Benchmark #1: /Library/Developer/Toolchains/swift-5.5-DEVELOPMENT-SNAPSHOT-2021-06-14-a.xctoolchain/usr/bin/swiftc -Xfrontend -disable-access-control -Xfrontend -disable-availability-checking -disallow-use-new-driver -typecheck b.swift
Time (mean ± σ): 129.6 ms ± 1.4 ms [User: 106.6 ms, System: 19.0 ms]
Range (min … max): 126.7 ms … 132.4 ms 22 runs
Benchmark #1: /Library/Developer/Toolchains/swift-5.5-DEVELOPMENT-SNAPSHOT-2021-06-14-a.xctoolchain/usr/bin/swiftc -Xfrontend -disable-access-control -Xfrontend -disable-availability-checking -disallow-use-new-driver -typecheck c.swift
Time (mean ± σ): 662.6 ms ± 5.4 ms [User: 623.5 ms, System: 33.9 ms]
Range (min … max): 655.7 ms … 672.9 ms 10 runs
Benchmark #1: /Library/Developer/Toolchains/swift-5.5-DEVELOPMENT-SNAPSHOT-2021-06-14-a.xctoolchain/usr/bin/swiftc -Xfrontend -disable-access-control -Xfrontend -disable-availability-checking -disallow-use-new-driver -typecheck d.swift
Time (mean ± σ): 118.6 ms ± 1.9 ms [User: 95.0 ms, System: 18.6 ms]
Range (min … max): 115.6 ms … 122.5 ms 24 runs
Local release build, no assertions:
Benchmark #1: /Users/jshier/Desktop/Code/swift/build/Ninja-Release/swift-macosx-x86_64/bin/swiftc -Xfrontend -disable-access-control -Xfrontend -disable-availability-checking -disallow-use-new-driver -typecheck a.swift
Time (mean ± σ): 60.1 ms ± 0.7 ms [User: 47.0 ms, System: 10.3 ms]
Range (min … max): 58.3 ms … 61.7 ms 48 runs
Benchmark #1: /Users/jshier/Desktop/Code/swift/build/Ninja-Release/swift-macosx-x86_64/bin/swiftc -Xfrontend -disable-access-control -Xfrontend -disable-availability-checking -disallow-use-new-driver -typecheck b.swift
Time (mean ± σ): 110.2 ms ± 1.3 ms [User: 96.1 ms, System: 11.2 ms]
Range (min … max): 108.0 ms … 113.7 ms 26 runs
Benchmark #1: /Users/jshier/Desktop/Code/swift/build/Ninja-Release/swift-macosx-x86_64/bin/swiftc -Xfrontend -disable-access-control -Xfrontend -disable-availability-checking -disallow-use-new-driver -typecheck c.swift
Time (mean ± σ): 557.6 ms ± 5.7 ms [User: 528.8 ms, System: 25.2 ms]
Range (min … max): 544.9 ms … 565.3 ms 10 runs
Benchmark #1: /Users/jshier/Desktop/Code/swift/build/Ninja-Release/swift-macosx-x86_64/bin/swiftc -Xfrontend -disable-access-control -Xfrontend -disable-availability-checking -disallow-use-new-driver -typecheck d.swift
Time (mean ± σ): 99.1 ms ± 1.3 ms [User: 84.2 ms, System: 12.2 ms]
Range (min … max): 97.2 ms … 102.8 ms 29 runs
Yeah, I'm trying to figure out why is so much time spent in TypeChecker::lookupMember everything else looks reasonable to me.
2 Likes
makes sense, and a good rule of thumb to remember, thanks!
1 Like
Is this still an issue today?
In general yes, but the specifics change over releases. The compiler's performance regresses a bit every release (at least as measured through Xcode), though specific things may improve. 5.9 especially regressed, both in CPU performance and overall memory usage, though at least part of that are bugs in Xcode and other tools like the iOS simulators. For my small test cases, the improvements to linking in Xcode 15 (great!) were more than offset by the loss in compiler performance (aww), so overall build performance dropped.
3 Likes
I think the experiment is not valid because the cases have different amount of content per file. It's better to reduce the number of generated lines per file and increase the number of runs (default is 10).
Here is how it looks (note number of "lines, number of runs" settings):
~/Work/_TMP/swift_performance_test $ swift test_swift_string.swift
=============================================
Testing: a = "...".
Using 5000 lines per file. Number of runs: 10
🐢 shell: hyperfine --warmup 1 -m 10 'xcrun swiftc -typecheck a.swift'
Benchmark 1: xcrun swiftc -typecheck a.swift
Time (mean ± σ): 271.6 ms ± 0.8 ms [User: 240.6 ms, System: 26.5 ms]
Range (min … max): 270.3 ms … 272.9 ms 10 runs
=============================================
Testing: a: String = "...".
Using 5000 lines per file. Number of runs: 10
🐢 shell: hyperfine --warmup 1 -m 10 'xcrun swiftc -typecheck b.swift'
Benchmark 1: xcrun swiftc -typecheck b.swift
Time (mean ± σ): 1.936 s ± 0.078 s [User: 1.900 s, System: 0.029 s]
Range (min … max): 1.879 s … 2.117 s 10 runs
~/Work/_TMP/swift_performance_test $ swift test_swift_string.swift
=============================================
Testing: a = "...".
Using 1 lines per file. Number of runs: 100
🐢 shell: hyperfine --warmup 1 -m 100 'xcrun swiftc -typecheck a.swift'
Benchmark 1: xcrun swiftc -typecheck a.swift
Time (mean ± σ): 60.9 ms ± 1.5 ms [User: 45.8 ms, System: 11.4 ms]
Range (min … max): 59.3 ms … 73.4 ms 100 runs
Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
=============================================
Testing: a: String = "...".
Using 1 lines per file. Number of runs: 100
🐢 shell: hyperfine --warmup 1 -m 100 'xcrun swiftc -typecheck b.swift'
Benchmark 1: xcrun swiftc -typecheck b.swift
Time (mean ± σ): 60.8 ms ± 0.8 ms [User: 45.9 ms, System: 11.4 ms]
Range (min … max): 59.5 ms … 64.1 ms 100 runs
Here is the script:
import Foundation
let filenames = ["a", "b"]
let info = ["a = \"...\"", "a: String = \"...\""]
let codes = [
"let a{} = \"hello, world!\"",
"let d{}: String = \"hello, world!\""
]
let lines = 1
let numberOfRuns = 100
for(i, name) in filenames.enumerated() {
var str = ""
for j in 0..<lines {
str += codes[i].replace("{}", withString: "\(j)") + "\n"
}
let data: Data = str.data(using: .utf8)!
let filename = "\(name).swift"
FileManager.default.createFile(atPath: "\(name).swift", contents: data, attributes: nil)
print("=============================================")
print("Testing: \(info[i]).\nUsing \(lines) lines per file. Number of runs: \(numberOfRuns)\n")
print(shell("hyperfine --warmup 1 -m \(numberOfRuns) 'xcrun swiftc -typecheck \(filename)'"))
}
public func shell(_ command: String) -> String {
print("🐢 shell: \(command)")
let task = Process()
let pipe = Pipe()
task.standardOutput = pipe
task.standardError = pipe
task.arguments = ["-c", command]
task.launchPath = "/bin/zsh"
task.standardInput = nil
task.launch()
let data = pipe.fileHandleForReading.readDataToEndOfFile()
let output = String(data: data, encoding: .utf8)!
return output
}
extension String {
public func replace(_ target: String, withString: String) -> String {
return self.replacingOccurrences(
of: target, with: withString,
options: NSString.CompareOptions.literal,
range: nil
)
}
}
While googling this I also found this which seems to say that using .init() is faster. But that article is using a struct rather than a String.