We also hit the linker error in this toolchain. 5 classes weren't visible for Objective-C when they should have been. Seen in all compilation modes.
Fascinating! Thank you so much for trying batch mode and reporting the results. I'm wondering what the clean build times are when they are broken down by module. Is batch slower than WMO on clean builds in every module?
Thank you for measuring and reporting. Great to read that you're seeing an improvement.
I believe we now have fixes in master for all the failures observed here; I'll ask about getting another snapshot for testing. Thanks everyone for being patient testing this!
1 Like
When we get the next snapshot I'll be sure to time clean builds per module instead of overall.
Thanks!
Hi,
I tried batch mode again today and it worked on my project 
So this is the results all the test have been run multiple times and from a clean build.
| Xcode | Toolchain | Mode | Time (seconds) |
|---|---|---|---|
| 9.3 | swift-DEVELOPMENT-SNAPSHOT-2018-04-21-a-osx | wmo | 175 |
| 9.3 | swift-DEVELOPMENT-SNAPSHOT-2018-04-21-a-osx | single file | 472 |
| 9.3 | swift-DEVELOPMENT-SNAPSHOT-2018-04-21-a-osx | batch | 369 |
| 9.3 | default | wmo | 121 |
| 9.3 | default | single file | 272 |
| 9.2 | default | wmo | 128 |
| 9.2 | default | single file | 454 |
Project CLOC
Main project
| Language | files | blank | comment | code |
|---|---|---|---|---|
| Swift | 799 | 19904 | 13022 | 55708 |
| Objective C | 460 | 18075 | 6339 | 53118 |
| JSON | 1129 | 0 | 0 | 24488 |
| C/C++ Header | 524 | 5666 | 9764 | 6387 |
| XML | 7 | 0 | 58 | 1384 |
| Markdown | 5 | 86 | 0 | 138 |
| -------- | -------- | -------- | -------- | -------- |
| SUM | 2924 | 43731 | 29183 | 141223 |
Modules
| Language | files | blank | comment | code |
|---|---|---|---|---|
| Objective C | 812 | 18578 | 11220 | 76644 |
| C/C++ Header | 1641 | 31524 | 62143 | 76259 |
| Swift | 570 | 13454 | 22316 | 47789 |
| JSON | 768 | 2 | 0 | 44738 |
| Objective C++ | 115 | 7971 | 5210 | 37815 |
| HTML | 39 | 678 | 0 | 13003 |
| C++ | 34 | 1593 | 1307 | 9264 |
| Markdown | 60 | 2659 | 0 | 6792 |
| Bourne Shell | 7 | 297 | 187 | 2369 |
| JavaScript | 3 | 175 | 429 | 973 |
| CSS | 2 | 42 | 59 | 436 |
| C | 1 | 103 | 45 | 389 |
| YAML | 1 | 0 | 0 | 5 |
| Bourne Again Shell | 1 | 2 | 1 | 3 |
| -------- | -------- | -------- | -------- | -------- |
| SUM | 4054 | 77078 | 102917 | 316479 |
Hope it helps,
1 Like
Seems tough to evaluate batch mode when the other compilation modes have regressed so severely. There are Swift performance tests that run for most PRs, and there are smaller compiler performance tests as well. Are there no full build tests that would indicate regressions like this?
I don't think this shows a regression. Without checking, it's likely that the difference is due to the toolchain being compiled with asserts on.
Ah, I thought they were still producing special builds for batch mode testing.
@Graydon_Hoare Can we get a new snapshot so we can run our tests as well? Thanks!
FWIW I’ve tested this successfully on the 4.2 branch snapshots.
1 Like
Agreed with @Keith the lasts snapshots works pretty well.
1 Like
Some early tests against our app:
- 5.6K Swift files, 590 KLOC
- 3.6K ObjC files, 250 KLOC
- 90 modules of different sizes
- 4 core machine (8 virtual)
Results (against 4.2 Snapshot 2018-04-23):
- wmo: 501 seconds
- single: 1,389 seconds
- batch: 520 seconds
I will also re-run these tests on a 2-core MacMini (our build machine) and update the report.
Thanks for working on improving builds perf!
3 Likes
Thanks for your testing and feedback everyone! Apologies in the delay.
Thanks to @mishal_shah there is now an updated no-asserts, LTO compiler snapshot available for testing with batch mode.
Several of the link-time bugs reported above have been fixed, as well as an issue with the way the driver consumes subprocess output; the latter adversely influenced performance when combined with some common diagnostic flags (in particular -Xfrontend -debug-time-function-bodies, which seems to be widely used). So hopefully this version is more stable and better-performing.
Please keep sharing your experiences trying it out! It's very helpful in evaluating how stable the feature is, and tracking down remaining corner cases in its behaviour.
(Also worth noting: if you're trying this under Xcode, there's a known issue with the way existing releases of Xcode process diagnostics coming back from a compiler running in batch mode, such that it's not entirely convenient to use this snapshot day-to-day yet: some diagnostics are not parsed correctly, and you'll wind up with either missing warnings or "compiler exited with nonzero status" in place of properly-parsed errors / issues in the Xcode UI. This is not a permanent state of affairs, just an artifact of using a newer Swift toolchain with an older Xcode.)
1 Like
Project size: 64'391 LOC of Swift over 666 files and 20 modules
Number of cores: 4
Standard mode compile time: 61s (3 samples)
Whole-module mode compile time: 52s (3 samples)
Batch mode compile time: 44s (3 samples)
Lines of Code
Language files blank comment code
--------------------------------------------------------------------------
Swift 1030 14704 1943 92383
Objective C 255 6684 1426 33411
C/C++ Header 265 1803 993 4014
--------------------------------------------------------------------------
SUM: 1550 23191 4362 129808
Timings:
| iMac Pro (10-core) | MBP (4-core)
--------|--------------------|--------------
WMO | 164.956 | 225.950
Batch | 137.533 | 188.954
This time includes a script that builds another project without these settings. (I'd guess it's 15-30s of the total.)
Our project doesn't build in single file mode (even with the default toolchain.) Some type names resolve differently, which causes errors.
This round is surprising! Definitely getting some improvements.
4.1.x from Xcode 9.4 beta 2 - debug - wholemodule: 168.38s
05-07 noassert toolchain - debug - wholemodule: 112.200s
05-07 noassert toolchain - debug - singlefile: 262.120s
05-07 noassert toolchain - debug - batch: 173.596s
I used the same code as before with just the toolchain change with swift-DEVELOPMENT-SNAPSHOT-NO_ASSERTIONS-LTO-2018-05-02-a.
Updated test results
| Xcode | Toolchain | Mode | Time (seconds) |
|---|---|---|---|
| 9.3 | swift-DEVELOPMENT-SNAPSHOT-NO_ASSERTIONS-LTO-2018-05-02-a | wmo | 110 |
| 9.3 | swift-DEVELOPMENT-SNAPSHOT-NO_ASSERTIONS-LTO-2018-05-02-a | single file | 225 |
| 9.3 | swift-DEVELOPMENT-SNAPSHOT-NO_ASSERTIONS-LTO-2018-05-02-a | batch | 145 |
| 9.3 | swift-DEVELOPMENT-SNAPSHOT-2018-04-21-a-osx | wmo | 175 |
| 9.3 | swift-DEVELOPMENT-SNAPSHOT-2018-04-21-a-osx | single file | 472 |
| 9.3 | swift-DEVELOPMENT-SNAPSHOT-2018-04-21-a-osx | batch | 369 |
| 9.3 | default | wmo | 121 |
| 9.3 | default | single file | 272 |
| 9.2 | default | wmo | 128 |
| 9.2 | default | single file | 454 |
Batch mode is almost as fast as wmo ! Bravo 
First of all, thanks so much for helping us test this before release! I've been tinkering with this for a few days now and I'm excited to report some numbers we've seen.
Xcode 9.3
- Single File: 425s
- Whole Module: 297s
LTO-2018-05-02
- Single File: 369s
- Whole Module: 285s
- Batch Mode: 298s
Source code stats:
Objective-C
- 744 files
- 124,021 lines of code
Swift
- 1150 files
- 111,594 lines of code
I was really interested to see the overall improvement even for Single File that this build showed. It seems like a lot of people have been reporting similar results. Is that improvement expected or is that the result of having assertions disabled?
1 Like