# Swift's unpredictable efficiency

**URL:** <https://forums.swift.org/t/swifts-unpredictable-efficiency/13506>\
**Category:** Using Swift\
**Tags:** performance\
**Created:** [June 10, 2018, 11:03pm UTC](https://forums.swift.org/t/swifts-unpredictable-efficiency/13506 "2018-06-10T23:03:52Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jens](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/jens/32/12903_2.png) [@Jens](https://forums.swift.org/u/Jens)\
**Post date:** [June 10, 2018, 11:03pm UTC](https://forums.swift.org/t/swifts-unpredictable-efficiency/13506/1 "2018-06-10T23:03:52Z")

</div>

I decided to start this topic after having tried out a series of slight modifications to the code example of [SR-7952](https://bugs.swift.org/browse/SR-7952), a bug that was reported in this thread:

> [@Trying to get a flat Set (actually just a Set\<Int\>)](https://forums.swift.org/t/trying-to-get-a-flat-set-actually-just-a-set-int/13469/14):
>
> BTW I filed the `compactMap` issue [as a bug](https://bugs.swift.org/browse/SR-7952) since if nothing else it's a regression from Swift 4.0.3

The demonstration program of that bug turned out to make for a good example of how seemingly insignificant modifications to the code can significantly impact the efficiency of the resulting executable, making it very hard to detect, isolate and work around performance issues.

The following is a step by step demonstration of this.

I will use the, as of today, most recent [development snapshot](https://swift.org/builds/development/xcode/swift-DEVELOPMENT-SNAPSHOT-2018-06-08-a/swift-DEVELOPMENT-SNAPSHOT-2018-06-08-a-osx.pkg) (although similar but not identical behavior can also be observed using eg the default toolchain of Xcode 9.4 and Xcode 10 beta).

* * *

## Program 1

Let's start with the following (a variation of the demonstration program of SR-7952):

```
import Dispatch

func benchmark<T>(_ function: () -> T) -> (T, Double) {
    let start = DispatchTime.now()
    let res = function()
    let end = DispatchTime.now()
    return (res, Double(end.uptimeNanoseconds - start.uptimeNanoseconds) / 1e9)
}

func runBenchmarks() {
    let base = 0 ..< 1_000_000

    let (cmR, cmT) = benchmark { base.lazy.compactMap({ $0 % 4 == 0 ? nil : $0 }).reduce(0, +) }
    let (fmR, fmT) = benchmark { base.lazy.filter({ $0 % 4 != 0 }).map({ $0 }).reduce(0, +) }

    print("compactMap: \(cmR), took \(cmT) s")
    print(" filterMap: \(fmR), took \(fmT) s")
}
runBenchmarks()

```

Compiling and running (a couple of times):

```
$ /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2018-06-08-a.xctoolchain/usr/bin/swiftc -O -whole-module-optimization -swift-version 4.2 -static-stdlib -sdk /Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -target x86_64-apple-macosx10.13 test.swift
$ ./test
compactMap: 375000000000, took 0.318972944 s
 filterMap: 375000000000, took 0.000626542 s
$ ./test
compactMap: 375000000000, took 0.317654136 s
 filterMap: 375000000000, took 0.000569041 s
$ ./test
compactMap: 375000000000, took 0.317911993 s
 filterMap: 375000000000, took 0.000626542 s

```

OK, so `compactMap` is much slower than `filterMap`.

* * *

## Program 2

But _how_ much slower is it? Let's add the following two lines to find out:

```
import Dispatch

func benchmark<T>(_ function: () -> T) -> (T, Double) {
    let start = DispatchTime.now()
    let res = function()
    let end = DispatchTime.now()
    return (res, Double(end.uptimeNanoseconds - start.uptimeNanoseconds) / 1e9)
}

func runBenchmarks() {
    let base = 0 ..< 1_000_000

    let (cmR, cmT) = benchmark { base.lazy.compactMap({ $0 % 4 == 0 ? nil : $0 }).reduce(0, +) }
    let (fmR, fmT) = benchmark { base.lazy.filter({ $0 % 4 != 0 }).map({ $0 }).reduce(0, +) }
    let ratio = Int((cmT/fmT).rounded()) // <-- Added this!

    print("compactMap: \(cmR), took \(cmT) s")
    print(" filterMap: \(fmR), took \(fmT) s")
    print("compactMap is \(ratio) times slower than filterMap") // <-- And this!
}
runBenchmarks()

```

Compiling and running:

```
$ /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2018-06-08-a.xctoolchain/usr/bin/swiftc -O -whole-module-optimization -swift-version 4.2 -static-stdlib -sdk /Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -target x86_64-apple-macosx10.13 test.swift
$ ./test
compactMap: 375000000000, took 0.324593056 s
 filterMap: 375000000000, took 0.00109638 s
compactMap is 296 times slower than filterMap
$ ./test
compactMap: 375000000000, took 0.332787167 s
 filterMap: 375000000000, took 0.001176485 s
compactMap is 283 times slower than filterMap
$ ./test
compactMap: 375000000000, took 0.323394991 s
 filterMap: 375000000000, took 0.001097987 s
compactMap is 295 times slower than filterMap

```

But wait a second! Why has `filterMap` suddenly become almost 2 times slower than it was before?

In Program 1 it took 0.0006 seconds but now it takes 0.0011 seconds. What's going on?

Inspecting the disassembly of Program 1 and 2 to explain the details behind this is left as an exercise for the reader.

* * *

## Program 3

As strange as the above issue might seem, there's an equally strange workaround, which is to just replace the string interpolation with a separate argument for `ratio`:

```
import Dispatch

func benchmark<T>(_ function: () -> T) -> (T, Double) {
    let start = DispatchTime.now()
    let res = function()
    let end = DispatchTime.now()
    return (res, Double(end.uptimeNanoseconds - start.uptimeNanoseconds) / 1e9)
}

func runBenchmarks() {
    let base = 0 ..< 1_000_000

    let (cmR, cmT) = benchmark { base.lazy.compactMap({ $0 % 4 == 0 ? nil : $0 }).reduce(0, +) }
    let (fmR, fmT) = benchmark { base.lazy.filter({ $0 % 4 != 0 }).map({ $0 }).reduce(0, +) }
    let ratio = Int((cmT/fmT).rounded())

    print("compactMap: \(cmR), took \(cmT) s")
    print(" filterMap: \(fmR), took \(fmT) s")
    print("compactMap is", ratio, "times slower than filterMap") // <-- Replaced string interpolated ratio with this!
}
runBenchmarks()

```

Compile and run:

```
$ /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2018-06-08-a.xctoolchain/usr/bin/swiftc -O -whole-module-optimization -swift-version 4.2 -static-stdlib -sdk /Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -target x86_64-apple-macosx10.13 test.swift
$ ./test
compactMap: 375000000000, took 0.322355613 s
 filterMap: 375000000000, took 0.000548265 s
compactMap is 588 times slower than filterMap
$ ./test
compactMap: 375000000000, took 0.324320271 s
 filterMap: 375000000000, took 0.000548251 s
compactMap is 592 times slower than filterMap
$ ./test
compactMap: 375000000000, took 0.323314934 s
 filterMap: 375000000000, took 0.00062654 s
compactMap is 516 times slower than filterMap
$ 

```

Ah! Now `filterMap` takes 0.0006 seconds again, so `compactMap` is about 550 times slower than `filterMap`, and not just 300 times slower as in Program 2.

* * *

## Program 4

Let's add a small improvement: Instead of running the program several times ourselves, we'll make the program do it for us:

```
import Dispatch

func benchmark<T>(_ function: () -> T) -> (T, Double) {
    let start = DispatchTime.now()
    let res = function()
    let end = DispatchTime.now()
    return (res, Double(end.uptimeNanoseconds - start.uptimeNanoseconds) / 1e9)
}

func runBenchmarks() {
    let base = 0 ..< 1_000_000

    for _ in 0 ..< 5 { // <-- Added this for-in loop!
        let (cmR, cmT) = benchmark { base.lazy.compactMap({ $0 % 4 == 0 ? nil : $0 }).reduce(0, +) }
        let (fmR, fmT) = benchmark { base.lazy.filter({ $0 % 4 != 0 }).map({ $0 }).reduce(0, +) }
        let ratio = Int((cmT/fmT).rounded())

        print("compactMap: \(cmR), took \(cmT) s")
        print(" filterMap: \(fmR), took \(fmT) s")
        print("compactMap is", ratio, "times slower than filterMap")
    }
}
runBenchmarks()

```

Compile and run (once):

```
$ /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2018-06-08-a.xctoolchain/usr/bin/swiftc -O -whole-module-optimization -swift-version 4.2 -static-stdlib -sdk /Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -target x86_64-apple-macosx10.13 test.swift
$ ./test
compactMap: 375000000000, took 0.32222596 s
 filterMap: 375000000000, took 3.8e-08 s
compactMap is 8479631 times slower than filterMap
compactMap: 375000000000, took 0.32233228 s
 filterMap: 375000000000, took 3.1e-08 s
compactMap is 10397815 times slower than filterMap
compactMap: 375000000000, took 0.320527889 s
 filterMap: 375000000000, took 3.1e-08 s
compactMap is 10339609 times slower than filterMap
compactMap: 375000000000, took 0.320475311 s
 filterMap: 375000000000, took 3.1e-08 s
compactMap is 10337913 times slower than filterMap
compactMap: 375000000000, took 0.329549344 s
 filterMap: 375000000000, took 3.2e-08 s
compactMap is 10298417 times slower than filterMap

```

Wow! `filterMap` now takes essentially no time at all (I guess it got optimized away somehow) which makes `compactMap` 10 _million_ times slower than `filterMap`!

* * *

## Program 5

As it turns out, this optimization will not happen if `base` is moved to top level:

```
import Dispatch

func benchmark<T>(_ function: () -> T) -> (T, Double) {
    let start = DispatchTime.now()
    let res = function()
    let end = DispatchTime.now()
    return (res, Double(end.uptimeNanoseconds - start.uptimeNanoseconds) / 1e9)
}

let base = 0 ..< 1_000_000 // <-- Moved this out of runBenchmarks().

func runBenchmarks() {
    for _ in 0 ..< 5 {
        let (cmR, cmT) = benchmark { base.lazy.compactMap({ $0 % 4 == 0 ? nil : $0 }).reduce(0, +) }
        let (fmR, fmT) = benchmark { base.lazy.filter({ $0 % 4 != 0 }).map({ $0 }).reduce(0, +) }
        let ratio = Int((cmT/fmT).rounded())

        print("compactMap: \(cmR), took \(cmT) s")
        print(" filterMap: \(fmR), took \(fmT) s")
        print("compactMap is", ratio, "times slower than filterMap")
    }
}
runBenchmarks()

```

Compile and run:

```
$ /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2018-06-08-a.xctoolchain/usr/bin/swiftc -O -whole-module-optimization -swift-version 4.2 -static-stdlib -sdk /Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -target x86_64-apple-macosx10.13 test.swift
$ ./test
compactMap: 375000000000, took 0.319917178 s
 filterMap: 375000000000, took 0.001174697 s
compactMap is 272 times slower than filterMap
compactMap: 375000000000, took 0.324600748 s
 filterMap: 375000000000, took 0.001174689 s
compactMap is 276 times slower than filterMap
compactMap: 375000000000, took 0.318163121 s
 filterMap: 375000000000, took 0.001174688 s
compactMap is 271 times slower than filterMap
compactMap: 375000000000, took 0.3232251 s
 filterMap: 375000000000, took 0.001174688 s
compactMap is 275 times slower than filterMap
compactMap: 375000000000, took 0.318713242 s
 filterMap: 375000000000, took 0.001174699 s
compactMap is 271 times slower than filterMap

```

OK. But look! `filterMap` is back at 0.0011s (or rather 0.0012 s) and not 0.0006 s which we know it can be.

* * *

## Program 6

The string-interpolation-trick comes to rescue, but now in reverse (by changing `ratio` _back_ into a string interpolation this time):

```
import Dispatch

func benchmark<T>(_ function: () -> T) -> (T, Double) {
    let start = DispatchTime.now()
    let res = function()
    let end = DispatchTime.now()
    return (res, Double(end.uptimeNanoseconds - start.uptimeNanoseconds) / 1e9)
}

let base = 0 ..< 1_000_000

func runBenchmarks() {
    for _ in 0 ..< 5 {
        let (cmR, cmT) = benchmark { base.lazy.compactMap({ $0 % 4 == 0 ? nil : $0 }).reduce(0, +) }
        let (fmR, fmT) = benchmark { base.lazy.filter({ $0 % 4 != 0 }).map({ $0 }).reduce(0, +) }
        let ratio = Int((cmT/fmT).rounded())

        print("compactMap: \(cmR), took \(cmT) s")
        print(" filterMap: \(fmR), took \(fmT) s")
        print("compactMap is \(ratio) times slower than filterMap") // <-- Undid the change we made in Program 3 here.
    }
}
runBenchmarks()

```

Compile and run:

```
$ /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2018-06-08-a.xctoolchain/usr/bin/swiftc -O -whole-module-optimization -swift-version 4.2 -static-stdlib -sdk /Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -target x86_64-apple-macosx10.13 test.swift
$ ./test
compactMap: 375000000000, took 0.322214887 s
 filterMap: 375000000000, took 0.000548238 s
compactMap is 588 times slower than filterMap
compactMap: 375000000000, took 0.320392592 s
 filterMap: 375000000000, took 0.000548245 s
compactMap is 584 times slower than filterMap
compactMap: 375000000000, took 0.319095109 s
 filterMap: 375000000000, took 0.000548243 s
compactMap is 582 times slower than filterMap
compactMap: 375000000000, took 0.325831477 s
 filterMap: 375000000000, took 0.000548246 s
compactMap is 594 times slower than filterMap
compactMap: 375000000000, took 0.320049229 s
 filterMap: 375000000000, took 0.000572327 s
compactMap is 559 times slower than filterMap

```

OK.

* * *

We'll stop here.

But note that _we didn't touch the actual code being benchmarked, nor the benchmark function_, the changes were only in the surrounding code.

* * *

The reason I went to the trouble of writing this post is to show what, according to my experience, is the rule rather than the exception, although you might not see it unless you're trying hard to optimize some performance critical part of your code.

So this was just a very small taste. In any given situation, there are all sorts of seemingly irrelevant but performance-impacting little changes that can be made, eg adding or removing a number of spaces at the start of a string used only for reporting the result of a test, switching between a for-in loop and the equivalent while loop, wrapping some piece of code in an immediately invoked closure (which can make that section of code _faster_ in some specific contexts), `&+` sometimes being _slower_ than `+` etc, etc.

* * *

Note also that these issues are impossible to represent/test/benchmark in eg XCTest and The Swift Benchmarking Suite. Why? Because the issues are all about _context_, and they are interacting in various funny ways as shown by eg the string-interpolation-trick working in reverse or not depending on whether `base` was declared in top-level or not. XCTest and The Swift Benchmarking Suite forces some layer(s) of context on the tested code, and thus their harness becomes part of the tested code.

---

<div class="post-metadata">

**Author:** ![sbromberger](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/sbromberger/32/3261_2.png) [@sbromberger](https://forums.swift.org/u/sbromberger)\
**Post date:** [June 11, 2018, 2:58am UTC](https://forums.swift.org/t/swifts-unpredictable-efficiency/13506/2 "2018-06-11T02:58:58Z")

</div>

I ran these tests using

```swift
Apple Swift version 4.1.2 (swiftlang-902.0.54 clang-902.0.39.2)
Target: x86_64-apple-darwin17.6.0

```

building with just `swiftc -O`. Here are my results (numbered the same way yours are):

```swift
$ ./main1
compactMap: 375000000000, took 0.289310122 s
 filterMap: 375000000000, took 0.000821735 s

$ ./main2
compactMap: 375000000000, took 0.316243825 s
 filterMap: 375000000000, took 0.000814479 s
compactMap is 388 times slower than filterMap

$ ./main3
compactMap: 375000000000, took 0.288270726 s
 filterMap: 375000000000, took 0.000808948 s
compactMap is 356 times slower than filterMap

$ ./main4
compactMap: 375000000000, took 0.288978948 s
 filterMap: 375000000000, took 0.001179241 s
compactMap is 245 times slower than filterMap
compactMap: 375000000000, took 0.295771218 s
 filterMap: 375000000000, took 0.00117926 s
compactMap is 251 times slower than filterMap
compactMap: 375000000000, took 0.298000228 s
 filterMap: 375000000000, took 0.001179243 s
compactMap is 253 times slower than filterMap
compactMap: 375000000000, took 0.287790936 s
 filterMap: 375000000000, took 0.00118008 s
compactMap is 244 times slower than filterMap
compactMap: 375000000000, took 0.290857785 s
 filterMap: 375000000000, took 0.001179366 s
compactMap is 247 times slower than filterMap

$ ./main5
compactMap: 375000000000, took 0.291611994 s
 filterMap: 375000000000, took 0.001179274 s
compactMap is 247 times slower than filterMap
compactMap: 375000000000, took 0.299940731 s
 filterMap: 375000000000, took 0.001179274 s
compactMap is 254 times slower than filterMap
compactMap: 375000000000, took 0.2925131 s
 filterMap: 375000000000, took 0.001179412 s
compactMap is 248 times slower than filterMap
compactMap: 375000000000, took 0.29145278 s
 filterMap: 375000000000, took 0.001179271 s
compactMap is 247 times slower than filterMap
compactMap: 375000000000, took 0.297499215 s
 filterMap: 375000000000, took 0.001179322 s
compactMap is 252 times slower than filterMap

$ ./main6
compactMap: 375000000000, took 0.309454594 s
 filterMap: 375000000000, took 0.000833356 s
compactMap is 371 times slower than filterMap
compactMap: 375000000000, took 0.319849605 s
 filterMap: 375000000000, took 0.000808931 s
compactMap is 395 times slower than filterMap
compactMap: 375000000000, took 0.31500959 s
 filterMap: 375000000000, took 0.000808922 s
compactMap is 389 times slower than filterMap
compactMap: 375000000000, took 0.298115479 s
 filterMap: 375000000000, took 0.000808922 s
compactMap is 369 times slower than filterMap
compactMap: 375000000000, took 0.300710155 s
 filterMap: 375000000000, took 0.000808922 s
compactMap is 372 times slower than filterMap

```

so it appears that I can't reproduce the slowdown between 1, 2, and 3, but the performance regression comes in for 4 and 5 and is again resolved in 6.

---

<div class="post-metadata">

**Author:** ![sbromberger](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/sbromberger/32/3261_2.png) [@sbromberger](https://forums.swift.org/u/sbromberger)\
**Post date:** [June 11, 2018, 3:08am UTC](https://forums.swift.org/t/swifts-unpredictable-efficiency/13506/3 "2018-06-11T03:08:51Z")

</div>

More interesting results (just for test 6):

```swift
$ swiftc -O -whole-module-optimization -static-stdlib -target x86_64-apple-macosx10.13 -o main6 main.swift
$ ./main6
compactMap: 375000000000, took 0.295983463 s
 filterMap: 375000000000, took 0.001179248 s
compactMap is 251 times slower than filterMap
compactMap: 375000000000, took 0.292354945 s
 filterMap: 375000000000, took 0.001179273 s
compactMap is 248 times slower than filterMap
compactMap: 375000000000, took 0.294831198 s
 filterMap: 375000000000, took 0.001179264 s
compactMap is 250 times slower than filterMap
compactMap: 375000000000, took 0.300107382 s
 filterMap: 375000000000, took 0.001179257 s
compactMap is 254 times slower than filterMap
compactMap: 375000000000, took 0.292348202 s
 filterMap: 375000000000, took 0.001179251 s
compactMap is 248 times slower than filterMap

$ swiftc -O -o main6 main.swift
$ ./main6
compactMap: 375000000000, took 0.304550801 s
 filterMap: 375000000000, took 0.000808928 s
compactMap is 376 times slower than filterMap
compactMap: 375000000000, took 0.307866093 s
 filterMap: 375000000000, took 0.000808922 s
compactMap is 381 times slower than filterMap
compactMap: 375000000000, took 0.299180831 s
 filterMap: 375000000000, took 0.000808924 s
compactMap is 370 times slower than filterMap
compactMap: 375000000000, took 0.30244097 s
 filterMap: 375000000000, took 0.000810153 s
compactMap is 373 times slower than filterMap
compactMap: 375000000000, took 0.302916451 s
 filterMap: 375000000000, took 0.000808916 s
compactMap is 374 times slower than filterMap

```

Further testing shows it's the `static-stdlib` that's causing it.

---

<div class="post-metadata">

**Author:** ![Jens](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/jens/32/12903_2.png) [@Jens](https://forums.swift.org/u/Jens)\
**Post date:** [June 11, 2018, 8:05am UTC](https://forums.swift.org/t/swifts-unpredictable-efficiency/13506/4 "2018-06-11T08:05:24Z")

</div>

Yes : ) As I mentioned in the original post, the specific effects of these slight-code-changes-things are highly dependent on all sorts of context, including compiler version, etc. And they will interact so that the same change might have the opposite effect or not given a certain context (see below, and eg the above mentioned "string-interpolation-trick" having opposite effect in Program 3 and Program 6, given that specific compiler version and flags of course).

* * *

Regarding `-static-stdlib`, I used that flag (among others) in my examples because Xcode does that by default, and I wanted the results of my examples to match what you'd get if using Xcode instead of the swiftc command. (Using Xcode in this case means doing a Release Build within Xcode 10 beta, with development snapshot 2018-06-08, using a command line app prj, default build settings).

* * *

As you demonstrated above:

When using the default toolchain of Xcode 9.4, ie:

```
$ swiftc --version
Apple Swift version 4.1.2 (swiftlang-902.0.54 clang-902.0.39.2)
Target: x86_64-apple-darwin17.5.0

```

Then, for Program 6, including `-static-stdlib` will make `filterMap` slower.

But! Note that for Program 1, `-static-stdlib` will have the _opposite_ effect, ie _not_ including it will make `filterMap` slower.

> **Detailed demonstration**
>
> ```
> $ swiftc --version
> Apple Swift version 4.1.2 (swiftlang-902.0.54 clang-902.0.39.2)
> Target: x86_64-apple-darwin17.5.0
> $
> $ swiftc -O Program6.swift
> $ ./Program6
> compactMap: 375000000000, took 0.304990868 s
> filterMap: 375000000000, took 0.001176455 s
> compactMap is 259 times slower than filterMap
> compactMap: 375000000000, took 0.302824142 s
> filterMap: 375000000000, took 0.000861471 s
> compactMap is 352 times slower than filterMap
> compactMap: 375000000000, took 0.303165661 s
> filterMap: 375000000000, took 0.000861489 s
> compactMap is 352 times slower than filterMap
> compactMap: 375000000000, took 0.302871331 s
> filterMap: 375000000000, took 0.00086148 s
> compactMap is 352 times slower than filterMap
> compactMap: 375000000000, took 0.302811901 s
> filterMap: 375000000000, took 0.000861474 s
> compactMap is 352 times slower than filterMap
> $
> $ swiftc -O -static-stdlib Program6.swift
> $ ./Program6
> compactMap: 375000000000, took 0.311002296 s
> filterMap: 375000000000, took 0.001297683 s
> compactMap is 240 times slower than filterMap
> compactMap: 375000000000, took 0.304119449 s
> filterMap: 375000000000, took 0.001255856 s
> compactMap is 242 times slower than filterMap
> compactMap: 375000000000, took 0.304179837 s
> filterMap: 375000000000, took 0.001253053 s
> compactMap is 243 times slower than filterMap
> compactMap: 375000000000, took 0.308980026 s
> filterMap: 375000000000, took 0.001255858 s
> compactMap is 246 times slower than filterMap
> compactMap: 375000000000, took 0.304566472 s
> filterMap: 375000000000, took 0.001255855 s
> compactMap is 243 times slower than filterMap
> $ 
> $ swiftc -O Program1.swift
> $ ./Program1
> compactMap: 375000000000, took 0.308087019 s
> filterMap: 375000000000, took 0.00086148 s
> $ swiftc -O -static-stdlib Program1.swift
> $ ./Program1
> compactMap: 375000000000, took 0.307667413 s
> filterMap: 375000000000, took 0.001255829 s
> 
> ```

So it's not as easy as "X will make Y slower", it's more like "X will make Y slower or faster depending on A, B, C, D, ...", and I guess this means that there might be situations where some specific performance-issue seems to have been "fixed" in a new version of the compiler, when it's really just because of some change in the interference of these things, and the same issue is still there, only in a different shape, manifesting itself in slightly different contexts.

---

<div class="post-metadata">

**Author:** ![sbromberger](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/sbromberger/32/3261_2.png) [@sbromberger](https://forums.swift.org/u/sbromberger)\
**Post date:** [June 11, 2018, 3:12pm UTC](https://forums.swift.org/t/swifts-unpredictable-efficiency/13506/5 "2018-06-11T15:12:49Z")

</div>

> [@Jens](#):
>
> So it's not as easy as "X will make Y slower", it's more like "X will make Y slower or faster depending on A, B, C, D, ...", and I guess this means that there might be situations where some specific performance-issue seems to have been "fixed" in a new version of the compiler, when it's really just because of some change in the interference of these things, and the same issue is still there, only in a different shape, manifesting itself in slightly different contexts.

As someone who's looking to use Swift for some high-performance computations, this scares the hell out of me.

---

<div class="post-metadata">

**Author:** ![Joe\_Groff](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/joe_groff/32/1929_2.png) [@Joe\_Groff](https://forums.swift.org/u/Joe_Groff)\
**Post date:** [June 11, 2018, 3:43pm UTC](https://forums.swift.org/t/swifts-unpredictable-efficiency/13506/6 "2018-06-11T15:43:26Z")

</div>

> [@Jens](#):
>
> Note also that these issues are impossible to represent/test/benchmark in eg XCTest and The Swift Benchmarking Suite. Why? Because the issues are all about _context_ , and they are interacting in various funny ways as shown by eg the string-interpolation-trick working in reverse or not depending on whether `base` was declared in top-level or not. XCTest and The Swift Benchmarking Suite forces some layer(s) of context on the tested code, and thus their harness becomes part of the tested code.

Nonetheless, we can still write regression tests for these cases in the optimizer's own test suite, to ensure that certain abstractions get broken down by the optimizer.

---

<div class="post-metadata">

**Author:** ![Jens](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/jens/32/12903_2.png) [@Jens](https://forums.swift.org/u/Jens)\
**Post date:** [June 11, 2018, 4:56pm UTC](https://forums.swift.org/t/swifts-unpredictable-efficiency/13506/7 "2018-06-11T16:56:36Z")

</div>

> [@Joe\_Groff](#):
>
> Nonetheless, we can still write regression tests for these cases in the optimizer's own test suite, to ensure that certain abstractions get broken down by the optimizer.

Ah, OK. Can the optimizer's test suite be found anywhere under [GitHub - apple/swift: The Swift Programming Language](https://github.com/apple/swift) ? (I tried but couldn't find it.)

* * *

Is there some strategy for dealing with cases like these, except trusting that enough people will find the urge to perform enough experiments like this and file enough bugs?

* * *

Assuming that there are more people like me out there, ie non-compiler hackers yearning for increased efficiency and predictability of the optimizer, willing to spend some but not too much of their time helping to improve the situation in some way, what are some recommended ways to contribute?

(I wish there is some other way than filing individual bugs for each specific case we can find (for the most recent snapshot I guess), which would for example mean at least 6 bugs for the little demonstration in the OP.)

---

<div class="post-metadata">

**Author:** ![Joe\_Groff](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/joe_groff/32/1929_2.png) [@Joe\_Groff](https://forums.swift.org/u/Joe_Groff)\
**Post date:** [June 11, 2018, 5:09pm UTC](https://forums.swift.org/t/swifts-unpredictable-efficiency/13506/8 "2018-06-11T17:09:15Z")

</div>

cc @Erik_Eckstein. Filing bugs for the individual issues you encounter is a good idea regardless. Ideally, we'd be able to use your examples as-is as regression tests to validate that compiler transforms continue to work on these abstractions without breakage due to standard library changes or changes elsewhere in the optimizer. Erik, would it be possible for the benchmark suite to work with ad-hoc tests like Jens' here without boilerplate, since at least some of these problems are context-dependent?

---

<div class="post-metadata">

**Author:** ![Michael\_Gottesman](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/michael_gottesman/32/280_2.png) [@Michael\_Gottesman](https://forums.swift.org/u/Michael_Gottesman)\
**Post date:** [June 11, 2018, 5:48pm UTC](https://forums.swift.org/t/swifts-unpredictable-efficiency/13506/9 "2018-06-11T17:48:12Z")

</div>

My suggestion would be to commit specific a-b benchmarks and then file an SR that states that the performance should be the same. Then we can validate when it is fixed and then maintain that performance over time.

---

<div class="post-metadata">

**Author:** ![John\_McCall](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/john_mccall/32/1351_2.png) [@John\_McCall](https://forums.swift.org/u/John_McCall)\
**Post date:** [June 11, 2018, 8:17pm UTC](https://forums.swift.org/t/swifts-unpredictable-efficiency/13506/10 "2018-06-11T20:17:38Z")

</div>

Let me try to tackle this as a higher-level point. There are optimizations that are inherently sensitive to the exact code sequence in the program, potentially including contextual information. These will always come across as "unpredictable" in the sense that most users won't intuitively understand why their program got slower; I don't think that's a good reason to not pursue them. Instead, I think we should empower users to take more control over the performance of their program, in the following ways:

- We should try to ensure that the optimizer isn't more powerful than the user: it should generally be possible to do important high-level optimizations (like minimizing ARC operations or specializing a generic algorithm) in source code rather than relying on the optimizer to independently discover the opportunity.

- We should publish "optimization remarks" so that users can see what optimizations have been applied to their code, assuming this is possible to do without overwhelming them; they can they use this in conjunction with the first point to reclaim performance that seems to be lost "unpredictably".

Part of the goal of the ownership features is to provide better tools for addressing the first point.

---

<div class="post-metadata">

**Author:** ![felix91gr](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/felix91gr/32/443_2.png) [@felix91gr](https://forums.swift.org/u/felix91gr)\
**Post date:** [June 11, 2018, 8:52pm UTC](https://forums.swift.org/t/swifts-unpredictable-efficiency/13506/11 "2018-06-11T20:52:02Z")

</div>

> [@John\_McCall](#):
>
> Part of the goal of the ownership features is to provide better tools for addressing the first point.

I was just gonna say the same when I read your first point xD  
I agree, most of what's related to memory overhead can be optimized with ownership by the user 🙂

I would in general however, try to stay away from telling the users to "optimize their code" if that would imply a less readable or less maintainable code. I'm not talking about ownership here, but rather about "hey, this struct layout hits an optimizer bug so try to go for this layout instead". When you have to change the meaning of your code to less precisely fit your algorithm, that's when things tend to go wrong from what I've seen.

---

<div class="post-metadata">

**Author:** ![John\_McCall](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/john_mccall/32/1351_2.png) [@John\_McCall](https://forums.swift.org/u/John_McCall)\
**Post date:** [June 11, 2018, 9:09pm UTC](https://forums.swift.org/t/swifts-unpredictable-efficiency/13506/12 "2018-06-11T21:09:16Z")

</div>

Right, I absolutely agree that we should continue to aim for the optimizer to optimize more cases. Test cases for that are always welcome.

The sorts of annotations we're talking about for ownership will hopefully not be too onerous to adopt in natural code, though.

---

<div class="post-metadata">

**Author:** ![reitzig](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/reitzig/32/2054_2.png) [@reitzig](https://forums.swift.org/u/reitzig)\
**Post date:** [June 20, 2018, 9:05am UTC](https://forums.swift.org/t/swifts-unpredictable-efficiency/13506/13 "2018-06-20T09:05:35Z")

</div>

> [@John\_McCall](#):
>
> We should publish "optimization remarks"

I think that would be most useful as a delta between revisions: "[piece of code] can no longer be optimized because [reason]".

---

<div class="post-metadata">

**Author:** ![SwiftStudies](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/swiftstudies/32/5077_2.png) [@SwiftStudies](https://forums.swift.org/u/SwiftStudies)\
**Post date:** [June 20, 2018, 10:37am UTC](https://forums.swift.org/t/swifts-unpredictable-efficiency/13506/14 "2018-06-20T10:37:37Z")

</div>

All the of the commentary here is great, and my subsequent comments are not intended to detract from it.

However, I do want to make a couple of points.

In my opinion, the compiler should be better than the developer at optimising. The set of assembly language routines I can write that run faster than compiled code has reduced to almost zero over the years to the point where the only ones left are small and dependant on contextual knowledge not captured in code. Long may that trend continue. Without me having to decorate code with annotations.

Finally, whilst these results should be addressed... my experience in writing OysterKit starting in Swift 1 through to now is that release after release performance has improved. Sometimes dramatically. These results look worry-some, but with such a small footprint any fluctuations are exaggerated. Across a complete application or framework my experience (and measurements) tell me things just get better release after release.

---

<div class="post-metadata">

**Author:** ![hlovatt](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/hlovatt/32/3044_2.png) [@hlovatt](https://forums.swift.org/u/hlovatt)\
**Post date:** [June 20, 2018, 11:23am UTC](https://forums.swift.org/t/swifts-unpredictable-efficiency/13506/15 "2018-06-20T11:23:02Z")

</div>

I am also worried about the number of annotations being added; it is a complete anathama for an easy to learn language to require annotations. I don’t buy that annotations are part of progressive disclosure when there so many annotations.

I am also concerned that annotations are becoming baked into the ABI. For example @inlineable should not be part of the ABI; ideally it would not exist, it’s the compiler’s job.

---

<div class="post-metadata">

**Author:** ![jawbroken](https://avatars.discourse-cdn.com/v4/letter/j/cc9497/32.png) [@jawbroken](https://forums.swift.org/u/jawbroken)\
**Post date:** [June 20, 2018, 2:38pm UTC](https://forums.swift.org/t/swifts-unpredictable-efficiency/13506/16 "2018-06-20T14:38:09Z")

</div>

Which attributes are required? I guess there's `@escaping`, then a few Objective-C and other interop ones (`@objc`, `@IB*`). What would the number of attributes have to do with whether they are progressively disclosed? Which algorithm should the compiler use to predict the future so it can work out which parts of one module are safe to inline into a different module?

---

<div class="post-metadata">

**Author:** ![torquato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/torquato/32/1203_2.png) [@torquato](https://forums.swift.org/u/torquato)\
**Post date:** [June 20, 2018, 3:07pm UTC](https://forums.swift.org/t/swifts-unpredictable-efficiency/13506/17 "2018-06-20T15:07:44Z")

</div>

> [@jawbroken](#):
>
> Which attributes are required? I guess … a few Objective-C and other interop ones (`@objc`, `@IB*`).

Strictly speaking, I think these are not attributes of the swift language itself, but of an Apple specific extension thereof, right?

---

<div class="post-metadata">

**Author:** ![Joe\_Groff](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/joe_groff/32/1929_2.png) [@Joe\_Groff](https://forums.swift.org/u/Joe_Groff)\
**Post date:** [June 20, 2018, 3:44pm UTC](https://forums.swift.org/t/swifts-unpredictable-efficiency/13506/18 "2018-06-20T15:44:47Z")

</div>

Inlining _is_ the compiler's job, but it's normally impossible to inline across ABI boundaries, and modules with long-term ABI compatibility concerns may not want inlining of old implementations to happen, so it's necessary for binary frameworks to opt in to allowing inlining. As Swift's build system improves to allow for cross-module optimization as part of its normal build process, `@inlinable` will become irrelevant for most Swift users.

I agree with your general sentiment that relying on annotations is unreasonable. I think there's a bare minimum expected level of optimization that we still aren't reliably reaching yet, and we should work on filling those gaps before dreaming up new language features.

---

<div class="post-metadata">

**Author:** ![hlovatt](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/hlovatt/32/3044_2.png) [@hlovatt](https://forums.swift.org/u/hlovatt)\
**Post date:** [June 20, 2018, 10:28pm UTC](https://forums.swift.org/t/swifts-unpredictable-efficiency/13506/19 "2018-06-20T22:28:37Z")

</div>

Two points re inlining:

1. You link your apps against known versions of the library (except for Apple's system libraries - which I will come back to) and therefore old code inlining is not a problem for non-system-3rd-party libraries. The compiler can decide that a function is potentially inlineable and automatically add `@inlineable` (and supply the source/AST/SIL/LLVM to inline) when compiling the library.
2. Apple supplied system libraries get changed from under you, i.e. a system software update happens. In this case I think a better solution is an on device relinking of the code including inlining. This works very well in the Java world including Android. An advantage of having control over the whole of the stack is that this is possible for Apple to do also.

---

<div class="post-metadata">

**Author:** ![Joe\_Groff](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/joe_groff/32/1929_2.png) [@Joe\_Groff](https://forums.swift.org/u/Joe_Groff)\
**Post date:** [June 20, 2018, 10:34pm UTC](https://forums.swift.org/t/swifts-unpredictable-efficiency/13506/20 "2018-06-20T22:34:44Z")

</div>

> [@hlovatt](#):
>
> You link your apps against known versions of the library (except for Apple's system libraries - which I will come back to) and therefore old code inlining is not a problem for non-system-3rd-party libraries. The compiler can decide that a function is potentially inlineable and automatically add `@inlineable` (and supply the source/AST/SIL/LLVM to inline) when compiling the library.

That's what I meant, sorry I wasn't clear. For most user modules, `@inlinable` should be irrelevant in the fullness of time.

> [@hlovatt](#):
>
> Apple supplied system libraries get changed from under you, i.e. a system software update happens. In this case I think a better solution is an on device relinking of the code including inlining. This works very well in the Java world including Android. An advantage of having control over the whole of the stack is that this is possible for Apple to do also.

That would be an interesting thing for Apple to explore, though there are many tradeoffs. "Works well" is subjective given Android's poor memory and energy efficiency compared to iOS. Apple's platform is fairly aggressively optimized for AOT compilation, and it's easier to optimize memory usage by sharing pages from dynamic libraries and pre-linking the system libraries into the shared cache. Some of that benefit could still be preserved with on-device recompilation, but the more you inline and specialize library code per application, the more per-application code size you pay for and less systemic savings you get.

[Next page](https://forums.swift.org/t/swifts-unpredictable-efficiency/13506.md?page=2)