Why compile an array size 3773 statically init each element very slow?

young · 2023-10-27T22:22:14.187Z

let allEmojis: [Emoji] = [
    .init(character: "😀", name: "Grinning face", category: .init(group: "Smileys & Emotion", subgroup: "face-smiling"), isNew: false, zwjGroup: nil),
// this repeat 3772 more time
// ...
]

this is making my compile taking a long time (several minutes) on a small SwiftUI project. I'm tempted to switch to json and parse it at runtime.

It slow every compile even when I only change other files.

tera · 2023-10-28T16:13:25.393Z

young:

this is making my compile taking a long time (several minutes) on a small SwiftUI project.

Can you plot a rough compilation time / count graph? Will it be linear, quadratic or what?

No idea why it takes long, but using JSON is a good workaround. Either store that JSON in an external file or in the source file:

let json = #"[
    {
        "character": "😀",
        "name": "Grinning face",
        "category": {
            "group": "Smileys & Emotion",
            "subgroup": "face-smiling" 
        },
        "isNew": false,
        "zwjGroup": nil
    },
    ...
]"#

let_zeppelin · 2023-10-28T17:17:15.833Z

Does writing Emoji instead of .init make a difference?

feifan · 2023-10-28T17:43:58.447Z

Large static arrays seem to be a known slow point for the compiler — see e.g. these issues

young · 2023-10-28T19:56:10.144Z

Since I have [Emoji] type annotation, I didn't think it would make any difference.

But I give it a try and very surprising result:

First, I change the first .init to Emoji of every element and the compile is about the same slow. Then I changed the second .init to Category and clean and. compile, first a window pop up saying:

"..run out of application memory. Quit some app ..."

Screenshot 2023-10-28 at 12.55.27922×786 99.2 KB

after a little while, my MacBook Air M2 with 24GB went black and shutdown.

I reboot back, clean and compile again an it's much much slower than before.

So whatever difference this make Xcode to consume all the memory.

tera · 2023-10-28T20:29:28.042Z

young:

First, I change the first .init to Emoji and the compile is about the same slow. Then I changed the second .init to Category and clean and. compile

What happens if you change all .inits to Emoji?

young · 2023-10-28T20:34:14.129Z

I did change all but there is another init to Catagory in the argument and I also changed that and Xcode code consume all memory

tera · 2023-10-28T22:15:03.908Z

young:

let allEmojis: [Emoji] = [
    .init(character: "😀", name: "Grinning face", category: .init(group: "Smileys & Emotion", subgroup: "face-smiling"), isNew: false, zwjGroup: nil),
// this repeat 3772 more time
// ...
]

Got you. That slowdown you saw before is probably due to VM trashing, just in that case the memory required didn't overflow available VM size (typically about machine RAM size x 2).

Another workaround that might work for you:

var allEmojis: [Emoji] = []

func initEmojis() {
    allEmojis.removeAll()
    allEmojis.append(.init(character: "😀", name: "Grinning face", category: .init(group: "Smileys & Emotion", subgroup: "face-smiling"), isNew: false, zwjGroup: nil))
    // repeat this 3772 more time
}

young · 2023-10-29T00:44:40.201Z

I did this:

var allEmojis: [Emoji] = {
    var result = [Emoji]()
    result.reserveCapacity(3773)
    result.append(.init(character: "😀", name: "Grinning face", category: .init(group: "Smileys & Emotion", subgroup: "face-smiling"), isNew: false, zwjGroup: nil))
    // ... repeat etc...

    return result
}()

Xcode compile took over 10 minutes now. And change some other file and recompile is much slower.

So my original way is the only way Xcode can finish compile, it's faster to compile than doing it as you suggest.

BTW, Xcode cannot edit my AllEmoji file, any change cause it to beach ball. I had to edit it with vim outside of Xcode.

Looks like parsing each

.init(...., category: .init(...))

expression is slow?

tera · 2023-10-29T01:16:27.856Z

I confirm your findings. Here's a nice quadratic timing graph I got:

image1940×1426 100 KB

x - number of appends, y - compilation time in seconds.

Raw timing data

young · 2023-10-29T01:27:24.109Z

Wonder what the memory curve is?

It’s probably the same quadratic and that’s why it’s slow?

tera · 2023-10-29T17:38:01.097Z

I didn't notice bad memory behaviour during that test.

tera:

Either store that JSON in an external file or in the source file:

let json = #"[
    {
        "character": "😀",
        "name": "Grinning face",
        "category": {
            "group": "Smileys & Emotion",
            "subgroup": "face-smiling" 
        },
        "isNew": false,
        "zwjGroup": nil
    },
    ...
]"#

I tried that approach, it took just 0.03 seconds to parse that JSON string during runtime, not a big deal.

young · 2023-10-29T19:08:43.789Z

I did this:

import Foundation

var allEmojis: [Emoji] = {
  // just let it crash if something is wrong but shouldn't because
  // we created the input and the output, everything should parse...
  try! JSONDecoder().decode([Emoji].self, from: allEmojisJson.data(using: .utf8)!)
}()
private let allEmojisJson = """
.
.
.

Now the command line tool that generates this is very slow to compile. But my app now compile super fast and app startup is instant on device even with debug.

tera · 2023-10-31T00:12:18.907Z

It's best to continue here to not mix the threads.

Anyway to do better: PhaseAnimator((1...45).map(Double.init)) { phase in

For my Emoji thread, it's very surprising that the way I was doing with just .init, it's actually the best case slow but can finish. When I change all the .init to the types, Swift consume all app memory popup and sometime after MacBook restarted.

Anyway to do better: PhaseAnimator((1...45).map(Double.init)) { phase in

My Emoji case already fully annotated with type:

let a: [Emoji = […]`

But swift still doing something very slow. There should not be any type inference here.

I confirm there's something odd going on here. Xcode consuming all those 10s or even 100s of GB and then crashing my computer (it did) is not good. I tried 4K of your lines with different permutations of explicit types vs ".init":

let emojis: [Emoji] = [
    .init(character: "😀", name: "Grinning face", category: .init(group: "Smileys & Emotion", subgroup: "face-smiling"), isNew: false, zwjGroup: nil),
// 4K of those
]

vs

let emojis: [Emoji] = [
    Emoji(character: "😀", name: "Grinning face", category: .init(group: "Smileys & Emotion", subgroup: "face-smiling"), isNew: false, zwjGroup: nil),
// 4K of those
]

vs

let emojis: [Emoji] = [
    .init(character: "😀", name: "Grinning face", category: Category(group: "Smileys & Emotion", subgroup: "face-smiling"), isNew: false, zwjGroup: nil),
// 4K of those
]

vs

let emojis: [Emoji] = [
    Emoji(character: "😀", name: "Grinning face", category: Category(group: "Smileys & Emotion", subgroup: "face-smiling"), isNew: false, zwjGroup: nil),
// 4K of those
]

the types:

struct Category {
    var group: String
    var subgroup: String
}

struct Emoji {
    var character: String
    var name: String
    var category: Category
    var isNew: Bool
    var zwjGroup: String?
}

It would be extremely rare anyone would want to write 4K of such lines, but still, quadratic CPU time complexity aside, quite questionable memory complexity (like 10 MB per expression?!) is ... unexpectable to put it mildly. Highly likely there's a memory leak in there. It's a bug worth filing.

young · 2023-10-31T00:17:07.565Z

Swift bug? It’s the same on command line.

It would be extremely rare anyone would want to write 4K of such lines

Not by hands but code gen tools can often produce large boilerplate source. My case is pure data. Storing in json and decode at runtime is better to avoid slow Swift compile.

However, it’s better if Swift can just compile such code and avoid the extra decode and large json string.

tera · 2023-11-01T18:44:17.934Z

There could also be the DoS angle here. For the record godbolt's compiler is killed with one error or another (when using explicit types vs .inits) - I guess in one case it's killed due to OOM situation and in another due to compiler spending too much time during the compilation.