In our build, we are seeing variability in the contents of generated .swiftmodules, which causes some problems. I would like to figure out what changes to make, either in our setup (if possible), or to the compiler, to ensure reproducible swiftmodules.
Two example cases:
We have recently started using a remote build cache, and this has stressed the importance reproducible .swiftmodules for us. The surprise to me was a whitespace-only change resulting in a non-identical swiftmodule (local compared remote cache), which caused all transitive dependencies to be rebuilt. (We have hundreds of modules in the build graphs of our apps.)
We have two CI jobs (one per app) that populated the cache. These jobs are built using identical code from the same git sha. However, in these two builds we see that modules shared between the two apps can end up with different swiftmodule contents. Whichever job finishes first is the one that populates the cache.
Some facts:
We use Bazel for our build
These are debug builds
Using -enable-batch-mode
Using -incremental
Builds are not clean builds (for speed)
The last two facts are what we initially suspect is the cause. Incremental builds from the same git sha can start from different states, depending what the previous build was on that machine.
I used llvm-bcanalyzer -dump to compare .swiftmodule contents. In most cases, the stats displayed were the same, seemingly only order of records within blocks looked different. However one case had different stats too, so more than record order issues, there were somehow more or less decls, or something. Does this indicate a deeper problem somewhere? A problem with our build? I'm not sure. I didn't keep the specific swiftmodules files for this one case.
Next, I'll see if I can figure out a reproducible test showing non-reproducible modules. But I wanted to ask the forum for any advice. Is there anything we should log or files to preserve from the build (swiftdeps maybe?) in order to solve this? Are there any tools we can use to compare better? Are there known causes of non-reproducible swiftmodules that we can avoid?
We have recently started using a remote build cache, and this has stressed the importance reproducible .swiftmodule s for us. The surprise to me was a whitespace-only change resulting in a non-identical swiftmodule (local compared remote cache)
Usually, the term "reproducible" is used to mean: given the exact same inputs, are the outputs exactly the same. In this case, you're changing two things, whitespace and the state of the cache. Have you tried narrowing it down to which factor is affecting things? If all both of these are the same, and you're still seeing non-reproducible swiftmodules, that's a much bigger issue.
The last two facts are what we initially suspect is the cause. Incremental builds from the same git sha can start from different states, depending what the previous build was on that machine.
Is it possible for you to sync the caches between the two machines (the simplest way is probably pick a "master copy" and overwrite the others) and then rerunning the builds? That would help isolate the issue a bit.
Alternately, you could use the same machine, once running the build with one cache, and once with another.
Next, I'll see if I can figure out a reproducible test showing non-reproducible modules.
In my checking, making a whitespace only change locally results in the exact same .swiftmodule. I think it's fair to say that "exact same" excludes trivial whitespace.
If we can ignore trivial whitespace, then what's happening is the same source inputs (from the same git sha), can produce different .swiftmodule outputs. The state of the previous incremental build can be different, so this input (.swiftdeps, .o, and .swiftmodule files) can be different. If incremental state is the cause of differing .swiftmodule files, then hopefully we can get a repro and a fix. If it's not incremental, then I don't have a lot of other possible causes to explore.
The builds use the same remote cache service. It's possible that we could try to sync incremental state, which is unique to each machine, or even clear it before each build, to see if that helps this particular case. And maybe we could also remove incremental state on local machines after each git pull for example. If incremental is the problem, that could be a solution.
I spent some time earlier this week, but have not yet come up with a reproducible case.
A whitespace-only change might reasonably result in different .swiftmodules to the extent that there are any source locations in the module file. But it's not that hard to have random differences between build environments affect module files, and I think we all agreed that those things should be sniffed out and eliminated.
Which source locations go in a module? In my testing, I had a reduced source file, made a module, then changed whitespace in ways that would change line numbers, but the before and after swiftmodules were the same. Basically I did this:
before:
public var prop1: Int = 0 {
didSet { self.doStuff() }
}
public var prop2: Int = 0 {
didSet { self.doStuff() }
}
after:
public var prop1: Int = 0 {
didSet {
self.doStuff()
}
}
public var prop2: Int = 0 {
didSet {
self.doStuff()
}
}
Are there any known or suspected leads you'd recommend I chase?