Switching branches and build cache invalidation

Development Compiler
1

In the development workflow I've found most success with, I generally try to avoid pulling in new changes from main for as long as reasonably possible. This is primarily because updating the main branch often seems to invalidate lots of things in my local build cache (ninja), incremental rebuilds frequently fail and clean rebuilds are very slow.

This approach has seemed to work reasonably well for me. However, periodically I must update to a more recent local checkout. This is what I've experienced recently:

  1. Do some work on branch some-branch off of main
  2. <time passes>
  3. Need to update main, so switch to main and git pull or run update-checkout or whatever
  4. (usually) Do a long rebuild because maybe llvm APIs changed or something and nothing builds incrementally anymore
  5. Great, main is now up-to-date!
  6. Switch to some-branch and git rebase main
  7. A bunch of stuff was invalidated by the branch change & rebase (???)
  8. Rebuild things (incremental builds usually work in this case, but can take a surprisingly long time)

I don't really understand why switching branches seems to invalidate so much of the local build cache (disclaimer: I don't know much about how CMake/ninja work). For example, if from a branch that I've recently built I switch to main and back without changing anything, it seems like all the files the branch modifies are considered invalidated and then rebuilt. Is that sort of thing expected? Is there some way to avoid invalidating the build cache when changing branches?

1 Like
2

By "build cache" here I assume you're referring to Ninja's incremental build and not e.g sccache?

I suspect this is the issue, by switching to some-branch you're checking out an older version of the repo that will cause a lot of files to be modified, even if you then rebase that on top of main those timestamps will have still changed so Ninja has to assume that all of the files need rebuilding. If you have sccache though that should be able to quickly retrieve cached outputs for any inputs that haven't actually changed though (you can enable this with the --sccache build-script flag).

Yes this is expected with Ninja since it only cares about the modification times of the files, not the actual contents.

AFAIK there's no way to avoid it at the Ninja level, but you can use sccache to make the rebuilds much quicker.

3

It has to do with timestamps: ninja and other build tools are usually just checking timestamps, which doesn't interoperate well with revision control systems like git that change those. Maybe sccache and other caching tools do a better job, as @hamishknight says, I haven't used those.

4

Yes, at least I believe so. I'm pretty sure I have sccache configured (I followed whatever is in the getting started instructions which I think suggests using that).

I see, thanks. Yeah the rebuilds do seem relatively quick, at least if a build has recently been performed. Anecdotally though if you're trying to, e.g. rebuild the stdlib after this sort of thing, linking still seems to take quite some time.

Thank you for the insight & references – that thread looks interesting!

1 Like
5

Yeah that's one limitation of sccache, it doesn't help for compilation of Swift sources or linking.

Locally I've recently been running caching using CAS (by using clang-cache and a Swift wrapper that forces -explicit-module-build -cache-compile-job). My setup is fairly hacked together though, I'm hoping to add support for it directly into build-script. This helps for the bits of the compiler written in Swift, but it doesn't really help with the stdlib since that needs to be built with the just-built compiler, so in principle any change to the compiler must require a rebuild of the standard library. Generally I would recommend directly invoking Ninja for incremental builds though, that way you only ever rebuild the compiler and not the standard library. When pulling latest main I would still rebuild that with build-script though.

1 Like
6

Reading your workflow purely from a git user's point of view; emphasis added by me:

I think what messes things up is the switching back to some-branch after a successful build on an updated main branch.

What you could do instead is a more streamlined step 6:

  1. While still on main, rebase the feature branch on top of it by running git rebase main some-branch.

That way, your feature branch commits will be applied one by one over main, without resetting everything else to the old main first. (After running the command, you'll be left on the updated some-branch again.) Hope this helps!

7

Thanks, I'll look into this. I always thought that rebase performs a checkout/switch first though, so I assumed explicitly switching branches first was effectively the same (if less efficient). The first thing in the help pages for it says:

If <branch> is specified, git rebase will perform an automatic git switch <branch> before doing anything else.

Edit: I did a basic test, and anecdotally it does seem to invalidate fewer things when rebasing in this way – thanks for the suggestion @pyrtsa!

1 Like
8

It's usually better to merge the default branch into your branch, so conflict resolution only happens once in a clearly marked commit instead of sprinkled over multiple different commits (that you never even tried to compile) whose original version will only be available in git reflog. The maintainers can always decide to squash all changes into a single commit if they want "clean" and linear history.

You don't need to update your local default branch or even have to remember it by name, you can just do this:

git fetch
git merge origin/HEAD

I sometimes have multiple clones of a repository, so I can instantly switch between them without having to worry about the switch adding to the incremental build time.

1 Like
9

You can also set the remote HEAD if you want to work against a different branch, like release/6.3 (--a resets it to the one defined on the server):

compdef _git gitSetHead=git-checkout
gitSetHead() {
	git remote set-head origin "${1:--a}"
}
10

I should note that for the Swift repo at least we prefer contributors to rebase their changes rather than merge when resolving conflicts, that way we can land a clean set of individual commits rather than being forced to squash them into a single commit

2 Likes