Compiling is how we’re allowed to slack off: xkcd: Compiling
A lot of work was done on improving build times for the upcoming releases. Try the binaries posted there, maybe it helps!
Also, I don’t know if those changes have been pushed for 4.1 (which was released today). Maybe they have been, maybe not. You could ask there... maybe some of the upgrades have been released already 