For what it's worth, I've filed FB9915302 about this. It's a major workflow issue for apps which need to be built with optimisations to run well enough to be tested – the compile times are very lengthy regardless and building for architectures you're not going to use only makes it worse.
I'll also add that per-target optimisation settings would be very useful – it's often the case that I'm working on a particular package and want that to be built in debug mode, but still need other packages it depends on to be in release mode so the app is usable.