I have managed to improve 32-bit build performance greatly by using combination of docker platform (linux/arm/v7) and a custom arch32v7 docker image with a patched uname. This gives us near native performance and what was taking 2 hours is now only taking 15 mins.
The server is up and running at https://ci.swiftlang.xyz/
Debian Bullseye Armv7 build job - swift-5.6.1-debian-bullseye-armv7 [Jenkins]
The Docker images are available here - Docker Hub
I have also setup a Github repository ci-swiftlang for the build jobs and will have more details on how to contribute patches and fixes soon, once I get PR testing setup.