I don't think compatibility of swiftly and the Swift toolchain build script has been tested. As discussed in a similar thread Swift compiler building error after cloning the repo - #26 by Max_Desiatov, it's best to replicate the CI environment as closely as possible, which does not use swiftly at all.
I'd recommend using the Docker image that CI uses for building the toolchain. If you prefer Ubuntu 24.04, it's this one: swift-docker/swift-ci/main/ubuntu/24.04/Dockerfile at main · swiftlang/swift-docker · GitHub