Xcbuild hangs when building SwiftPM from source

Hey guys, when I was trying to build Swift toolchain on my mac(apple silicon), I found that the build will hang at "Building SwiftPM" stage.

Specifically, it stops at the swift-build command that builds the SwiftPM itself:

swift-build --disable-sandbox --force-resolved-versions --build-path /Users/whoami/code/toolchain/swift-project/build/buildbot_osx/swiftpm-macosx-x86_64 --configuration release -Xlinker -rpath -Xlinker @executable_path/../lib/swift/macosx --arch x86_64 --arch arm64 -Xswiftc -module-cache-path -Xswiftc /Users/whoami/code/toolchain/swift-project/build/buildbot_osx/swiftpm-macosx-x86_64/module-cache -Xmanifest -module-cache-path -Xmanifest /Users/whoami/code/toolchain/swift-project/build/buildbot_osx/swiftpm-macosx-x86_64/module-cache

I put an -v to this swift-build to print more info and it appeared to actually stop at xcbuild:

/Applications/Xcode.app/Contents/SharedFrameworks/XCBuild.framework/Versions/A/Support/xcbuild build /Users/whoami/code/toolchain/swift-project/build/buildbot_osx/swiftpm-macosx-x86_64/manifest.pif --configuration Release --derivedDataPath /Users/whoami/code/toolchain/swift-project/build/buildbot_osx/swiftpm-macosx-x86_64/apple --target AllExcludingTests --buildParametersFile /var/folders/30/cmv9c_5j3mq_kthx63sb1t5c0000gn/T/TemporaryFile.FOMrAW

However if I execute swift-build or xcbuild manually with sudo, it will proceeds.

sudo swift-build ...
sudo xcbuild ...

Any clue? Thanks!