Hi @bclee ! Yes it was your patch I remember fixing the issue. The problem was that I didn't really want to spend a lot of time as 5.8.1 was the last version that didn't need an existing Swift install (until @Finagolfin mentioned a workaround that I didn't know about).
While looking for your patch/info I added a tiny patch to set the build to debug, and not release, and that actually made it build under aarch64; my goal is to immediately follow that up with going straight to 6.0.3-RELEASE so theoretically the changes of someone running an aarch64 version of CentOS Stream and installing Swift are likely pretty small.
It will be great to have EPEL 10 finally part of the release chain when deploying a new version.