As @odmir says, this has been discussed before. If it is a particular concern for you, perhaps you can chip in with slimming that down.
On the other hand, if you're deploying on the server, it shouldn't be hard to make sure that your linux distro has the system Swift package installed or to otherwise provision it with the Swift runtime as shared libraries that your app uses, assuming you have some control of that server.
It is not clear why you insist on the minority usecase of building a tiny static executable, which Go and Rust have focused on but Swift hasn't.