It’s not exactly what you're asking for, but it is possible to specify a .json destination file via the --destination file.json argument to swift build. In that, you can configure flags for the linker, swiftc, and clang.
I can't find the documentation right now, but the source file within swiftpm is here, and all of the members of the Destination struct can be set through the json file.
It's also possible to set Xcode-specific configuration flags through the --xcconfig-overrides argument to swift build, using an xcconfig file.
An example json destination file looks something like this:
{
"version": 1,
"sdk": "/swift/usr/bin/swift",
"toolchain-bin-dir": "/swift/usr/bin",
"target": "x86_64-unknown-windows-msvc",
"dynamic-library-extension": "lib",
"extra-cc-flags": [
],
"extra-swiftc-flags": [
"-static-stdlib",
"-use-ld=lld",
"-ILibraries/CVulkan/"
],
"extra-cpp-flags": []
}