AFAIK, Swift packages in Xcode are built for the active architecture in debug mode and universally in release mode and there's no way to control the behavior. You may try changing the setting in your project, but I don't believe Xcode's SPM integration passes that setting through.
That is correct, packages behave like projects do by default here and there's no control over the build setting. Build settings of client projects generally don't apply to packages in any way.
For what it's worth, I've filed FB9915302 about this. It's a major workflow issue for apps which need to be built with optimisations to run well enough to be tested – the compile times are very lengthy regardless and building for architectures you're not going to use only makes it worse.
I'll also add that per-target optimisation settings would be very useful – it's often the case that I'm working on a particular package and want that to be built in debug mode, but still need other packages it depends on to be in release mode so the app is usable.