Search paths in for .systemLibrary SwiftPM?

I’ve got a simple Swift wrapper around the libmodbus C library. It uses an umbrella header like this:

module.modulemap:

module libmodbus
{
	umbrella header "libmodbus.h"
	link "modbus"
}	

libmodbus.h:

#include <modbus.h>

This worked fine on my Intel Mac, where Homebrew installed libmodbus in /usr/local, but on Apple Silicon, it installs everything in /opt/homebrew/includes/modbus/ and /opt/homebrew/lib/.

I’d like for this to work in any environment (Intel, Linux), where the location of modbus.h (and the libraries) can be in different locations. How can I tell Swift where to look for .systemLibrary headers and libraries?

Update: It seems swift build works, but building the package in Xcode does not. Ah! Closing and re-opening the project, and then selecting "My Mac" as the target (instead of "My Mac (Rosetta)" allowed Xcode to build.