How to use system libraries in a Clang target?

I have a Swift target RaylibKit wrapping a C library target raylib which depends on a system library X11. I can make the system target and use it in Swift but raylib still can't find the headers (or PC file).

Here's the system target definition:

.systemLibrary(name: "X11", pkgConfig: "X11", provides: [
    // ...
]),

Here's the X11 module map:

module X11 [system] {
    module Xlib {
        header "/usr/include/X11/Xlib.h"
    }
    module X {
        header "/usr/include/X11/X.h"
    }
    link "X11"
}

Anyone successfully done this *without* using unsafeFlags? Until their removal is in stable toolchains and Xcode I don't want to rely on them.