I think that the error isn't saying what you think it's saying. I think that PYTHON_LIBRARY isn't the path to the external pip package libraries - it's looking for the path to the library for Python itself.
Anyway - you may have better luck trying the other approach (use the version of Python that PythonKit is already finding). It worked for me.
When I installed Python via brew and attempted to install some packages via pip I was warned that:
× This environment is externally managed
If you wish to install a Python library that isn't in Homebrew,
use a virtual environment:
So I made a virtual environment (which is a separate location for installed packages etc...)
Then I needed to add that location to the PATH for the instance of Python that PythonKit was using.
In my Swift test app I added the following:
let os = Python.import("os")
let sys = Python.import("sys")
sys.path.append(os.path.abspath("/Users/diggory/python/venv/lib/python3.12/site-packages"))
That path is only valid on my machine (obviously). If you open the terminal and in the Python interpreter type
import sys
sys.path
It will show you the paths that it has. One of which is where your pip packages have been installed.
n.b. This isn't very portable as you are hard-coding a local path into your code (but then you are relying on pip installed packages anyway, so I assume that this project is just for your own machine and you're not expecting anyone else to run your app.)
It's also fragile as the python version is hard-coded into the path.
Also - this means that you are using packages installed by a different version of Python that the one that is using them, which could cause problems...