PythonKit not finding Python 3 library

Hi. :wave:

I'm experimenting with @pvieito's great looking PythonKit. I'm having a weird dlopen/search path issue.
I'm wondering if anyone can advise?

Problem

Following the project README I have a very simple hello world setup that imports PythonKit, sets sys.path to find my local script, imports that, and calls a function that prints a Hello, World!.

Thus far, all great. Out of the box, PythonKit is finding Python 2.7, so (again following the README) I try setting the PYTHON_VERSION and PYTHON_LIBRARY environment variables. (I also set PYTHON_LOADER_LOGGING for good measure.)

The issue is PYTHON_LIBRARY seems correct — it reports as loading correctly — but Python 2.7 is still being loaded.

This is the sample code from the README:

let sys = Python.import("sys")
print("Python \(sys.version_info.major).\(sys.version_info.minor)")
print("Python Version: \(sys.version)")
print("Python Encoding: \(sys.getdefaultencoding().upper())")

Which (with PYTHON_LIBRARY set) outputs:

Loading symbol 'Py_Initialize' from the Python library...
Trying to load library at '/opt/boxen/homebrew/Cellar/python@3.8/3.8.7/Frameworks/Python.framework/Python'...
Library at '/opt/boxen/homebrew/Cellar/python@3.8/3.8.7/Frameworks/Python.framework/Python' was sucessfully loaded.
Loading symbol 'PyEval_GetBuiltins' from the Python library...
Loading symbol 'Py_IncRef' from the Python library...
Loading symbol 'PyRun_SimpleString' from the Python library...
Loading symbol 'PyImport_ImportModule' from the Python library...
Loading symbol 'PyObject_GetAttrString' from the Python library...
Loading symbol 'Py_DecRef' from the Python library...
Loaded legacy Python library, using legacy symbols...
Loading symbol 'PyString_FromStringAndSize' from the Python library...
Loading symbol 'PyObject_GetItem' from the Python library...
Loading symbol 'PyErr_Occurred' from the Python library...
Loading symbol 'PyTuple_New' from the Python library...
Loading symbol 'PyTuple_SetItem' from the Python library...
Loading symbol 'PyObject_CallObject' from the Python library...
Loading symbol 'PyString_AsString' from the Python library...
Python 2.7
Python Version: 2.7.16 (default, Jan 22 2021, 05:09:26) 
[GCC Apple LLVM 12.0.5 (clang-1205.0.19.9) [+internal-os, ptrauth-isa=deploymen
Python Encoding: ASCII

With the Library at <PATH> was sucessfully loaded., which is the 3rd line of the output above , I'm expecting Python 3.8, but 2.7 is in fact what we get.

With a test script such as:

print("Hello, world! (From Python 💃)")

... we get a crash, without declaring the file encoding — Ah, yes, definitely Python 2… :slightly_smiling_face:

I'm not quite sure (yet) how to get the planets aligned here. How can I get PythonKit to pick up the right Python version? Is there something obvious I'm missing? (Quite likely.)

Many thanks if you're to advise! I know this sort of thing is difficult to comment on.
Kind Regards,

Carlton

Possibly a related post in another thread suggesting the need to edit the PythonLibrary.swift file to set the search paths correctly. :thinking:

Hi! That may be related to the Python 2 library being linked to the executable, you can check that with otool:

$ otool -L path_to_your_executable
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1770.255.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1770.255.0)
	@rpath/libswiftCore.dylib (compatibility version 1.0.0, current version 1200.2.41)
	@rpath/libswiftCoreFoundation.dylib (compatibility version 1.0.0, current version 1.6.0, weak)
	@rpath/libswiftCoreGraphics.dylib (compatibility version 1.0.0, current version 2.0.0, weak)
	@rpath/libswiftDarwin.dylib (compatibility version 1.0.0, current version 0.0.0)
	@rpath/libswiftDispatch.dylib (compatibility version 1.0.0, current version 4.40.2, weak)
	@rpath/libswiftFoundation.dylib (compatibility version 1.0.0, current version 20.0.0)
	@rpath/libswiftIOKit.dylib (compatibility version 1.0.0, current version 1.0.0, weak)
	@rpath/libswiftObjectiveC.dylib (compatibility version 1.0.0, current version 1.0.0, weak)
	@rpath/libswiftXPC.dylib (compatibility version 1.0.0, current version 1.1.0, weak)

Make sure no references are made to Python and that you are not linking nor importing the Python module in your code.

1 Like

Hi @pvieito — thanks for the quick response!

I can't see that I'm linking against Python...

$ otool -L /Users/carlton/Library/Developer/Xcode/DerivedData/FromPython-axlvqoykyiwdfwddzgxlgofxeral/Build/Products/Debug/FromPython
/Users/carlton/Library/Developer/Xcode/DerivedData/FromPython-axlvqoykyiwdfwddzgxlgofxeral/Build/Products/Debug/FromPython:
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1770.255.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
	/usr/lib/swift/libswiftCore.dylib (compatibility version 1.0.0, current version 1200.2.41)
	/usr/lib/swift/libswiftCoreFoundation.dylib (compatibility version 1.0.0, current version 1.6.0, weak)
	/usr/lib/swift/libswiftCoreGraphics.dylib (compatibility version 1.0.0, current version 2.0.0, weak)
	/usr/lib/swift/libswiftDarwin.dylib (compatibility version 1.0.0, current version 0.0.0)
	/usr/lib/swift/libswiftDispatch.dylib (compatibility version 1.0.0, current version 4.40.2, weak)
	/usr/lib/swift/libswiftFoundation.dylib (compatibility version 1.0.0, current version 20.0.0, weak)
	/usr/lib/swift/libswiftIOKit.dylib (compatibility version 1.0.0, current version 1.0.0, weak)
	/usr/lib/swift/libswiftObjectiveC.dylib (compatibility version 1.0.0, current version 1.0.0, weak)
	/usr/lib/swift/libswiftXPC.dylib (compatibility version 1.0.0, current version 1.1.0, weak)

:thinking: Still thinking.

Hey @pvieito — it's always SIP :grinning:

Set "Enable Hardened Runtime" to "No" and I get the correct Python load.
:dancer: — I'm off. :tada:

I'll experiment but, this makes me wonder if static linking the libpython would work (despite the dlopen)... — that would be quite handy. Gonna have a read of Use RTDL_SELF to load embedded symbols by kewlbear · Pull Request #30 · pvieito/PythonKit · GitHub.

Thanks so much for your efforts on this lib. Very exciting! (I see you don't have Issues or Discussions enabled, so you prefer it here yes?)

Have a great rest of your weekend, and thank you for the help! :medal_sports:

Oh, gotcha! Yes, with the “Hardened Runtime” enabled the process won't load libraries not signed by Apple or the developer. You can embed the Python framework in the app and sign it with your developer certificate.

1 Like

Do you know the exact steps to accomplish that (embed the Python framework in the app)?

I'm having very weird problems loading python today. It's been loading fine until recently, and I'm really not sure what could have changed. My python lib is from Conda so I point to that in my PythonLibrary.useLibrary(at:) call. The PYTHON_LOADER_LOGGING indicates that it is trying to load the intended version. I'm running the app on macOS and I've disabled App Sandbox and Hardened Runtime in the Build Settings:

Loading symbol 'Py_Initialize' from the Python library...
Trying to load library at '/Users/xxxx/anaconda3/envs/torch_17/lib/libpython3.8.dylib'...
PythonKit/PythonLibrary.swift:46: Fatal error: Python library not found. Set the PYTHON_LIBRARY environment variable with the path to a Python library.
2021-10-08 16:15:09.479242-0700 SpliqGenerate[4532:37353] PythonKit/PythonLibrary.swift:46: Fatal error: Python library not found. Set the PYTHON_LIBRARY environment variable with the path to a Python library.

Based on the error I also added PYTHON_LIBRARY to my scheme's Environment Variables, but it makes no difference.

Any ideas? (Particularly curious what might have made a working lib suddenly stop working.)

PS — Just running python at the terminal in this Conda env works as expected.

UPDATE: The one thing that's changed is that I've added other package dependencies to the project. Is it possible that one of these is somehow preventing Python from loading?

A little further info: it is able to load my non-anaconda (brew, I think) system python3 (3.8.1), so something specifically related to conda?...

UPDATE 2: Tried a new basic test app, with no dependencies, and it's the same thing. Seems to be isolated to Conda env python versions. Weird. It was working... (I suppose I could install all my dependencies into my brew python3, but I don't love the idea...)
I've tried other envs, too, with no success.

UPDATE 3: Finally figured this out. It wasn't clear to me the difference between Enable App Sandbox in Build Settings and the actual App Sandbox entitlement in my app's .entitlements file. It seems the file has to have its App Sandbox key set to NO. It's very strange to me that Xcode doesn't include that in the actual app GUI, but I suppose they're actively discouraging people from changing it... who knows...??

1 Like

Confirmed, to get PythonKit in my swift app to load my macports python library I had to:

  1. Set environment variable PYTHON_LIBRARY = "/opt/local/Library/Frameworks/Python.framework/Python" under Scheme->Run
  2. Remove App Sandbox section from build settings -> Signing and capabilities (click the trashcan icon next to the whole section)
1 Like

Works when I set the environment variable PYTHON_LIBRARY but mine's in a different path. Had to run "which -a python" to find the path

To my knowledge, PYTHON_LIBRARY should reference the Python library file and not the directory where Python is installed.