I guess in order to call a Python function via PythonKit multiple times in parallel, you should use parallelization within the according Python script e.g. by using the „multiprocessing“ module? Or better calling Python.attemptImport(pythonModule) several times, is this possible?
There is another forums topic with this question that just says no (calls to Python should be sequentialized).
The question was also discussed in the TensorFlow group.
I guess you could do something with Python‘s multiprocessing module (then different Python interpreters are started), but then the elegant and easy possibility to directly call differently Python functions would be gone.
So for me, it is also a „no“. PythonKit is great, but I will start a separate Swift program for each work item / request.
Update:
I am thinking about using Cython (not: CPython) for my purpose. Has anyone tried Cython together with Swift? If successful, I will make a little demo project and post an according topic.
Found:
Cython or Pypy probably will work, since they don't have GIL, but AFAICT, nobody did serious work to integrate with these niche Python platforms. With CPython, You have to use multiprocess package if you want to run Python in parallel. However, to run your Swift code in parallel with Python, or calling Python code from a dispatch queue / Task, you can use save / restore primitive. I have that in my PythonKit fork: Add support to release GIL if needed. · liuliu/PythonKit@99a298f · GitHub
Sounds good, thanks. Do you have some sample code, or a simple explanation how to use it (and why it‘s working)? Thanks.
BTW, wouldn‘t it be good to get your code (+ some explanation) into the original PythonKit then?
BTW, how would you judge IronPython in this regard (being a niche), meaning: are the alternative Python distributions „serious“ replacements for CPython?
Oh, I meant in Swift context as "nobody did serious work to integrate these with Swift". Not saying these are not serious work in itself! These are impressive works.
It is the same as in the Python doc: Initialization, Finalization, and Threads — Python 3.12.0 documentation Just exposed in Swift interface.
Now I remember that I didn't implement Initialization, Finalization, and Threads — Python 3.12.0 documentation, so it means you can run Swift code in parallel with whatever you run in Python's Thread: threading — Thread-based parallelism — Python 3.12.0 documentation, not that you can setup Python-aware thread (which requires you to call PyGILState_Ensure / PyGILState_Release pair).
Forgive me my ignorance: I would like to call Python functions from several different threads in Swift, e.g. when having to call Python functions during different requests in a web application, how would I do that? Thanks again.
From my understanding, the one Swift thread would then have to wait until the Python call of the other thread has finished? (This edit has already been answered, see blow.)
Each thread need to call PyGILState_Ensure first, and then you can call Python functions. After done with Python functions, you call PyGILState_Release.
This setup makes sure the thread aware of the GIL and acquire them properly when interpreting Python code (actually, when calling PyGILState_Ensure, it will acquire GIL). So the Python code doesn't really run in "parallel", they serialized against the GIL, but at least now you can execute your Python code from different Swift threads.
Yes, unless these Python functions internally called their C extension and release GIL due to blocking IOs. This is the fundamental limitation of Python because of GIL.
To run it truly parallel, use the multiprocess package. It works fine from Swift side.
From what I read, they both do have a GIL.
CPython might gain the ability to run without the GIL, see e.g. there, but from my understanding not before version 3.12 (or maybe much later).
Maybe I use a bridge to .NET (see also there) and use IronPython (or just wait for a GIL-free CPython).