Embedding python in multithreaded C application

I had exactly the same problem and it is now solved by using PyEval_SaveThread() immediately after PyEval_InitThreads(), as you suggest above. However, my actual problem was that I used PyEval_InitThreads() after PyInitialise() which then caused PyGILState_Ensure() to block when called from different, subsequent native threads. In summary, this is what I do now:

1. There is global variable:

   static int gil_init = 0; 

2. From a main thread load the native C extension and start the Python interpreter:

   Py_Initialize() 

3. From multiple other threads my app concurrently makes a lot of calls into the Python/C API:

   if (!gil_init) {    gil_init = 1;    PyEval_InitThreads();    PyEval_SaveThread();}state = PyGILState_Ensure();// Call Python/C API functions...    PyGILState_Release(state);

4. From the main thread stop the Python interpreter

   Py_Finalize()

All other solutions I've tried either caused random Python sigfaults or deadlock/blocking using PyGILState_Ensure().

The Python documentation really should be more clear on this and at least provide an example for both the embedding and extension use cases.

Eventually I figured it out.
After

PyEval_InitThreads();

You need to call

PyEval_SaveThread();

While properly release the GIL for the main thread.

Having a multi-threaded C app trying to communicate from multiple threads to multiple Python threads of a single CPython instance looks risky to me.

As long as only one C thread communicates with Python you should not have to worry about locking even if the Python application is multi-threading.If you need multiple python threads you can set the application up this way and have multiple C threads communicate via a queue with that single C thread that farms them out to multiple Python threads.

An alternative that might work for you is to have multiple CPython instances one for each C thread that needs it (of course communication between Python programs should be via the C program).

Another alternative might the Stackless Python interpreter. That does away with the GIL, but I am not sure you run into other problems binding it to multiple threads. stackless was a drop-in replacement for my (single-threaded) C application.