PyEval_InitThreads in Python 3: How/when to call it? (the saga continues ad nauseam)

Your understanding is correct: invoking PyEval_InitThreads does, among other things, acquire the GIL. In a correctly written Python/C application, this is not an issue because the GIL will be unlocked in time, either automatically or manually.

If the main thread goes on to run Python code, there is nothing special to do, because Python interpreter will automatically relinquish the GIL after a number of instructions have been executed (allowing another thread to acquire it, which will relinquish it again, and so on). Additionally, whenever Python is about to invoke a blocking system call, e.g. to read from the network or write to a file, it will release the GIL around the call.

The original version of this answer pretty much ended here. But there is one more thing to take into account: the embedding scenario.

When embedding Python, the main thread often initializes Python and goes on to execute other, non-Python-related tasks. In that scenario there is nothing that will automatically release the GIL, so this must be done by the thread itself. That is in no way specific to the call that calls PyEval_InitThreads, it is expected of all Python/C code invoked with the GIL acquired.

For example, the main() might contain code like this:

Py_Initialize();PyEval_InitThreads();Py_BEGIN_ALLOW_THREADS... call the non-Python part of the application here ...Py_END_ALLOW_THREADSPy_Finalize();

If your code creates threads manually, they need to acquire the GIL before doing anything Python-related, even as simple as Py_INCREF. To do so, use the following:

// Acquire the GILPyGILState_STATE gstate;gstate = PyGILState_Ensure();... call Python code here ...// Release the GIL. No Python API allowed beyond this point.PyGILState_Release(gstate);

I have seen symptoms similar to yours: deadlocks if I only call PyEval_InitThreads(), because my main thread never calls anything from Python again, and segfaults if I unconditionally call something like PyEval_SaveThread(). The symptoms depend on the version of Python and on the situation: I am developing a plug-in that embeds Python for a library that can be loaded as part of a Python extension. The code needs therefore to run independent of whether it is loaded by Python as main.

The following worked for be with both python2.7 and python3.4, and with my library running within Python and outside of Python. In my plug-in init routine, which is executed in the main thread, I run:

  Py_InitializeEx(0);  if (!PyEval_ThreadsInitialized()) {    PyEval_InitThreads();    PyThreadState* mainPyThread = PyEval_SaveThread();  }

(mainPyThread is actually some static variable, but I don't think that matters as I never need to use it again).

Then I create threads using pthreads, and in each function that needs to access the Python API, I use:

  PyGILState_STATE gstate;  gstate = PyGILState_Ensure();  // Python C API calls  PyGILState_Release(gstate);

There are two methods of multi threading while executing C/Python API.

1.Execution of different threads with same interpreter - We can execute a Python interpreter and share the same interpreter over the different threads.

The coding will be as follows.

main(){     //initialize PythonPy_Initialize();PyRun_SimpleString("from time import time,ctime\n"    "print 'In Main, Today is',ctime(time())\n");//to Initialize and acquire the global interpreter lockPyEval_InitThreads();//release the lock  PyThreadState *_save;_save = PyEval_SaveThread();// Create threads.for (int i = 0; i<MAX_THREADS; i++){       hThreadArray[i] = CreateThread    //(...        MyThreadFunction,       // thread function name    //...)} // End of main thread creation loop.// Wait until all threads have terminated.//...//Close all thread handles and free memory allocations.//...//end python here//but need to check for GIL here tooPyEval_RestoreThread(_save);Py_Finalize();return 0;}//the thread functionDWORD WINAPI MyThreadFunction(LPVOID lpParam){//non Pythonic activity//...//check for the state of Python GILPyGILState_STATE gilState;gilState = PyGILState_Ensure();//execute Python herePyRun_SimpleString("from time import time,ctime\n"    "print 'In Thread Today is',ctime(time())\n");//release the GIL           PyGILState_Release(gilState);   //other non Pythonic activity//...return 0;}

2. Another method is that, we can execute a Python interpreter in the main thread and, to each thread we can give its own sub interpreter. Thus every thread runs with its own separate , independent versions of all imported modules, including the fundamental modules - builtins, __main__ and sys.

The code is as follows

int main(){// Initialize the main interpreterPy_Initialize();// Initialize and acquire the global interpreter lockPyEval_InitThreads();// Release the lock     PyThreadState *_save;_save = PyEval_SaveThread();// create threadsfor (int i = 0; i<MAX_THREADS; i++){    // Create the thread to begin execution on its own.    hThreadArray[i] = CreateThread    //(...        MyThreadFunction,       // thread function name    //...);   // returns the thread identifier } // End of main thread creation loop.  // Wait until all threads have terminated.WaitForMultipleObjects(MAX_THREADS, hThreadArray, TRUE, INFINITE);// Close all thread handles and free memory allocations.// ...//end python here//but need to check for GIL here too//re capture the lockPyEval_RestoreThread(_save);//end python interpreterPy_Finalize();return 0;}//the thread functionsDWORD WINAPI MyThreadFunction(LPVOID lpParam){// Non Pythonic activity// ...//create a new interpreterPyEval_AcquireLock(); // acquire lock on the GILPyThreadState* pThreadState = Py_NewInterpreter();assert(pThreadState != NULL); // check for failurePyEval_ReleaseThread(pThreadState); // release the GIL// switch in current interpreterPyEval_AcquireThread(pThreadState);//execute python codePyRun_SimpleString("from time import time,ctime\n" "print\n"    "print 'Today is',ctime(time())\n");// release current interpreterPyEval_ReleaseThread(pThreadState);//now to end the interpreterPyEval_AcquireThread(pThreadState); // lock the GILPy_EndInterpreter(pThreadState);PyEval_ReleaseLock(); // release the GIL// Other non Pythonic activityreturn 0;}

It is necessary to note that the Global Interpreter Lock still persists and, in spite of giving individual interpreters to each thread, when it comes to python execution, we can still execute only one thread at a time. GIL is UNIQUE to PROCESS, so in spite of providing unique sub interpreter to each thread, we cannot have simultaneous execution of threads

Sources: Executing a Python interpreter in the main thread and, to each thread we can give its own sub interpreter

Multi threading tutorial (msdn)