Python 2.7 concurrent.futures.ThreadPoolExecutor does not parallelize Python 2.7 concurrent.futures.ThreadPoolExecutor does not parallelize linux linux

Python 2.7 concurrent.futures.ThreadPoolExecutor does not parallelize


It sounds like you're seeing the results of Python's Global Interpreter Lock (a.k.a GIL).

In CPython, the global interpreter lock, or GIL, is a mutex that prevents multiple native threads from executing Python bytecodes at once.

As all your threads are running pure Python code, only one of them can actually run in parallel. That should cause only one CPU to be active and matches your description of the problem.

You can get around it by using a multiple processes with ProcessPoolExecutor from the same module. Other solutions include switching to Jython or IronPython which don't have GIL.

The ProcessPoolExecutor class is an Executor subclass that uses a pool of processes to execute calls asynchronously. ProcessPoolExecutor uses the multiprocessing module, which allows it to side-step the Global Interpreter Lock but also means that only picklable objects can be executed and returned.