Python, multithreading too slow, multiprocess Python, multithreading too slow, multiprocess multithreading multithreading

Python, multithreading too slow, multiprocess


Just replace threading with multiprocessing and Thread with Process. Threads in Pyton are (almost) never used to gain performance because of the big bad GIL! I explained it in an another SO-post with some links to documentation and a great talk about threading in python.

But the multiprocessing module is intentionally very similar to the threading module. You can almost use it as an drop-in replacement!

The multiprocessing module doesn't AFAIK offer a functionality to enforce the use of a specific amount of cores. It relies on the OS-implementation. You could use the Pool object and limit the worker-onjects to the core-count. Or you could look for an other MPI library like pypar. Under Linux you could use a pipe under the shell to start multiple instances on different cores