Multiprocessing with "fork" context blocks on Linux/Intel Xeon with Python 3.6.1? Multiprocessing with "fork" context blocks on Linux/Intel Xeon with Python 3.6.1? python-3.x python-3.x

Multiprocessing with "fork" context blocks on Linux/Intel Xeon with Python 3.6.1?


As @jxh hinted, the differences between fork and spawn are important. The documentation on multiprocessing indicates in section 17.2.1.2 that the difference is: forking preserves the environment and things like stdin/out, whereas spawn just makes a fresh new process. I think you maybe have something in your environment which causes issues for the worker function, likely in the code behind your comments about other processing. Spawning gives you a clean slate and things are running fine under those conditions.

To determine what is going on, I would have each worker print diagnostic messages, probably written to a file unique to each worker. open/close that file each time you want to write a message, so the contents gets updated/flushed.

Fork should not be faster than spawn, because fork needs to copy the environment information to the new process. In any case, this is only the startup cost which is minimal, I assume, because the worker needs to do some computationally or I/O bound work that you want to parallelize.