python multiprocessing Pool not always using all workers

The multiprocessing.Pool relies on a single os.pipe to deliver the tasks to the workers.

Usually on Unix, the default pipe size range from 4 to 64 Kb in size. If the JSONs you are delivering are large in size, you might get the pipe clogged at any given point in time.

This means that, while one of the workers is busy reading the large JSON from the pipe, all the other workers will starve.

It is generally a bad practice to share large data via IPC as it leads to bad performance. This is even underlined in the multiprocessing programming guidelines.

Avoid shared state

As far as possible one should try to avoid shifting large amounts of data between processes.

Instead of reading the JSON files in the main process, just send the workers their file names and let them open and read the content. You will surely notice an improvement in performance because you are moving the JSON loading phase in the concurrent domain as well.

Note that the same is true also for the results. A single os.pipe is used to return the results to the main process as well. If one or more workers clog the results pipe then you will get all the processes waiting for the main one to drain it. Large results should be written to files as well. You can then leverage multithreading on the main process to quickly read back the results from the files.