Python Process Pool non-daemonic?
The multiprocessing.pool.Pool
class creates the worker processes in its __init__
method, makes them daemonic and starts them, and it is not possible to re-set their daemon
attribute to False
before they are started (and afterwards it's not allowed anymore). But you can create your own sub-class of multiprocesing.pool.Pool
(multiprocessing.Pool
is just a wrapper function) and substitute your own multiprocessing.Process
sub-class, which is always non-daemonic, to be used for the worker processes.
Here's a full example of how to do this. The important parts are the two classes NoDaemonProcess
and MyPool
at the top and to call pool.close()
and pool.join()
on your MyPool
instance at the end.
#!/usr/bin/env python# -*- coding: UTF-8 -*-import multiprocessing# We must import this explicitly, it is not imported by the top-level# multiprocessing module.import multiprocessing.poolimport timefrom random import randintclass NoDaemonProcess(multiprocessing.Process): # make 'daemon' attribute always return False def _get_daemon(self): return False def _set_daemon(self, value): pass daemon = property(_get_daemon, _set_daemon)# We sub-class multiprocessing.pool.Pool instead of multiprocessing.Pool# because the latter is only a wrapper function, not a proper class.class MyPool(multiprocessing.pool.Pool): Process = NoDaemonProcessdef sleepawhile(t): print("Sleeping %i seconds..." % t) time.sleep(t) return tdef work(num_procs): print("Creating %i (daemon) workers and jobs in child." % num_procs) pool = multiprocessing.Pool(num_procs) result = pool.map(sleepawhile, [randint(1, 5) for x in range(num_procs)]) # The following is not really needed, since the (daemon) workers of the # child's pool are killed when the child is terminated, but it's good # practice to cleanup after ourselves anyway. pool.close() pool.join() return resultdef test(): print("Creating 5 (non-daemon) workers and jobs in main process.") pool = MyPool(5) result = pool.map(work, [randint(1, 5) for x in range(5)]) pool.close() pool.join() print(result)if __name__ == '__main__': test()
I had the necessity to employ a non-daemonic pool in Python 3.7 and ended up adapting the code posted in the accepted answer. Below there's the snippet that creates the non-daemonic pool:
import multiprocessing.poolclass NoDaemonProcess(multiprocessing.Process): @property def daemon(self): return False @daemon.setter def daemon(self, value): passclass NoDaemonContext(type(multiprocessing.get_context())): Process = NoDaemonProcess# We sub-class multiprocessing.pool.Pool instead of multiprocessing.Pool# because the latter is only a wrapper function, not a proper class.class NestablePool(multiprocessing.pool.Pool): def __init__(self, *args, **kwargs): kwargs['context'] = NoDaemonContext() super(NestablePool, self).__init__(*args, **kwargs)
As the current implementation of multiprocessing
has been extensively refactored to be based on contexts, we need to provide a NoDaemonContext
class that has our NoDaemonProcess
as attribute. NestablePool
will then use that context instead of the default one.
That said, I should warn that there are at least two caveats to this approach:
- It still depends on implementation details of the
multiprocessing
package, and could therefore break at any time. - There are valid reasons why
multiprocessing
made it so hard to use non-daemonic processes, many of which are explained here. The most compelling in my opinion is:
As for allowing children threads to spawn off children of its own usingsubprocess runs the risk of creating a little army of zombie'grandchildren' if either the parent or child threads terminate beforethe subprocess completes and returns.
The multiprocessing module has a nice interface to use pools with processes or threads. Depending on your current use case, you might consider using multiprocessing.pool.ThreadPool
for your outer Pool, which will result in threads (that allow to spawn processes from within) as opposed to processes.
It might be limited by the GIL, but in my particular case (I tested both), the startup time for the processes from the outer Pool
as created here far outweighed the solution with ThreadPool
.
It's really easy to swap Processes
for Threads
. Read more about how to use a ThreadPool
solution here or here.