Python sharing a lock between processes
You can't pass normal multiprocessing.Lock
objects to Pool
methods, because they can't be pickled. There are two ways to get around this. One is to create Manager()
and pass a Manager.Lock()
:
def main(): iterable = [1, 2, 3, 4, 5] pool = multiprocessing.Pool() m = multiprocessing.Manager() l = m.Lock() func = partial(target, l) pool.map(func, iterable) pool.close() pool.join()
This is a little bit heavyweight, though; using a Manager
requires spawning another process to host the Manager
server. And all calls to acquire
/release
the lock have to be sent to that server via IPC.
The other option is to pass the regular multiprocessing.Lock()
at Pool creation time, using the initializer
kwarg. This will make your lock instance global in all the child workers:
def target(iterable_item): for item in items: # Do cool stuff if (... some condition here ...): lock.acquire() # Write to stdout or logfile, etc. lock.release()def init(l): global lock lock = ldef main(): iterable = [1, 2, 3, 4, 5] l = multiprocessing.Lock() pool = multiprocessing.Pool(initializer=init, initargs=(l,)) pool.map(target, iterable) pool.close() pool.join()
The second solution has the side-effect of no longer requiring partial
.
Here's a version (using Barrier
instead of Lock
, but you get the idea) which would also work on Windows (where the missing fork
is causing additional troubles):
import multiprocessing as mpdef procs(uid_barrier): uid, barrier = uid_barrier print(uid, 'waiting') barrier.wait() print(uid, 'past barrier') def main(): N_PROCS = 10 with mp.Manager() as man: barrier = man.Barrier(N_PROCS) with mp.Pool(N_PROCS) as p: p.map(procs, ((uid, barrier) for uid in range(N_PROCS)))if __name__ == '__main__': mp.freeze_support() main()