How do you pass a Queue reference to a function managed by pool.map_async()?
The following code seems to work:
import multiprocessing, timedef task(args): count = args[0] queue = args[1] for i in xrange(count): queue.put("%d mississippi" % i) return "Done"def main(): manager = multiprocessing.Manager() q = manager.Queue() pool = multiprocessing.Pool() result = pool.map_async(task, [(x, q) for x in range(10)]) time.sleep(1) while not q.empty(): print q.get() print result.get()if __name__ == "__main__": main()
Note that the Queue is got from a manager.Queue() rather than multiprocessing.Queue(). Thanks Alex for pointing me in this direction.
Making q
global works...:
import multiprocessing, timeq = multiprocessing.Queue()def task(count): for i in xrange(count): q.put("%d mississippi" % i) return "Done"def main(): pool = multiprocessing.Pool() result = pool.map_async(task, range(10)) time.sleep(1) while not q.empty(): print q.get() print result.get()if __name__ == "__main__": main()
If you need multiple queues, e.g. to avoid mixing up the progress of the various pool processes, a global list of queues should work (of course, each process will then need to know what index in the list to use, but that's OK to pass as an argument;-).