How do you pass a Queue reference to a function managed by pool.map_async()? How do you pass a Queue reference to a function managed by pool.map_async()? python python

How do you pass a Queue reference to a function managed by pool.map_async()?


The following code seems to work:

import multiprocessing, timedef task(args):    count = args[0]    queue = args[1]    for i in xrange(count):        queue.put("%d mississippi" % i)    return "Done"def main():    manager = multiprocessing.Manager()    q = manager.Queue()    pool = multiprocessing.Pool()    result = pool.map_async(task, [(x, q) for x in range(10)])    time.sleep(1)    while not q.empty():        print q.get()    print result.get()if __name__ == "__main__":    main()

Note that the Queue is got from a manager.Queue() rather than multiprocessing.Queue(). Thanks Alex for pointing me in this direction.


Making q global works...:

import multiprocessing, timeq = multiprocessing.Queue()def task(count):    for i in xrange(count):        q.put("%d mississippi" % i)    return "Done"def main():    pool = multiprocessing.Pool()    result = pool.map_async(task, range(10))    time.sleep(1)    while not q.empty():        print q.get()    print result.get()if __name__ == "__main__":    main()

If you need multiple queues, e.g. to avoid mixing up the progress of the various pool processes, a global list of queues should work (of course, each process will then need to know what index in the list to use, but that's OK to pass as an argument;-).