How to obtain the results from a pool of threads in python?
Python actually has a built-in thread pool you can use, its just not well documented:
from multiprocessing.pool import ThreadPooldef foo(word, number): print (word * number) r[(word,number)] = number return numberwords = ['hello', 'world', 'test', 'word', 'another test']numbers = [1,2,3,4,5]pool = ThreadPool(5)results = []for i in range(0, len(words)): results.append(pool.apply_async(foo, args=(words[i], numbers[i])))pool.close()pool.join()results = [r.get() for r in results]print results
Or (using map
instead of apply_async
):
from multiprocessing.pool import ThreadPooldef foo(word, number): print word*number return numberdef starfoo(args): """ We need this because map only supports calling functions with one arg. We need to pass two args, so we use this little wrapper function to expand a zipped list of all our arguments. """ return foo(*args)words = ['hello', 'world', 'test', 'word', 'another test']numbers = [1,2,3,4,5]pool = ThreadPool(5)# We need to zip together the two lists because map only supports calling functions# with one argument. In Python 3.3+, you can use starmap instead.results = pool.map(starfoo, zip(words, numbers))print resultspool.close()pool.join()