Threading pool similar to the multiprocessing Pool?
I just found out that there actually is a thread-based Pool interface in the multiprocessing module, however it is hidden somewhat and not properly documented.
It can be imported via
from multiprocessing.pool import ThreadPoolIt is implemented using a dummy Process class wrapping a python thread. This thread-based Process class can be found in multiprocessing.dummy which is mentioned briefly in the docs. This dummy module supposedly provides the whole multiprocessing interface based on threads.
In Python 3 you can use concurrent.futures.ThreadPoolExecutor, i.e.:
executor = ThreadPoolExecutor(max_workers=10)a = executor.submit(my_function)See the docs for more info and examples.
Yes, and it seems to have (more or less) the same API.
import multiprocessingdef worker(lnk): .... def start_process(): .........if(PROCESS): pool = multiprocessing.Pool(processes=POOL_SIZE, initializer=start_process)else: pool = multiprocessing.pool.ThreadPool(processes=POOL_SIZE, initializer=start_process)pool.map(worker, inputs)....