Multiprocessing : use tqdm to display a progress bar
Use imap instead of map, which returns an iterator of processed values.
from multiprocessing import Poolimport tqdmimport timedef _foo(my_number): square = my_number * my_number time.sleep(1) return square if __name__ == '__main__': with Pool(2) as p: r = list(tqdm.tqdm(p.imap(_foo, range(30)), total=30))
Sorry for being late but if all you need is a concurrent map, I added this functionality in tqdm>=4.42.0
:
from tqdm.contrib.concurrent import process_map # or thread_mapimport timedef _foo(my_number): square = my_number * my_number time.sleep(1) return square if __name__ == '__main__': r = process_map(_foo, range(0, 30), max_workers=2)
References: https://tqdm.github.io/docs/contrib.concurrent/ and https://github.com/tqdm/tqdm/blob/master/examples/parallel_bars.py
It supports max_workers
and chunksize
and you can also easily switch from process_map
to thread_map
.
Solution Found : Be careful! Due to multiprocessing, estimation time (iteration per loop, total time, etc.) could be unstable, but the progress bar works perfectly.
Note: Context manager for Pool is only available from Python version 3.3
from multiprocessing import Poolimport timefrom tqdm import *def _foo(my_number): square = my_number * my_number time.sleep(1) return square if __name__ == '__main__': with Pool(processes=2) as p: max_ = 30 with tqdm(total=max_) as pbar: for i, _ in enumerate(p.imap_unordered(_foo, range(0, max_))): pbar.update()