Multiprocessing : use tqdm to display a progress bar Multiprocessing : use tqdm to display a progress bar python python

Multiprocessing : use tqdm to display a progress bar


Use imap instead of map, which returns an iterator of processed values.

from multiprocessing import Poolimport tqdmimport timedef _foo(my_number):   square = my_number * my_number   time.sleep(1)   return square if __name__ == '__main__':   with Pool(2) as p:      r = list(tqdm.tqdm(p.imap(_foo, range(30)), total=30))


Sorry for being late but if all you need is a concurrent map, I added this functionality in tqdm>=4.42.0:

from tqdm.contrib.concurrent import process_map  # or thread_mapimport timedef _foo(my_number):   square = my_number * my_number   time.sleep(1)   return square if __name__ == '__main__':   r = process_map(_foo, range(0, 30), max_workers=2)

References: https://tqdm.github.io/docs/contrib.concurrent/ and https://github.com/tqdm/tqdm/blob/master/examples/parallel_bars.py

It supports max_workers and chunksize and you can also easily switch from process_map to thread_map.


Solution Found : Be careful! Due to multiprocessing, estimation time (iteration per loop, total time, etc.) could be unstable, but the progress bar works perfectly.

Note: Context manager for Pool is only available from Python version 3.3

from multiprocessing import Poolimport timefrom tqdm import *def _foo(my_number):   square = my_number * my_number   time.sleep(1)   return square if __name__ == '__main__':    with Pool(processes=2) as p:        max_ = 30        with tqdm(total=max_) as pbar:            for i, _ in enumerate(p.imap_unordered(_foo, range(0, max_))):                pbar.update()