How can we use tqdm in a parallel execution with joblib?

Just put range(10) inside tqdm(...)! It probably seemed too good to be true for you, but it really works (on my machine):

from math import sqrtfrom joblib import Parallel, delayed  from tqdm import tqdm  result = Parallel(n_jobs=2)(delayed(sqrt)(i ** 2) for i in tqdm(range(100000)))

I've created pqdm a parallel tqdm wrapper with concurrent futures to comfortably get this done, give it a try!

To install

pip install pqdm

and use

from pqdm.processes import pqdm# If you want threads instead:# from pqdm.threads import pqdmargs = [1, 2, 3, 4, 5]# args = range(1,6) would also workdef square(a):    return a*aresult = pqdm(args, square, n_jobs=2)

Modifying nth's great answer to permit a dynamic flag to use TQDM or not and to specify the total ahead of time so that the status bar fills in correctly.

from tqdm.auto import tqdmfrom joblib import Parallelclass ProgressParallel(Parallel):    def __init__(self, use_tqdm=True, total=None, *args, **kwargs):        self._use_tqdm = use_tqdm        self._total = total        super().__init__(*args, **kwargs)    def __call__(self, *args, **kwargs):        with tqdm(disable=not self._use_tqdm, total=self._total) as self._pbar:            return Parallel.__call__(self, *args, **kwargs)    def print_progress(self):        if self._total is None:            self._pbar.total = self.n_dispatched_tasks        self._pbar.n = self.n_completed_tasks        self._pbar.refresh()