How can we use tqdm in a parallel execution with joblib?

Question

I want to run a function in parallel, and wait until all parallel nodes are done, using joblib. Like in the example:

from math import sqrt from joblib import Parallel, delayed Parallel(n_jobs=2)(delayed(sqrt)(i ** 2) for i in range(10)) 

But, I want that the execution will be seen in a single progressbar like with tqdm, showing how many jobs has been completed.

How would you do that?

Answer 1

Just put range(10) inside tqdm(...)! It probably seemed too good to be true for you, but it really works (on my machine):

from math import sqrt from joblib import Parallel, delayed   from tqdm import tqdm   result = Parallel(n_jobs=2)(delayed(sqrt)(i ** 2) for i in tqdm(range(100000))) 

Answer 2

I've created pqdm a parallel tqdm wrapper with concurrent futures to comfortably get this done, give it a try!

To install

pip install pqdm 

and use

from pqdm.processes import pqdm # If you want threads instead: # from pqdm.threads import pqdm  args = [1, 2, 3, 4, 5] # args = range(1,6) would also work  def square(a):     return a*a  result = pqdm(args, square, n_jobs=2)