Tracking progress in python list(map(...))

Question

I am attempting to track progress in the following code:

from toolz import compose
calculator = compose(my_function, list, my_dict.get, tuple)
result = list(zip(*map(calculator, my_values)))

my_values is a list of length ~1mio. My first attempt is to add a counter to my_function that increments and print it out when a multiple of X (e.g. X==500) is reached.

Is there a pythonic or cleaner way to achieve this, i.e. without adding lots of counters to various loops? A progress bar in jupyter notebook would work too.

Answer 1

If a progress bar in Jupyter will work, I like to use tqdm, as it works for any iterable. Here is some sample code (slightly simplified from your example since I had to write my_function, my_values, etc):

def my_function(x):
    yield x + 2

my_values = range(1000000)

result = list(zip(*map(my_function, my_values))) 

Now just add tqdm on my_values (no progress checkers/counters clogging up your code!) to get a nice progress bar:

from tqdm import tqdm

def my_function(x):
    yield x + 2

my_values = tqdm(range(1000000))

result = list(zip(*map(my_function, my_values)))

which rolls through the awesome tqdm progress bar:

100%|██████████| 1000000/1000000 [00:04<00:00, 210661.41it/s]

Note I have nothing to do with the tqdm project; I just like using it. https://github.com/tqdm/tqdm