#297: Progress Bars Everywhere With Tqdm

Since we used progressbar2 in Python Friday #71 a lot changed and even with basic things like a progress bar we can find many new libraries. Some already faded away, while others give us new and helpful features. In this post we explore the tqdm library that is a quick and easy replacement for all the places we used progressbar2.

Installation

According to the documentation, the name tqdm derives from the Arabic word taqaddum which can be translated as "progress", just in case you wondered if the name means something. We can install this library with this command:

pip install tqdm

Basic usage

To get a progress bar for any iterable, we can wrap tqdm() around it to get our progress bar:

import time
from tqdm import tqdm

for i in tqdm(range(100), desc="Processing"):
    time.sleep(0.05)

We now have a progress bar that grows with each iteration of the loop.

The main point that I noticed was the items per second, that I found especially helpful when I wanted to know how much items are currently processed.

For unknown sizes of work

When we do not know the number of items, we can use tqdm without any modifications. However, to get a slightly more useful output, we can use the desc= parameter and set a description for our progress bar:

from tqdm import tqdm
import time, random

def random_gen():
    while True:
        yield random.random()

for x in tqdm(random_gen(), desc="Working on"):
    time.sleep(0.01)

We now have an indicator that shows some work is in progress with the current item number, the duration, and the items per second.

The indicator now shows that something is happening and how fast, but without a known size, we cannot determine the end time or the percentage of work completed.

Integration with Pandas

Especially for long running transformations in Pandas it is sometimes a question if the code is still running and how far we processed the data. Tqdm allows us to integrate our progress bar with Pandas and answer this question:

import pandas as pd
from tqdm import tqdm
import time

tqdm.pandas(desc="Applying function")
df = pd.DataFrame({"A": range(5000)})
df['B'] = df['A'].progress_apply(lambda x: time.sleep(0.001) or x**2)

We now get a progress bar on our transformation in the DataFrame of Pandas.

Integration into Jupyter Notebooks

Most progress bars fail in Jupyter Notebooks because we lack a command line and use a different method to produce output. But with tqdm we can use the same progress bar, all we need to do is to import an additional module:

from tqdm.notebook import trange, tqdm
from time import sleep

for i in trange(5, desc='1st loop'):
    for j in tqdm(range(100), desc=f'    inner loop #{i}'):
        sleep(0.01)

Now we have a progress bar inside a Jupyter Notebook.

We get the same experience with the progress bar in a Jupyter Notebook as we do in the CLI.