While our application waits on IO (Input / Output), our CPUs idle around. If we would have a way to let the program step aside and let other code run while we wait on a file or a response, we could get more work done at the same time. That is the basic idea behind asynchronous programming that we can do with coroutines and the async and await keywords in Python.
If you need to run a task every few seconds or create a report every week, you can use Celery and its feature for periodic tasks – at least if we use Linux or WSL on Windows.
So far our Celery tasks where fire-and-forget jobs. We put a task in a queue and go to the next activity. However, sometimes we need to know if a task succeeded before we can do more work. Let us take a look on how Celery tracks tasks.
Flask is a great web framework. The only problem, as with all web applications, is the processing of long-running tasks. Luckily for us, we can use our knowledge of Celery and combine it with Flask to process the long-running tasks asynchronously. Even better, we already know everything we need to do this.
Errors happen, especially when you have a distributed application. Let's look how Celery allows us to log messages into the Celery logger.
One big benefit of asynchronous jobs is that they do not need to be processed right away. Let's look how we can tell Celery to keep our jobs around for a while before starting the work.
If we try to run Celery on Windows, we will run into a problem: Windows is not officially supported by Celery. However, that does not mean that it will not work. Let's look what we need to do to run Celery on Windows.
RabbitMQ is a great tool for asynchronous jobs. It works well but we need to do all the infrastructure parts on our own. Celery can run on top of RabbitMQ and offers us a lot more comfort for our asynchronous jobs. However, that comfort comes with the price of additional complexity that we need to handle. In this post we start with the basic set-up and over the next weeks we take a deeper look at the features of Celery.
The series on Tweepy showed my that I have a big gap in my Python knowledge: How can I separate work in an asynchronous way? In this post I look at RabbitMQ and how we can use it to get more flexibility in our Python applications.