Real Python: Asynchronous Tasks With Django and Celery

You’ve built a shiny Django app and want to release it to the public, but you’re worried about time-intensive tasks that are part of your app’s workflow. You don’t want your users to have a negative experience navigating your app. You can integrate Celery to help with that.

Celery is a distributed task queue for UNIX systems. It allows you to offload work from your Python app. Once you integrate Celery into your app, you can send time-intensive tasks to Celery’s task queue. That way, your web app can continue to respond quickly to users while Celery completes expensive operations asynchronously in the background.

In this tutorial, you’ll learn how to:

• Recognize effective use cases for Celery
• Differentiate between Celery beat and Celery workers
• Integrate Celery and Redis in a Django project
• Set up asynchronous tasks that run independently of your Django app
• Refactor Django code to run a task with Celery instead

If you’ve never used Celery in a Django app before, or if you’ve peeked into Celery’s documentation but couldn’t find your way around, then you’re in the right place. You’ll learn all you need to start running asynchronous tasks with Django and Celery.

You’ll integrate Celery into an existing Django app. Go ahead and download the code for that app so you can follow along:

Python Celery Basics

Celery is a distributed task queue that can collect, record, schedule, and perform tasks outside of your main program.

To receive tasks from your program and send results to a back end, Celery requires a message broker for communication. Redis and RabbitMQ are two message brokers that developers often use together with Celery.

In this tutorial, you’ll use Redis as the message broker. To challenge yourself, you can stray from the instructions and use RabbitMQ as a message broker instead.

If you want to keep track of the results of your task runs, then you also need to set up a results back end database.

You can use many different databases to keep track of Celery task results. In this tutorial, you’ll work with Redis both as a message broker and as a results back end. By using Redis, you limit the dependencies that you need to install because it can take on both roles.

You won’t do any work with the recorded task results in the scope of this tutorial. However, as a next step, you could inspect the results with the Redis command-line interface (CLI) or pull information into a dedicated page in your Django project.

Why Use Celery?

There are two main reasons why most developers want to start using Celery:

1. Offloading work from your app to distributed processes that can run independently of your app
2. Scheduling task execution at a specific time, sometimes as recurring events

Celery is an excellent choice for both of these use cases. It defines itself as “a task queue with focus on real-time processing, while also supporting task scheduling” (Source).

Even though both of these functionalities are part of Celery, they’re often addressed separately:

1. Celery workers are worker processes that run tasks independently from one another and outside the context of your main service.
2. Celery beat is a scheduler that orchestrates when to run tasks. You can use it to schedule periodic tasks as well.

Celery workers are the backbone of Celery. Even if you aim to schedule recurring tasks using Celery beat, a Celery worker will pick up your instructions and handle them at the scheduled time. What Celery beat adds to the mix is a time-based scheduler for Celery workers.

In this tutorial, you’ll learn how to integrate Celery with Django to perform operations asynchronously from the main execution thread of your app using Celery workers.

You won’t tackle task scheduling with Celery beat in this tutorial, but once you understand the basics of Celery tasks, you’ll be well equipped to set up periodic tasks with Celery beat.