While doing web applications, you usually need to run some operations in the background to improve the application performance, or because a job really needs to run outside of the application environment. In both cases, if you are on Django, you are in good hands because you have Celery, the Distributed Task Queue written in Python. Celery is a tiny but complete project. You can find more information on the project page. In this post, we will see how it’s easy to integrate Celery with an existing project, and although we are focusing on Django here, creating a standalone Celery worker is a very similar process.
Installing Celery
The first step we will see is how to install Celery. If you already have it, please move to the next section and follow the next step!
As every good Python package, Celery is distributed on pip. You can install it just by entering:
pip install celery
Choosing a message broker
The second step is about choosing a message broker to act as the job queue. Celery can talk with a great variety of brokers; the main ones are:
Check for support on other brokers here. If you’re already using any of these brokers for other purposes, choose it as your primary option.
In this section there is nothing more you have to do. Celery is very transparent and does not require any source modification to move from a broker to another, so feel free to try more than one after we end here.
Ok let’s move on, but first do not forget to look the little notes below.
¹: For Redis (a great choice in my opinion), you have to install the celery[redis] package.
²: Celery has great features like web monitoring that do not work with this broker.
Celery worker entrypoint
When running Celery on a directory it will search for a file called celery.py, which is the application entrypoint, where the configs are loaded and the application object resides.
Working with Django, this file is commonly stored on the project directory, along with the settings.py file; your file structure should look like this:
- your_project_name
- your_project_name
- __init__.py
- settings.py
- urls.py
- wsgi.py
- celery.py
- your_app_name
- __init__.py
- models.py
- views.py
- ….
- your_project_name
The settings read by that file will be on the same settings.py file that Django uses. At this point we can take a look at the official documentation celery.py file example. This code is basically the same for every project; just replace proj by your project name and save that file. Each part is described in the file comments.
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')
app = Celery('proj')
# Using a string here means the worker don't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')
# Load task modules from all registered Django app configs.
# This is not required, but as you can have more than one app
# with tasks it’s better to do the autoload than declaring all tasks
# in this same file.
app.autodiscover_tasks()
Settings
By default, Celery depends only on the broker_url setting to work. As we’ve seen in the previous session, your settings will be stored alongside the Django ones but with the 0‘CELERY_’ prefix.
The broker_url format is as follows:
CELERY_BROKER_URL = ‘broker://[[user]:[password]@]host[:port[/resource]]’
Where broker is an identifier that specifies the chosen broker, like amqp or redis; user and password are the authentication to the service. If needed, host and port are the addresses of the service and resource is a broker-specific path to the component resource. For example, if you’ve chosen a local Redis as your broker, your broker URL will be:
CELERY_BROKER_URL = ‘redis://localhost:6379/0’ ¹
1: Considering a default Redis installation with the database 0 being used.
Doing this we have a functioning celery worker. How lucky! It’s so simple! But wait, what about the tasks? How do we write and execute them? Let’s see.
Creating and running tasks
Because of the superpowers Celery has, it can autoload tasks from Django app directories as we’ve seen before; you just have to declare your app tasks in a file called tasks.py in the app dir:
- your_project_name
- your_project_name
- __init__.py
- settings.py
- urls.py
- wsgi.py
- celery.py
- your_app_name
- __init__.py
- models.py
- views.py
- tasks.py
- ….
- your_project_name
In that file you just need to put functions decorated with the celery.shared_task decorator. So suppose we want to do a background mailer; the source will be like this:
from __future__ import absolute_import, unicode_literals
from celery import shared_task
from django.core.mail import send_mail
@shared_task
def mailer(subject, message, recipient_list, from=’[email protected]’):
send_mail(subject, message, recipient_list, from)
Then on the Django application, on any place you have to send an e-mail on background, just do the following:
from __future__ import absolute_import
from app.tasks import mailer
….
def send_email_to_user(request):
if request.user:
mailer.delay(‘Alert Foo’, ‘The foo message’, [request.user.email])
delay is probably the most used way to submit a job to a Celery worker, but is not the only one. Check this reference to see what is possible to do. There are many features like task chaining, with future schedules and more!
As you can have noticed, in a great majority of the files, we have used the from __future__ import absolute_import statement. This is very important, mainly with Python 2, because of the way Celery serializes messages to post tasks on brokers. You need to follow the same convention when creating and using tasks, as otherwise the namespace of the task will differ and the task will not get executed. The absolute import module forces you to use absolute imports, so you will avoid these problems. Check this link for more information.
Running the worker
If you get the source code above, put anything in the right place and run the Django development server to test your background jobs, they will not work! Wait. This is because you don’t have a Celery worker started yet. To start it, do a cd to the project main directory (the same as you run python manage.py runserver for example) and run:
celery -A your_project_name worker -l infoReplace your_project_name with your project and info with the desired log level. Keep this process running, start the Django server, and yes. Now you can see that anything works!
Where to go now?
Explore the Celery documentation and see all the available features, caveats, and help you can get from it.
There is also an example project on the Celery GitHub page that you can use as a template for new projects or a guide to add celery to your existing project.
Summary
We’ve seen how to install and configure Celery to run alongside a new or existing Django project. We explored some of the broker options we have, and how simple it is to change between them. There are some hints about brokers that don’t offer all of the features Celery has.
We have seen an example of a mailer task, and how it was created and called from the Django application. Finally I provided instructions to start the worker to get the things done.
References
[1] – Django project documentation
[2] – Celery project documentation
[3] – Redis project page
[4] – RabbitMQ project page
[5] – Amazon SQS page
About the author
Jean Jung is a Brazilian developer passionate about technology. He is currently a system analyst at EBANX, an international payment processing company for Latin America. He’s very interested in Python and artificial intelligence, specifically machine learning, compilers and operational systems. As a hobby, he’s always looking for IoT projects with Arduino.
