Delayed Jobs in Django with Celery and Redis

     

       Celery is an asynchronous task queue used for distributed commands and different processing of tasks which don’t need to be under normal request and response lifecycle, for example: sending an email, or processing an uploaded csv file and after saving it in model , using celery for these kind of tasks runs application smoothly and quickly.

          Task queues are basically used to distribute the work load on machine, In this blog my focus is on setup Celery with redis for queue instead of RabbitMQ which is default for celery and execution of asynchronous tasks in Django app.

Getting start with example on how to use celery with Django

First let’s setup Django Project:

           So let’s get started from scratch with new Django project. First, create virtualenv and activate it then install django and celery.

virtualenv env

source env/bin/activate

pip install django

pip install celery

pip install celery[redis]

# Setup new project with single app.

django-admin.py startproject sample

cd sample

python manage.py startapp celeryapp

After generating the app, add the celeryapp in installed_apps, then sync database for the first time:

INSTALLED_APPS = [

       ...,

       “celeryapp”,

]

python manage.py migrate

Now update the settings.py file with addition of BROKER_URL and BROKER_TRANSPORT

BROKER_URL = ‘redis//127.0.0.1:6379/0’

BROKER_TRANSPORT = ‘redis’

Celery requires a messaging agent for handling requests from external source s which is referred to as broker, it is simply a location where we store all our tasks that need to be executed in queue, here we are using redis as our broker, 6379 is the port for redis and 0 is indication that we are using the first database of redis.

Let’s create our celery.py file.

import os

import django

from celery import Celery

from django.conf import settings

os.environ.setdefault('DJANGO_SETTINGS_MODULE','sample.settings')

django.setup()

app = Celery(‘sample’)

app.config_from_object(‘django.conf:settings’)

app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

      Here we have done few imports and set default environment variable DJANGO_SETTINGS_MODULE to default django settings, where sample being our project name . This is important because we are bootstrapping django so that we can use django. Next thing is django setup required in django1.7 and above versions because they use new app registry system.

               In next line we are instantiating our celery object so that we have app running on celery, celery is actually dependent upon its own application. In next line we are configuring same project settings file instead of creating another file so that all our configuration remains at same place.

              Finally, we are setting autodiscovering tasks and looking into all apps of our project that we have into INSTALLED_APPS and looks for tasks.py file inside it for the tasks that need to be executed.

Now let’s create a celeryapp/task.py file inside our celeryapp and see how it works.

from sample.celery import app

@app.task

def test_celery():

     print "Its working!!"

       Here we have imported the app from celery which we created in last step, and create a function test_celery with decorator @app.task for executing print statement. This function is like a normal function and can be execute normally or can be executed as a task for celery.

   Lets go to celeryapp/views.py and create another function home with simple HttpResponse `Celery Demo` and calls our test_celery function inside the function to test our celery task.

from django.http import HttpResponse

from sampleapp.tasks import test_celery

def home(request):

     test_celery.delay()

     return HttpResponse("Celery Demo!!")

     Here we will imported our test_celery function and called it with delay(), which indicates that we want to use it with celery, we can run without delay also as a normal function also which will simply execute print statement.

Now register the function in urls file.

from django.conf.urls import url

from django.contrib import admin

from sampleapp import views

urlpatterns = [

    url(r'^admin/', admin.site.urls),

    url(r'$', views.home),

]

To test the function in celery we need to start celery worker in another terminal using :

`celery -A sample worker -l debug`

Here we have use sample as project name and set log level to debug to get more info in console.

This is the response from home view, now lets check our celery task execution from console.

    

       You can see in the console the print statement got executed, now lets remove that delay and call the function normally and test the response in console.

Here you can see the print statement got executed in terminal.