Introduction:
In
this article I will explain you how to handle priority background
jobs in django celery. Django celery queues all the jobs and works on
FIFO, if there are already some tasks lined up in queue, than new
task which is of higher priority will not be executed until all the
jobs are executed which are scheduled earlier.
For
example, consider that we run one management command which does
something for all users in our platform, say for example it fetches
mails from users account as per our requirement, now it will queue
the same task for all all users which does the work for us, now in
between if any new user register’s at our platform and we are
sending the welcome mail to him in delayed job which is of higher
priority, than in this case it will get delayed until all pre
assigned jobs are completed.
If
you have not gone through the previous blogs on celery please go
through it before proceeding further Delayed Jobs in Django Celery ,
I am continuing from sample blog.
Handling
Priority Tasks in Celery:
Django
celery does not provides any option to prioritise the task in queue,
so we need to create multiple queues to handle the jobs for us. We
can create one `default` to run the normal tasks and one
`priority_tasks` to deal with priority tasks, we can create `n`
number of queues depending on the priority of tasks and our
requirements and than we can queue it accordingly.
Configuring
Multiple Queues:
For
running multiple queues, we need to define those queues and tell
about the default queue in settings. So, update the settings.py file
with following code, where we are defining two queues, default and
priority_tasks, where default will server all normal tasks.
from
kombu import Queue, Exchange
CELERY_QUEUES
= (
Queue('default',
Exchange('default'), routing_key='default'),
Queue('priority_tasks',
Exchange('priority_tasks'), routing_key='priority_tasks'),
)
CELERY_DEFAULT_QUEUE
= 'default'
CELERY_DEFAULT_EXCHANGE
= 'default'
CELERY_DEFAULT_ROUTING_KEY
= 'default'
CELERY_ROUTES
= {
'sampleapp.tasks.hello_world':
{'queue': 'default'},
'sampleapp.tasks.test_celery':
{'queue': 'priority_tasks'},
}
We
can route the tasks as well, like by default `hello_world` will be
queued in default and `test_celery` will be queued in priority_tasks
queue until we specify explicitly.
Running
Multiple Queues:
Now,
as we have defined two queues, so we need to run two different
workers for the queues, so to run them use the following command,
where sample is our project name.
`celery
-A sample worker -E -l INFO -n worker.default -Q default` , and
`celery
-A sample worker -E -l INFO -n worker.priority_tasks -Q
priority_tasks`
-
Now, update the sampleapp/view.py with following code, where we have
defined these test_celery and hello_world tasks in previous blog.
from
django.http import HttpResponse
from
sampleapp.tasks import test_celery, hello_world
def
home(request):
hello_world.apply_async()
# Will run it in default queue
test_celery.apply_async(queue='priority_tasks')
# Will run it in priority_tasks queue.
return
HttpResponse("Celery Demo!!")
-
Now as you will run the server, workers and hit the landing url, you
will see the tasks running in different queues.
Priority Tasks Worker
Default Tasks Worker
- sampleapp/views.py,
where it will return `Celery Demo!!` and queue hello_world to
default, and test_celery to priority_tasks queue.
Landing Page
`hello_world`
task executed in default queue.
`test_celery`
task executed in priority_tasks.
Conclusion:
We
can create different queues for different delayed tasks depending
upon their priorities, we can create n number of queues depending on
our requirements, like low priority, medium priority and high
priority etc.
Code
can be obtained from github repo
`https://github.com/pawan3103/priorityDelayedJobsDjango`
for reference.
0 Comments