Django asynchronous requests to remote api

Question

I have the following code (simplified version)

for data in my_data_array:
  res = api_request(data)
  #write result to db

These request may take some time and there are a lot of them. How can I make each iteration of loop asynchronous and send progress with percentage of completed requests to the front-end with Django. If I have to use Tornado or Celery, please give me the links with information how to integrate Django with them.

Answer 1

You will need Celery (or other async task queue). To integrate it with Django, see http://celery.readthedocs.org/en/latest/django/first-steps-with-django.html. I recommend to use Celery with Redis, because Redis is often used as a cache, so you don't need to install another backend for Celery (mostly RabbitMQ).

To get the progress bar, count total number of tasks (len(my_data_array)), store the value in cache (e.g. key total_count) and add the second key (e.g. complete_count) with zero value. In every task that completes, increase the complete_count value.

Last step is to query the status. It is just a simple view that loads these two values from cache and returns to the user (html/json).