Handling long requests

Question

I'm working on a long request to a django app (nginx reverse proxy, mysql db, celery-rabbitMQ-redis set) and have some doubts about the solution i should apply :

Functionning : One functionality of the app allows users to migrate thousands of objects from one system to another. Each migration is logged into a db, and the users are provided the possibility to get in a csv format the history of the migration : which objects have been migrated, which status (success, errors, ...)

To get the history, a get request is sent to a django view, which returns, after serialization and rendering into csv, the download response.

Problem : the serialisation and rendering processes, for a large set of objects (e.g. 160 000) are quite long and the request times out.

Some solutions I was thinking about/found thanks to pervious search are :

• Increasing the amount of time before timeout : easy, but I saw everywhere that this is a global nginx setting and would affect every requests on the server.
• Using an asynchronous task handled by celery : the concept would be to make an initial request to the server, which would launch the serializing and rendering task with celery, and give a special httpresponse to the client. Then the client would regularly ask the server if the job is done, and the server would deliver the history at the end of processing. I like this one but I'm not sure about how to technically implement that.
• Creating and temporarily storing the csv file on the server, and give the user a way to access it & to download it. I'm not a big fan of that one.

So my question is : has anyone already faced a similar question ? Do you have advises for the technical implementation of the solution (#2), or a better solution to propose me ?

Thqnks !

Answer 1

Clearly you should use Celery + RabbitMQ/REDIS. If you look at the docs it´s not that hard to setup.

The first question is whether to use RabbitMQ or Redis. There are many SO questions about this with good information about pros/cons.

The implementation in django is really simple. You can just wrap django functions with celery tasks (with @task attribute) and it´ll become async, so this is the easy part.

The problem I see in your project is that the server who is handling http traffic is the same server running the long process. That can affect performance and user experience even if celery is running on the background. Of course that depends on how much traffic you are expecting on that machine and how many migrations can run at the same time.

One of the things you setup on Celery is the number of workers (concurrent processing units) available. So the number of cores in your machine will matter.

If you need to handle http calls quickly I would suggest to delegate the migration process to another machine. Celery/REDIS can be configured that way. Let´s say you´ve got 2 servers. One would handle only normal django calls (no celery) and trigger celery tasks on the other server (the one who actually runs the migration process). Both servers can connect to the same database. But this is just an infrastructure optimization and you may not need it.

I hope this answers your question. If you have specific Celery issues it would be better to create another question.