Celery - Having sequential tasks rather than concurrent?

Question

I have a few tasks which use the Amazon API which is limited to 1 req/second.

Therefore I would like to group up tasks which use the Amazon API and have them lined up in a sequential queue.

All I've figured out is how to associate queues to each task. But no idea on how to remove concurrency in a specific queue.

Any ideas?

Thanks a lot! :)

Answer 1

Single worker consuming from a queue with concurrency equals to one ensures that the tasks will be processed in sequential order. In other words you can create a special queue and run only one celery worker with concurrency equals to one:

celery -A tasks worker -Q amazon_queue -c 1

And submit tasks to that queue:

tasks.add.apply_async(args=[1,2], kwargs={}, queue='amazon_queue')

Or use automatic routing for certain task types.