Forking processes for every task in Celery

Question

I currently use a C extension library for Python, but it seems to have memory leaks. Tasks that are run on my celeryd do something using this C extension library, and celeryd eats a lot of memory about a hour later. I cannot patch this C extension library in many reasons, but instead I want to fork processes for every task in Celery. Are there any such options for Celery?

Answer 1

You can use CELERYD_MAX_TASKS_PER_CHILD option or --maxtasksperchild celeryd switch.

To restart worker processes after every task:

CELERYD_MAX_TASKS_PER_CHILD=1

https://celery.readthedocs.org/en/latest/userguide/workers.html#max-tasks-per-child-setting