Why does Celery revoke all scheduled tasks and discard them?

Question

I have a problem with Celery: from a certain point (the Celery worker logs do not show the root cause), all tasks (scheduled by celerybeat every 5 minutes) are being revoked and discarded by the workers.

There is no clue in the logs of celerybeat and celeryd (the workers) that could indicate why this happens. I noticed that from a certain point, 1 / 50 task gets revoked, the frequency of the revoked tasks keeps going up until all tasks are systematically revoked 24 hours later.

I am using Redis as the broker, the problem occurs on a Linux deployment of Celery.

Do you know what could be the cause of the issue? If you need more info, don't hesitate to ask!

Answer 1

I figured out what happened after looking at the task UUIDs in the logs:

[2015-10-27 05:27:45,879: INFO/MainProcess] Received task: hidden_task_name[29e47148-8865-4d46-a7ef-f8a0d625fe05] expires:[2015-10-27 05:57:45.870869+00:00]
[2015-10-27 06:28:23,696: INFO/MainProcess] Discarding revoked task: hidden_task_name[29e47148-8865-4d46-a7ef-f8a0d625fe05]

The tasks are discarded because of the task expiration date I have set using CELERYBEAT_SCHEDULE:

CELERYBEAT_SCHEDULE['---'] = {
        'task': ...,
        'schedule': ...,
        'args': (),
        'options': {
            # Do not run the task if it starts 30 minutes after it is
            # scheduled. This is useful if the Celery workers go down,
            # celerybeat will keep adding tasks to the queue.
            'expires': 1800
        }
    }

My bad :)