Avoiding duplicate jobs in delayed_job queue

Question

I'm having an issue avoiding duplicate jobs being added to my applications the delayed job queue. I've done a bit of searching, but all have been fruitless. Any suggestions on how to implement a solution would be appreciated.

Links I've stumbled upon, but offer no solutions.

1. https://github.com/collectiveidea/delayed_job/issues/72
2. https://groups.google.com/forum/#!topic/delayed_job/gZ9bFCdZrsk

Answer 1

I would not focus on avoiding duplicate jobs. In my opinion, a job queue should be dump and not hold a state. Adding a job to the queue should be as fast as possible. Any check upfront would slow down the response time.

Instead of checking for duplicates upfront, the job itself should be clever enough to figure out if it is still appropriate to run. The job could for example first check a status, a flag, or a timestamp on a model. And only if that precondition is still valid start with the actual processing.

Answer 2

Following what @spickermann was saying, any duplicate checking should be happening in the background not at dump time.

The way I solved this problem is I created 2 new fields on the delayed_jobs table: signature and args. Signature will be either "Class#method" or "Object:id#method". Args will just be the args from the payload object. By adding these fields, you can now query the jobs table to see if there is a duplicate.

For our application, if there is a dupe, we will cancel the current job allowing the more recent call to go through. However, this approach my lead to starvation if newer duplicate jobs keep getting added to the queue so you may want to do the reverse approach where if there are duplicates, you destroy all future calls other than the current.

Here is a gist of how to accomplish this via a delayed job plugin: https://gist.github.com/synth/fba7baeffd083a931184