Sidekiq enqueue job after a wait time

Question

The problem is that the sidekiq worker that process the object runs before the object exists in the database. The job is sending to the queue in after_commit callback in the object model. It's posible because I have two replicated databases one for reads and other for inserts. So the time that the process is from enqueue to fail is minor to the time that the data is replicated in the database.

What is the best approach for the solution? I was thinking to add some wait time between enqueue and process to ensure that the data is in the slave database. Is it possible in sidekiq configuration or something like that?

Answer 1

You could do a few things:

1. Implement a check in the worker to make sure the object exists; otherwise, re-enqueue the job. Probably want to think about this to make sure you don't accidentally re-enqueue bad jobs forever, but this seems like a good sanity check for you.

2. Introduce a delay. In particular, sidekiq can wait to pull jobs from the queue until a specified time.

   "Sidekiq allows you to schedule the time when a job will be executed. You use perform_in(interval, *args) or perform_at(timestamp, *args) rather than the standard perform_async(*args):

   MyWorker.perform_in(3.hours, 'mike', 1) MyWorker.perform_at(3.hours.from_now, 'mike', 1)"

   See https://github.com/mperham/sidekiq/wiki/Scheduled-Jobs for more details on this option.

Personally I would go for #1 but #2 might be a quicker fix if you're desperate.