Sidekiq and concurrency of a single queue

Question

I have 3 Queues running

User
Follower
Default

The User can import his Data which is then done by the User queue. That actually creates tons of followers (up to millions) and each of them is having an after_create to update theirself, in the Follower queue.

The Problem here is, that 50 Users can create their imports right now, but i want only 1 import to be done at the same time. First in First out - while the Follower-Queue should be still working with 100 simultanously workers. example: 1 user imports 500.000 followers. then the next user imports 5 million. the next imports 400. meanwhile the followers of user 1 are loaded.

is it possible to do?

my YAML looks like this

:queues:
  - user
  - follower
  - default

the docs are telling me now, that the user queue is high prioritized. but how to tell that only 1 job from this queue is done at the same time?

Answer 1

This feature is not supported by the sidekiq core (and the author does not want to add such feature to it). However, there is an external middleware that seems to support setting worker limits per queue, the sidekiq-limit_fetch gem. Using it you should be able to easily set what you need:

# sidekiq.yml
:limits:
  user: 1
  follower: 100
  default: 10 # or whatever you like

The gem code is a little older though, you mileage may vary...