RabbitMQ: Priority queue with changing priority

Question

I want to implement a priority work queue, in which the priority of a group of messages can change once they are in the queue. Since it is a work queue with variable processing time, the messages are not assigned using round-robin algorithm, but are pulled from the queue when a resource is free (using per-consumer limit).

I came up with 2 ideas for implementation:

1. Use priority queue from RabbitMQ, and when a request for priority change comes, read messages with this priority from the queue and re-send them with different priority. (I am not sure this is a good approach, given the O(n) complexity.)
2. Use several queues with distinct names for each group of messages, and use a separate queue to communicate the current priority list (ordered list of queue names) to workers. (Using this approach, I am not sure how to make the list of priorities "persistent", so that newly joined worker knows what is the current priority list.)

How would you implement it? Is RabbitMQ viable option for this use case?

Answer 1

your idea "priority of a message can change once they are in the queue" IMO is not possible with rabbitmq because rabbitmq only allows you to get messages from the head of a queue.

for example:

• you have N queues each used for a different priority
• each queue has 100+ messages
• your idea requires you to reach into the middle of a queue to get a specific message but this is not possible with rabbitmq so the thought experiment stops here because you can only get messages at the head of a queue

your idea IMO would require using something else besides rabbitmq.

a quick and dirty idea that would work with rabbitmq now and is similar to your idea:

• create one rabbitmq queue with N priorities
• submit a message with priority x
• if you need to change the priority to higher priority like priority y then you could send the same message again but with a new higher priority y
• this would ensure the new message is processed faster
• the side effect is that you may process the same request twice
  • you could fix the side effect in your design by having a some sort of database for synchronization to keep track of what jobs are completed and then this could avoid processing the job twice
  • there are many other details that would need to be addressed like keeping the original message around somehow outside of rabbitmq, concurrency, etc, etc,