Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Email message queue - X workers and stopping overlap

I've got a large queue of emails that are loaded in a MySQL table as events occur around my website. At anytime there maybe 1k-2k waiting to be sent at anytime.

I've been working on batching sets of emails and sending them out via one worker. Now I'm looking to add X workers to the system as load increases.

My question is about not allowing workers to grab the same email. I've heard the following options.

  1. Use SELECT FOR UPDATE to lock the row
  2. Use a timestamp to let the workers know it's been taken and when
  3. Have a parent that tracks the children and delegates the work

I see value in each of these but also a lot of reading / writing to the db. So what would you do?

Doing this via PHP and MySQL

like image 856
Tim Avatar asked Dec 08 '25 09:12

Tim


1 Answers

Another quick and dirty option, assuming you have a primary key which it either random or structured, is letting each of N workers pick up only mails where key%N==i with i the worker id.

You could of course also take some other hash value which is sufficiently random. No need to lock, no extra coordination, just a tiny problem when one worker dies and you don't notice...

like image 179
mvds Avatar answered Dec 09 '25 23:12

mvds



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!