Laravel horizon long queue time

Question

I have problems with queue time in Laravel Horizon..

Case: I have an exam proctoring software that take picture from ~1500 students webcam every 30 seconds. Then I put it on the queue to broadcast via websocket the latest image to the teachers.

My server has 32 CPU and 192GB RAM.

Here is my horizon configuration:

And I receive some emails similar like this from the system. Varying from 1000 seconds to >30k seconds like this one.

The queue processing feels like very slow even when I put 100k maxProcesses..

Can someone help me with this? Thanks in advance..

EDIT: To make the question clearer here is the summary:

• My server has 32 CPU and 192GB RAM (Ubuntu 18.04).

• I allocated 100k max processes in horizon config (I even changed it yesterday to 1.000.000 max processes)

• BUT the queue processing is not as fast as I think it would. One of the queue even have 30k+ seconds wait time.

• How to make the queue processing faster? Is there any horizon/server configuration that I missed? Some limit configuration maybe in PHP Limit/Redis Limit/any limit?

Answer 1

The following method is from horizon repository

/**
 * Handle the event.
 *
 * @param  \Laravel\Horizon\Events\SupervisorLooped  $event
 * @return void
 */
public function handle(SupervisorLooped $event)
{
    if (! $this->dueToMonitor()) {
        return;
    }

    // Here we will calculate the wait time in seconds for each of the queues that
    // the application is working. Then, we will filter the results to find the
    // queues with the longest wait times and raise events for each of these.
    $results = app(WaitTimeCalculator::class)->calculate();

    $long = collect($results)->filter(function ($wait, $queue) {
        return $wait > (config("horizon.waits.{$queue}") ?? 60);
    });

    // Once we have determined which queues have long wait times we will raise the
    // events for each of the queues. We'll need to separate the connection and
    // queue names into their own strings before we will fire off the events.
    $long->each(function ($wait, $queue) {
        [$connection, $queue] = explode(':', $queue, 2);

        event(new LongWaitDetected($connection, $queue, $wait));
    });
}

LongWaitDetected event triggering emails. "31941 seconds" is the waiting time of your webcam queue. Whenever supervisor starts looping it triggers MonitorWaitTimes and the code above works.

If you don't want to get this email, add your queue name into the your horizon config in the waits array with a huge number.

'waits' => [
    'redis:default' => 60,
    'redis:webcam' => 1514124141
],

But keep in mind that; it may be a sign that some of the processes/jobs are not working. 30K seconds may be an indication for that.