[Redis]When the number of tasks in the queue exceeds 10,000, the speed is slow and errors may occur.

[vistart]

What steps will reproduce the problem?

I am using a redis queue.
When the number of tasks in the queue exceeds 10,000, the worker gets the task very slowly, and may also encounter the following error:

PHP Warning 'yii\base\ErrorException' with message 'A non-numeric value encountered'

in /var/www/rho.one/vendor/yiisoft/yii2-queue/src/drivers/redis/Queue.php:153

Stack trace:
#0 /var/www/rho.one/vendor/yiisoft/yii2-queue/src/cli/Queue.php(117): yii\queue\redis\Queue->yii\queue\redis\{closure}()
#1 /var/www/rho.one/vendor/yiisoft/yii2-queue/src/cli/Queue.php(117): ::call_user_func:{/var/www/rho.one/vendor/yiisoft/yii2-queue/src/cli/Queue.php:117}()
#2 /var/www/rho.one/vendor/yiisoft/yii2-queue/src/drivers/redis/Queue.php(68): yii\queue\redis\Queue->runWorker()
#3 /var/www/rho.one/vendor/yiisoft/yii2-queue/src/drivers/redis/Command.php(76): yii\queue\redis\Queue->run()
#4 /var/www/rho.one/vendor/yiisoft/yii2/base/InlineAction.php(57): yii\queue\redis\Command->actionListen()
#5 /var/www/rho.one/vendor/yiisoft/yii2/base/InlineAction.php(57): ::call_user_func_array:{/var/www/rho.one/vendor/yiisoft/yii2/base/InlineAction.php:57}()
#6 /var/www/rho.one/vendor/yiisoft/yii2/base/Controller.php(157): yii\base\InlineAction->runWithParams()
#7 /var/www/rho.one/vendor/yiisoft/yii2/console/Controller.php(148): yii\queue\redis\Command->runAction()
#8 /var/www/rho.one/vendor/yiisoft/yii2/base/Module.php(528): yii\queue\redis\Command->runAction()
#9 /var/www/rho.one/vendor/yiisoft/yii2/console/Application.php(180): yii\console\Application->runAction()
#10 /var/www/rho.one/vendor/yiisoft/yii2/console/Application.php(147): yii\console\Application->runAction()
#11 /var/www/rho.one/vendor/yiisoft/yii2/base/Application.php(386): yii\console\Application->handleRequest()
#12 /var/www/rho.one/yii(27): yii\console\Application->run()
#13 {main}

The problem comes from here:
https://github.com/yiisoft/yii2-queue/blob/master/src/drivers/redis/Queue.php#L153

The $ttr variable may not be a number in some cases.

What's expected?

Solve the above errors and improve performance.

What do you get instead?

Additional info

Q	A
Yii version	2.0.16.1
PHP version	7.2.14
Operating system	Ubuntu 18.04.1
redis	5.0.3

[samdark]

The $ttr variable may not be a number in some cases.

What cases?

[vistart]

The $ttr variable may not be a number in some cases.

What cases?

There are 10,000+ download jobs for downloading webpages in the queue and there are 20 workers work at the same time..

[zhuravljov]

There are 10,000+ download jobs for downloading webpages in the queue and there are 20 workers work at the same time..

The $ttr variable may not be a number in some cases.

Excuse me, how first is related with second?

TTR always must be integer.

[vistart]

There are 10,000+ download jobs for downloading webpages in the queue and there are 20 workers work at the same time..

The $ttr variable may not be a number in some cases.

Excuse me, how first is related with second?

TTR always must be integer.

The first paragraph just wants to emphasize that there are a lot of tasks to be carried out, and there are also many workers working at the same time.

This problem hardly occurs because there are too few tasks or too few workers.

I added a statement that outputs the content and length of $ttr variable before this line of code, and I got the following output:

Obviously, the $ttr output from the first two tasks is correct, ie 3.

Further, I added the code: when the $ttr length is 0, it outputs the $payload variable. The result is as follows:

At this point $payload is NULL.

[vistart]

I personally guess: it is likely that the acquisition of queue messages does not guarantee complete isolation.

[vistart]

I missed a detail:
I push all the tasks to the queue first, then open the workers one by one for listening.
If an exception occurs, at least two workers will throw an exception at the same time.

[vistart]

It is also possible that the performance of the operating environment is too bad, causing the mutex to fail.

[mikk150]

related #218