RabbitMQ pre_fetch_count in common cases in PHP
Let’s talk about batch message processing in a PHP consumer application. In most cases, we use a library provided by your framework and we’ve not had to worry about connection settings because all work fine out of the box. In some cases, you need to use your wrapper onphp-amqplib and now let’s figure it out.
RabbitMQ has parameters named QOS which means the quality of service where the second parameter is pre_fetch_count. When we set it to 10, it means RabbitMQ sends to consumer 10 messages and will wait for an ack to each message in this batch, and only when RabbitMQ gets ack/no-ack/requeue for all messages He will send to a consumer the next batch of messages.
PHP is not multithreaded. Let’s look into the code block.
When a consumer gets messages he will process them successively. One after the one, not parallel.
Let’s imagine a case, see the picture. Your consumer has business logic that requires sending data to another system, let it be some bank. The first and second messages in the batch were handled successfully on the third we got a connection problem with bank systems and marked this message as no-ack and when we handled message number four, the consumer is failing to fatal error and stopped work. All messages in the batch that are sent to the consumer will return to the queue without acks or no-acks obtained while messages were handled and when the consumer starts again messages were already handed will try handling them again. If we made some operations with money transfer or other sensitive operations we can make it twice.
Сertainly your code may have some handler to resolve double handling cases but many developers don’t do it before problems were gained. To minimize problems in batch processing like in my example you should set pre_fetch_count to 1 and handle per one message on a consumer, it will be slowly but safer instead of a batch with 100 messages that can be rehandled many times.
Disclaimer: I think GO/C#/Java or other multithreaded language is much better for queueing and batch tasks.
