Magento Message Queue Consumers: Keeping RabbitMQ and db Workers Alive With Supervisor and cron:consumers_runner

On Magento and Adobe Commerce, message queue consumers are the background workers that process asynchronous jobs: order confirmation emails, bulk product updates, inventory exports, and reindex messages. When they stop, the storefront looks fine but work silently piles up. This guide covers running them reliably with the consumers_runner cron job or a process manager like Supervisor.

If your store “feels slow” in ways that have nothing to do with page speed (emails arrive late, bulk attribute updates never finish, async indexing lags), the cause is usually consumers that are not running, dying early, or misconfigured. The fix is understanding how consumers are supposed to stay alive and choosing the right mechanism for your environment.

What message queue consumers actually do

Magento uses a message queue framework to move slow or high-volume work off the request that triggered it. A shopper places an order; the order saves immediately, but the confirmation email, the async indexer update, and any ERP sync get published as messages. Consumers are long-running PHP processes that pull those messages off the queue and process them.

Out of the box, Magento defines many consumers: async.operations.all for bulk operations, product_action_attribute.update for mass attribute changes, consumers for the asynchronous and bulk REST APIs, and more. If no process is consuming a given queue, those messages sit there forever. The store does not throw an error. It just quietly stops doing the background work you assumed was happening.

This is the same class of “invisible backlog” problem as a stalled indexer or cron group. We cover the related indexer side in why your Magento store feels slow and how to fix the queue, and the two often fail together on an under-provisioned box.

MySQL vs RabbitMQ as the queue backend

Magento can run its queues on MySQL (the default) or on a dedicated message broker. The choice shapes how you should run consumers.

The MySQL-backed queue stores messages in database tables and is fine for low volume and single-consumer setups. Adobe’s own guidance advises against running multiple consumers against a MySQL-operated queue, because the database was never meant to be a high-throughput broker. For anything with real volume, the Adobe message queue configuration docs point to a proper broker: RabbitMQ (AMQP) or ActiveMQ Artemis (STOMP).

RabbitMQ is the common choice and is required for Adobe Commerce Cloud. It handles concurrency properly, survives restarts, and lets you scale consumers horizontally. If you are planning to run multiple processes per consumer or push significant bulk volume, treat RabbitMQ as a prerequisite, not an optimization. This is one of the integration decisions our team scopes early on a build, alongside the rest of the technology partner stack a store depends on.

Option 1: the consumers_runner cron job

The default way to run consumers is a cron job that Magento ships with, called consumers_runner. It runs every minute, starts each defined consumer, lets it process a set number of messages, and exits. The next cron run starts them again. You configure it under cron_consumers_runner in app/etc/env.php, and the Adobe message queues reference documents the options.

'cron_consumers_runner' => [
    'cron_run' => true,
    'max_messages' => 10000,
    'consumers' => [],
    'multiple_processes' => [
        'product_action_attribute.update' => 2,
    ],
],

The options that matter:

• cron_run (default true): turns the consumers_runner cron job on or off. Set it to false only when you are running consumers another way, for example under Supervisor.
• max_messages (default 10000): how many messages each consumer processes before it terminates. Cron then restarts it on the next run. Setting it to 0 stops consumers from terminating, which Adobe explicitly does not recommend for the cron approach, because a never-terminating process under cron defeats the point.
• consumers: an array of specific consumer names to run. An empty array runs all of them. Narrow this when you want certain consumers handled separately.
• multiple_processes: the number of parallel processes per consumer, supported from Commerce 2.4.4 onward. Use it to give heavy consumers more throughput.

The cron model is self-healing: because each consumer exits after max_messages and cron brings it back every minute, a crashed consumer recovers within a minute without anyone watching. That is its real strength. Its weakness is latency and ceiling: a consumer that exits at 10,000 messages and waits for the next cron tick is not ideal when you need sustained, high-throughput processing.

Option 2: Supervisor keeping consumers alive

For high-volume stores, many teams run consumers as persistent processes under a process manager instead of cron. Adobe’s documentation notes you can use a process manager such as Supervisor to monitor consumer processes and restart them. Supervisor is a mature, widely used choice; systemd services work too.

The pattern is: disable the cron runner ('cron_run' => false), then define a Supervisor program per consumer that runs queue:consumers:start and let Supervisor restart it whenever it exits.

[program:magento-async-operations]
command=/usr/bin/php /var/www/magento/bin/magento queue:consumers:start async.operations.all --max-messages=50000
numprocs=1
autostart=true
autorestart=true
startretries=10
user=www-data

With autorestart=true, Supervisor immediately relaunches a consumer that exits, so you get continuous processing without waiting for a cron tick. You still set a finite --max-messages so each process recycles periodically and releases memory, which matters because long-lived PHP workers leak over time. Supervisor handles the restart; max-messages handles the memory hygiene. That combination, a bounded worker plus a supervisor that always brings it back, is the production default for stores with real queue volume.

A note on process recycling and deploys: persistent consumers hold old code until restarted, so your release process must restart them. We fold that into the deploy flow described in zero-downtime Magento deploys with Deployer, because a consumer still running last release’s code after a schema change is a classic cause of mysterious post-deploy errors.

queue:consumers:start and its flags

Whether cron or Supervisor launches them, consumers run through the same CLI command. The flags you will use most:

• --max-messages=N: process N messages, then exit. Bounds memory and lets the manager recycle the worker.
• --batch-size=N: how many messages to pull per batch, for broker-backed queues.
• --single-thread: force one message at a time.
• --multi-process=N: run N parallel processes for the consumer.
• --area-code: set the area for the consumer when a job needs a specific one.

You can list all defined consumers with bin/magento queue:consumers:list, which is the first command to run when you are not sure what should be processing.

Which approach to choose

For most small and mid-sized stores, the cron runner is the right default: it is built in, self-healing, and needs no extra infrastructure. Move to Supervisor when queue volume is high enough that tick-bound processing causes real backlog, which usually coincides with having already moved the queue to RabbitMQ. This is the same tradeoff we weigh when scaling Magento for flash sales and product drops, where bulk and async processing spike hard and fast.

Diagnosing stuck or dying consumers

When background work is not happening, work through this order:

1. Are consumers even running? Check your process list for queue:consumers:start. If nothing is there and cron_run is false, you disabled the runner and never set up the replacement.
2. Is cron healthy? The consumers_runner job depends on Magento cron running every minute. A dead cron means dead consumers. Confirm the base cron is installed and the consumers group is scheduled.
3. Are messages accumulating? On RabbitMQ, the management UI shows queue depth. Growing, unconsumed queues point straight at a missing or crashed consumer.
4. Are consumers dying early? Check logs for memory exhaustion or uncaught exceptions. A consumer that hits a PHP memory limit mid-batch will exit and, under cron, silently retry the same poisoned message next run.
5. Wrong backend for the load? Multiple consumers on a MySQL queue will contend and stall. If you are scaling, the backend is the problem, not the runner.

Async bulk operations are the most common place this surfaces, because a stuck async.operations.all consumer makes mass actions and the bulk API appear broken. Our guide to async bulk APIs and RabbitMQ for high-volume catalog imports covers that path in depth, and the indexer update-on-schedule model explains the reindex messages those consumers carry.

Where this fits in a healthy Magento store

Consumer reliability is infrastructure plumbing, but it decides whether features your merchants paid for actually run. It is part of the operational depth that separates a store that is merely live from one that is dependable, which is the standard we hold across every Magento and Adobe Commerce build and every Hyva frontend we ship. The same operational care applies to the other platforms we work on, whether that is Shopify, Shopware, or BigCommerce; the queue mechanics differ, but the discipline does not.

Bemeir runs Magento support and infrastructure work as a core service. You can read more about Bemeir or start from the Bemeir homepage if you want help auditing why your background jobs are not finishing.

FAQ

How do I know if my Magento consumers are running?

Check the server process list for queue:consumers:start processes, and run bin/magento queue:consumers:list to see what should exist. On RabbitMQ, watch queue depth in the management UI: queues that only grow mean nothing is consuming them. If the consumers_runner cron job is enabled, confirm base Magento cron is firing every minute.

Should I use cron or Supervisor for consumers?

Use the built-in consumers_runner cron job for small and mid-sized stores; it is self-healing and needs no extra setup. Use Supervisor or systemd for high-volume stores that need continuous, high-throughput processing, which usually means you have already moved the queue off MySQL to RabbitMQ.

What does max_messages do, and should I set it to 0?

max_messages is how many messages a consumer processes before terminating, defaulting to 10000. Setting it to 0 means the consumer never terminates. Adobe does not recommend 0 with the cron runner. Under Supervisor, keep a finite value so each worker recycles and releases memory, and let Supervisor restart it.

Why do my order emails and bulk updates arrive late or never?

Those are queued jobs handled by consumers. Late or missing emails and bulk attribute updates almost always mean the relevant consumer is not running, is crashing on a memory limit, or is overwhelmed on a MySQL-backed queue. Confirm the consumer exists, check logs for memory errors, and consider moving to RabbitMQ.

Do I need RabbitMQ, or is MySQL enough?

MySQL is fine for low volume and a single consumer. For concurrency, multiple processes, or significant bulk and async volume, use RabbitMQ. Adobe advises against running multiple consumers on a MySQL queue, and Adobe Commerce Cloud requires RabbitMQ.