Adobe Commerce Async Bulk APIs and RabbitMQ for High-Volume Catalog Imports on Magento

The Adobe Commerce async bulk API is the right way to load a large catalog into Magento. You POST an array of products to an /async/bulk endpoint, get back an HTTP 202 with a bulk_uuid, and RabbitMQ consumers process the work in the background while you poll for status.

We build Magento and Adobe Commerce integrations where catalog and inventory data arrive from an ERP or PIM at volume. The parts that decide whether such a pipeline survives production are consumer supervision, status polling, and retry logic by status code, which most guides skip, so this is written for that operational reality rather than the happy path.

The three ways to call the Web API

Magento exposes the same REST resources through three delivery modes, and choosing the wrong one is the classic performance failure.

Note the /async and /async/bulk segments come before /V1 on self-hosted Adobe Commerce. On Adobe Commerce as a Cloud Service, the SaaS offering, the order inverts to /V1/async/bulk/..., so confirm which platform you are targeting. Async methods support POST, PUT, DELETE, and PATCH, but not GET. These routes are documented on Adobe’s bulk endpoints reference and asynchronous web endpoints reference.

The async bulk mode is the recommended path for importing thousands of SKUs, because the request returns immediately, the work is decoupled through a message queue, and you can source data from anything that can build JSON, not just a CSV. It replaces the pattern of hammering the synchronous endpoint one product at a time, which is where integrations fall over.

RabbitMQ, or the database queue

Adobe states that before using the Bulk API you must install and configure RabbitMQ, which is the default message broker. There is nuance worth knowing: the broker selection logic says that if AMQP or STOMP is configured for a queue, that connection is used, otherwise the database connection is used. So async bulk can technically run on the MySQL-backed database queue as a fallback.

The line to draw is operational. The database queue is acceptable only for low volume with a single consumer. RabbitMQ becomes mandatory once you need multiple parallel consumers, guaranteed delivery, dead-letter handling, or message priorities. Adobe explicitly warns against running multiple consumers against a MySQL-operated queue. For a real catalog integration, use RabbitMQ. This is the same message-queue backbone that carries production-grade Magento B2B integration patterns for ERP, CRM, and POS.

Enable it in env.php:

'queue' => [
    'amqp' => [
        'host' => 'rabbitmq.internal', 'port' => '5672',
        'user' => 'magento', 'password' => '***', 'virtualhost' => '/',
    ],
    'consumers_wait_for_messages' => 0,
],

Then run bin/magento setup:upgrade, which creates the exchanges, queues, and bindings, including the magento exchange and the async.operations.all queue that carries every async and bulk message. Confirm with bin/magento queue:consumers:list, which should show async.operations.all in the list.

Submitting a bulk catalog import

The body is a JSON array of the same request bodies you would send synchronously, one per product.

POST /rest/all/async/bulk/V1/products
Authorization: Bearer <admin token>
Content-Type: application/json

[
  { "product": { "sku": "SKU-1", "name": "Widget 1", "attribute_set_id": 4, "price": 9.99, "type_id": "simple" } },
  { "product": { "sku": "SKU-2", "name": "Widget 2", "attribute_set_id": 4, "price": 19.99, "type_id": "simple" } }
]

The response is HTTP 202 Accepted with a bulk_uuid and a request_items array, each item carrying an id, a data_hash (a SHA256 of the message content), and a status:

{ "bulk_uuid": "bdb2b78a-...", "request_items": [
    { "id": 0, "data_hash": "...", "status": "accepted" },
    { "id": 1, "data_hash": "...", "status": "accepted" } ],
  "errors": false }

One important subtlety: the submit response’s status field is currently always “accepted” and errors is always false, both reserved for future use. You cannot judge success from the submit response. You must poll the bulk status endpoint. Split a large catalog into batches and track one bulk_uuid per batch. A practical batch size is in the low hundreds of products per request; see the regression note below before going to a thousand or more per call.

Running and supervising consumers

Consumers are the part that gets neglected, and it is why integrations look fine in testing and stall in production.

By default, the consumers_runner cron job is enabled and starts all consumers, each processing 10,000 messages and then terminating. Consumers are designed to exit so they recycle memory, which means something must restart them. For manual or development runs:

bin/magento queue:consumers:start async.operations.all --max-messages=1000

For production, supervise the process with a manager so it restarts after each recycle and after a crash. A Supervisor program using the PID file path Adobe supports looks like:

command=php /var/www/magento/bin/magento queue:consumers:start async.operations.all
        --max-messages=10000 --pid-file-path=/var/run/mage-async.pid
autostart=true
autorestart=true
numprocs=4

The cron_consumers_runner block in env.php controls the built-in runner, with max_messages defaulting to 10,000 (setting 0 for never-terminate is not recommended), and multiple_processes mapping a consumer to a process count on Commerce 2.4.4 and later. Adobe’s manage message queues guide documents these keys, and the async configuration reference documents the broker selection rule and the async.operations.all consumer. Use RabbitMQ, not the database queue, when you run several consumers in parallel like this.

Polling status and handling failures

This is the logic that separates a resilient importer from one that silently drops data.

Check a batch with the status endpoint, and diagnose failures with detailed status:

GET /rest/all/V1/bulk/bdb2b78a-.../status
GET /rest/all/V1/bulk/bdb2b78a-.../detailed-status
GET /rest/all/V1/bulk/bdb2b78a-.../operation-status/3

The status response returns an operations_list with a per-operation integer status. The operation status endpoints reference defines the codes:

The retry policy follows directly from these. Treat status 2 as automatically retryable with exponential backoff. Treat status 3 as a data problem to surface to your ETL layer, correct, and resubmit, never a blind retry. Treat status 5 as terminal. Poll /status until no operation is still Open (status 4), with a timeout guard so a stuck batch does not loop forever, and use /detailed-status to pull the topic name and serialized data so you can replay the exact failed message.

Idempotency makes retries safe. Product create and update keyed on SKU is naturally idempotent, so a PUT /async/bulk/V1/products/bySku can replay a whole failed batch without creating duplicates, and the data_hash from the submit response lets you de-duplicate on your side. Path parameters like :sku become bySku in the async routes, with the first letter uppercased.

Monitoring the pipeline in production

Status polling tells you about one batch. Running a catalog integration day after day needs two standing signals, because a healthy submit path with stalled processing is the failure mode that hides longest.

The first signal is RabbitMQ queue depth on the async.operations.all queue. If depth climbs and does not drain, your consumers are not keeping up or have died without being restarted, and no amount of status polling on individual batches will surface that until every batch times out. Watch the queue length and the consumer count directly in the RabbitMQ management interface, and alert when depth crosses a threshold you set from normal throughput. A queue that grows during a nightly ERP push and drains by morning is fine; one that never returns to near zero is a supervision problem.

The second signal is inside Magento. The admin exposes bulk operations under System, in the Action Logs and Bulk Actions area, where a store manager can see recent bulk jobs and their success and failure counts without touching the API. This is the human-readable complement to the status endpoints, and it is where a merchandiser first notices that last night’s price update only half-applied.

Treat the two as a pair. RabbitMQ tells you whether messages are moving; the Magento bulk view and the status endpoints tell you whether they succeeded. An integration that only checks the submit response, and neither of these, will appear to work for weeks and then quietly drop a batch when a consumer dies during a deploy. Build the queue-depth alert and the status-polling loop before you trust the pipeline with a full catalog, the same way you would not run webhooks and out-of-process extensibility without visibility into what fired and what failed.

A regression worth patching around

One current, concrete gotcha: the APSB25-08 security patch introduced a regression where POST .../async/bulk/V1/products requests with 1,000 or more entries take significantly longer, affecting a range of 2.4.4 through 2.4.8 patch levels. The fix is Adobe patch AC-14078. If your imports are slow after a security patch, check whether you are on an affected version before rearchitecting, and keep batch sizes conservative until the patch is applied. Keeping this kind of async processing healthy is the same discipline as keeping cron and indexers healthy: the queue is machinery that needs monitoring, not a fire-and-forget.

FAQ

Do I need RabbitMQ to use the async bulk API?

For a real high-volume integration, yes. Async bulk can fall back to the MySQL database queue with a single consumer, but Adobe warns against running multiple consumers on the database queue. RabbitMQ is required once you need parallel consumers, guaranteed delivery, dead-letter handling, or priorities.

How do I know if my bulk import succeeded?

Not from the submit response, whose status is always “accepted.” Poll GET /V1/bulk/{bulkUuid}/status and read the per-operation integer codes: 1 is complete, 4 is still open, 2 and 3 are failures, and 5 is rejected. Use /detailed-status to see why an operation failed.

Why do my consumers stop processing after a while?

By design. Each consumer processes a set number of messages, 10,000 by default, then terminates to recycle memory. In production you must run them under a process manager like Supervisor so they restart automatically, rather than relying on a manual start.

How large should each bulk request be?

Batch into the low hundreds of products per request and track one bulk_uuid per batch. Avoid very large arrays, especially 1,000 or more entries on versions affected by the APSB25-08 regression, until Adobe patch AC-14078 is applied.

Is async bulk better than the native CSV import?

For automated, high-volume, or non-CSV data sources, yes. The native admin CSV import is good for one-off full loads but is format-locked and blocking. Async bulk accepts array payloads from any system, returns immediately, and decouples the work through the queue, which is what an ERP or PIM integration needs.

Where this fits

Bemeir is a Brooklyn Magento and Adobe Commerce agency with a strong B2B and enterprise integration focus and a deep technology partner ecosystem across ERP, PIM, and middleware. We connect catalogs and inventory to fast Hyva storefronts without the import layer becoming the bottleneck. We also deliver on Shopify, Shopware, and BigCommerce, so our integration advice is grounded in how each platform actually behaves at volume. To scope a data pipeline that holds under load, read about Bemeir or talk to the team.