The conventional process for picking a Magento agency tends to produce conventional results. The buyer collects pitch decks from three agencies, runs reference calls that are too short to surface anything important, and chooses the firm with the best logos. Three quarters in, the engagement is either working or it is not, and the cost of finding out is borne entirely by the buyer.
This is not how you would evaluate any other engineering supplier. When you evaluate a payment processor or a hosting partner, you look at engineering signals, uptime reports, security disclosures, incident response data, team depth in specific specialties. Magento agencies are an engineering supplier, and they should be evaluated on engineering signals. This article is a framework for doing that, built from years of competitive evaluations and second-opinion engagements at Bemeir’s Magento development practice.
Why logos and case studies aren’t enough
Logos prove that an agency once won a deal with a known brand. They do not prove anything about the engineering practices, the team depth, the on-call maturity, or the current state of the relationship. Case studies are marketing artifacts written by the agency about themselves; they describe success without surfacing the cost, the friction, or the things that went wrong. Reference calls are usually thirty minutes long, conducted with a hand-picked happy client, and produce almost no useful signal.
What you actually want to evaluate is whether this agency, with the team they will actually staff on your account, can do the engineering work your platform needs over the next three years. That is an engineering evaluation, and it deserves engineering rigor.
Six engineering signals that actually matter
The signals below are ones we have seen consistently separate strong Adobe Commerce agencies from weak ones. None of them is fully revealed in a pitch deck. All of them are accessible if the buyer asks the right questions in the right order.
1. Code review discipline
Ask the agency to walk you through how a pull request gets approved on a production Magento codebase. The answer should include: at least one named senior reviewer per PR, automated linting and static analysis gates, a test suite that runs on every PR, and a clear policy on what classes of change require additional review (database schema changes, payment flow changes, security-adjacent changes). If the answer is vague, or if “code review” means a senior person occasionally looks at things, you are looking at an agency that ships code without engineering hygiene.
2. Test coverage on custom modules
Magento has a complicated relationship with test coverage. Core Magento tests are extensive but slow; custom module tests are usually thin. A serious agency has a documented testing approach for the custom modules they write, at minimum, unit tests for business logic and integration tests for the most critical workflows. Ask to see a representative custom module they’ve written and look at the tests folder. If it’s empty, the agency does not test their own code.
3. Architecture decision records
Mature engineering organizations document the decisions they make. The mechanism is usually called an ADR (Architecture Decision Record), and a well-run Adobe Commerce engagement will have several dozen of them after a couple of years. Ask the agency to show you an example ADR from a client engagement, redacted for client confidentiality. If they have never heard of the concept, or if “architectural decisions” live entirely in a senior engineer’s head, you are looking at an agency that has no institutional memory.
4. On-call and incident response
Production Magento environments have incidents. The right question is not whether they happen but how the agency handles them. Ask: what is your on-call rotation, what is your incident response runbook, what is your time-to-acknowledge SLA, and can you show me a redacted postmortem from a recent incident? The answer should be specific and quantitative. If the answer is “we have great engineers who respond when needed,” you are looking at an agency that does not have a real on-call practice.
| Engineering capability | Strong signal | Weak signal |
|---|---|---|
| Code review | Mandatory senior review, automated gates, documented policy | “Senior devs check things” |
| Custom module tests | Coverage targets, representative test files visible on request | Empty test folders in their own modules |
| Architecture decisions | ADR repository, examples available redacted | Tribal knowledge, no documentation |
| On-call response | Rotation schedule, time-to-acknowledge SLA, postmortem examples | “We respond fast when something breaks” |
| Security practice | Patch cadence committed to SLA, security disclosures handled | “We apply patches when needed” |
| Team depth | Named architect per account, bench of senior engineers, hiring plan | Single senior person serving multiple accounts |
5. Security patch cadence
The Adobe Commerce security patch cadence is well-defined and the patches themselves are well-documented. A serious agency commits to applying critical security patches within a specific number of days of release, typically 7-14 days for the highest severity, 30 days for moderate severity. Ask for that SLA in writing, ask for evidence of recent compliance, and ask what their process is for patches that introduce regressions. According to the Adobe Commerce security center, patches are rated by severity and most include a recommended deployment timeline; the agency’s commitment should map to that timeline.
6. Team depth and named-individual continuity
The single biggest hidden cost of a Magento agency relationship is staff turnover at the agency. The lead architect you meet during the sales process is not always the architect who ends up running your account, and even if they are, they may not stay there. Ask for the agency’s average tenure, the named individuals who will be on your account, and what happens if a key engineer leaves. A serious agency has bench depth and a documented handover process. A weak one has a single senior engineer covering three accounts and no plan for their departure.
The technical due diligence pass
Beyond the questions above, a buyer who is serious about evaluating an Adobe Commerce agency should commission a technical due diligence pass: a paid, scoped exercise where the candidate agency reviews a portion of the buyer’s existing codebase and produces a written report. The report should include findings on code quality, security posture, performance hot spots, and architectural recommendations. The cost is typically a few thousand dollars; the signal is enormous.
This is the single most useful evaluation tool we know of. The reasons are clean: it surfaces how the agency actually thinks about a Magento codebase, the depth of their team’s expertise, and their ability to communicate technical findings to non-technical stakeholders. It also reveals their commercial behavior, does the report try to upsell at every opportunity, or is it candid about what should be left alone? At Bemeir’s Adobe Commerce team, we have lost engagements during this kind of due diligence pass, and we have won engagements precisely because the buyer compared our report to one written by a competitor and chose accordingly.
What about Shopify, Shopware, and BigCommerce?
For retailers evaluating non-Magento platforms, the framework is identical but the specifics shift. A Shopify agency should be evaluated on app development discipline, Shopify Plus expertise, and headless capability. A Shopware agency should be evaluated on European compliance experience and ERP integration depth. The buyer’s job is the same: ignore the logos, focus on the engineering signals, and commission paid due diligence before signing a long-term contract. Bemeir’s Shopify Plus practice follows the same evaluation methodology that the Magento practice does, because the underlying question is the same: can this team actually do the engineering work?
Industry references
Gartner’s research on eCommerce agency selection consistently identifies engineering discipline and team continuity as the largest differentiators between successful and unsuccessful agency engagements. Internet Retailer’s annual agency survey shows that mid-market retailers who switched agencies cite “engineering quality” and “responsiveness” as the top two reasons; logos and case studies do not appear in the top five reasons retailers stay with an agency longer than three years.
The buyer’s job
The retailers who get agency selection right do not treat it as a procurement exercise. They treat it as an engineering hire, with all the rigor that implies. They ask hard questions. They commission paid technical due diligence. They look past the sales process to the actual engineering team that will run their account. And they walk away from agencies that cannot answer the engineering questions with engineering specificity.
The cost of doing this work is real. The cost of skipping it is much larger and shows up six to twelve months into an engagement that should have been caught at evaluation. The best agencies in the Magento ecosystem welcome this kind of rigor, because they know they will be the ones who pass it. The buyer’s job is to apply it consistently.
