The decision to engage a Magento agency is one of the most consequential technology choices a mid-market retailer makes, and most evaluation processes are not rigorous enough to surface the differences between agencies that look similar on paper. The default pattern is to issue an RFP, take three pitches, ask for references, and pick the agency whose sales process felt strongest. That process produces predictable disappointments because the things that distinguish a great Magento partner from a mediocre one are not visible in pitch decks.
A better evaluation process treats the agency selection like a technical hire, not a procurement decision. The agency is going to operate inside your codebase, your data, and your release cadence for months or years. The friction of switching later is high enough that doing the work upfront is worth it. The framework below is the one we use when CTOs ask us how to choose between us and someone else, and it is the same framework we recommend even when the answer is going to be that they hire a different agency.
Start With the Codebase Conversation, Not the Pitch Deck
The first qualifying conversation should not be about your roadmap or your business goals. It should be about your codebase. Send a sanitized composer.json, a list of installed modules, and a description of your current Magento version, theme, hosting setup, and integration list. Then ask each agency to walk you through what they see, what they would want to look at next, and which decisions look risky.
The agencies that produce useful conversation from this exercise will identify specific modules they have seen cause issues, ask targeted questions about how customizations were authored, and surface dependencies that affect upgrade or migration paths. The agencies that produce generic responses will say things like “we’d want to do a full audit” without telling you what they expect to find. The difference is not subtle once you do the comparison side by side.
This step also surfaces whether the agency’s technical leadership is going to be involved in your engagement or whether you are talking to sales staff who will hand off to delivery once the contract is signed. The right answer is that the people who walked through your composer.json are the people who will be in your standups three months later. Bemeir’s Magento engineering team operates this way deliberately – the architect on the call is the architect on the project.
Ask for Proof of Hyvä, B2B, and Performance Depth
Most agencies will claim Magento depth. The areas where claims and reality diverge are Hyvä theme implementation, Adobe Commerce B2B, headless and PWA Studio work, and Core Web Vitals optimization. These are the four specializations where mid-market projects most often go wrong, and they are the four where most agencies are still ramping up.
For Hyvä, ask the agency to walk through a recent migration with specific module compatibility decisions, checkout customizations they had to rebuild, and CWV measurements before and after. For B2B, ask about company hierarchies, shared catalogs, approval workflows, punchout integrations, and ERP sync patterns. For headless and PWA Studio, ask which projects they have shipped to production and how they handled SEO, personalization, and content management. For CWV, ask for a recent INP improvement project with concrete metrics on LCP, INP, and CLS.
The agencies with real depth will give you specific stories with named modules, real numbers, and clear opinions about trade-offs. The agencies without depth will speak in generalities. This is the single best filter for separating boutique specialists from generalist shops that happen to take Magento work.
Validate the Resourcing Plan Against the Estimate
The proposal will name an architect, a lead developer, a project manager, and probably a frontend specialist. Ask for each named resource: what percentage of their time is dedicated to your project, which other projects they are on, who their substitute is if they become unavailable, and what their tenure has been at the agency. Then ask for the contract to lock in named resources with substitution requiring your approval.
Agencies that resist this language are telling you something important. The agencies that name their best architects in the proposal and then assign your project to whoever has bandwidth are the ones that produce the largest gap between what was sold and what was delivered. The agencies that are willing to lock in resourcing in writing are usually the ones whose resourcing is actually stable.
The estimate itself should be decomposable. Ask the agency to show you hours by phase, hours by role, and the assumptions that drive each estimate. An agency that gives you a confident single number without derivation is negotiating, not estimating. An agency that can defend each line item against your scrutiny is doing real estimation. The contract you sign should reflect the latter.
Pressure-Test the Reference Conversations
Agency references are one of the most underused evaluation tools because most retailers ask the wrong questions. The references the agency provides are going to be positive – that is why the agency provided them. The valuable information is in the texture of the answers, not the headline sentiment.
The questions that surface the texture are: who actually did the work compared to who was named in the original proposal, what changed during the project that was hard to manage, how did the agency behave when something went wrong, what is the current state of the post-launch relationship, and would you hire them again knowing what you know now. The last question often produces the most honest answer because it forces a forward-looking judgment rather than a retrospective justification.
Ask for two references the agency picks and two references the agency does not pick – projects you find through case studies, LinkedIn, or industry conversations. The agency-picked references give you the agency’s best face; the independently-found references give you the texture across their portfolio. The difference between the two sets is usually informative.
The Cultural and Operational Fit Questions
The technical evaluation will get you to a shortlist. The cultural and operational fit will determine which shortlisted agency you should actually pick. The questions here are softer but they predict the long-term success of the engagement more than the technical answers do.
| Evaluation Dimension | Strong Signal | Warning Signal |
|---|---|---|
| Architect involvement | Same person on pitch, kickoff, and ongoing standups | Pitch architect disappears after contract signature |
| Communication cadence | Standing daily or twice-weekly working sessions with named owners | Weekly status meetings owned by project managers only |
| Conflict handling | References describe specific disagreements that were resolved constructively | References describe smooth projects with no friction (suspicious) |
| Estimate derivation | Hours by phase and role, documented assumptions, named contingency | Confident single number, “trust us” framing, no derivation |
| Resourcing transparency | Named resources with availability percentages, substitution rules | Generic team description, “we’ll assign the right people” |
| Knowledge transfer | Documentation deliverables specified, runbook included in scope | Knowledge stays with the agency, you stay dependent |
The fit questions should also include how the agency handles disagreement. Ask each finalist to describe a recent project where they pushed back on a client request, what the disagreement was about, and how it was resolved. The agencies that cannot remember a recent pushback are either telling you they are order-takers or are not being honest. The agencies that can describe a substantive pushback and a constructive resolution are showing you the kind of partner they will be when your project hits a hard decision.
The Contract Clauses That Matter
The contract is where evaluation becomes commitment, and the language that goes into the contract is what you will live with for the duration of the engagement. The clauses that matter most are named-resource lock-in with substitution rules, IP ownership of all code and documentation produced under the engagement, knowledge transfer requirements at each phase boundary, escalation paths for both technical and commercial disputes, and termination terms that allow you to exit cleanly if the engagement is not working.
The pricing model matters less than the discipline around scope. Fixed-price contracts work when the scope is well-defined and unlikely to change; time-and-materials works when discovery is going to surface unknowns; a hybrid structure with fixed price for known scope and explicit T&M handling for change orders usually produces the best alignment. The structure that produces friction is fixed-price contracts with unclear scope – the agency ends up arguing every change order and the relationship deteriorates.
The change management process should be specified in the contract itself, not deferred to “we’ll figure it out.” Lightweight email approval works for small changes; documented change orders work for substantive ones. The contract should say that only changes processed through the documented change management process count as authorized changes. This language prevents the informal scope creep that produces unrecognized timeline slip.
When to Walk Away
Some signals during evaluation should produce a walk-away decision rather than a deeper evaluation. The signals include refusal to lock in named resources, inability to walk through a recent CWV or Hyvä project with specifics, references that describe smooth projects with no friction (which usually means the references are coached), confident single-number estimates with no derivation, and unwillingness to commit to documentation and knowledge transfer as project deliverables.
The cost of switching agencies mid-project is high enough that the right move when these signals appear is to keep evaluating, not to compromise on the warning signs. The agencies that look like they will be hard to work with usually are, and the cost of discovering that confirms it after signing is much higher than the cost of taking an extra two weeks on evaluation.
The retailers who follow a rigorous evaluation process find that the shortlist of qualified Magento agencies is shorter than they expected. The depth required to handle Hyvä, B2B, and CWV at the level mid-market work requires lives in a handful of agencies. Boutique specialists like Bemeir’s Brooklyn-based Magento practice sit in that shortlist alongside two or three peers, and the comparison between them is the right comparison to make rather than comparing them against generalist shops that lack the specialization.
For deeper reference on agency selection practice, the Adobe Commerce Solutions Partner directory shows the partners that have been validated by Adobe, and the Magento Open Source community maintains contributor lists that show which agencies are investing in the platform itself. Industry research from Gartner on digital commerce providers and Forrester’s commerce services landscape provides additional structured frameworks for evaluating eCommerce technology partners.
