Reference checks are the most underrated step in an Adobe Commerce agency evaluation. CTOs spend weeks on technical interviews, then accept three glowing 20-minute calls with references the agency hand-picked, and consider the diligence complete. That is not a reference check. That is a sales call with a different person on the line.
A real reference check is a 45-minute conversation with someone who has finished an engagement, sometimes painfully, and who has the distance to be honest about what they would do differently. You will not get that conversation by accident. You have to design for it. The right questions, the right tone, and the right follow-ups produce a level of insight that no proposal document will ever match.
Bemeir runs the same exercise on the way in when we are being evaluated, and we sit through the exercise on the way out when our clients refer us to other founders. Both sides of the table tell you the same thing: the questions that actually reveal who an agency is are not on the list the agency sent you.
Why the standard reference list is broken
The default reference experience goes like this. The agency sends you three names. You email each of them. Two reply within a day with positive availability. You schedule 20-minute calls. On the call you ask whether they were satisfied, whether the agency hit deadlines, and whether they would recommend them. All three say yes. You close the call feeling reassured, when in fact you have just confirmed that the agency knows how to pick references.
The problem is not that the references are lying. They are usually not. The problem is that the questions are too soft to surface the things that matter. “Were you satisfied” is a five-second question. The interesting answer is in the 30 seconds after that, which most reference calls never reach because the conversation moves on.
Design the call so that the interesting answer is the entire goal of the meeting. That requires three changes: longer calls, sharper questions, and a willingness to find your own references, not just take the agency’s curated list.
Find your own references
The references the agency provides are useful but insufficient. The references you find on your own are where the truth lives. There are four channels worth using.
The first is LinkedIn. Search for ex-clients of the agency by looking at their case studies and finding the directors of engineering or VPs of eCommerce who were at the company when the work was done. Reach out for a 30-minute conversation. The second is industry community. The Magento Stack Exchange and the Magento Reddit community have long memories. Ask about agency reputation in those forums and you will get honest signal, especially in DMs. The third is partner ecosystem. Hyvä, Klaviyo, Yotpo, and ERP vendors keep informal track of which agencies their joint customers struggle with. A 10-minute call with a senior account executive at a partner often reveals more than two hours with the agency directly. The fourth is conference and meetup history. Look at who spoke at Adobe Summit or Meet Magento on behalf of which agency and follow up with the audience members who asked sharp questions. Those are the people who care about technical depth, and they will give you a candid read.
The questions to actually ask
A real reference call has a shape. Spend the first 10 minutes building rapport and getting the project context. Spend the next 25 minutes on the questions below. Spend the last 10 minutes on follow-ups and the off-the-record portion.
The questions below are the ones we have seen produce honest answers. They are designed to make it easier for the reference to volunteer real information without feeling disloyal.
- What surprised you about the engagement, positively or negatively, that you did not see coming during the sales process?
- Walk me through the moment the project felt the most off-track. What was happening and how did the agency respond?
- If you were running this same project again, what would you ask the agency to do differently from day one?
- Who on the agency’s team would you specifically request again, and who would you not?
- What is a category of work where this agency clearly outperforms peers, and what is a category where they should not be your first choice?
- How did the relationship feel at month 12 compared to month two, and what drove the change?
- What was the conversation like when scope changed or budget got tight, and how did they handle it?
- If you had to predict where they would struggle on our specific project, where would that be?
Question 8 is the killer. It forces the reference to apply their experience to your situation, which is what you actually came to learn. It also gives them an honest opt-out: they can say they are not sure, which itself is useful data.
What body language tells you on a call
You cannot literally see body language on a Zoom call, but you can hear the equivalent. Pauses, sighs, qualifying language, and word choice all carry signal. Pay attention to four patterns.
The first is the diplomatic detour. A reference who answers a sharp question with a generic answer about how the agency is “really collaborative” is usually hiding a specific story. Follow up with a more specific question, not a softer one. The second is the rate of speech change. People speed up when they are reciting and slow down when they are remembering. The slow-down moments are where the real material lives. The third is the qualifier creep. Watch for phrases like “for the most part” or “generally” or “in our case.” Those are markers of partial agreement. Probe gently. The fourth is the off-record offer. A reference who says “happy to chat more directly if you want to grab 15 minutes” is signaling they have more to say than they will on a recorded or formal call. Take them up on it.
The table below summarizes the signals we coach in-house teams to listen for during a reference call, alongside what the signal usually means.
| Signal | What it usually means | Recommended follow-up |
|---|---|---|
| Long pause before answer | They are deciding how candid to be | Stay silent for five extra seconds |
| Generic positive answer | They are diplomatically hiding a story | Ask for a specific example |
| Soft qualifiers (“for the most part”) | Partial endorsement | Probe what falls outside |
| Offer to chat more later | They have material they will not share on this call | Schedule the second call same week |
The off-the-record portion
Save the last five minutes of the call for the question that gets the most useful answer of the entire conversation. The phrasing is simple. “Off the record, if your closest friend was about to sign a contract with this agency, what would you tell them?” The framing matters. You are not asking them to publicly bad-mouth a vendor. You are asking them to advise a friend, which is a role most people take seriously.
You will get one of three responses. Either a clear positive endorsement, which is real because it costs them something to give. Or a clear concern, which is gold. Or a hedged answer, which itself tells you the engagement was middling and you should keep looking. We have hired and recommended teams based on this single question, and we have walked away from agencies based on this single question. The signal-to-noise ratio is unusually high.
What to do with what you learn
Triangulate. Do not over-weight one reference call, positive or negative. Look for patterns across three or four conversations. If the same complaint shows up twice, that is a pattern. If the same praise shows up twice, that is also a pattern. Single data points are anecdotes. Patterns are evidence.
When you sit down to make the final decision, the reference-check notes should sit next to the technical interview notes and the commercial proposal. Bemeir’s experience is that founders who weight reference checks at roughly 30 percent of the final decision tend to make better hires than founders who weight them at 5 percent. The technical interview tells you what the agency can do. The reference check tells you what they will actually do, which is not always the same thing.
If you are running parallel evaluations across Magento and Adobe Commerce, Shopify Plus, Shopware, and BigCommerce agencies, run the same reference framework against each. The questions translate cleanly across platforms because they are about partnership behavior, not technical specifics. You will see the platform-agnostic patterns immediately, and that signal is more durable than any single technical recommendation.
The agencies that earn their reputation know they will be reference-checked seriously, and they prepare their clients accordingly. The agencies that fade after the sales process do not. Designing the reference-check phase as its own thoughtful exercise, rather than a formality at the end of procurement, is the single highest-leverage diligence move available to a CTO making this decision. Take the time. The 90 minutes you spend on three real reference conversations will save you the 9 months it takes to unwind a wrong choice.
