How to Check an AI Software Vendor's Customer References

To check an AI software vendor's customer references, ask for two or three customers who match your size and use case and have been live for at least six months, call them yourself with prepared questions on accuracy, set-up effort, support and surprise costs, then find one more customer the vendor didn't choose. Weigh specifics over praise.

The unchosen reference matters because a vendor will naturally put forward its happiest customers, sometimes ones who had extra onboarding help or a discount for agreeing to take calls. With AI tools the gap between the best customer and a typical one can be wide, since accuracy depends on how clean each customer's data and set-up are. A studio that runs the product on tidy data with a dedicated account manager tells you little about how it will behave on your messy inbox.

Follow me on Instagram@sagnikteaches

References are one part of a wider decision, alongside the demo, the security paperwork and the contract. If you haven't scored the vendors on those yet, the small business scorecard for evaluating AI vendors covers the full picture; here we stay with the reference check itself, step by step, using one running example: a four-person photography studio choosing an AI assistant that answers website enquiries and books consultation calls.

Connect on LinkedInSagnik Bhattacharya

Ask for references that match your size, job and plan

A reference is only useful if their situation resembles yours. A 200-person agency on an enterprise contract will have a different product experience from a small studio on the entry plan, even with the same software. Send the vendor your matching criteria in writing, ideally the day after the demo while they're keen. (If the demo itself felt scripted, the tutorial on running an AI software demo so you see the real product is worth reading before you go further.)

Subscribe on YouTube@codingliquids

Here is the request the studio sent, filled in. Notice that it asks openly about incentives: many vendors give references a discount or free months, which is fine, but you should know about it when you weigh what they say.

Subject: Customer references before we decide

Hi [first name],

Thanks for Tuesday's demo. Before we decide, we'd like to speak to
two or three of your customers. Ideally they would:

- be a studio or small creative business with under 10 staff;
- use the assistant for website chat replies AND consultation
  bookings, not only the gallery features;
- have been live for at least six months;
- be on the Growth plan (the one you've quoted us) and connected
  to Google Calendar.

If one of them had a problem with the assistant that you helped
fix, we'd especially like to talk to them.

Could you also tell us whether any of these customers receive a
discount or another benefit for acting as a reference? That's
completely fine; we'd just like to know.

Thanks,
[your name]

The vendor's reply is itself evidence. A quick answer with three well-matched names is a good sign. A two-week delay, an offer of written testimonials instead of calls, or three references who are all far larger than you are all worth noting on your scorecard. So is a reference list that contains only the customers named in the vendor's case studies, because those accounts were chosen for marketing.

Find at least one customer the vendor didn't hand-pick

This is the backchannel check, and it is where most of the useful surprises come from. You're looking for an ordinary customer, on an ordinary plan, who has no reason to perform. Four places tend to work.

• Review sites. Use them to find people, not scores. G2 says it clearly labels reviews that came with an incentive, and its tags show whether the reviewer is a current user and how the review was sourced. Capterra invites some reviewers with a nominal incentive, paid whatever rating they give, and its reviewer profiles show organisation size and how long they've used the product. Filter for businesses your size, read the two- and three-star reviews first, and note the date of each review against the vendor's last big AI update.
• The vendor's own case studies. These name real customers. Contact them directly rather than through the vendor, and you'll often get a franker account than the published one.
• Trade groups and communities. Photographers, wedding suppliers and most trades have forums and social groups where people discuss their software. A search for the product name inside those groups often turns up exactly the complaint you need to hear.
• Your own network. Your accountant, IT support firm or a supplier may know three other businesses using the product.

Keep the approach short and make it easy to say yes. The studio sent this message to a wedding photographer it found through a review site:

Hi [first name], I run a four-person photography studio and we're
considering [product] for website enquiries and consultation
bookings. I saw from your review that your studio uses it.

Would you have ten minutes for a call this week, or be willing to
answer three quick questions by message?

1. How accurate are its replies now, compared with the first month?
2. What took longest to set up?
3. Is there anything you'd check before signing?

Happy to return the favour any time. Thanks, [your name]

Two of the three people the studio contacted replied. One offered a call; the other answered by message, and that written answer turned out to be the most useful of the week, because it mentioned something no vendor-supplied reference had: on the entry plan, support was by email only.

A 20-minute reference call script

Prepared questions stop the call drifting into friendly chat. Ask for numbers and recent examples rather than opinions, and if the reference is willing, ask them to share their screen for two minutes so you can see the product's review queue or settings as they actually use them.

CONTEXT (2 minutes)
- What do you use it for, and what did you use before?
- Roughly how many conversations or items does it handle a month?

ACCURACY (6 minutes)
- Of its last 50 replies or actions, about how many needed correcting?
- What was the last mistake you caught? How did you spot it?
- Did accuracy change after an update? Were you told in advance?
- What does it do when it doesn't know the answer?

EFFORT (4 minutes)
- How many hours did set-up take, and how many per week now?
- What did you have to write or tidy first (FAQs, prices, rules)?

SUPPORT (3 minutes)
- Last time something broke, how long until a useful answer?
- Who answered: a named person, a ticket queue or a chatbot?

MONEY (3 minutes)
- Has your bill changed since you signed? Usage charges? Add-ons?
- Did you get a discount or free months for being a reference?

EXIT (2 minutes)
- Could you export your data and settings if you left? Tried it?
- Would you buy it again at today's price?

Write the answers down during the call, in the reference's words. "About one in twenty needs a tweak, usually prices" is far more useful later than your summary of "accuracy good".

Questions that expose AI-specific weak spots

Ordinary software references focus on bugs and support. AI tools add failure modes that a happy customer may not think to mention unless you ask: confident wrong answers, quiet changes in behaviour when the vendor swaps the underlying model, and extra review work that eats the time the tool was meant to save. For a longer list of questions to put to the vendor itself, see questions to ask an AI vendor before you sign anything; with references, focus on what they have seen happen.

The difference between a weak and a strong answer is usually obvious once you hear both. These two answers to "how accurate is it?" are illustrative, but typical of what you'll get:

Weak: "Oh, it's really accurate. We love it. Clients don't even know it's a bot."

Strong: "We read about 20 chats a week. Roughly one in fifteen needs a correction, mostly package prices right after we change them. The vendor added a setting that hands over to us whenever someone asks about a price, and that cut the errors a lot."

The strong answer tells you the error rate, the kind of error, the fix, and that the customer checks the output. The weak one tells you the customer likes the product, and also, worryingly, that clients may not know they are talking to software. Follow that up: if you sell to customers in the EU, the AI Act's transparency rules already require telling people when they're talking to a chatbot.

Three more AI-specific probes are worth asking every reference:

• "What changed after the last big update?" Vendors update models often. A reference who noticed replies getting longer, vaguer or more formal overnight has seen how the vendor handles change.
• "How much of your week goes on checking it?" If the answer is three hours to save four, the maths is thinner than the sales deck suggested.
• "Has it ever said something you had to apologise for?" Nearly every honest reference has one story. The absence of any story after a year is less reassuring than a story with a good fix.

Reading hesitation, hedges and too-perfect answers

Much of a reference check is listening for what isn't said. These signals don't prove a problem, but each one deserves a follow-up question.

• They keep mentioning "our account manager". Ask which plan they're on. Dedicated success managers usually come with larger contracts, and you may get email support instead.
• "We're still rolling it out." After six months, that often means adoption stalled. Ask what's holding it back.
• No numbers at all. If they can't estimate an error rate or hours saved, they probably aren't measuring, which means nobody knows whether it works.
• A relationship you didn't know about. Resellers, integration partners and former employees of the vendor make poor references. Ask how they came to use the product.
• Everything is perfect. Real software has rough edges. Ask directly: "If you could change one thing, what would it be?"

Here is how skipping this step shows up later, in an illustrative case. A wedding planner checked two glowing references for an AI enquiry tool and signed a year's contract on the entry plan. Both references, it turned out, were on the top tier with a named success manager who tuned their set-up monthly. On the entry plan, support was a ticket form with two-day replies, and the planner's first busy season ran on default settings that nobody had time to adjust. One question, "which plan are you on, and who helps you when it misbehaves?", would have caught it.

Worked example: a photography studio scores four references

Back to the studio. It handles around 180 website enquiries a month and was choosing between two AI enquiry assistants, Vendor A and Vendor B, at similar list prices. It spoke to four references in total: two from each vendor, one of them found through a review site. The figures below are illustrative, but they're the kind of answers you should be collecting.

The studio scored each vendor out of 5 on the five things it cared about, weighted by importance: accuracy (weight 3), weekly effort (2), price stability (2), support (1) and exit (1). Vendor A averaged 4 for accuracy, 4 for effort, 5 for price stability, 3 for support and 4 for exit: a weighted total of 37 out of 45. Vendor B scored 2, 2, 2, 5 and 2: a total of 21. Vendor B's quicker set-up and faster support were real advantages, but the correction rate and the renewal jumps outweighed them.

A quick sum shows why five hours of calls was time well spent. Say a poor tool left 5% of the studio's 180 monthly enquiries unanswered or mishandled for a month while it was being fixed: that's 9 enquiries. If one in three would normally have become a booking worth around $600, the studio loses about $1,800 in a month, before counting the time spent migrating to something else. The reference calls cost an afternoon.

The studio chose Vendor A, but it didn't stop there. The independent reference had mentioned that support was email-only on the entry plan, so the studio asked for a named contact during the first 60 days and got it in writing.

Turn what references told you into contract terms

Every warning a reference gives you is a clause you can ask for. Vendors will often agree to terms a reference prompted, because the request sounds specific and reasonable rather than defensive.

The exit terms deserve more attention than most owners give them. AI products do disappear: Clockwise, an AI calendar tool, shut down on 27 March 2026 and deleted user data rather than transferring it, and OpenAI discontinued Sora entirely in 2026. Ask each reference whether they have ever exported their data, then ask the vendor to show you. The tutorial on what happens if your AI vendor shuts down covers the checks in full.

Using an AI assistant to compare your call notes

After four calls your notes will be uneven, and it's tempting to rely on whichever conversation felt best. A general AI assistant on a business plan can line the evidence up for you, as long as you forbid it from filling gaps. The studio used a prompt like this:

I'm comparing reference calls for two AI enquiry assistants.
Below are my raw notes from four calls.

Build a table with one row per reference and these columns:
vendor, business type and size, months live, plan tier,
corrections needed (as stated), set-up hours, weekly checking
hours, support speed, price changes, export tested, would buy again.

Use only what is in the notes. Where the notes don't cover a
column, write "not asked". Do not estimate or round numbers.

Then list up to three follow-up questions for each vendor,
based only on gaps or warnings in the notes.

[paste notes]

The illustrative output below shows the two typical problems to look for:

| Ref | Vendor | Business        | Months | Plan   | Corrections | Export tested |
| 1   | A      | Studio, 3 staff | 14     | Growth | ~1 in 20    | No            |
| 2   | A      | Wedding photog. | 9      | Growth | ~1 in 15    | Yes (CSV)     |
| 3   | B      | Portrait studio | 7      | Pro    | ~1 in 8     | No            |
| 4   | B      | Events studio   | 11     | Growth | ~1 in 25    | No            |

Follow-ups for Vendor B:
1. Can you confirm renewal pricing for the Growth plan?
2. What support is included on Growth?
3. How do customers export their data?

Two fixes were needed. For reference 4, the notes said "a few a week, out of maybe 40", and the assistant converted that into "about 1 in 25", which is a guess dressed as a number; it should say "a few a week" or "not measured". And "Export tested: No" for references 1, 3 and 4 hides the fact that the studio never asked references 3 and 4; the honest entry is "not asked". The follow-up questions were sensible but generic, so the studio rewrote them around what the references actually said: "Reference 3 saw a 30% renewal rise. What will our renewal price be?"

Keeping the reference check proportionate

Not every purchase needs four calls. A $20-a-month writing tool on monthly billing that you can cancel in a click needs a trial and perhaps a glance at recent reviews. A tool that talks to your customers, touches client data, or locks you into an annual contract deserves the full process, because unwinding it costs far more than the calls. A reasonable rule: the more the tool acts on its own, and the harder it is to leave, the more references you want and the more of them should be ones you found yourself.

Finally, watch for the reverse problem: a product so new that no matched references exist. That isn't automatically disqualifying, but it shifts the burden onto a trial or a paid pilot with written success criteria, and onto monthly billing until the product has earned a longer commitment. It's also worth checking how much of the "AI" is the vendor's own work; the tutorial on spotting software that oversells its AI explains how to tell.

Reference checks: the follow-up questions

How many references should I speak to before buying AI software?

Three is a sensible minimum for a tool that touches customers or money: two the vendor supplies and at least one you find yourself. For a cheap tool on monthly billing that you can cancel in a click, one matched reference plus a proper trial is usually enough. Scale the effort to what a bad choice would cost you to unwind.

What if the vendor won't give me any references?

Ask why. A genuinely new product may have few customers, in which case ask for a longer trial, a paid pilot with written success criteria, or monthly billing until it proves itself. A vendor that has been selling for years and still refuses, or offers only written testimonials, is telling you something, and you should weigh that heavily.

Is it acceptable to contact a vendor's customers without asking the vendor?

Yes, as long as you're polite, brief and honest about why you're asking. Many owners will happily give a peer ten minutes. Don't press them for confidential details such as their contract price, and never repeat what they said to the vendor without their permission, because that ends the goodwill that makes backchannel checks work.

Should star ratings on review sites influence my decision?

Use review sites to find patterns and people, not to decide. Filter for businesses your size, read the two- and three-star reviews first, and check the labels showing whether a reviewer received an incentive or was invited. A handful of detailed reviews from the last six months beats a high average built up before the product's latest AI changes.

Further reads

• SOC 2 and ISO 27001 Explained: Checking an AI Vendor's Security — Check the security claims a reference can't vouch for.
• What to Check in an AI Vendor's Data Processing Agreement — What the data terms should say once references check out.
• AI Software Contracts: Auto-Renewals, Price Rises, Notice Periods — Turn reference warnings about renewals into contract wording.
• How to Negotiate AI Software Prices and Seat Numbers — Use what references said about real usage to size your seats.
• AI Vendor Lock-In: How to Keep Your Data and Prompts Portable — Plan the exit before you need it.
• How to Vet an AI Consultant's Case Studies and References — The same checks, applied to consultants instead of software.
• Choosing a Mortgage CRM With AI Built In: What Actually Matters — A demo-ready checklist for mortgage CRMs with AI: document extraction, chasing, client-bank alerts, compliance logs, integrations and exit terms.
• AI Video Survey Tools for Movers Compared: Cost and Accuracy (2026) — Six video survey tools for removal firms compared on published price, pricing model and what their accuracy claims actually measure.
• How to Run a Two-Week AI Tool Trial Before You Commit — A day-by-day plan for trialling an AI tool on real work, with a test-case list, a daily log, a filled-in scorecard and the cancellation steps people forget.
• Is That AI Tool a ChatGPT Wrapper? How to Check Before You Pay — Five checks that reveal which model an AI tool runs on, a worked sum of what the model really costs the vendor, and when a wrapper is worth it.
• What Does an AI SDR Cost, and What Does It Actually Do? — Published AI SDR prices from about $1 a lead to $3,750 a month, the mailbox and staff costs around them, and how to work out cost per booked meeting.
• AI Tools and AI Development: The Complete 2026 Guide — the AI hub, including every tutorial in the AI-for-business series.

Sources: G2 'Review validity' page; Capterra 'How Capterra collects and verifies reviews'; OpenAI's help article on the Sora discontinuation; the series fact sheet for the Clockwise shutdown date.