Often, but rarely from the wording alone. Research has found people poor at spotting AI-written text by style, yet customers quickly notice when a reply ignores their question, gets a detail wrong, over-apologises or reads like a template. Specific, accurate replies that sound like your business are rarely questioned, whoever drafted them.
So the useful question isn't "will they know it was AI?" but "will this reply make them trust us less?" A customer who gets a vague, over-polished answer to a specific question doesn't need to identify the author to feel fobbed off. Fixing that problem is also what stops replies reading as AI-written, so the two questions have the same answer.
The main example is an illustrative sign maker with eight staff that started drafting quote follow-ups and customer replies with an AI assistant, and found out the hard way which replies gave themselves away. There are also examples from a software reseller, an events company, a print shop and an online clothing shop.
What the research says about spotting AI-written text
The best-known study is a 2023 paper in PNAS (Proceedings of the National Academy of Sciences) titled "Human heuristics for AI-generated language are flawed". Across six experiments with 4,600 participants, people could not reliably tell whether short self-descriptions in professional, hospitality and dating settings had been written by a person or by an AI model. They relied on cues that felt sensible but didn't work, such as treating first-person pronouns, contractions or mentions of family as signs of a human writer.
Two caveats matter for a business. The study used AI models from before 2023 and short self-presentations rather than customer service replies. And "people can't tell from style" isn't the same as "people can't tell". Customers judge a reply against what they asked and what they know about you, which a stranger in an experiment can't do.
Tools don't do much better. OpenAI launched an AI text classifier in early 2023 and withdrew it that July, citing its low rate of accuracy. Detectors still exist, and some customers use them, but a short email gives them very little to work with, and false results in both directions are common.
What customers actually notice (it isn't the vocabulary)
Customers don't analyse sentence structure. They notice when a reply fails them. These are the signals that make people suspect, or simply feel, that no one really read their message:
- The question wasn't answered. They asked whether the sign can be fitted before a particular Saturday; the reply talks about "our commitment to timely installation".
- A detail is wrong. The reply mentions a product they didn't order or a conversation that didn't happen. Invented specifics give AI away faster than anything else, because a person who read the message wouldn't make that mistake.
- Generic sympathy. "I completely understand your frustration" in reply to a question that contained no frustration.
- A sudden change of voice. A regular customer who is used to "Hi Sam, yes we can do that" receives "Dear valued customer, thank you for reaching out".
- Too long for the question. A one-line question answered with four paragraphs and a bulleted summary.
- Identical wording to two people. Customers compare. Two committee members of the same club received word-for-word identical "personalised" replies from a print shop and mentioned it to each other, and then to the shop.
- Stock phrases. Some customers now look for phrases they associate with AI, such as "I hope this email finds you well" or "rest assured". The phrases are empty whoever writes them, so cutting them costs nothing.
Notice that almost every signal is a quality problem first. A human-written reply with the same faults would get the same reaction; AI just produces these faults faster and more often when nobody checks.
Four factors that decide whether a customer notices
Whether a particular customer will notice, and whether it matters if they do, depends on the situation. These four factors decide it, with rough thresholds to help you judge.
| Factor | Lower risk | Higher risk |
|---|---|---|
| Emotional stakes | Routine questions: hours, delivery, how to order | Complaints, bereavement, money disputes, anything the customer is upset about |
| How well they know you | First contact from a new enquiry | A customer who has written to the same person three or more times and knows their style |
| How specific the question is | General questions with standard answers | Questions that depend on their job, their order, their history with you |
| Channel and timing | Email answered in working hours | A long, polished reply arriving seconds after a message at 2am, which reads as automated |
A reply in the left-hand column all the way down can be AI-drafted with a quick check and nobody will think twice. A reply with two or more factors in the right-hand column needs a person to write it, or at least to rewrite the draft properly. The tutorial on handling sensitive customer conversations without AI covers the high-stakes end in detail.
A sign maker's quote follow-up, before and after
Here is the main worked example. The sign maker used an AI assistant to draft follow-ups to quotes that hadn't been accepted after a week. The first batch went out barely edited, and one of the replies it got back was from a café owner who had dealt with the business for years: "Is this a bot? You've never called me 'valued customer' before."
Before (AI draft, sent): Dear Valued Customer, I hope this email finds you well. I wanted to follow up regarding the signage quotation we recently provided. We understand that choosing the right signage solution is an important decision for your business. Please rest assured that our team is committed to delivering exceptional quality and service. Should you have any questions or require any further information, please do not hesitate to reach out. We look forward to the opportunity of working with you.
Everything in it is polite and nothing in it is specific: no mention of the sign, the price, the café or the one question the owner had actually asked on the phone (whether the fitting could happen on a Sunday when the café is closed).
After (edited): Hi [first name], just checking in on the quote for the new hanging sign and the window lettering ($1,240 fitted). You asked about fitting on a Sunday when you're closed: yes, we can do that, at no extra charge. If you're happy with the design, we could fit on either of the next two Sundays. Any changes to the lettering, just send them over and I'll redo the proof. Thanks, [owner's first name]
The after version answers the open question, mentions the actual job and price, offers two concrete dates and sounds like the person the customer knows. The owner accepted the quote the next day. Nobody asked who wrote it.
The sign maker's fix for the whole batch took a change of method rather than more editing. The assistant now drafts only from a short set of notes for each quote (what the job is, the price, what the customer asked, anything promised on the phone), and the prompt forbids the stock phrases. Editing time went from about six minutes per follow-up, most of it spent rewriting, to about two, most of it spent checking facts. Across roughly 30 follow-ups a month, that's two hours back, and the replies no longer read like circulars. The same approach, turned into reusable structures, is covered in writing reply templates that keep AI replies on-script.
The drafting prompt that stopped the circular-letter replies
This is the prompt the sign maker now uses, with the notes pasted in for each quote. It works because the AI can only use what's in the notes, and because it is told what not to write.
Draft a short follow-up email to a customer about a quote.
Use ONLY these notes. If something isn't in the notes, leave it out.
Notes: [customer first name] | [job: what we're making] | [price, fitted or not]
| [what they asked us] | [anything promised on the phone] | [next available dates]
Rules:
- Under 90 words. Start with "Hi [first name]," and end with "Thanks, [my first name]".
- Answer their question in the first two sentences.
- Mention the job and the price exactly as written in the notes.
- Offer one clear next step.
- Never write: "I hope this email finds you well", "valued customer",
"rest assured", "do not hesitate", "solution", "exceptional".An illustrative output for a different quote, a vinyl-lettered van for a plumbing firm, with notes saying the customer had asked whether the old lettering could be removed first:
Sample output: Hi [first name], following up on the quote for lettering your van ($680 fitted). You asked about the old lettering: yes, we'll remove it first and clean the panels, and that's included in the price. We can do the job at our workshop on Tuesday or Thursday next week, and it takes about half a day. Want me to pencil one in? Thanks, [owner's first name]
What still needed checking: "about half a day" wasn't in the notes. It happened to be right, but it was the AI filling a gap, so the owner added typical job durations to the notes template rather than trusting the guess next time. Everything else came straight from the notes, which is why it reads like a person who knows the job.
A five-minute blind test for your own replies
If you want to know whether your replies give themselves away, test them the way customers read them. Take eight recent replies, some AI-drafted and some written from scratch, remove names, and ask a colleague or a trusted regular customer to sort them into "AI" and "person" and say why for each one.
The guesses matter less than the reasons. When the sign maker ran this, the reader's reasons for calling a reply "AI" were "doesn't answer what they asked", "too formal for us" and "says 'solution' twice". Two of the replies she labelled AI had been written by a person in a hurry. The reasons became the checklist the team uses before sending, for every reply, however it was drafted. Run the test again every few months, especially after changing tools or prompts.
Where replies gave themselves away in other businesses
- A software reseller's support ticket. A customer described an error in a specific version of an accounting app's add-in. The AI-drafted reply gave steps for a different version and closed with "I hope this helps resolve your issue!" The customer's response was "Did anyone actually read my ticket?" The problem wasn't the style; it was a reply that didn't match the question. The reseller now requires the agent to paste the customer's exact error message into the draft prompt and to check the steps against it.
- An events company and a cancelled wedding. A couple cancelled because of a family bereavement. The AI draft was kind in a generic way and three paragraphs long, with a bulleted list of "next steps" for the refund. The owner deleted it and wrote four short sentences herself. Some messages should never start as an AI draft, and a bereavement is one of them.
- An online clothing shop's apology loop. Three emails to the same customer about a delayed parcel all opened with "I completely understand your frustration". By the third, the customer wrote back "please stop understanding my frustration and tell me where my parcel is". A banned-phrase list and a rule to lead with the tracking information fixed it.
- A print shop's 2am replies. An AI assistant answering website messages overnight replied instantly with long, polished answers. Customers weren't fooled and some felt misled, because nothing said it was automated. Adding "You're chatting with our AI assistant; a person will pick this up from 8am" at the start made the same answers acceptable.
Should you tell customers AI helped?
There are two different situations here, and the answer differs.
When AI talks to customers directly, such as a website chatbot or automatic replies in messaging apps, tell them. Meta's Business Agent, for example, can now answer WhatsApp, Instagram and Messenger messages on a business's behalf; customers deserve to know when that's what is replying. If you sell to customers in the EU, the EU AI Act's transparency duties, in force since 2 August 2026, require that people are told when they're interacting with an AI system. The tutorial on what to tell customers at the start of a chat has wording you can use.
When a person uses AI to draft a reply that they read, correct and send under their own name, there's no universal rule, and most businesses treat it like any other writing aid. Many choose to explain their general use of AI in a short statement on their website rather than labelling individual emails. What matters more is the honesty of the content: a reply shouldn't claim personal knowledge ("I remember your last order well") that the sender doesn't have. The fuller discussion is in should you tell customers you use AI.
Humaniser tools and detectors don't solve this
A whole category of tools promises to make AI text "undetectable" by rewriting it. They target the wrong problem. A reply that fails to answer the question, invents a detail or opens with stock sympathy is still a bad reply after it has been reworded to fool a detector, and customers aren't running detectors on your emails anyway. The tutorial on whether an AI humaniser tool is worth paying for goes into the risks, including rewrites that introduce new errors.
The same logic applies in reverse. If a customer tells you a detector says your email was AI-written, don't argue about the detector. Ask whether the reply answered their question and put it right if it didn't.
A rule for which replies AI may draft
Here is the sign maker's rule, which works for most small businesses. It sorts replies by the four factors above rather than by how "AI-sounding" they might be.
| Type of message | Who writes it | Check before sending |
|---|---|---|
| Routine questions (hours, lead times, how to order, delivery) | AI drafts from notes or templates | Facts correct; question answered; no stock phrases |
| Quote follow-ups and job updates | AI drafts from job notes; person edits | Price, dates and anything promised in person are right |
| Complaints and disputes | Person writes; AI may help summarise the history | Tone read by a second person if money or blame is involved |
| Bereavement, illness, anything personal | Person writes, from scratch | None beyond care |
| Replies sent automatically with no person involved | AI, clearly labelled as AI | Weekly sample read; hand-over path to a person |
The short version: let AI draft what's routine, keep people on what's personal, label what's automatic, and check every draft for the things customers actually notice. Do that, and whether a customer could tell stops mattering much, because the reply does its job either way.
More questions about AI-drafted replies
Is it dishonest to send an AI-drafted email under my name?
Not if you have read it, corrected it and stand behind it, in the same way a letter drafted by an assistant and signed by a manager isn't dishonest. It becomes a problem when nobody checks, when the reply claims personal knowledge it doesn't have, or when a customer believes they're talking to a person but a bot is replying on its own.
Do AI detector tools work on customer emails?
Not reliably. Short texts are especially hard to judge, and OpenAI withdrew its own AI text classifier in 2023 because of its low accuracy. A customer running your reply through a detector may get a false result either way, which is another reason to judge replies by accuracy and fit rather than by how human they sound.
Should our chatbot pretend to be a person?
No. Tell people at the start of the chat that they're talking to an AI assistant and how to reach a person. If you sell to customers in the EU, the EU AI Act's transparency duties require people to be told they're interacting with an AI system, and customers who discover a disguised bot tend to trust everything else you say less.
Which words make an email sound AI-written?
No single word proves anything, but stock phrases such as 'I hope this email finds you well', 'rest assured', 'I understand your frustration' and 'please do not hesitate to reach out' have become associated with AI drafts. Cut them because they're empty, whether a person or a model wrote them.
Further reads
- Tone of Voice Examples: AI Copy Before and After Editing — Before-and-after edits that make AI drafts sound like your business.
- AI Words to Avoid: A Banned-Phrase List for Your Brand — Build a banned-phrase list so stock AI wording never goes out.
- Should You Tell Customers When You Use AI in Your Marketing? — Whether to tell customers when AI helped with your marketing.
- How to Write an AI Disclosure Statement for Your Website — Write a short statement explaining how your business uses AI.
- How to Draft Customer Email Replies With AI That Sound Like You — Prompts for customer email replies that sound like you.
- How to Spot-Check AI Support Replies: A Weekly Sampling Routine — A weekly routine for checking AI replies before patterns set in.
- AI Tools and AI Development: The Complete 2026 Guide — the AI hub, including every tutorial in the AI-for-business series.
Sources: Jakesch, Hancock and Naaman, Human heuristics for AI-generated language are flawed, PNAS (2023), via the authors' lab summary; OpenAI's 2023 note withdrawing its AI text classifier for low accuracy; series fact sheet for EU AI Act Article 50 transparency duties and Meta Business Agent (checked September 2026).