In week one, collect your real customer questions and write the answers down. In week two, let AI draft replies that a person checks and sends. In week three, switch on automatic answers for order tracking, delivery times and returns only. In week four, review a sample of conversations, fix wrong answers and decide what comes next. Complaints and refunds stay with a person throughout.
That order matters more than the tool you pick. Most shops that regret AI support switched on a chatbot in week one and let customers discover what it got wrong. Below is the month day by day, with a filled-in answer bank, an edit log, handover rules and the numbers to check at the end.
Before day one: decide where the AI will sit
Pick the simplest place that covers where your questions arrive:
- Shopify Inbox, if you are on Shopify and most questions come through website chat. Its instant answers show preset questions and answers in the chat window (you can display up to 100), and Shopify Magic can suggest instant answers from your store policies (English only) and suggest replies during live conversations. There is a default "Track my order" answer you cannot delete, which is a hint about what shoppers ask most. Inbox now also includes a free AI agent that replies to customers on its own (Basic plan or above, with new customer accounts). It can fall back on web search when your store content doesn't cover a question, which is why the day-14 test below matters.
- A helpdesk with an AI agent, if you get a steady flow of email, chat and social messages and want them in one place. These usually charge per resolved conversation; Gorgias, for example, lists $0.90 per resolved interaction on most plans and $1 on Starter, on top of the helpdesk plan.
- A general assistant for drafting, such as ChatGPT or Claude, if most questions come by email and volumes are modest. Nothing customer-facing is automated; you paste in a message and get a draft.
For a side-by-side of chat tools, see the best AI chatbots for small Shopify stores. For the first month, the choice matters less than the plan below, and all three routes follow it.
Days 1 to 7: build the answer bank from real questions
Export the last three months of emails, chats and social messages. Remove customer names, addresses and order numbers, then ask an AI assistant to group them:
Below are customer messages to an online shop, with personal
details removed. Group them into topics. For each topic give:
the number of messages, two typical examples quoted exactly, and
whether answering needs order-specific information (yes/no).
Sort by number of messages. Do not invent topics.
[paste]An illustrative extract of the output, for a home-care provider's online shop selling daily-living aids:
1. Where is my order? (61) - order info needed: yes
"Ordered Monday, no tracking yet?"
2. Will it fit / is it suitable? (38) - order info: no
"Will the raised toilet seat fit an oval toilet?"
3. Delivery times and costs (27) - order info: no
4. Returns and exchanges (19) - order info: sometimes
5. Damaged or faulty item (11) - order info: yes
6. Product advice for a condition (9) - order info: no
"What would help my dad with Parkinson's eat more easily?"What to check: the AI placed three "damaged item" messages under returns, which matters because damage needs a person. And topic 6 is not a product question at all; it is a request for health advice, which the shop should route to a person who can suggest talking to an occupational therapist rather than have AI recommend products for a condition.
Now write the answer bank: one entry per topic, with the true answer, where the facts come from, and who owns keeping it up to date. A filled-in section:
| Topic | Answer (short version) | Source of truth | AI may answer? |
|---|---|---|---|
| Where is my order? | Dispatched within 1 working day; tracking email sent on dispatch; link to tracking page | Order system | Yes, via order lookup |
| Delivery times | 2-3 working days standard; $6, free over $50 | Shipping settings | Yes |
| Returns | 30 days, unused and in packaging; hygiene items excluded; how to start a return | Returns policy | Yes, process only |
| Will it fit? | Give the product's measurements and the measuring guide; never say "fits all" | Product specs | Draft only, person checks |
| Damaged item | Apologise, ask for a photo, person arranges replacement | Person | No |
| Advice for a condition | Person replies; suggest a professional assessment | Person | No |
Expect this week to take four to six hours. The tutorial on building the FAQ your chatbot needs before launch goes deeper on wording each answer.
Days 8 to 14: AI drafts, people send
For a fortnight, every AI reply is a draft. In Shopify Inbox that means using suggested replies; with a general assistant it means pasting each message in with the answer bank attached. Whoever sends the reply records any edit in a simple log. Running AI this way, where it does the work but customers never see its unchecked output, is what we call piloting in shadow mode.
A filled-in edit log after five days (illustrative):
| Topic | Drafts | Sent unchanged | Typical edit |
|---|---|---|---|
| Where is my order? | 22 | 19 | Added the courier's name |
| Delivery times | 9 | 9 | None |
| Returns | 7 | 4 | AI said hygiene items could be returned unopened; they can't |
| Will it fit? | 14 | 5 | AI said "fits most toilets"; replaced with measurements |
If you draft with a general assistant, keep the instruction identical all fortnight so the drafts are comparable:
You draft replies for a small online shop. Use ONLY the answer
bank below. If it doesn't cover the question, reply "NOT COVERED"
and nothing else. Under 90 words, plain and friendly, no
exclamation marks. Never promise a refund, a replacement or a
delivery date.
ANSWER BANK:
[paste]
CUSTOMER MESSAGE (name and order number removed):
[paste]An illustrative draft, for "I ordered the bath board on Thursday, can I swap it for the wider one?":
Thanks for getting in touch. You can exchange any unused item within 30 days, as long as it's in its original packaging. Start the exchange from our returns page and choose the wider board, and we'll send it as soon as the original is on its way back.
A person would change two things. The order was placed the day before, so it may not have left the warehouse, and the quicker fix is to amend the order before dispatch; the answer bank had no entry for that, so the AI answered a neighbouring question instead of saying "NOT COVERED". And "we'll send it as soon as the original is on its way back" is a promise the policy never makes, despite the instruction. Both go in the log, and "changing an order before dispatch" becomes a new answer-bank entry.
The log makes the week-three decision for you. Tracking and delivery drafts are nearly always right. Returns had a policy gap the answer bank needed to fix. Fit questions are still wrong more often than not, so they stay as drafts.
The day-14 test before anything goes live
On the last day of shadow mode, put twelve test questions to the assistant exactly as a customer would type them: eight the answer bank covers, and four it doesn't. The four are the real test, because a good set-up says it doesn't know and offers a person. An illustrative extract of the aids shop's test sheet:
| Test question | Expected | What came back | Result |
|---|---|---|---|
| "How much is delivery on a $42 order?" | $6, free over $50 | $6, free over $50 | Pass |
| "Can I return a commode I've used once?" | No, hygiene exclusion; offer a person | "Most retailers accept returns within 14 days" | Fail |
| "Do you do trade accounts for care homes?" | Not covered; hand to a person | Handed over | Pass |
| "Order came, but the grab rail is bent" | Damage trigger; hand over at once | Sent the returns steps first | Fail |
The commode answer is the web-search fallback at work: the store content didn't say "used", so the assistant filled the gap with a generic rule from elsewhere. The fix was one sentence in the returns entry ("hygiene items can't be returned once used or unsealed"). The bent grab rail needed "bent" and "broken" added to the damage trigger words. Rerun the failed questions after each fix, and don't go live until all twelve pass.
Days 15 to 21: automate three safe topics, and nothing else
Switch on automatic answers for the topics your edit log shows are reliable: typically order tracking, delivery times and costs, and the returns process. Everything else goes to a person, with the AI's draft attached if that helps.
Three things to set before switching on:
- A clear opening line saying the assistant is automated and a person can take over. If you sell to customers in the EU, chatbot transparency duties under the AI Act make this a requirement rather than a courtesy. Compare a vague greeting, "Hi there! Ask me anything.", with one that sets expectations: "Hi, I'm the shop's automated assistant. I can track orders and answer delivery and returns questions. For anything else, type 'person' and one of the team will reply, usually within four working hours, Monday to Friday."
- An easy way out: a "talk to a person" option that works, with honest response times.
- Handover triggers, written down (below).
Our tutorial on whether a chatbot can handle order tracking and returns covers the order-lookup side in more detail.
Handover rules, filled in
Hand to a person immediately if the customer:
- mentions damage, a fault, injury or a safety worry
- asks for a refund, or has already asked once
- asks about suitability for a health condition
- uses angry language or says they will complain or review
- asks the same question twice in one conversation
- mentions an order over $150
Hand to a person at the end of the day if:
- the AI answered but the customer did not confirm it helpedOrder lookup has two edge cases worth planning for on day 15. Guest checkouts with a mistyped email address can't be matched, so the assistant should ask once for the order number and hand over if that fails too, rather than asking again. And orders that ship in two parcels confuse customers when the first tracking link shows "delivered" while the second part is still on its way. Add a line to the tracking answer ("if your order came in more than one parcel, each has its own tracking link in your dispatch email") and a trigger for "the parcel arrived but something's missing", which goes to a person.
Adapt the thresholds to your shop; the tutorial on when a chatbot should hand over to a human explains the reasoning behind each.
The 15-minute daily check while automation beds in
For the first week of automatic answers, spend a quarter of an hour each morning on three things: read every conversation that ended in a handover (was the trigger right?), read five that did not (was the answer right?), and scan for any question the assistant could not match to the answer bank. Those unmatched questions are next month's answer-bank entries. After a week, the aids shop's list read:
- "Can I leave a note for the delivery driver?" (asked 4 times): new entry, pointing to the order-notes box at checkout.
- "Do you sell replacement suction pads for the bath seat?" (3): new entry with the spare-part link; the shop also added the part to the product page.
- "Can a care home pay by invoice?" (2): kept as a handover, since trade terms are agreed case by case.
An illustrative before and after from this check, on the most common topic. The first automated tracking answer read:
Thank you for reaching out! We understand how important your order is. Orders are typically processed and shipped promptly. Please check your email for updates.
It sounded polite and answered nothing. After a rewrite in the answer bank:
Orders placed before 2pm on a working day are dispatched the same day, and you'll get a tracking link by email when it leaves us. If it's been more than 3 working days since you ordered and you have no tracking email, reply "person" and one of us will check it today.
The second version gives the rule, what the customer should have received, and exactly when to ask for a person. Handovers on tracking fell once it went in, because customers could see whether their order was actually late.
Days 22 to 30: review, fix and decide
Take 30 conversations from the automated topics and read them in full. For each, note: was the answer correct, did it solve the problem, and did the customer need a person anyway? Scoring them in a sheet keeps the review honest. Three illustrative rows:
| Conversation | Correct? | Solved? | Needed a person? | Action |
|---|---|---|---|---|
| Tracking, order 3 days old | Yes | Yes | No | None |
| Delivery cost, order just under $50 | Yes | No: customer wanted to know what to add to qualify | No | Link popular items under $10 in the delivery answer |
| Weekend dispatch | No: said Saturday dispatch | No | Yes | Fix the old cut-off text in the policy page |
A conversation can be correct and still not solved, like the second row; those are where the answer bank gains most. Then compare the month's numbers with the three months before:
| Measure | How to get it | What good looks like at day 30 |
|---|---|---|
| First response time | Inbox or helpdesk report, or a sample | Instant for automated topics; unchanged or better for the rest |
| Resolved without a person | Automated conversations with no handover | Most tracking and delivery questions |
| Wrong answers | From the 30-conversation review | Zero on automated topics; any error triggers a fix |
| Handovers | Count by trigger | Mostly damage, refunds and suitability, as designed |
| Staff time on support | A week's rough log, before and after | Down, with time moved to the hard questions |
From month two, keep the weekly habit going with a spot-check routine for AI support replies, and add one new automated topic at a time, only when its drafts have been sent unchanged for two weeks.
Worked example: the aids shop's month in numbers
For the illustrative aids shop from week one, the month looks like this. It gets about 220 enquiries a month, answered by an office manager alongside other work, taking roughly 12 hours a month.
- Week 1: five hours building the answer bank; two policy gaps found (hygiene returns, weekend dispatch).
- Week 2: 96 AI drafts; 63 sent unchanged, all but four of them on tracking, delivery and returns.
- Week 3: those three topics automated in Shopify Inbox. About 120 of the month's enquiries fall into them.
- Week 4: the 30-conversation review finds one wrong answer (an old delivery cut-off time still in a policy page), fixed the same day.
By the end of the month, support takes about seven hours instead of twelve, and the hours left are spent on fit and suitability questions, which is where the shop's knowledge actually helps people. Cost: nothing extra beyond the Shopify plan, plus ChatGPT Plus at $20 during the answer-bank week.
A quick sum shows when a paid AI agent would make sense instead. If roughly 100 of those 120 automated enquiries were resolved without a person, a helpdesk charging $0.90 per resolution would add about $90 a month on top of its plan, to save around five hours. At this shop's volume, with nearly everything arriving through website chat, the free Inbox tools win. The sum changes when enquiries run into the thousands or arrive across email, chat and social channels, because then the time saved grows faster than the per-resolution bill.
The same month for a florist
A florist's safe topics are different: delivery windows, card-message character limits and care instructions automate well, but anything on the delivery day itself ("it hasn't arrived and the funeral is at two") goes straight to a person, with a phone number in the reply. The answer bank also needs a substitution policy, because "will my bouquet look like the photo?" is its most common question and the honest answer involves seasonal flowers.
A realistic mistake to avoid in week three
A shop automated its delivery answer on day 15 and changed courier on day 20. The answer bank still said "next working day", the new courier took two to three days, and for a week the assistant promised something the shop could not deliver. Nobody had connected "changing courier" with "update the chatbot". The fix is an owner for each answer-bank entry and a line in the checklist for any business change (prices, courier, returns window, opening hours): "Does the AI's answer bank need updating?"
Questions about the first month of AI support
Do I have to tell customers they're talking to AI?
It is good practice everywhere, and if you sell to customers in the EU, the AI Act's transparency duties for chatbots mean people should know they are dealing with an automated system. A short line at the start of the chat is enough, such as saying the assistant is automated and a person can take over. Customers mind being misled far more than they mind automation.
What if the AI gives a customer a wrong answer?
Correct it with the customer quickly and plainly, by a person, and honour anything reasonable the AI promised if the customer relied on it. Then fix the source: usually a missing or outdated entry in the answer bank. Log it in the weekly review so you can see whether the same topic keeps going wrong, which is a sign it should go back to a person.
Can I skip the shadow-mode fortnight if I'm short of time?
You can shorten it, but not skip it. Even five days of AI drafts that a person checks will show you which answers the AI gets wrong for your shop, and those are rarely the ones you would guess. Going straight to automatic replies means customers find those errors for you.
How much does this cost in the first month?
It can be close to nothing. Shopify Inbox is Shopify's own chat tool, and its Shopify Magic suggestions are free, while a ChatGPT Plus subscription for drafting is $20 a month. Helpdesks with AI agents usually charge per resolved conversation on top of a plan, so estimate your monthly volume before choosing one.
Further reads
- Best AI Customer Support Software for Small Teams in 2026 — Compare helpdesks with AI features once you outgrow the basics.
- How to Measure Whether Your AI Chatbot Is Actually Working — A fuller set of measures for month two onwards.
- How to Automate Returns and Refunds With Clear AI Rules — Rules to follow before letting AI touch refunds.
- How to Draft Customer Email Replies With AI That Sound Like You — Draft email replies that still sound like your shop.
- How to Set a Baseline Before You Introduce AI — Record where you are now so the month's results mean something.
- How to Train an AI Chatbot on Your FAQs, Policies, and Prices — Load your policies and prices into a chatbot properly.
- Taking a Local Shop Online With AI: Product Listings in a Weekend — A Friday-to-Sunday plan for getting a shop's first 30-80 products online, with the spreadsheet, prompt and checks that keep AI listings accurate.
- How to Set Up AI Abandoned Cart Emails for a Small Online Store — Turn on the free built-in checkout reminder, add two follow-ups, and let AI write the copy while plain rules decide who is emailed and when.
- Online Shop AI Mistakes That Hurt Trust and Conversions — Eight AI mistakes that quietly raise returns and lower sales in small online shops, each with a real-looking example and the fix.
- Can a Small Online Shop Use AI to Compete With Bigger Brands? — Compete through clearer product choices and dependable service, with a practical AI trial and an honest margin calculation.
- AI Customer Service for Small Businesses: What to Automate First — Rank your customer messages by volume and risk, then automate in order: sorting, drafted replies, proactive messages, FAQ answers, and only then actions.
- Freshdesk vs Zendesk for a Small Support Team Using AI — Freshdesk bills AI per session and Zendesk per verified resolution. A costed three-person team shows which helpdesk fits, and what to ask sales.
- How to Write Reply Templates That Keep AI Replies On-Script — Rebuild your canned responses so AI fills them in without drifting: locked lines, marked slots, never-say lists and a clear rule for when to hand over.
- AI or Outsourced Customer Service: Which Costs Less? — A bike shop's 380 monthly calls and messages, costed with a human answering service, an AI first line and a cheaper third route most owners miss.
- AI Tools and AI Development: The Complete 2026 Guide — the AI hub, including every tutorial in the AI-for-business series.
Sources: Shopify help pages on Shopify Inbox instant answers and Shopify Magic suggested replies; Gorgias AI Agent pricing page (May 2026); site fact sheet for ChatGPT pricing and EU AI Act transparency duties.