Shopify Inbox AI Agent: Test It on Your Policy Questions First

Test the Shopify Inbox AI agent by running 30 to 40 questions from your own policies and past chats through your store's chat in a short, watched window, scoring each answer right, partly right or wrong, and fixing the source content. Only let it answer customers when the wrong answers on money, delivery and returns are zero.

Shopify's help pages say the agent answers from your product catalogue, store policies, pages and any Knowledge Base content you add, and that you are responsible for the accuracy of what it tells customers. So the real test is not whether the agent is clever. It is whether your own policy text is clear enough that a machine reading it cannot get it wrong.

Follow me on Instagram@sagnikteaches

What the agent is, and what Shopify says about it

Inbox is Shopify's chat tool for the online store. The Inbox agent is an opt-in AI feature inside it that chats with shoppers about products, shipping, returns, sizing and policies. From the Shopify help pages, as fetched in November 2026:

Connect on LinkedInSagnik Bhattacharya

• Plans: Basic, Grow, Advanced or Shopify Plus, with new customer accounts and an active online store. Stores on legacy customer accounts keep conversations with staff until they upgrade. The Inbox app listing shows Inbox as free; confirm the current terms for your store.
• Where it works: only in your online store's chat widget.
• Sources: your product catalogue, store policies, pages and Knowledge Base content. Web search results are used only as a secondary source for product-adjacent content, and are not presented as your store's own information.
• Order lookups: need the customer to be signed in.
• Handoff: complex or sensitive issues go to staff, or customers are given your sender email if nobody is available.
• Notice: customers always see that the chat is AI-powered, and you cannot remove or change that notice.
• Assignment: in the admin, go to Sales channels, then Inbox; the Inbox agent section turns the agent on or off. Options are All conversations, Only when your staff is unavailable (the agent works only outside the availability hours you set), or Never. Never is the default for stores that installed Inbox before the agent was available.
• Language: the agent replies in the language the customer writes in, but product titles and other catalogue details may still show in your store's language.
• Testing: the help pages describe no separate preview chat. The Training area has Practice scenarios, generated conversations in which you pick the better of two agent replies to shape its voice, but there is no documented way to type your own questions there. The persona page says you can test the persona before you start using the agent without saying how, so look on your Inbox page for a test option first; if there is none, use the watched window below.

Every item on that list is a thing to test. The web-search point is the one most owners miss: the agent may say something about a product that is not on your pages, because it found a review somewhere. That might be fine, or it might contradict your own description, so your test should include a few questions that tempt it.

Subscribe on YouTube@sagnikteaches

Set up a short, watched test window

You want to test the agent without a live customer receiving a wrong answer, and with the assignment on Never the agent answers nobody, including you. Shopify documents no sandbox for this, so the live test has to be short and supervised. Work in this order.

1. Read the current Assignment setting. Write down what it is, because you will return to it, and leave it on Never while you prepare.
2. Finish the content the agent reads. Policies pages (shipping, returns, privacy), product descriptions with sizes and materials, and the Knowledge Base facts and files. Do this before testing, or you will test an empty cupboard.
3. Run the Practice scenarios first. Under Training, each generated scenario shows two candidate replies. They will not cover your 40 questions, but a reply that invents a policy here tells you which content to fix before the live test.
4. Pick a quiet hour. Look at when your chats usually arrive and choose the emptiest hour of the week. For that hour only, switch the assignment to All conversations, or set your availability hours so the hour falls outside them and choose Only when your staff is unavailable. Any real shopper who opens the chat in that hour also reaches the agent, which is why the window stays short. If 40 questions will not fit, split them across two quiet sessions.
5. Test in a private browser window, as a visitor, using the chat widget on your own store. Use a second window signed in to a customer account that has a real order, to test order lookups.
6. Watch the Inbox while you test. Keep the Inbox app open and tell staff which conversations are yours. If a real customer's chat appears during the window, read it at once and follow up yourself, after the handoff or by email, if the agent got anything wrong. When the hour ends, set the assignment back to what you wrote down.

The persona settings sit on the same Inbox page, with two kinds of guidance: Tone and personality, and Service style. Shopify's persona page says facts belong in your catalogue, policies, pages or Knowledge Base rather than in the persona, because facts written only into the persona might not be used consistently. So use the persona for voice, and treat voice as the last thing to polish. Accuracy comes first.

Build the test script from real questions

A good test set is 30 to 40 questions, drawn from what customers actually ask. Pull them from four places: your last two months of chat and email, your returns inbox, your reviews, and the questions your staff say they get every day. An assistant can sort and de-duplicate that pile, but feed it real text, not a request to "make up typical questions".

Below are 120 customer messages from my store's last two months.
Group them into topics (shipping, returns, sizing, order status, payment,
product care, discounts, other). For each topic, pick the 5 most common
distinct questions and rewrite each as a short, neutral question a customer
might type. Keep the original wording where it is clear.
Do not invent questions that are not represented in the messages.
Remove personal details such as names, emails and order numbers.

[paste messages]

The final line matters. Strip customer details before pasting anywhere, and check your policies for what you may do with customer messages. From the grouped list, build your script of 30 to 40 questions across the groups below.

Group 1: policy questions with a clear written answer (12 questions)

Examples, using an illustrative store:

• "How long do you take to deliver?"
• "Can I return something I've opened?"
• "Who pays for return postage?"
• "Do you ship to [a country you do not serve]?"
• "Can I change my order after I've paid?"
• "What payment methods do you take?"

For each, write the correct answer from your policy next to the question. This is your answer key.

Group 2: edge cases where the policy has a condition (8 questions)

These are the ones that go wrong. "My item arrived damaged four weeks ago and I only just opened the parcel. Can I return it?" "I bought in the sale. Does the 30-day return apply?" "Can I get a refund for the postage as well?" The answer key should state the condition and what a person must decide.

Group 3: product questions (8 questions)

Size, fit, material, care, compatibility. Pick products with thin descriptions and a few with thorough ones. Include "Is this good quality?" and "What do reviews say?" to see what the agent does with web search results. Shopify says web search is secondary and limited to product-adjacent content, so check that what comes back is consistent with your own product pages.

Group 4: things it must not answer (6 questions)

"I want a refund for $120 right now, my card was charged twice." "Your product burnt my skin." "I'm going to dispute this with my bank." "Can you give me a discount code?" "Who is the owner?" The correct behaviour here is handoff to a person, not an attempt to resolve. Score a polite handoff as right, any promise of money as wrong.

Group 5: tricks and drift (4 to 6 questions)

Questions in a mixed or non-English language, a question with a typo, one that mixes two topics, one asking for something unrelated ("recommend a good restaurant"), and one that tries to get the agent to say your policy is different. Also include any language your store serves, in that language.

Score each answer, then look at the totals

Use a simple sheet with these columns: number, group, question, expected answer, what the agent said (paste), score, source of the error, fix.

Set a go-live rule in advance, before you see any results, so you do not bargain with yourself afterwards. A reasonable rule: no 0 in groups 1, 2 or 4, at least 90 per cent of group 1 scored 2, and every group 4 question handed to a person. Do not accept an average across all questions, because one wrong refund promise outweighs ten correct size answers.

An illustrative result for a small homewares store: of 38 questions, 29 scored 2, 6 scored 1, 2 scored 0, 1 was a handoff. The two zeros were "Do you ship to [a country]?", which the agent answered yes because a product page mentioned international delivery while the shipping policy listed exclusions, and "Can I return a candle I've lit?", which it answered yes because the returns page said "unused" without defining it. Both were content errors, not agent errors, which is the usual pattern.

Fix the content, not the agent

The fastest improvement comes from changing the pages the agent reads. For each 0 or 1, find its source of the error.

• Policy text is ambiguous. "Unused" means unopened? Unlit? Write the definition and the exception. Short, plain sentences work better than legal-sounding ones.
• Two pages disagree. A product page and a policy page say different things. Pick one truth and update the other. This is the single most common cause.
• The answer is missing. The agent guessed because there was nothing to read. Add it to the Knowledge Base as a fact: one question, one answer, in your words.
• The agent took a web source. If an answer cited outside content that conflicts with your page, tighten your product text, and consider whether you want the agent giving opinions on reviews at all. Test again.
• Persona or tone issue. The agent was too eager, too apologetic, or promised things. Adjust the persona instructions in the Inbox settings, and test again.

Retest the failed questions only after each fix, then the full script once at the end. Keep the sheet; it becomes your regression test whenever you change a policy. The wider idea of writing the FAQ before you launch is covered in building the FAQ your AI chatbot needs before launch.

Two exchanges, scored

Seeing a scored exchange makes the sheet concrete. Both below are invented and illustrative, using a made-up homewares store whose returns policy allows 30 days for unused items.

An answer that scores 0

Customer: I lit the candle once and it smells different from the photo. Can I return it?
Agent: Yes, you can return it within 30 days for a full refund. Please start a return from your account.

The policy says "unused", and a lit candle is not unused. The agent read the 30-day window and ignored the condition. Score 0, because it promised a refund the store may not owe. The fix is in the policy text: "Candles and other consumables can only be returned unopened and unlit. If an item is faulty, contact us and we will look at it." After the edit, run the question again and look for an answer that states the condition and offers a route for faults.

An answer that scores 2

Customer: I lit the candle once and it smells different from the photo. Can I return it?
Agent: Thanks for telling us. Our returns policy covers unused items, so a candle that has been lit can't be returned for a change of mind. If you think it's faulty, tell me what you noticed and I'll pass it to the team to look at.

It states the rule, does not promise anything, and offers a path to a person. The tone could be warmer, but a persona setting handles tone. Accuracy is what you scored.

Knowledge Base facts: before and after

The agent also draws on facts and files you add. Short, single-purpose facts work better than a long document, because each can be checked on its own. Compare:

Before: "We aim to dispatch orders quickly and delivery times vary."
After:  "Orders paid before 14:00 on a weekday are dispatched the same day.
         Standard delivery takes 3 to 5 working days after dispatch.
         We do not give delivery dates for specific orders; tracking shows the latest scan."

The "after" version has a number, a cut-off time and a rule about what the agent must not promise. Write the facts that answer your group 1 questions in this style, then retest. As an illustration of effort, a store with 15 policy questions might write 15 facts of about 40 words each, which is roughly an hour of work.

A quick sum: is it worth the effort?

Illustrative numbers: a store receives 60 chats a week and a person spends about four minutes on each, 240 minutes or four hours. If the agent resolves half of them correctly, that frees two hours a week. Against that, a monthly read of ten conversations and a quarterly test costs about 30 minutes a week averaged out. The saving is real but not large, so the more important benefit is cover outside staff hours, where the alternative is an unanswered chat. The risk side is bigger than the time saving: one wrong promise about a refund can cost more than a month of saved minutes. That is why the test comes first.

Locale and language checks

If your store serves more than one language or currency, repeat a slice of the script in each. Shopify documents one locale behaviour worth testing: the agent replies in the language the customer uses, but product titles and other catalogue details may still appear in your store's language. I found no verified reports of other locale bugs, so test rather than assume either way. Things to look at in each locale:

• Does the agent reply in the language the customer used?
• Are prices, currency symbols and dates shown in the format your store uses for that market?
• Do shipping and returns answers match the translated policy page, not just the original one?
• Are product names or sizes mis-translated in a way that changes meaning, such as a size chart converted wrongly?
• When it hands off to a person, does the message make sense, and does a person who reads that language exist?

Test five or six questions per language, drawn from the groups above. If a language fails badly, fix the translated policy and product text first. The assignment page describes no per-language setting, so you cannot switch the agent off for one language alone; until that language passes, keep the agent on Only when your staff is unavailable, or on Never.

Turning it on gradually

Once the score meets your rule, do not switch to All conversations on day one. Shopify's assignment options give you a ladder:

1. Week 1: Only when your staff is unavailable. The agent covers the hours when nobody would answer anyway, so it can only improve on silence. Read every conversation each morning.
2. Week 2 to 3: keep that setting, count handoffs and complaints, and add Knowledge Base facts for every question you see twice.
3. Week 4 onward: consider All conversations if the first weeks produced no wrong answers on money, delivery or returns. Keep reading a sample each week.

This mirrors the shadow approach described in piloting AI in shadow mode before customers see it: let the AI act where the downside is smallest, then widen. When something does go wrong with a customer, what to do when AI gets something wrong gives the steps for the reply and the repair.

Reading real conversations after launch

Shopify's help pages say you can review completed conversations on the Inbox page in your admin and reply from the Inbox desktop and mobile apps. Under Training, Real conversations lets you rate each agent reply with a thumbs up or down; the ratings feed persona suggestions but do not change what the customer already received. Block fifteen minutes each morning for the first month. Read ten conversations: three handled by the agent, three handed off, and four chosen at random. For each, ask: was the answer right, was the tone right, and was the handoff timely? Log wrong answers in the same sheet as the test, with the date. Anything wrong on a policy goes to the fix list the same day, and the full test is rerun once a month.

An assistant can summarise a week of conversations, if you paste them with names removed: "list the questions the agent answered, the ones it handed off, and any answer that contradicts the policy text below". Do not rely on that summary alone for compliance. Read the actual conversations that it flags.

Where a person must stay in the loop

Keep these with staff regardless of how well the test goes: refund decisions above a threshold, any claim of injury or harm, anything involving legal threats, chargebacks and disputes, custom orders and wholesale pricing, and any customer who says they are upset. Write a one-line rule for each in your handoff notes, and test that the agent hands them off. Remember also that you cannot remove the AI notice that customers see, so do not design answers that pretend a person wrote them.

The cost side is mostly your time: a first test of 40 questions takes about two hours to build and two hours to run and score, plus whatever it takes to fix the content. Repeat runs take under an hour. For an overview of what else adding AI to a store costs, see the cost of adding AI to a Shopify store, and for how the agent compares with a paid option, Shopify Inbox against Tidio Lyro sets out the differences.

Questions about the Inbox agent before you switch it on

Does the Inbox agent cost anything?

Shopify's app listing shows Inbox as free, and the help pages list the agent as available on Basic, Grow, Advanced and Plus plans. Pricing and terms change, so read the current listing and your plan page before you rely on it.

Can I test it without customers seeing it?

Not completely. Shopify's help pages describe no separate preview chat, and on Never the agent answers nobody, including you. Use the Practice scenarios under Training first, then run your script in the store chat during a short, quiet window with the agent switched on, watch the Inbox throughout, and switch it back afterwards.

Why does it say it cannot see my order?

Order lookups need the customer to be signed in, and the agent does not work with legacy customer accounts. If your store uses legacy accounts, the agent will not take over chats until you move to new customer accounts.

Will it use information from the internet?

Shopify says web search is a secondary source, used only for product-adjacent content such as reviews of a catalogue product, and not presented as your store's information. Test for it by asking about reviews and sizing, and judge whether you are comfortable with the answers.

How often should I retest?

Retest whenever you change a policy, a shipping rate, a product range or a theme, and at least once a quarter. Keep the same 40 questions so you can compare scores over time.

Further reads

• Best AI Chatbots for Small Shopify Stores (2026) — Other chatbots worth a trial on a small store.
• Ecommerce AI Chatbot Examples: 7 Store Conversations to Copy — Store conversations to copy into your tests.
• How to Train an AI Chatbot on Your FAQs, Policies, and Prices — Train a chatbot on FAQs, policies and prices.
• Can an AI Chatbot Handle Order Tracking and Returns? — Whether a chatbot can handle tracking and returns.
• Dropshipping Returns and Refunds: Policy Wording and AI Replies — Returns wording that a bot can quote without error.
• Click and Collect for a Small Shop: Setup, Messages and Rules With AI — Switch on pickup in Shopify or Square, then write the confirmation, ready, reminder and no-show messages and a collection rule customers accept.
• Shopify Dropshipping Automation: What AI Runs and What Breaks — Which dropshipping jobs to automate on Shopify, which to keep manual, and the daily and weekly checks that catch silent failures.
• Shopify Sidekick Skills: Worked Recipes for Repeat Store Tasks — Five copy-ready Sidekick skills for the jobs you repeat every week, with the instructions, sample output and what to check.
• Applications of AI in E-Commerce: 9 Uses Mapped to a Small Online Store — Nine uses of AI for a small store, ranked by weekly hours saved and matched to Sidekick, Shopify Inbox, Klaviyo and Gorgias, with plan costs.
• Does Shopify Have AI? What Is Built In and What Costs Extra — Which Shopify AI features come with Basic, Grow, Advanced and Plus, at what price, and where the extra costs begin.
• llms.txt for Shopify: Test Whether AI Bots Fetch It and Send Traffic — How to run a before-and-after test of llms.txt on your own Shopify store, what Shopify serves by default, and the product data that matters more.
• Dropshipping Customer Service With AI: Can It Handle Slow Delivery? — How to let AI answer slow-delivery messages with honest windows, tracking facts, refund triggers and a clear point where a person takes over.
• AI Tools and AI Development: The Complete 2026 Guide — the AI hub, including every tutorial in the AI-for-business series.

Sources: Shopify Help Centre pages for the Inbox agent, assigning it to conversations, its persona, training it and managing conversations (read 5 November 2026), and the Shopify Inbox app listing; the test questions, scores and store details are illustrative.