Pick the best AI chatbot for a small hotel website by testing, not by reading rankings. Collect 40 real guest questions from past messages, run every candidate through the same questions during its free trial, and score accuracy, hand-off, languages and booking-engine link. Weight accuracy highest, and reject any chatbot that gets a policy or price wrong.
Vendor rankings tell you what vendors sell. They cannot tell you whether a tool knows your check-in time, your pet policy or the difference between your two room types. Your inbox can. Forty past messages, a one-line correct answer for each and three trap questions make a scorecard you can finish in an afternoon, and one invented amenity is enough to rule a candidate out.
Why a test beats a feature list
Every hotel chatbot page claims the same things: answers guests 24/7, books rooms, speaks many languages. A feature list is a promise. The test is the proof, and it costs only your time. It also protects you from a subtle problem: a chatbot that sounds helpful and gets facts wrong is worse than no chatbot, because guests believe it, and you inherit the dispute.
The test has four parts: collect the questions, write the answer sheet, run each candidate through it, and score. Allow about three hours for a first pass on two or three candidates, plus a short follow-up on the winner.
Step one: build the question set from past messages
Export or copy 40 guest messages from the last six to twelve months, across email, your website form, social messages and, if you keep notes, calls. Remove names and booking details. Then sort them into groups, and make sure the set mirrors your real mix. A 14-room guesthouse might end up with this split:
| Group | Count | Example (illustrative) |
|---|---|---|
| Availability and rates | 8 | "Anything free for the 12th to 14th, two adults?" |
| Policies (cancellation, deposit, children, pets) | 8 | "Can we bring a small dog? Is there a charge?" |
| Arrival (check-in time, parking, directions) | 6 | "We land at 11 pm, can we still check in?" |
| Food (breakfast, dietary, restaurants) | 5 | "Do you do a gluten-free breakfast?" |
| Rooms and facilities | 5 | "Is there a lift? Which rooms have a bath?" |
| Changes and cancellations | 4 | "Can I move my booking by a day?" |
| Complaints and awkward messages | 4 | "The room was freezing and nobody came." |
Include a few messy ones: no punctuation, two questions in one, a message with a typo, and one that is not really a question. Guests write like that, and demos do not.
Add three traps on purpose
Write three questions the chatbot should not be able to answer, so you can see what it does. For example: "Is the pool open?" if you have no pool, "Do you do airport pick-ups?" if you do not, and "Can I get a discount if I book direct?" if you have no such offer. A good chatbot says it does not know or hands off. A bad one invents. Invention is the single most important thing the test is looking for.
Step two: write the answer sheet
For each question, write the correct answer in one line, the way you would reply. This is your marking key and also the content you will give the chatbot. Doing it first has a second benefit: you will find places where your own policies are unclear.
Q: We land at 11 pm, can we still check in?
A: Yes, until 1 am with notice. Message us your arrival time. After 1 am by arrangement only.
Q: Can we bring a small dog?
A: Dogs welcome in two ground-floor rooms, $15 a night, max one dog. Not allowed in the dining room.
Q: Is the pool open?
A: We have no pool. Nearest is not something we can advise on.
(TRAP: chatbot must not invent a pool.)If you cannot write a clear answer for a question, fix the policy first. The chatbot will repeat whatever ambiguity you give it, with confidence.
Using an AI assistant to sort the messages
If your 40 messages come from a messy inbox, a chat assistant can sort them for you. Strip names and booking numbers first, then paste the messages into an assistant on a business plan, or one with training switched off. An illustrative prompt:
Here are 40 anonymised guest messages sent to a small hotel. Group each into one of:
availability/rates, policies, arrival, food, rooms/facilities, changes/cancellations,
complaints. Return a table with the message number, the group, and a one-line
summary of what the guest wants. Flag any message that contains two questions.A typical result is a clean table, with a few mistakes: a question about "late checkout" tagged as arrival, or a pet question tagged as rooms. Fix those by hand, since the grouping decides whether your test set reflects reality. Do not let the assistant write the correct answers; those must come from you.
Severity: what counts as a critical error
Not every wrong answer is equal. Agree the levels before you start marking, so you are not tempted to go easy on a tool you like.
| Level | Meaning | Example | Effect on score |
|---|---|---|---|
| Critical | Wrong fact that changes what the guest does or pays | Wrong check-in time, wrong cancellation window, invented amenity | Caps accuracy; candidate fails until fixed and retested |
| Major | Incomplete or misleading, but not harmful | Says breakfast is included and omits the dietary note | Half marks |
| Minor | Tone or format problem | Too long, or an odd greeting | Full marks; note it |
Example: a complaint, before and after tuning
The message: "The room was freezing and nobody came. This is not what I paid for." Before tuning, the chatbot replies: "I am sorry to hear that! Our rooms have heating controls on the wall. Is there anything else I can help with?" That is a fail: it lectures the guest and does not hand off. After you add a rule ("any complaint, apology and transfer to a person, no advice"), it replies: "I am sorry about that, [first name]. I have passed this to a member of the team, who will reply within the hour. Could you tell me your room number?" Same tool, different instruction. Check that rules like this can be set by you, without a support ticket, and count that as part of the score.
Step three: run every candidate through the same questions
Pick two or three candidates to test. For pricing context, Hyperleap lists plans from $40 a month for 3,000 AI replies, with a 7-day free trial; Tidio lists a free plan with 50 billable conversations a month and 50 lifetime AI conversations; Botpress lists a free plan with 100 conversations a month; Chatbase lists paid plans from $32 a month and no free plan; Intercom's Fin charges $0.99 for each resolved outcome. Those figures date from September and October 2026 and move, so confirm on each page. They are examples of the market, not a ranking.
For each trial, load your answer sheet into the chatbot in whatever way it offers: a document upload, pasted text, or the link to your website. Then ask all 40 questions, one at a time, copying each reply into a spreadsheet. Do not coach it between questions. Use a fresh conversation for each, or a clean session, so earlier answers do not leak into later ones.
Example: a typical reply and how it is marked
Question: "We land at 11 pm, can we still check in?" Chatbot A replies: "Yes! Our reception closes at 6 pm, but late arrivals are welcome. Just let us know your arrival time." Correct answer: until 1 am with notice. The reply is friendly, and wrong about the closing time. It is a policy error, which is critical on this scorecard, because the guest may arrive believing the wrong thing. Chatbot B replies: "Late check-in is available until 1 am if you tell us your arrival time. After 1 am, please message us to arrange." That scores full marks.
Step four: the scorecard
Score each candidate on six criteria. Weighting is yours to set; these weights suit a small hotel where wrong information costs most.
| Criterion | Weight | How to score it from the test |
|---|---|---|
| Accuracy on the 40 questions | 35 | Right answers divided by 40, times 35. Any critical error caps this at 15. |
| Behaviour on the 3 traps | 15 | 5 points per trap that says "I do not know" or hands off |
| Hand-off to a person | 20 | Does it alert a named person, and how fast? Test it live. |
| Languages guests use | 10 | Score a native reader's marks on 5 messages per language |
| Booking-engine link | 15 | Live availability = 15; link with dates prefilled = 10; generic link = 5 |
| Cost against limits | 5 | Is the plan enough for your monthly message count? |
Hand-off and the booking-engine link are covered in more detail below. Cost sits low at 5 because a $40 difference matters far less than a wrong cancellation policy, but it is not zero: a plan that runs out mid-month is its own failure.
Example: two candidates scored
Say Candidate A answers 33 of 40 correctly, makes one critical error (the closing-time reply above), invents a pool, and hands off slowly. Candidate B answers 36 of 40, makes no critical errors, says "I do not know" on all three traps, and alerts staff within a minute.
| Criterion | Candidate A | Candidate B |
|---|---|---|
| Accuracy (35) | 15 (capped by critical error) | 31.5 |
| Traps (15) | 10 | 15 |
| Hand-off (20) | 8 | 18 |
| Languages (10) | 8 | 6 |
| Booking link (15) | 15 | 10 |
| Cost (5) | 5 | 4 |
| Total (100) | 61 | 84.5 |
Candidate A is better on languages and the booking link, and loses because of one wrong policy answer and an invented pool. That is the point of the critical-error cap: the features can be good, and the answer sheet still wins.
Testing hand-off properly
Every chatbot hands off in the demo. Test yours with three messages: an angry complaint, a request to change a booking, and a question the chatbot cannot answer. Watch three things: whether the guest is told a person will reply and when, whether a real person is notified, and where. An email to an inbox nobody reads until Monday is not hand-off.
Check also what the guest sees afterwards. Ideally they leave their email or number, and the transcript goes with the alert so you do not ask them to repeat themselves. Then do the dull but decisive test: send yourself a message at 9 pm, and see who is told. If the answer is "nobody until morning", write that expected delay into the chatbot's own reply, so the guest knows.
Testing the booking-engine link
There are three levels, and the difference matters:
- Live availability and rates in chat: the chatbot connects to your booking engine and shows what is free. Powerful, and the riskiest if rates are stale.
- Link with dates prefilled: the chatbot collects dates and guests, then opens your booking page with those fields filled in. Good balance for most small hotels.
- Generic link: "You can book on our website." Lowest effort, and it loses guests who ask a question and then have to start again.
Hyperleap's pricing page, for instance, separates plans on this point: its Plus plan shares your booking link, and in-chat booking with Calendly or Cal.com arrives on Pro. Whether a given tool talks to your hotel's booking engine is not something a generic page tells you; ask the vendor for the engine's name on their integrations list, and check that it supports your room types and extras. For how the pieces fit, read the tutorial on whether a hotel chatbot should take bookings or only answer questions.
Example: a stale-rate mistake
A chatbot reads a rate from a pasted sheet that was right in March. In June a guest asks the price for the weekend and is quoted $95. The real rate is $120. The guest books direct at the lower figure, and you honour it. Prevention: never paste rates into the chatbot's knowledge. Tell it "do not quote prices; send to the booking page" or connect it to live rates. Add a test question to your set: "How much for the 20th to 22nd?" and check what happens.
Languages: test with a reader, not a hunch
If a fifth of your guests write in another language, add five questions in that language to the set, with a fluent reader to mark them. Check five things: it understands the question, replies in the same language, gets dates and times right, keeps the policy facts the same as in English, and hands off when unsure. A chatbot that gives a different cancellation rule in another language is a serious fault; the multilingual support tutorial explains the translation checks.
If you have no fluent reader, give the check to a friendly guest or a colleague, or leave that language off the chatbot and use a fallback message in it. Better a plain "please email us" than confident mistakes.
Privacy, data and what you tell guests
A guest chat may contain names, dates of stay, phone numbers and, sometimes, health or family details ("my mother uses a wheelchair"). Before choosing, find out whether the vendor uses chats to train models, how long transcripts are kept, whether you can delete them, and where the data is held. Tell guests they are talking to an automated assistant, and give them the route to a person. If you have guests in the EU, ask for the vendor's data-protection terms.
The same caution applies if you build a chatbot yourself from a general assistant. The custom ChatGPT embed tutorial covers the options, now that Custom GPTs are being retired, and the scorecard here works on those too.
Counting your real message volume against plan limits
Plans are metered in conversations, replies or resolutions. Count your own: say 160 chatbot conversations a month with an average of six AI replies each is 960 replies, well inside a 3,000-reply plan, while a per-resolution price of $0.99 on 100 resolved conversations comes to $99. A free tier with 50 conversations would not last a week of a busy season. For a detailed look at fees, the hotel chatbot cost tutorial goes line by line. Remember also that some metered products count a conversation differently when a guest returns later.
Re-running the test when the vendor changes something
Chatbot products change monthly: new models, new features, new limits. A reply that passed in March may fail in September because a model behind it was swapped. Keep the question set and the answer sheet in a shared spreadsheet, and rerun the full 40 after any announced change, after any edit to your policies, and in any case once a quarter. It takes thirty minutes the second time. If you ever switch vendors, the same test lets you compare the new one fairly against the old.
Making the decision, and checking it after launch
Pick the highest scorer with no critical errors, then run it live for two weeks in a limited way: on one page, or only outside reception hours. Each morning, read the transcripts for wrong answers and unanswered questions, and fix the sheet. After two weeks, repeat the 40-question test on the live version. If the score has dropped, something changed, either in your policies or in how the tool reads them.
Then review monthly: number of conversations, number handed off, number of wrong answers found, and the number of direct bookings that began in chat. If direct bookings are the goal, how independent hotels win direct bookings with AI shows where a chatbot fits among the other steps. A scorecard that took three hours to build will keep saving you time every time a vendor updates its product and you need to test it again.
Questions hotel owners ask before choosing a chatbot
How many past guest messages do I need for the test?
Forty is enough for a small hotel, if they cover your real mix: availability, policies, directions, breakfast, parking, pets and complaints. Fewer than 25 hides weak spots. More than 60 takes longer to score than it adds in confidence.
Is a free chatbot good enough for a small hotel?
It can be, for a trial or a very quiet property. Free tiers cap conversations or reduce features, and some free agents are deleted after inactivity. Run your test on the free tier, and price the paid plan before you rely on it for guests.
Should the chatbot take bookings itself?
Only if it connects to your booking engine and shows live availability and rates. Otherwise it should hand the guest to your booking page with dates filled in, which is safer than a chat quoting a stale price.
What should I do about guests who write in another language?
Test it in each language with a fluent reader and score the answers as you would in English. A chatbot may reply in a language it handles poorly, so decide what it does when unsure: switch to a short fallback message and offer a person.
Further reads
- How to Test a Customer Chatbot Before It Goes Live — The general pre-launch test, beyond hotels.
- AI Concierge for a Small Hotel: Local Tips Without Wrong Answers — Local tips from a concierge bot without wrong answers.
- Hotel AI Guest Messaging Tools Compared for Independent Properties — Guest messaging tools that go beyond a website chat.
- How Small Hotels Use AI to Answer Guest Messages Around the Clock — Round-the-clock replies across email and messaging.
- AI Receptionist for Hotels: Overnight Phone Cover vs a Night Porter — Phone cover at night, priced against a night porter.
- AI Visibility for Hotels: Get Recommended in ChatGPT Answers — Free checks and a fix list for a small hotel: what ChatGPT and Google AI say about you today, where they get it, and what you can and cannot change.
- AI Agents for Hotels: What They Can Do Today and Where They Fail — What a hotel AI agent can handle now, where it fails on bookings, guest messages and housekeeping, and the tests and hand-off rules to set first.
- AI Chatbot for Vacation Rentals: What It Should Not Answer — A three-question test, a sorted list of guest messages, hand-off wording and twelve test messages for a rental chatbot that knows its limits.
- Travel Industry AI Chatbot: What a Tour Operator Website Should Answer — What a tour operator's website chatbot should answer, look up and hand to staff: meeting points, cancellation wording, availability, refunds and safety.
- AI Tools and AI Development: The Complete 2026 Guide — the AI hub, including every tutorial in the AI-for-business series.
Sources: Hyperleap, Tidio and Botpress pricing pages, Chatbase and Intercom Fin pricing pages (checked September and October 2026). Hotel size, message counts and scores are illustrative.