Bookkeeping AI Reviews: How to Test a Tool on Your Own Client Files

Take 100 to 200 transactions from a closed period for one client who has agreed, with categories you have already reviewed. Run them through the tool blind, then tally exact matches, harmless differences, near-misses and costly errors. Weigh errors by cost, not count, and repeat on a messier second client before trusting any online review.

A first pass takes half a day, and the result is worth more than any star rating, because accuracy depends on your clients' payees, charts of accounts and habits. Get permission before any real data leaves your system. All the tallies and timings that follow are invented to show the method.

Follow me on Instagram@sagnikteaches

What reviews can tell you and what they cannot

Reviews are good evidence on support response, billing disputes, outages and whether you can get your data out. They are weak evidence on accuracy, for three reasons. The reviewer's clients are not yours, so a tool that handles a retailer's card feed well may stumble on a tradesperson's mixed personal and business spending. Reviewers rarely define "accurate" the same way you would. And reviews date quickly, because vendors change features monthly and some stay in beta for a long time. Intuit says its Accounting AI is on the Essentials, Plus and Advanced plans but not Simple Start, and Xero's automatic reconciliation carried a beta label in its August 2026 update but shows none on its pricing page now, so a review from last spring may describe a different product.

Connect on LinkedInSagnik Bhattacharya

Use reviews as a screen for the first and last of those risks. Read the one-star and three-star entries for patterns: repeated billing surprises, slow support in busy periods, trouble exporting. Then use your own test for accuracy. The two sources answer different questions, and mixing them up is how a firm ends up choosing on stars and discovering the error rate in March.

Subscribe on YouTube@sagnikteaches

Get permission and protect the data first

Real client transactions are confidential. Uploading them to a vendor's system, or pasting them into a chatbot, is a new use of that data, so treat it that way. Three steps keep you on solid ground.

1. Ask each client. Name the purpose (testing software to speed up their bookkeeping), the data (a stated date range of transactions), the vendor, and how long you will keep the test copy. A short email with a yes in reply is a start.
2. Strip what you can. Remove account numbers and personal names, and replace payees that identify a private person with a code. You rarely need them to judge categorisation.
3. Read the vendor's terms on training, retention and sub-processors. Business plans of ChatGPT and Claude do not train on your content by default, while consumer plans have a model-training switch you must turn off yourself. For a dedicated bookkeeping tool, find the data-processing terms and ask the same question in writing.

Here is wording you can adapt for the permission email. Have a qualified adviser check it against your own engagement letter.

Hello [first name],

I am testing software that categorises bank transactions automatically, to see whether it could speed up your bookkeeping and cut errors. I would like to use [number] of your transactions from [month to month] with account numbers and personal names removed. The copy will be deleted within [number] days of the test ending, and nothing will be used to train anyone's AI models. Please reply yes if that is fine, or tell me if you would rather I did not.

Thank you,
[your name]

Build a 150-transaction set that can embarrass the tool

A random month will flatter any tool, because most bank lines are the same twelve suppliers. Build the set on purpose. Pick a closed period whose books you have already reviewed and signed off, so you have an answer key you trust. Then stack the set with the lines that cause real rework.

Keep your own categories hidden in a separate column or file so you do not nudge the tool, and so the comparison afterwards is honest. If the tool learns from history, run one client with years of history and one brand-new client with none. The cold-start result tells you what week one looks like. The history result tells you what month six might look like.

Use different kinds of client where you can. Illustratively, a café's bank feed is full of daily card settlements and produce suppliers. A bicycle repair shop buys parts from a few wholesalers and pays a lot of small online marketplace charges. A bakery's flour and packaging orders arrive in odd amounts. A dog groomer's mobile-van fuel is half business, half household. Each profile stresses a different group in the table above.

Run it twice: cold, then after you correct it

The first run answers "what does this do with no coaching". The second shows whether it learns. Follow this order and time every step.

1. Import the 150 lines with no rules, no history and no corrections. Do not touch anything. Record the start and end time.
2. Export or screenshot the tool's answers before you change a single line. Your tally uses this snapshot.
3. Compare each line to your answer key and classify it (the next section defines the four classes).
4. Correct 30 of the wrong lines the way you normally would, which teaches rules in most tools.
5. Import a second, comparable batch of 50 lines and see how many of the old mistake types repeat. A tool that repeats the mistakes you just fixed is a different proposition from one that stops.

Note the date, the plan tier and any feature labelled beta or new in a one-line header on the log. Features move in and out of beta, and the version you tested may be the version you pay for in three months. That line makes your result reproducible.

The accuracy tally, filled in

Classify each of the 150 lines into one of four classes. Exact means the right account. Harmless means a different account that lands on the same line of the accounts you will report, so nothing changes. Near-miss means a fix is needed but the amount ends up in the right area. Costly means an error that distorts profit, drawings or a balance if it goes unnoticed, such as a transfer posted as income. Here is an invented tally for one tool.

The headline is 103 of 150 exact, which is 68.7 per cent, and 116 of 150 acceptable if you count harmless differences, or 77.3 per cent. But the headline hides the shape. The tool is excellent on repeat suppliers (38 of 40 exact, none costly) and poor on transfers and owner spending, where 8 of 35 lines were costly. A single accuracy figure would have told you to reject it or accept it wholesale. The grouped table tells you to let it run unattended on repeat suppliers and keep everything else in front of a person.

Turn the tally into minutes and money

Accuracy matters because it sets your review time. Say you categorise by hand at 40 seconds a line, so 150 lines take 100 minutes. If you let the tool answer and then review every line at 15 seconds, that is 37.5 minutes, plus 90 seconds to fix each of the 34 wrong lines (19 near-misses and 15 costly), which is another 51 minutes. The total is 88.5 minutes before setup, only about 11 per cent better than doing it yourself, and reviewing every line wastes the point of the tool.

The grouped result gives a better plan. Let the tool run on repeat suppliers and spot-check 5 seconds a line (40 lines, 3.3 minutes), and do the other 110 lines by hand at 40 seconds (73.3 minutes). The batch takes about 76.7 minutes, a saving of 23.3 minutes or about 23 per cent once setup is done. Over a client with 600 lines a month in the same mix, that is roughly 93 minutes saved. At $50 an hour of your time, that is about $78 a month, so a tool priced at $25 a month for that client breaks even if it saves 30 minutes. A real month usually holds far more repeat-supplier lines than this deliberately hard set, so redo the sum with the client's actual mix: if 420 of their 600 lines are repeat suppliers, the unattended group saves 35 seconds on each, about 245 minutes or $204 a month at the same rate. Every figure here is an illustration, so rerun it with your own rates.

One more number matters more than the percentage: did the tool know when it was unsure? If it flags lines for review, count how many of the 34 wrong lines it flagged. Say it flagged 20, which is 58.8 per cent, and the 14 unflagged errors included 6 costly ones. Those 6 silent costly errors, 4 per cent of the set, are what would reach a client's accounts unseen. A tool that flags 90 per cent of its own errors is safer than a more accurate one that flags none.

Keep an exception log that explains the misses

The tally says how often. The log says why, and it is what you take back to the vendor. One row per wrong line, with the payee anonymised.

Two patterns jump out of six lines: transfers and credits that carry the wrong sign or meaning, and merchants whose right answer depends on information the bank feed does not hold. Those become your questions. Can I define my own-account transfer rules? What does the tool do with credits? Can I attach a receipt so that a split is possible? Vendors answer specific questions with a yes or no, and a vague demo answer is itself a result.

Sort the log with an assistant, without the client's data

With a larger log, ask a general assistant to group it. Paste only the anonymised columns, use a business plan whose data is not used for training, and never include account numbers or names.

Below are anonymised exceptions from a test of an AI bookkeeping tool. Group them by likely cause, count each group, and suggest one question per group I should ask the vendor. Do not invent causes; say "unclear" where the data does not show one.

17 | transfer to savings | Subscriptions | Transfer | Costly
42 | online marketplace | Office supplies | Packaging | Near-miss
58 | supermarket | Staff welfare | Cost of sales | Near-miss
77 | supplier credit | Sales income | Purchases reduction | Costly
96 | fuel | Motor 100% | Motor 60% | Near-miss
121 | lunch counter | Meals | Personal | Near-miss

An illustrative reply grouped the lines into three causes: merchants that mix purposes (42, 58, 121), items that need a rule the tool lacked (96) and direction or type errors (17, 77). Its questions were sensible: "Can transfers between the firm's own accounts be defined as a rule?" and "How are credits and refunds classified?" It also claimed the set showed an overall accuracy of "about 85 per cent", which the data cannot support, since six rows are only the exceptions. I would delete that line and keep the groups. Always check an assistant's counts against your own spreadsheet.

Run two or three tools through the identical set

The test is most useful when it picks between tools. Use the same 150 lines, in the same order, cold, for each one, and write the results in one sheet. Add columns for what accuracy does not capture: whether you can undo a bulk change, whether there is an audit trail of who or what changed a line, whether rules are editable, whether the export is clean, and how long the run took. The table below is invented, to show a decision that a single accuracy figure would get wrong.

Tool C looks best on exact matches and is quickest, and it is the one I would not let near a client's books unattended: nine of its fourteen costly errors went through silently, and it keeps no trail of changes. Tool B is the least accurate and the safest, with a single silent costly error. The choice depends on how you will use it. If a person reviews everything, C's speed wins. If you want parts of the work to run on their own, B's honesty about its own mistakes wins. The accuracy column alone would have told you the opposite.

Check the dollar effect on the client's profit

Counting errors treats a $2,000 mistake like a $2 mistake. Rebuild the profit and loss for the test period from the tool's answers and compare it with the version you signed off. Say the signed-off net profit was $41,260 and the tool-based figure came out at $38,950. The difference is $2,310, or 5.6 per cent of profit. The single transfer on line 17 explains $2,000 of it, which is 87 per cent. That is the line to fix first, with an own-account transfer rule, before you worry about the supermarket lines that moved a few dollars between cost headings.

Do the same for the balance sheet: check that bank, loan and drawings balances match the signed-off versions. Errors often cancel in profit and still leave a bad balance. If the totals match but the detail does not, the tool is still giving you rework even though the report looks right.

Set pass marks before you look at the results

Decide what "good enough" means before the numbers arrive, because afterwards it is easy to argue yourself into any answer. These are my rules of thumb, not an industry standard, so adjust them to your risk appetite.

• Unattended group: a transaction group may run without review only if the tool got at least 95 per cent exact on at least 30 lines of that group and made no costly errors.
• Always-reviewed group: any group with a costly error stays with a person until the cause is fixed and re-tested.
• Flagging: at least 80 per cent of the tool's wrong lines should be flagged for review, or there is no safe way to sample.
• Cold start: note how many corrections it took before the second batch improved. If the answer is "hundreds", that is a cost.
• Re-test: run the set again after any major vendor update and each quarter in the first year.

In live use, spot-check 20 lines per client each week for the first month, and suspend a group the moment a costly error appears in it. That one rule is what keeps an early success from turning into a month-end surprise. The wider practice questions, such as how to brief the team and when to tell clients, are covered in the rollout guide for a bookkeeping practice.

A test that looked good for the wrong reason

A realistic mistake: a bookkeeper tests on a bakery client's quiet January, when 80 per cent of the lines are the same flour, packaging and utility suppliers. The tool scores 94 per cent and she signs up. In December the same client has dozens of one-off event suppliers, extra marketplace purchases and a loan refinance, and the tool's accuracy drops sharply. It showed up as bank reconciliation differences on the 3rd of the month. The test set should always include a busy period and the odd lines, which is why the table of groups above is built the way it is.

Where this test sits in the wider choice

Your tally answers one question: would this tool cope with my clients? It does not tell you which platform to build a practice around. For that, read how a practice chooses an AI bookkeeping platform, and for the major accounting products compare them in the QuickBooks, Xero and Sage agents comparison. If your main use is receipts and card feeds, the setup in AI receipt capture and bank categorisation shows what to configure first. And before you put any client data in any tool, run through the questions to ask before buying AI that touches client data.

Keep the sheet. Next year it is the baseline against which the next version of the tool, or its competitor, is measured, and that is worth more than a review written by someone with different clients.

Testing AI bookkeeping tools: practical questions

How many transactions do I need for a fair test?

About 150 is enough to see patterns without taking a whole day, as long as the set is deliberately mixed: easy repeat vendors, mixed-use merchants, transfers, refunds, split purchases and personal spending. A random 150 from a quiet month will flatter the tool because most lines are easy. Repeat on a second client before you decide.

Do I need client permission to test an AI tool?

Usually yes, or at least a clear basis for it. Real client transactions are confidential, and uploading them to a vendor's system is a new use. Ask each client in writing, anonymise payee and account details where you can, and check the vendor's terms on training and retention. If your engagement letter is silent, ask a qualified adviser what to add.

What error rate is acceptable?

There is no standard figure, so set your own before testing. I would weigh errors by cost: harmless differences count for nothing, near-misses count once, and costly errors such as a transfer treated as income count five times. Then decide which transaction groups the tool may handle unattended and which always go to a person.

Should I trust a vendor's accuracy percentage?

Treat it as a marketing figure until you have measured your own. A vendor's number depends on its sample, how it defines a correct answer and which clients it included. Your tally, on your clients and your chart of accounts, is the only number that predicts your review time.

Further reads

• Can AI Do My Bookkeeping? What Still Needs an Accountant — What still needs an accountant once the tool is running.
• AI Bookkeeping Software vs a Human Bookkeeper: Accuracy and Cost — Accuracy and cost side by side with human work.
• Xero vs QuickBooks AI: Which Saves More Bookkeeping Time? — Two platforms worth running your test set through.
• How Much Does AI Bookkeeping Software Cost per Month? — Put a price next to your tally once you have one.
• AI Bookkeeping for Accounting Firms: Pricing It as a Service Line — Turn time saved into a priced service line.
• Is AI Worth It for a One-Person Bookkeeping Business? — A sole practitioner's view on whether it pays.
• Free ChatGPT for Accountants: When Client Work Needs a Paid Plan — What free ChatGPT gives an accountant, why ads and training defaults matter for client data, and the rule for moving client work to a paid business plan.
• Claude vs ChatGPT for Bookkeeping: A Test You Can Run on Bank Data — A 26-line bank export with a known answer key, planted duplicates and a refund trap, plus a scoring card to compare Claude and ChatGPT yourself.
• Dext Pricing for Accountants: Essentials, Advanced and AI Assist — Dext practice pricing per client, how its five-client blocks work, and a payback grid for the $7 AI Assist add-on.
• Dext vs AutoEntry for a Small Practice: Costs, Accuracy Claims and How to Check Them — Run a 30-document accuracy test on Dext and AutoEntry, then compare cost per client with cost per credit at your own volumes.
• Double (Formerly Keeper): AI Bookkeeping Features and Cost per Client — What Double charges per client on Core, Plus and Scale, which AI features each tier adds, and a 40-client monthly bill worked through line by line.
• Free AI Tools for Accounting: Which Jobs Free Plans Handle — Job by job: what free ChatGPT, Claude, Gemini, Copilot Chat and notetaker plans do for an accounting practice, and where client data or accuracy demands more.
• AI Bookkeeping Agents: What They Do Alone and What You Must Review — A task-by-task review checklist for AI bookkeeping agents: what each may post alone, what must queue for a person, and how big a sample to check.
• AI Bookkeeping in Canada: GST/HST Coding and Remittance Checks — A bookkeeper's review routine for GST/HST coding and payroll remittances: formulas for the maths, AI for the judgement calls, CRA pages for the dates.
• AI Bookkeeping in the UK: What Works With MTD and VAT — Four MTD and VAT tests for any AI bookkeeping tool, applied to Xero, QuickBooks, Sage and Dext, plus a test pack that catches silent coding errors.
• Catch-Up Bookkeeping With AI: Rescue Months of Untouched Books — A bookkeeper's month-by-month recovery method, with exception lists, client queries, reconciliation checks and a scoped clean-up quote.
• AI Tools and AI Development: The Complete 2026 Guide — the AI hub, including every tutorial in the AI-for-business series.

Sources: Intuit and Xero help and product pages as recorded in the series fact base (checked September to October 2026), Xero's pricing page (read 1 November 2026); ChatGPT and Claude plan privacy defaults from the same base; no vendor accuracy claims are quoted.