A small law firm should run the trial for two weeks with three realistic tasks, score every output on accuracy, confidentiality, supervision and cost, and set pass marks before starting. Week one uses dummy documents and tests confidentiality first; week two tests real workflow. Any confidentiality failure ends the trial, whatever the other scores say.
Buying after a demo means finding the problems after the contract is signed. A two-week trial reverses that: you decide what good looks like, give the tool work it will actually face, and let a lawyer mark the results against an answer key written in advance. The pass marks matter more than the tool's polish, so write them down before anyone logs in.
Set the rules of the trial before anyone logs in
Write one page, signed off by the partner who is accountable, that covers who is in the trial, what material may be used, what may not, and what counts as a pass. Without it, the trial turns into people trying the tool on whatever is on their desk. That is how client documents end up in the wrong plan.
Items for the page:
- The people. One accountable partner, two fee earners who will use it, one person who marks outputs. Three to five in total.
- The material. Week one: dummy or fully anonymised documents only. Week two: real material only if the confidentiality gate has been passed and your professional rules and client terms allow it.
- The plan being tested. The exact plan you would pay for, with its settings recorded. Free plans often differ in data handling.
- The pass marks. Written now, not after seeing the results.
- The stop rule. Any confidentiality gate failure, or any output sent to a client or court without review.
Rules on using AI with client material differ by profession and place, and they change. Check your professional body's current guidance and your insurer's position before the trial, and keep a copy with the trial page. For a view on the confidentiality question in general, can solicitors use ChatGPT without breaching confidentiality is the companion tutorial.
The Complete Excel Course with AI Integration
From formulas and pivot tables to Power Query, DAX, VBA and AI with Copilot, ChatGPT and Claude — hands-on from the first lesson.
By Sagnik Bhattacharya and Virginia Thorn, Primetime Emmy Award winner
Buy nowThe scorecard: four criteria, weights and a gate
Four things matter, and they do not carry equal weight. Confidentiality is a gate rather than a score: a tool either meets your minimum or it does not, and no accuracy makes up for a failure. The other three are scored.
| Criterion | Type | Weight | What you measure |
|---|---|---|---|
| Confidentiality | Pass or fail gate | Gate | Where data goes, whether it trains models, retention, deletion, access controls, sub-processors |
| Accuracy | Scored 0 to 5 per task | 45 | Facts right, citations real, nothing invented, nothing missing that matters |
| Supervision | Scored 0 to 5 | 25 | Can a lawyer see sources, check quickly, log what was done and keep control |
| Cost and time | Scored 0 to 5 | 30 | Price per user, time saved minus time spent correcting, admin effort |
Turn the scores into a total out of 100: accuracy (average of the three tasks, divided by 5, times 45), plus supervision (divided by 5, times 25), plus cost and time (divided by 5, times 30). A reasonable pass mark is 70, with no accuracy task below 3. Those numbers are suggestions; the point is to choose yours before you see the results.
Illustrative total: accuracy tasks score 4, 3 and 2, average 3, giving 27 of 45. Supervision scores 4, giving 20 of 25. Cost and time scores 3, giving 18 of 30. Total 65. That misses a pass mark of 70, and it would also fail the rule that no accuracy task falls below 3, because the research task scored 2. The result is "not yet", with a clear reason.
The confidentiality gate, item by item
Run this gate in the first two days, before any realistic material goes near the tool. For each item, find the answer in the vendor's own terms, privacy policy, security page or admin settings, not in a sales conversation, and write down where you found it.
- Training use. Does the vendor use your inputs or outputs to train models on the plan you will buy? What is the setting, and who can change it?
- Retention. How long are prompts, documents and outputs kept? Can you delete them, and does deletion remove back-ups?
- Who can see it. Can vendor staff read your content? Under what conditions?
- Sub-processors. Which other companies handle the data, such as the model provider and cloud host? Are they listed?
- Location. Where is data stored and processed? Does that fit your client contracts?
- Access control. Single sign-on or two-factor authentication, user roles, the ability to remove a leaver at once.
- Data processing terms. Does the vendor offer a data processing agreement, and does it say what you need it to say?
- Breach notice. How and when will they tell you?
- Client position. Do your engagement terms or your clients' own requirements restrict the use of outside tools? Do any clients need to be told?
If any item has no clear answer, treat it as a fail until the vendor puts the answer in writing. A vendor that cannot say whether its tool trains on your data has told you something. Keep a screenshot or a saved copy of each page you relied on, dated, because terms change.
Do not trust a slide that says "secure" or "enterprise-grade". Those are descriptions, not terms.
Three sample tasks that match what a small firm does
Pick tasks the firm does every week, so that the trial tests the work and not a demonstration. Three that cover the range, each with a way to mark it:
Task 1: summarise a contract or lease
Take a dummy 12-page commercial lease, written for the trial or fully anonymised, with a known set of terms: rent, term, break date, repairing obligations, rent review, assignment restrictions and a deposit clause. Ask the tool for a one-page summary for a client who is not a lawyer, and for a list of the dates and money terms.
Mark it against an answer key you wrote before the trial. Score 5 if every key term is right and nothing is invented; deduct one point for each missed key term, two for each wrong term and three for anything invented. A realistic failure: the summary gets the rent right but states a break clause that is not in the lease, because it assumed the market norm. That is a 2, however polished the text looks.
Task 2: draft a client update letter from file notes
Give the tool short, anonymised file notes: a stage reached, a delay, a next step and a cost estimate. Ask for a letter to the client in plain English, under 250 words, in the firm's tone, using an example letter you supply.
Mark on four points: facts match the notes, tone matches, nothing is promised that the notes do not support, and the length is right. The usual faults are a promised completion date that is not in the notes, and a reassurance such as "this is straightforward" that nobody wrote. Record the minutes a fee earner spends fixing the draft, because that goes into the cost score.
An illustrative paragraph from a draft shows both faults. The file notes said the other side had replied, disputed two invoice items, and that the firm's response was due by a set date. The draft read: "Good news: the other side has replied and the matter is straightforward, so we expect it to be resolved by the end of the month. Our costs remain within the estimate of [amount]." Fixed, it reads: "The other side has replied and disputes two of the invoice items. We will send our response by [date] and update you then. Our costs remain within the estimate of [amount]." That draft fails two of the four marking points, the facts and the promise, and the seven minutes spent fixing it go into the cost score.
Task 3: research a point and show its sources
Ask a question the firm can answer itself, such as a settled point in a field it practises, and ask the tool to give the rule and its sources. Then check every citation in a legal database or primary source.
This is the task that most often fails. Assistants can produce a case name, a court and a year that look right and do not exist. Score 5 only if every cited source is real, says what the tool claims, and is correctly cited. Score 0 if any source is invented, and note it. For the routine to check this, see how to stop AI inventing case law.
Run each task with five different documents or questions, so that you have 15 outputs in all. Give each a score of 0 to 5 and a note. Ask two people to mark a sample of the same outputs independently, and compare, because a tool marked by one friendly colleague looks better than it is.
A two-week timetable that fits around billable work
A trial that needs a free fortnight will never happen. This timetable assumes about three hours a week from each of four people.
- Days 1 and 2. Write the trial page. Run the confidentiality gate. Prepare the dummy documents and the answer keys.
- Days 3 to 5. Run task 1 on five lease summaries and task 2 on five letters, using dummy material. Time every correction.
- Days 6 and 7. Run task 3 on five research questions. Check every citation.
- Days 8 to 10. If the gate is passed and rules allow, repeat one task on real, low-risk material with the supervising lawyer reading every output. If not, repeat with more dummy material.
- Days 11 and 12. Mark everything. Tally scores, minutes and costs.
- Days 13 and 14. The partner reads the results and decides: yes, no or not yet, with reasons.
Keep a one-line log each day: who used it, for what, what went wrong. Small things, such as formatting that breaks when pasted into a Word template, add up to a reason for a no.
A filled-in marking sheet for one output
A blank scorecard invites vague marking, so here is one completed line, as an illustration, for a lease summary from task 1.
Output 3 of 5, lease summary. Key terms in the answer key: 9. Correct in the summary: 6. Missed: the rent review date and the restriction on assigning the lease. Invented: none. Wrong: the deposit stated as three months, lease says two. Score: 5, less 2 for the missed terms, less 2 for the wrong term, giving 1 of 5. Time to correct: 18 minutes. Reviewer note: reads well, which makes the deposit error easy to miss. Source references: none shown, so every figure had to be checked against the lease by hand.
Two points from that sheet carry over. A fluent output deserves more suspicion rather than less, and a tool that shows no source references pushes the checking cost onto the lawyer. Both belong in the supervision score. A summary with page references would have turned an 18-minute check into five.
The prompt makes a difference too, so test two versions of the same task and score both. Before: "Summarise this lease." After: "Summarise this lease for a client who is not a lawyer, in under 300 words. List rent, term, break dates, repairing obligations, rent review, assignment restrictions and deposit. For each, quote the clause number. If a term is not in the lease, write 'not stated' rather than assuming." The second usually scores better, and the difference tells you how much of the result depends on the firm's prompting skill rather than the tool, which matters if the people using it will not write careful prompts.
Supervision: how a lawyer stays in charge
Supervision is where small firms are most exposed, because a tool that is usually right makes people stop checking. The question is not whether the tool can be wrong; it is whether your process catches it when it is.
Score supervision on five things: can the user see where the tool got a fact, such as a quote from the document with a page reference; can the output be traced to an input and a user; does the tool warn about uncertainty or does it state everything with the same confidence; is there a way to record that a lawyer reviewed it; and can the firm turn features off. A tool that shows its sources scores higher than one that does not, because it makes a check take 20 seconds rather than ten minutes.
Write the supervision rule in one sentence and test it: "Nothing generated goes to a client, a court or another party until a named lawyer has read it against the source." Then see whether the trial respected it. Ask each participant whether they ever sent something unchecked, and make it a safe question to answer honestly. If the answer is yes, that is a finding about the tool's design or the firm's habits, and either one affects whether you buy.
Cost: the sum that decides it
Price per seat is the least informative cost figure. What counts is price per seat plus the time spent correcting, against the time saved. Use your trial data.
As an illustration, say the vendor quotes $60 per user per month and five fee earners would use it, which is $300 a month, or $3,600 a year. In the trial, task 2 letters took on average 6 minutes to fix against 20 minutes to write from scratch, saving 14 minutes. If each fee earner writes 8 such letters a week, that is 112 minutes a week each, or about 1.9 hours. Across five people and 46 working weeks, that is about 430 hours. At an internal cost of $50 an hour, that is $21,500 against $3,600.
Now the check. Task 1 summaries took 15 minutes to correct against 40 from scratch, but task 3 research took 25 minutes to verify and sometimes produced nothing usable. If research is not a use you will rely on, count only tasks 1 and 2. And if the trial shows that two of five people never used it, count the saving for three people: about 258 hours, or $12,900 a year, still set against $3,600 if you pay for five seats. A realistic result is a lower figure than the headline, which is why you measure rather than assume. For a view of pricing models, per-seat or pay-as-you-go legal AI pricing for small firms compares the structures, and ChatGPT or a legal AI tool helps decide what type of tool to trial.
Also add the costs outside the invoice: setup time, training time, any extra review step and any extra insurance conversation. Some vendors tie advanced features to higher tiers; check which tier you tested.
Naming a tool: what to verify on the vendor page
If your shortlist includes a named product, such as the AI features inside your practice management system, verify the following on the vendor's own pages during the trial, and write down the date you checked. Clio's assistant, for instance, is now called Manage AI (formerly Clio Duo), and features and plans change often, so check the current name, what the plan includes, whether the AI features are in your tier or an add-on, and what the vendor says about data use. Do not rely on a review site or a reseller for this.
Ask for the same things in writing from each vendor on your shortlist, and compare the answers side by side. A one-page table with a row per vendor and a column per gate item takes an hour and reduces most of the argument about which to choose.
Reading the result: yes, no or not yet
Three outcomes, each with a next action. Yes means the gate passed, the score cleared the pass mark and no task fell below its floor. Buy for the same users and plan you tested, write the supervision rule into your office manual, and set a review at three months. No means the gate failed or the scores were well below the mark. Record why, and keep the document for the next time a vendor calls. Not yet means the tool was close but one task failed or one question is open. Name what must change, such as a setting the vendor can alter or a feature on a higher tier, and rerun only that part.
Whatever you decide, keep the trial pack: the one-page rules, the gate answers with dates, the scored outputs and the cost sum. It is what you will show an insurer, a regulator or a client who asks how the firm decided. For a wider plan after a yes, the first 90 days of an AI implementation plan for a small law firm picks up where this leaves off.
If you would rather design the trial with someone, that is what my AI implementation consultation covers; the contact page has the details.
Questions law firms ask before a trial
Can we trial an AI app on real client matters?
Not at first. Use dummy or fully anonymised documents in the first week. Move to real matters only once the confidentiality gate is passed, the client-consent position is settled and the supervising lawyer has agreed the rules in writing. Check your professional body's guidance before using client material.
How many people should take part in the trial?
Three to five is enough: one partner who will be accountable, one or two fee earners who will use it, and one person who will mark results. A trial run by one enthusiast tends to measure enthusiasm rather than the tool.
What score should stop us buying?
Any fail on the confidentiality gate stops the purchase, whatever the other scores. For the rest, set a threshold before you start, such as 70 out of 100 overall and no task scoring below 3 out of 5 on accuracy.
Is a free plan good enough for the trial?
A free consumer plan is useful for learning but is rarely the one you would buy. Trial the plan you would actually pay for, because retention, training and admin settings often differ between free, individual and business plans. Read the plan's data terms before uploading anything.
What if the tool is accurate but our staff dislike it?
Record that as a finding, not a failure. Low adoption is a real cost. Ask users what slowed them down, such as extra clicks or poor formatting, then weigh it against the time saved. A tool nobody opens saves no money.
Further reads
- How Small Law Firms Use AI to Draft Letters and Routine Documents — Use the trial's drafting task on routine letters.
- AI Client Intake for Law Firms: Qualify Enquiries Out of Hours — Qualify enquiries out of hours, a second use case to test.
- AI Time Capture for Solicitors: Recover Hours You Never Billed — Recover unbilled time; another task to put through the scorecard.
- AI Readiness Checklist for Accountants, Solicitors, Consultants — Check the firm is ready before any trial starts.
- What an AI Consultant Does for a Law Firm, and What It Costs — What outside help with a trial costs a law firm.
- AI Contract Review Software for Lawyers: Spellbook vs Gavel Exec — Compare Spellbook, Gavel Exec and Copilot in Word on pricing, playbooks and data handling, then test all three on one planted-error contract.
- AI Demand Letters for Personal Injury: EvenUp, Supio or ChatGPT? — Test EvenUp, Supio and a general chatbot on one closed case, then verify every special damages figure against the records before you sign.
- AI Paralegal Software: What a Small Firm Should Test Before Paying — Three kinds of AI paralegal software, a six-test trial run on your closed matters, and the questions to ask when pricing is quote-only.
- AI Software for Estate Planning Attorneys: Automation or Generative? — Rules-based document assembly against generative drafting for wills and trusts: how each fails, what each costs and the review step that suits each.
- Best AI Tools for Attorneys: Seven Picks for Research, Drafting and Intake — Seven AI tools for a small law practice, mapped job by job, with a worked example and the checks to run before you pay.
- Best AI for Australian Lawyers: Tools, Privilege and Practice Rules — Legal AI against general assistants for an Australian firm, with a privilege test, a court-by-court map of AI rules and a supervision routine.
- Best AI for Immigration Lawyers: Intake to Filing, Stage by Stage — A nine-stage map of an immigration case with the AI tool type, the data check and the manual step for each, plus a costed worked example.
- Claude or ChatGPT for a Law Firm: Which One to Pay For — Team seat costs, long-document limits, privacy defaults and Claude for Legal plugins compared, with a weighted three-task trial a small firm can run.
- Clio Manage AI Pricing: What It Costs and How to Switch It On — Manage AI (formerly Clio Duo) is bundled into Core and above. What Clio publishes, what needs a quote, and the admin steps to switch it on.
- Clio vs Smokeball: Which AI Assistant Suits a Small Firm? — Eight tasks to run on both Clio Manage AI and Smokeball's Archie, a sample scorecard, and the cost caveats that feature lists skip.
- AI Tools and AI Development: The Complete 2026 Guide — the AI hub, including every tutorial in the AI-for-business series.
Sources: Clio product pages for the Manage AI name (formerly Clio Duo), checked September 2026; vendor terms must be read directly for any tool you trial. No legal advice is given here.