A small business can use AI at three points in recruiting: write a scorecard first, draft the job ad from it, then have AI score masked applications against the same scorecard. A person reads every application and makes every decision, and you run a simple bias check on who gets shortlisted.
The order matters more than the tool. Most owners start by pasting a stack of CVs into a chat window and asking "who is best?", which gives a confident ranking built on criteria nobody wrote down. Write the criteria first and everything after it gets easier to defend, to repeat and to audit when somebody asks why a candidate was not called.
Why the scorecard has to exist before the advert
A scorecard is a one-page list of what the person must do well, each item weighted, with a plain description of what a strong, acceptable and weak answer looks like. It becomes the brief for the advert, the marking guide for the first sift and the structure for interviews. If you only write it after seeing applications, you will unconsciously build it around whoever impressed you first.
Keep it to five or six criteria. More than that and the weights blur. Separate the things the person must have on day one (a licence, a qualification the role legally needs) from things they could learn in three months. Learnable skills belong in the scorecard with a low weight, not in a "must have" filter that silently removes good people.
A filled-in scorecard for the main example
Take a law firm with four fee earners that needs a legal secretary and case administrator. Illustrative scorecard, with weights adding to 100:
| Criterion | Weight | Strong looks like | Weak looks like |
|---|---|---|---|
| Accurate document handling | 25 | Describes a checking routine for dates, names and amounts | Says "attention to detail" with no example |
| Diary and deadline control | 20 | Has run a multi-fee-earner diary and chased missed items | Only managed their own calendar |
| Client communication | 20 | Examples of calm, clear written updates to worried clients | No client-facing work in any role |
| Practice software | 15 | Used any case-management system; names what they did in it | None, and no sign of learning new tools |
| Confidentiality habits | 10 | Gives a specific example of handling sensitive files | Generic statement only |
| Availability and notice period | 10 | Can start inside eight weeks | Over six months |
Notice what is missing: years of experience, a particular degree, "native-level" anything, a named previous employer. Those are proxies, and proxies are where bias gets in. Ask what the person can show they have done, not how long or where.
The Complete Excel Course with AI Integration
From formulas and pivot tables to Power Query, DAX, VBA and AI with Copilot, ChatGPT and Claude — hands-on from the first lesson.
By Sagnik Bhattacharya and Virginia Thorn, Primetime Emmy Award winner
Buy nowDrafting the advert from the scorecard
Give the AI the scorecard, not a one-line request. It will turn each criterion into a duty or a requirement, which keeps the advert honest about what the job is. Then ask it to flag wording that narrows the pool.
You are helping a four-fee-earner law firm write a job advert for a legal secretary and case administrator.
Use only the scorecard below. Write: a 2-sentence summary of the role, 6 duties, 5 requirements
(split into "needed from day one" and "we will train"), what we offer, and how to apply.
Rules: no gendered words, no "young and energetic", no "native speaker", no years-of-experience
requirement unless I gave one, no "ninja/rockstar" language. Salary range: [range]. Hours: [hours].
After the advert, list any phrase that could put off a qualified applicant and suggest a plain alternative.
SCORECARD:
[paste table]A realistic first output reads well but contains traps. Illustrative sample line from the draft: "We are looking for a dynamic, proactive self-starter with 3+ years in a busy legal environment." Three problems: "dynamic" is vague and age-coded, "self-starter" tells nobody what to do, and the three-years requirement was never in the scorecard. The fixed line: "You will manage diaries for four fee earners and chase anything that slips. Experience of a busy office helps, but we care more that you can describe how you check your own work."
Two more checks before it goes live. Read the requirements against the scorecard and delete anything that is not there. Then look at the salary line: stating a range in the advert lets more people self-select accurately, and it saves both sides a call that was going nowhere. For a wider guide to wording, see how to write job adverts with ChatGPT without biased language.
Choosing what applicants send you
Ask for what the scorecard needs, and no more. A CV plus three short written answers is usually enough: "Describe a time you caught an error before it reached a client", "How do you keep a deadline list when five people ask for things at once?", "What would you want to know in your first week?". The answers are the best material for the AI to score because they are tied directly to criteria, whereas CVs differ wildly in layout and length.
Use a form or a shared mailbox so you hold the data in one place. Put a short notice on the form saying that applications are reviewed with the help of software, including AI tools, and that a person makes the decision. That wording protects the candidate and you.
Masking before anything touches an AI tool
Candidate data is personal data. Before pasting, remove what the scorecard does not need and what invites bias: name, photo, address, date of birth, graduation years that reveal age, nationality, marital details, and the names of schools if you are not testing for a specific qualification. Replace the name with a number such as C-017. Keep a private key sheet that links numbers to people; the AI never sees it.
Use a business plan where your content is not used for training by default (ChatGPT Business, Claude Team, Gemini in Workspace and Microsoft 365 Copilot all work that way, per their vendor pages; Microsoft's documentation now calls the last one Microsoft Copilot), or switch off training on an individual plan. ChatGPT Business and Claude Team both need two seats, about $40 to $50 a month in all, so a one-person hirer may do better on an individual plan with training switched off and with masked text only. For a fuller routine, see how to anonymise data before you paste it into AI.
Before and after masking
Illustrative extract from a CV. Before: "[first name] [surname], [home address], born [year], [university] [graduation year], legal secretary at [firm] 2010 to 2016, maintained diaries for 6 solicitors." After: "C-017. Legal secretary 2010 to 2016: maintained diaries for 6 solicitors." The AI loses nothing it needs and gains nothing it should not use. Your job-title history and what the person did stay in; the rest goes.
A structured first pass: scoring each application the same way
Ask for one fixed format for every candidate, quoting evidence from the text for every score. Evidence quotes are your safeguard: you can check each one in seconds, and any score without a quote gets ignored.
Score this masked application against the scorecard. For each criterion give: score 0-5,
a direct quote from the application that supports it (or "no evidence"), and one sentence of reasoning.
Do not guess at anything not written. Do not use age, gender, ethnicity, nationality, family status,
disability, accent or the prestige of any school or employer as a factor.
Finish with: total out of 100 using the weights, and one question I should ask this person.
Return a table. Do not recommend hire or reject.
SCORECARD: [paste]
APPLICATION C-017: [paste masked CV and answers]Illustrative output for one candidate, trimmed: "Accurate document handling: 4. Quote: 'I check every date against the court notice before filing.' Reasoning: names a specific routine. Diary and deadline control: 5. Quote: 'maintained diaries for 6 solicitors.' Practice software: 2. Quote: no evidence of case-management software; mentions Word only. Total 78. Question: what did you do when two deadlines collided?" You would read the whole application before agreeing with a 2 on software, because the candidate may have written it under another word.
What goes wrong in the first pass, and how to spot it
Four failures appear again and again. Each is detectable if you look.
- Keyword worship. The AI rewards the scorecard's own words. A candidate who writes "case management system" scores higher than one who names a product you have not heard of. Spot it by reading three low-scoring applications that mention tools.
- Gap penalties nobody asked for. Models sometimes treat a career gap as a negative even when told not to. Search the reasoning column for "gap", "break" and "inconsistent".
- Drift between runs. Score the same masked application twice in separate chats. A swing of more than 8 or 10 points means your scorecard descriptions are too loose.
- Invented evidence. A quote that is not in the application. Search for the quoted phrase in the source. If the quote is missing, discard that row and rerun.
Run a calibration step before real scoring: take three applications you have already read and scored yourself, let the AI score them, and compare. If you and the AI differ by more than about ten points on any of them, fix the scorecard wording and try again. The aim is not that AI agrees with you; it is that you understand where and why it differs.
Cutting from sixty-four applications to six
Say the law firm gets 64 applications. Illustrative time: reading and scoring by hand at 10 minutes each is about 10 hours and 40 minutes spread over several evenings, and the last twenty get less attention than the first. With masking (about 2 minutes each, 2 hours in all) and an AI first pass (about 4 minutes of your checking each, around 4 hours), you are at roughly 6 hours, with every candidate scored against identical criteria. The saving is real but moderate; the bigger gain is consistency.
The cut itself: sort by total, then read every application in the top 15 and every application scoring within 6 points below your cut line, and a random sample of five from below that. You shortlist six. Reading the near-misses and a random sample is how you find the AI's blind spots. In this example the random sample surfaces one candidate who scored 58 because her experience was in a hospital records office rather than a law firm; her checking routine was the best in the pile. She goes to the shortlist.
Running a bias check on your own shortlist
You do not need a statistician. After the shortlist, compare outcomes for groups where you hold information legitimately, for example if applicants voluntarily completed an equal-opportunities form kept separate from the application. Count how many in each group applied and how many were shortlisted, then divide to get a selection rate. A long-standing guideline from employment-selection rules, the four-fifths rule, treats a group's selection rate below 80% of the highest group's rate as a sign of possible adverse impact, so that is your cue to look closer. The same guideline says smaller gaps can still matter, so 80% is a trigger for a second look, not a pass mark.
Illustrative figures: 40 applicants in group A, 12 shortlisted (30%); 24 applicants in group B, 4 shortlisted (about 17%). 16.7 divided by 30 is about 0.56, well under 0.8. With numbers this small it may be chance, but it is a reason to read group B's low-scoring applications again and ask which criterion cost them marks. With fewer than ten in a group the figures say almost nothing; use judgement and read the applications.
Other cheap checks to add:
- Swap test: change a masked candidate's details to a different placeholder, or swap a school's name, and rescore. A different score means the tool is reacting to something it should not.
- Criteria audit: for every scorecard line, ask whether it measures the job or a lookalike (a degree standing in for ability).
- Decision log: keep one line per candidate saying who decided and why, in your own words.
The fuller routine is in the bias checks every recruiter should run on AI screening.
Interviews use the same scorecard with new evidence
Shortlisted candidates should be asked the same core questions in the same order, with the scorecard criteria as the marking guide. Ask the AI to turn each criterion into two behavioural questions and a short scoring guide, then edit them. The guide to building interview questions and scorecards with AI goes further. If you use an AI note-taker, ask each candidate for consent first and tell them where the recording is stored. Never let a tool score a candidate's voice, face or "energy"; it has no place in a fair process.
After each interview, mark the scorecard yourself before you look at any AI summary, so the summary does not anchor you. Then compare notes with a second interviewer if you have one. The aim is two independent sets of marks that you reconcile in conversation.
A short work sample beats a longer CV review
For a role like this one, a twenty-minute task tells you more than a third read of a CV. Give each shortlisted candidate the same small exercise: a messy client email and a diary extract with two clashing deadlines, and ask them to write a two-line reply and say which item they would move. Mark it on the scorecard criteria it touches (document accuracy, deadline control, client communication). Use AI only to tidy your marking notes, not to mark the exercise. Candidates who suspect a machine is grading them often write differently, and you want their normal judgement.
Illustrative sum: six shortlisted, twenty minutes each to sit the task while you supervise and ten minutes for you to mark it is three hours of your time in all. It replaces an hour-long first interview for anyone who clearly falls short, which saves roughly three further hours and treats everyone identically.
Rejection messages that stay honest and kind
Drafting the messages is the easiest win. Ask for a short, kind rejection for candidates who were not shortlisted, and a longer one with a specific line of feedback for those who interviewed. Illustrative output to keep: "Thank you for applying and for the time you put into your answers. We have decided to progress with candidates whose experience of running multi-person diaries was closer to this role. We will keep your details for six months unless you ask us not to." Check the retention sentence matches what you actually do. Do not promise feedback you will not give.
An evidence trail you can show a candidate
Keep five things for each vacancy: the signed-off scorecard, the final advert, the masked scoring outputs, your reasons for who went forward, and the bias check. Store them with the key sheet, access restricted, and delete on the schedule in your recruitment privacy notice. If someone later asks why they were not shortlisted, you can answer with their scorecard marks rather than a shrug.
If you hire in the EU, note that the EU AI Act lists AI systems used to place targeted job adverts, analyse and filter applications or evaluate candidates as high-risk, and some other places require a bias audit of automated hiring tools or advance notice to candidates. That covers the job-board side too: if a platform decides who sees your advert, ask it what the targeting is based on. Check the current rules and timing for where you recruit with a qualified adviser. A small firm using AI only to summarise and sort, with a person deciding, is on safer ground than one that auto-rejects.
Where this process stops fitting
This process is built for roughly 20 to 200 applications. For a very senior hire, a handful of referrals or a role needing a licence check, do it by hand. If you recruit often, the next step is an applicant-tracking system with its own screening features; see how to evaluate the AI features in recruitment software, and setting up an AI-assisted hiring process for a small team for the operating routine. If you are not sure the role needs a hire at all, settle that question before you write the scorecard.
One last example of a realistic failure. A four-person electrical contractor asked AI to rank 30 applicants for an estimator job and told it to prefer "stability". The tool leaned heavily on tenure, so a candidate with three short contracts, each ended when the employer closed, ranked near the bottom. Reading the reasoning column caught it, the criterion was rewritten as "completed estimates for projects of a similar size", and the candidate reached interview. The tool did what it was asked; the scorecard asked the wrong thing.
Questions hiring owners ask before they paste a CV into AI
Can I paste candidates' CVs into ChatGPT or Claude?
Only on a plan where your content is not used for training, and only after you have removed names, contact details, photos and dates of birth. CVs are personal data. Tell applicants in your recruitment privacy notice that AI-assisted tools help you review applications, and keep the originals in your own system.
Should AI reject candidates automatically?
No. Let it sort and summarise, then have a person read every application, including the ones the AI scored low. Automatic rejection with no human review is the setup most likely to cause unfair outcomes, and in some places it is regulated or restricted.
How many applications is too few to bother with AI?
Under about fifteen, reading them yourself against the scorecard is quicker. AI starts paying for itself from roughly forty applications, when the first pass takes an hour or more by hand and tiredness makes your own scoring drift.
What if the AI scores a strong candidate low?
Treat it as a prompt to check your scorecard, not the candidate. Read the application yourself, find which criterion cost them marks, and ask whether it is a genuine requirement. Often the scorecard rewarded a keyword the job does not need.
Further reads
- AI CV Screening for Recruitment Agencies: Setup and Safeguards — How agencies set up CV screening and its safeguards.
- How to Test Job Candidates' AI Skills in an Interview — Check candidates' own AI skills fairly.
- AI in Recruitment and Hiring: A Bias Test Checklist for Small Agencies — A longer bias test checklist from the agency side.
- AI Bias in Small Business Decisions: Hiring, Pricing and Credit — How bias creeps into hiring, pricing and credit.
- How to Use AI in Hiring for a Small Shop: Ads, Questions, Scorecards — The same idea for a shop floor hire.
- AI or a New Hire? How to Decide Before You Recruit — Check whether the work needs a hire at all.
- Plumber Job Description: Write the Ad and a Practical Test With AI — Write the plumber ad from the real working week, then add a short practical test scored by rubric, with AI drafting and a qualified plumber checking.
- AI Boolean Search Generator for Recruiters: Prompts and Checks — How to brief an AI for a Boolean search string, what a hiring manager should review, and how to test the string on three sites and log false positives.
- AI Recruitment Platform for Employer Branding: What to Look For — What a small employer should check in an AI recruitment platform: careers page, job-ad consistency, review replies, AI switches and price shape.
- Caregiver Recruiting Strategies: AI for Ads, Screening and Follow-Up — Draft caregiver job ads, build fair screening questions and re-engage lapsed applicants with AI, with a human making every hiring decision.
- AI Tools and AI Development: The Complete 2026 Guide — the AI hub, including every tutorial in the AI-for-business series.
Sources: OpenAI ChatGPT Business pricing page and Claude pricing page (seat minimums, no training by default); Google Workspace generative AI privacy hub; Microsoft Copilot data, privacy and security documentation; EU AI Act Annex III, point 4; the four-fifths rule in the Uniform Guidelines on Employee Selection Procedures.