Test only the skills the job needs, give every candidate the same task and scoring guide, keep a person responsible for each decision, offer adjustments on request, and check the results two ways: whether pass rates differ between groups (adverse impact, often screened with the four-fifths rule) and whether scores predict how placements go (validity).
A test can look rigorous and still be neither fair nor useful. Marking is the easy part for AI, and it is also where problems hide, because a fluent score feels more objective than it is. The steps below start from the job, build a work-sample test, mark it against a rubric, and then check the numbers before anyone relies on them.
Decide what the job needs before choosing a test
Ask the client for the five tasks a temp will actually do in the first fortnight, and the systems they will use. That list, not a vendor catalogue, decides what to test. A short job analysis is enough: task, how often, what a good result looks like, and what goes wrong when it is done badly. If a skill does not appear in the list, it does not belong on the test, however easy it is to measure.
| Temp role | Task that mirrors the job | Where AI helps | Avoid |
|---|---|---|---|
| Accounts payable | Match 20 invoices to purchase orders with five planted discrepancies | Generating versions and the answer key | A generic personality profile |
| Customer service | Written replies to three mock queries, scored on a rubric | Marking against the rubric | Scoring voice, accent or tone of speech |
| Warehouse | Scenario questions on safety, labelling and pick accuracy; licences checked as documents | Question variants | Timed reaction games, unless speed is a real requirement |
| IT support | Triage five mock tickets in priority order, with reasons | Ticket generation | Trivia about obscure products |
| General admin | Format a document to a house style; sort a mock inbox | Variation | A typing-speed cut-off, unless the job is typing |
| Any role | None | None | "Culture fit" or video-based mood scoring |
A 2024 government guide to responsible AI in recruitment is blunt about the last row and about some personality and game-based tools. It says many psychometric tests and games lack scientific validity, can be inaccessible to neurodivergent candidates or people using assistive technology, and that there is little to no scientific consensus on inferring emotion from faces in video interviews. If a vendor sells a score you cannot connect to a task on your list, ask for the validation study, and treat silence as an answer. Building an AI skills test for applicants covers the construction side for employers generally.
Build a work-sample test and its rubric with AI
Work samples are the easiest tests to defend, because the link to the job is visible. AI saves the dull part: producing versions, answer keys and rubric wording. A prompt for the accounts payable test:
I run a staffing agency placing accounts payable temps. Job analysis from the client: [paste the five tasks and the systems used]. Create a 30-minute work-sample test: (1) a table of 20 invoices and 20 purchase orders with exactly five planted discrepancies of different types (price, quantity, duplicate invoice, wrong supplier, missing purchase order); (2) an answer key listing each discrepancy; (3) three short written scenarios a temp would face, each with a 0 to 4 scoring rubric and sample answers worth 0, 2 and 4. Use fictional suppliers. Nothing may depend on speed, accent, age, gender or cultural references.Expect to fix the output, always. In an illustrative run the table contained six discrepancies, not five, because the model made an arithmetic slip while building a line total; two invoices carried dates after their payment dates; one scenario used a regional term most candidates would not know; and the rubric for the third scenario asked for "professional tone", which is subjective. Count the planted discrepancies yourself, check every total with a calculator, strip regional language, and rewrite subjective criteria as observable ones: "states the next step and a date" instead of "professional tone". Then run the finished test yourself, timing it, before any candidate sees it.
Pilot it on ten temps you already know: five who were placed successfully and five who struggled. If the test cannot separate those two groups, it is not measuring what the job needs, and no amount of rubric polishing will fix that. This pilot also gives you a first pass mark based on evidence, not on a number that sounded right.
The Complete Excel Course with AI Integration
From formulas and pivot tables to Power Query, DAX, VBA and AI with Copilot, ChatGPT and Claude — hands-on from the first lesson.
By Sagnik Bhattacharya and Virginia Thorn, Primetime Emmy Award winner
Buy nowMark against the rubric, and review the borderline band by hand
Objective parts, such as the discrepancy count, are marked by the answer key. Free-text answers can be marked by AI against the rubric, with two safeguards: it marks only what the rubric asks for, and a person reviews anything near the line. A marking prompt:
You are marking a candidate's answer to a work-sample scenario. Score it 0 to 4 using only the rubric below. For each point awarded, quote the words in the answer that earned it. Do not consider spelling, grammar, writing style or anything not in the rubric. If the answer is unclear, give the lower score and write REVIEW.
RUBRIC: 4 = identifies the duplicate, says it must not be paid, states who to tell and by when; 2 = identifies the duplicate but gives no next step; 0 = does not identify it.
ANSWER: [paste]The "do not consider spelling or style" line matters: marking down spelling or grammar penalises dyslexic candidates and people writing in a second language, for a skill the job may not require. A typical output reads "Score 3. Earned: 'this is a duplicate of invoice 4471' (identifies it), 'hold payment' (do not pay), 'I would tell the supplier ledger owner' (who, but no deadline). REVIEW: no." Check that the quoted words are really in the answer, because a model can quote loosely.
Set three rules before the first candidate. First, run a marking check on 30 answers: a person and the AI each mark them, and the two should be within one point on at least 90% of answers; anything two or more points apart goes to a second human. Second, a borderline band: if the pass mark is 70 out of 100, every score from 60 to 75 gets a consultant reading the actual answers, not just the number. Third, nobody is rejected on the score alone; the consultant records a reason code and the decision. A checklist for placements and candidate data sets out the same human step at each stage of a placement.
An illustrative agreement check shows what you are looking for. A person and the AI marked the same 30 answers. Twenty-eight pairs were within one point (93%), so the 90% bar was met, and two pairs were two points apart, which sent both to a second human. In one of them the AI gave 1 and the person gave 3. The candidate had written "I'd flag it to my team lead today", and the rubric asked for "who to tell", naming only the supplier ledger owner. The AI read the rubric literally; the human read the intent. The fix was to the rubric, which now accepts any named role with authority over payments, and to the marks already given, which were re-run. Disagreements are the cheapest way to find ambiguous rubric wording, so keep a list of them.
Check adverse impact with a four-fifths table
Adverse impact means a test passes some groups at a noticeably lower rate than others. The usual first screen compares each group's pass rate with the highest group's rate. The Uniform Guidelines on Employee Selection Procedures say a rate below four-fifths (80%) of the highest group's rate will generally be regarded as evidence of adverse impact. They add that smaller differences can count where they are significant in statistical and practical terms, and that larger differences may not where numbers are small and not statistically significant. It is a screen, not a verdict. Three formulas cover it:
Pass rate for a group: =COUNTIFS(Group,"B",Result,"Pass")/COUNTIFS(Group,"B")
Impact ratio: =PassRate_B/MAX(all pass rates)
Two-proportion z-score: =(p1-p2)/SQRT(p*(1-p)*(1/n1+1/n2)) where p = total passes / total testedWorked example, with illustrative figures. An agency tested 180 accounts payable candidates over six months with a version of the work sample above that also included a timed data-entry block, and a pass mark of 70 out of 100. Group labels stand for any two groups you monitor, such as two age bands.
| Group | Tested | Passed | Pass rate | Impact ratio |
|---|---|---|---|---|
| A | 100 | 62 | 62.0% | 1.00 |
| B | 80 | 36 | 45.0% | 0.73 |
The impact ratio is 45 divided by 62, which is 0.73: below 0.80. The z-score is (0.62 - 0.45) divided by the standard error of 0.075, which gives 2.28, beyond the roughly 1.96 that is conventionally treated as significant at the 5% level. Both signals point the same way, so investigate. Do not drop the test on the spot and do not explain it away. Look at the average score by section. In this example the gap was 3 points on invoice matching (74 against 71), 3 on the written scenarios (70 against 67) and 20 on the timed data-entry block (72 against 52). The timed block was driving the difference.
Now ask the job-analysis question: does the client's role need speed? The client's real task is accuracy, and speed was not on the five-task list. The agency removed the time limit, kept the accuracy scoring, and re-scored the same answers. Group A passed 60 of 100 (60%) and group B 41 of 80 (51.25%), an impact ratio of 0.85 and a z-score of 1.18. The check on validity matters here, because a fairer test that no longer predicts anything is no improvement. The correlation between score and the client's four-week rating of 64 placed temps was 0.37 before and 0.38 after, so nothing was lost. Where two procedures are about equally valid, the Guidelines say to use the one with less adverse impact.
Small groups need caution. A third group, C, has only 12 candidates and 6 of them passed (50%). Against group A's 62% that is a ratio of 0.81. One more pass in group C (7 of 12, 58%) lifts it to 0.94, and one fewer (5 of 12, 42%) drops it to 0.67. A single person moves the verdict from comfortable to alarming. With groups that small, pool six to twelve months, keep the same test version, and treat a flag as a reason to look at the answers, not as a conclusion. Paste only counts into an AI assistant if you want help with the sums; the analysis needs no names. A related routine for CV screening is in bias checks every recruiter should run.
Check validity: do scores predict how the placement goes?
Passing a fairness check says nothing about whether the test is any use. Validity asks whether higher scores go with better outcomes. The simple version for an agency is to collect a rating from the client after four weeks (say 1 to 5, using the same question every time), match each rating to the temp's test score, and calculate the correlation with =CORREL(scores, ratings).
- How big must it be? With 30 rated placements the correlation needs to reach about 0.36 to be statistically distinguishable from chance at the 5% level. With 50 the figure is about 0.28, and with 100 about 0.20. A result below those numbers is not evidence that the test works.
- Remember the range problem. You only see outcomes for people you placed, and placed people scored well. That compresses the range and makes the true correlation look weaker than it is, so a modest figure is not a reason to throw the test out, and a figure near zero is a reason to look harder.
- Check the rater, not just the temp. Client ratings carry their own bias. Use specific questions, such as "turned up on time" and "completed tasks accurately", alongside the overall one.
- Re-check each year. Jobs, systems and candidate pools change. A test that predicted well on last year's accounts payable systems may not on this year's.
Until you have the placements to calculate this, use the ten-temp pilot as an interim check and say honestly in your records that full validation is pending.
Candidate notice and adjustments, in words you can copy
Tell candidates what the exercise is, how it is marked, who decides and how to ask for adjustments, before they begin. Plan adjustments in advance; the government guide cited above says they should be considered and planned before a tool is deployed, and that AI use should be signposted so applicants know they may need adjustments. Wording to adapt:
About this exercise
This 30-minute exercise is a sample of the work in the role you applied for. It is the same for every candidate and is marked against a written scoring guide. AI software helps with marking, and a consultant reviews every result before any decision is made. No one is rejected by software alone.
If you need adjustments, such as extra time, a different format, assistive technology or a quiet space, tell us before you start by replying to this message or phoning [number]. We will arrange them, and asking will not count against you.
We keep your answers for [period]. You can ask to see your result, or to have a person look at it again. [your name]Keep a note of every adjustment request and what was done. It is the best evidence that the process works for people who do not fit the default, and it shows where the test itself needs changing. If three people in a quarter ask for the same thing, such as screen-reader access to one question type, change the question.
An example of an adjustment in practice: a candidate replies that they use a screen reader. The agency sends the scenarios as a plain document instead of a web form, supplies the invoice and purchase order lists as a spreadsheet with column headings, and marks the answers against the same rubric with no change to the pass mark. Nothing about the content of the test changed, so the result stays comparable, which is the standard to hold adjustments to. A candidate who asks for more time gets the same test with the time limit relaxed, and the consultant notes it on the record, not in the score.
Two legal regimes in outline: Title VII and the Equality Act
The table is orientation, not legal advice. Rules differ by place, individual states and cities add their own, and the details are changing, so ask a qualified adviser about wherever you recruit.
| Topic | Title VII, the ADA and the Uniform Guidelines | Equality Act 2010 and data-protection law |
|---|---|---|
| Core test | A selection practice with a disparate impact must be job related for the position and consistent with business necessity (Title VII) | A neutral test that puts people with a protected characteristic at a disadvantage is indirect discrimination unless objectively justified (Equality Act 2010) |
| Agencies covered? | Title VII has a specific provision on employment agencies referring or classifying people | Section 55 covers employment service-providers, and section 56 defines an employment service to include finding work for people and supplying employers with workers, which describes most agencies |
| Numeric screen | Four-fifths rule of thumb in the Uniform Guidelines, with statistical and practical significance and small-number cautions | No numeric threshold in the Act; use the same screen as an internal alarm, not a legal test |
| Disability | Tests must not screen out disabled applicants unless job related and consistent with business necessity, and must be given so results reflect the skill, not the impairment (Americans with Disabilities Act) | Reasonable adjustments are a legal duty (section 20, applied to employment service-providers by section 55(6)); the government guide says to plan them before deployment |
| AI-specific rules | Some cities and states regulate automated hiring tools. One city law covers employers and employment agencies, needs a bias audit within a year of use, a public summary and notice to candidates 10 business days before use | Data-protection law sets rules, including safeguards such as human involvement, for decisions taken solely by automated means that significantly affect people; the government guide tells recruiters to consider whether they apply, and whether a data protection impact assessment is needed |
| Records | Users should keep records showing impact by race, sex and ethnic group, and may sample if the pool is large | Keep monitoring data apart from selection decisions and ask an adviser how to collect it lawfully |
For the city law in the AI-specific row, an explainer for small staffing agencies goes through the audit and notice steps. If you place people with employers in the EU, the AI Act lists recruitment systems as high-risk. The official EU AI Act implementation timeline, updated after a 2026 amendment, now shows those obligations applying from 2 December 2027; the date has moved once, so check it, and give that third regime its own conversation with an adviser.
Where AI skill tests go wrong in practice
- The pass mark chosen by feel. A cut-off of 70 "because it sounded right" has no job behind it. Set it from the ten-temp pilot, then review it after your first 30 placements.
- A vendor black box. If the supplier cannot give a validation report, adverse impact data by group, an accessibility statement and a clear account of what happens to candidate data, the score is a number you cannot defend to a candidate or a client.
- Answers written by AI. Two candidates submit scenario answers with near-identical phrasing. Do not reject on a detector's verdict. Add a two-minute live follow-up question about their answer, which a candidate who wrote it can answer easily, and rotate versions so nobody can share a single answer sheet.
- Test leakage. The same 20 invoices for six months will appear in a forum. Generate three versions with different figures and rotate them; the AI is quick at that, and the answer keys can be checked in minutes.
- An inaccessible item. A drag-and-drop question that a screen-reader user cannot complete will show up as a low score. Try each item with assistive technology before launch, and keep a plain-text alternative.
- Forgetting to re-check. The first check is easy and the fifth never happens. Put a six-month review in the diary with the pass-rate table and the correlation as its two agenda items.
Questions agencies ask about AI skill tests
Can we buy a vendor's personality or cognitive test instead?
You can, if the vendor shows a validation study for roles like yours, adverse impact data by group, an accessibility statement and a clear account of how candidate data is used and kept. Without those you are buying a number you cannot defend. A work sample built from the client's job analysis is usually easier to defend, and a vendor test can be added once the pilot shows it separates strong temps from weak ones.
Should skill test results be shared with the client?
Share what is relevant to the role, such as completed the invoice-matching task with 18 of 20 correct, not a raw score without context, and only if the candidate's notice said results may go to clients. Keep full answers on your own record, and stop clients applying their own cut-off to a score your test was never validated for.
Do we need candidates' protected characteristics to run the check?
An impact check needs group data, which candidates can give voluntarily on a separate monitoring form kept apart from selection decisions. Rules on collecting it differ by place and some limit it, so confirm with an adviser before adding the questions. Without group data you can still check item-level problems, accessibility feedback and complaint patterns.
What do we say when a candidate asks why they did not pass?
Say what the exercise measured and where their answers scored lower against the written guide, in plain language, and offer to have a person read the answers again. Do not forward a raw model explanation you have not checked. Log the request, who reviewed it and what you said, and treat a pattern of similar questions as a signal to review the test.
Further reads
- AI in Recruitment and Hiring: A Bias Test Checklist for Small Agencies — A monthly bias test routine for small agencies.
- Can AI Screen CVs Fairly? What Small Employers Need to Know — What small employers should know about AI and CV screening.
- How to Evaluate the AI Features in Your Recruitment Software — Judge the AI features already in your recruitment software.
- AI Screening: Bolt It Onto Your ATS or Switch Systems? — Bolt screening onto your system or switch platforms.
- AI Staffing Tools for Workforce Planning: Forecast Client Demand — Forecast client demand so you know which roles to assess.
- AI Staffing Tools for Employee Engagement: Keep Temps on Assignment — A check-in cadence for temp assignments: opt-in texts, AI reply sorting into fine, call and urgent, and a same-day route to a consultant.
- AI Tools and AI Development: The Complete 2026 Guide — the AI hub, including every tutorial in the AI-for-business series.
Sources: the Uniform Guidelines on Employee Selection Procedures (29 CFR 1607), the Title VII and Americans with Disabilities Act provisions on employment agencies and tests, a city agency's page on its automated employment decision tool law, a 2024 government guide to responsible AI in recruitment, Equality Act 2010 sections 19, 20, 55 and 56, UK GDPR Article 22C, and the official EU AI Act implementation timeline.