Questions to Ask a Recruitment Agency About Its AI Screening

Ask which AI tools touch candidates at each stage, whether any tool rejects applicants before a person reads their application, how the agency tests those tools for bias, what candidates are told, and who handles a challenge if someone disputes a decision. Get the answers in writing and add them to your terms of business before you sign.

The questions to ask a recruitment agency as an employer used to be about fees, guarantees and time to hire. Those still matter, but the agency's screening software now decides which applicants you ever hear about. A filter that quietly rejects good people costs you hires you never knew were possible, and in some places it can create legal exposure for you as well as for the agency. You can't outsource the responsibility completely, so you need to understand the process well enough to trust it.

Follow me on Instagram@sagnikteaches

The running example is a dog-grooming salon with three grooming tables and a busy front desk, hiring two qualified groomers through an agency because its own adverts had produced few suitable applicants. The salon's owner knew grooming inside out and knew almost nothing about recruitment software, which is the usual position for a small employer.

Connect on LinkedInSagnik Bhattacharya

Why an employer can't leave AI screening entirely to the agency

Three reasons, in rising order of seriousness. First, missed candidates: every applicant rejected by software is invisible to you, so you can't judge whether the filter is too strict. Second, reputation: to the candidate, a curt automated rejection comes from your business as much as the agency's. Third, regulation, which varies by where you hire, so treat what follows as background and ask an employment adviser about your own position.

Subscribe on YouTube@codingliquids

• The EU AI Act lists as high-risk any "AI systems intended to be used for the recruitment or selection of natural persons, in particular to place targeted job advertisements, to analyse and filter job applications, and to evaluate candidates". Most of the obligations for stand-alone high-risk systems like these have been deferred to 2 December 2027, but the direction is clear, and agencies with EU candidates or clients should already be preparing.
• Emotion recognition is already banned. Since 2 February 2025 the same Act has prohibited AI that infers people's emotions in the workplace, and that includes recruitment. A video-interview tool that claims to read enthusiasm or honesty from faces is a serious red flag if you hire in the EU.
• Data-protection law such as the GDPR restricts decisions based solely on automated processing that significantly affect people, and turning someone down for a job can fall into that category. Candidates also have rights to know how their data is used.
• Some local laws go further, requiring independent bias audits of automated hiring tools, advance notice to candidates, or consent before AI analyses a video interview. Your adviser can tell you whether any apply where you recruit.

The practical upshot is the same everywhere: you want a person, not a score, making every rejection decision that matters, and you want evidence the tools have been checked. For the broader picture of how bias creeps into automated decisions, see AI bias in small business decisions.

Questions about job adverts and sourcing

AI often enters before anyone applies: in writing the advert, deciding who sees it, and searching databases or social profiles for people to approach.

• "Do you use AI to write or target the job advert? Can we approve the final text?" AI-written adverts can carry biased language, such as phrases that put off older applicants or parents. Ask to approve it, and see writing job adverts with AI without biased language for what to look for.
• "Where will the advert run, and does an algorithm decide who sees it?" Targeted advertising on job boards and social platforms can narrow the audience in ways nobody chose deliberately.
• "Do you use AI tools to search for passive candidates? What do they search on?" Tools that rank people by profile keywords can favour those who write polished online profiles over skilled people who don't.

Questions about the first sift of applications

This is where most of the risk sits, because it's where the volume is and where software is most likely to act alone. Ask these in full, and don't accept "our system shortlists the best candidates" as an answer.

1. "Does any tool reject or down-rank applicants before a person reads their application?" The answer you want: software may sort or highlight, but a consultant reads every application, or at least every one they reject.
2. "What are the knockout questions or hard filters for our role?" These are yes/no rules, such as "holds qualification X" or "lives within 20 miles". They're often the biggest source of wrongly rejected candidates. Ask to see and approve them.
3. "If the tool gives a score, what drives it and where's the cut-off?" A consultant who can explain what the score rewards (keywords, job titles, years of experience) understands their tool. One who can't, doesn't.
4. "Can we see a random sample of rejected applications?" This single question is the most useful on the list. Anonymised if necessary, ten rejected CVs will tell you more about the filter than any sales description.
5. "How many applications did the software handle, and how many did a person read?" Ask for the actual numbers for your role after the first week.

Many agencies screen perfectly well with AI as a sorting aid and a human reading the results. The tutorial on whether AI can screen CVs fairly explains the difference between sorting and deciding, which is the distinction these questions are designed to uncover.

Questions about chatbots, assessments and video interviews

Some agencies use a chatbot to pre-screen applicants, online tests scored automatically, or one-way video interviews assessed by software. Each deserves its own questions.

• "What does the chatbot ask, and can we see the script?" Chatbots can drift into questions you'd never ask yourself. In one illustrative case, an agency's pre-screening bot asked every applicant what year they left school, which is effectively a question about age.
• "Are test results used to reject people automatically?" Tests can be useful, but a hard pass mark set by the vendor may not suit your role.
• "If you use video interviews, does software score them? On what?" Transcripts scored against the answers you care about are one thing; any claim to read personality, honesty or emotion from faces and voices is another, and it's the kind of analysis the EU AI Act now bans at work.
• "Can candidates ask for a person instead?" A good agency offers an alternative for anyone who needs it, including applicants with disabilities that make timed or recorded formats unfair.

Questions about bias testing and records

Any agency using AI screening should be checking whether it treats groups of applicants differently. The method needn't be sophisticated, but it has to exist. The checks recruiters should be running are set out in bias checks every recruiter should run on AI CV screening; your questions are about whether they happen.

• "How do you test your screening tools for bias, and how often?" Good: "We compare pass rates across groups each quarter, using the equality data applicants volunteer, and we review any gap with the vendor." Weak: "The vendor handles that."
• "What did the last test find?" A real answer includes a problem found and fixed. "Nothing, ever" suggests nobody looked.
• "What records do you keep of screening decisions, and for how long?" If a candidate challenges a rejection, someone needs to be able to show why it happened.
• "Has your vendor been independently audited?" Not every small agency's vendor will have been, but the question shows whether the agency has asked.

Questions about candidates' rights and data

• "What are candidates told about AI in your process?" Ask to see the wording in the privacy notice or application page.
• "How does a candidate ask for a human review of a decision?" There should be a named route, and you should know about any request involving your role.
• "How long do you keep applicants' data, and does your vendor use it to train its models?" Candidate CVs used to improve a vendor's product is a use applicants may not have agreed to.
• "Who do we call if a candidate complains about the process?" Decide this before it happens, not after.

Sending the questions: the salon's email

Questions asked on a sales call get sales answers. Put them in writing. This is the email the salon sent to two agencies, filled in:

Subject: Your screening process for our two groomer roles

Hi [first name],

Before we agree terms, could you answer these in writing?

1. Which AI or automated tools will touch applicants for our roles
   (advert targeting, CV sorting, chatbot, tests, video)?
2. Will any tool reject or down-rank applicants before a person
   reads their application?
3. What knockout questions or filters would you set? We'd like to
   approve them. Please note we value salon experience as much as
   any specific grooming certificate.
4. After the first week, can you send us numbers (applications
   received, applications read by a person) and five anonymised
   rejected applications?
5. How do you test your tools for bias, and what did the last
   check find?
6. What are candidates told about AI, and how can they ask for a
   person to review a decision?
7. How long do you keep applicants' data, and does any vendor use
   it to train its models?

Thanks,
[your name], [salon name]

Grading the answers you get back

You won't get perfect answers from every agency, especially small ones. Grade each answer as good, follow-up needed, or walk away.

One agency's reply to question 2 read, in part: "Our platform uses AI to identify top matches, and applicants below the match threshold receive an automated update so they aren't left waiting." Politely phrased, that means software rejects people without a person reading their application. The salon asked a follow-up: could a consultant read everything below the threshold for these two roles? The agency agreed, which moved the answer from "walk away" to "follow up", and the salon added it to the terms.

If two or three agencies have replied at length, a general AI assistant on a business plan can line their answers up against your questions, as long as you tell it not to fill gaps. The salon used this prompt:

Below are two recruitment agencies' written answers to my seven
questions about AI screening. For each question, quote the relevant
sentence from each agency, then grade it Good, Follow up or Walk
away using this rule: Good = a person reads every rejection and the
detail is specific; Follow up = partly answered or vague; Walk away
= automated rejection, refusal to share, or emotion analysis.
If an agency didn't answer a question, write "no answer".
Don't soften or reinterpret their wording.

[paste the questions and both replies]

The output was a useful first pass with one telling mistake. For question 2, the assistant graded the agency whose applicants "below the match threshold receive an automated update" as Good, reading "update" as a status message rather than a rejection. The quoted sentence sat right next to the grade in its own table, which made the error easy to catch. That's the reason to insist on quotations: you can check every grade against the agency's actual words.

Worked example: the salon checks the rejected pile

With terms agreed, the agency advertised both roles. After ten days it reported 140 applications. Its software had ranked them, a consultant had read the top 25, and six people were shortlisted. The remaining 115 had been sorted as "below threshold", awaiting the consultant's promised read-through.

The salon asked for its sample of ten rejected applications, anonymised. Two stood out immediately: applicants with six and nine years of salon grooming experience who had trained on the job rather than holding the particular certificate the agency's template treated as essential. Both had scored low because the certificate's name didn't appear in their CVs. The salon interviewed both, and one was hired.

A quick sum shows what the sample suggested about the whole pile. If two in ten of the rejected sample were worth interviewing, the 115 below the threshold might have held twenty or more people worth a look. Even if the real figure was half that, the filter was discarding more good candidates than it shortlisted. The agency removed the certificate from the knockout rules, re-ran the ranking, and the consultant's read-through of the remaining applications produced three further interviews. The salon filled both roles, and the second hire came from the re-run.

Without the sample, the salon would have interviewed six people chosen by a rule it never approved, concluded that experienced groomers were scarce, and perhaps raised its pay offer to attract people who were already in the rejected pile.

Apply through the process yourself

Written answers describe the process the agency intends; applying to your own advert shows you the one candidates actually get. Ask a friend to apply, or use a spare email address with an honest note in the application saying it's a test, and go through every step: the advert, the form, any chatbot, and whatever emails follow.

The salon's owner did this with a deliberately thin application. The automated rejection email arrived four minutes after she applied, at 11.20 at night, signed "The team at [salon name]". That told her three things at once. No person had read the application, despite the agency's written answer. The rejection went out under her business's name, not the agency's. And the wording gave applicants no way to ask for a review. She raised all three, and the agency held rejections for a consultant's review, changed the sign-off to its own name, and added a line inviting applicants to reply if they wanted a person to look again.

Keep the test honest and small. You're checking the experience, not trying to catch out a consultant, and one test application per role is plenty.

Terms to add before you sign

Turn the answers into commitments. A short addendum to the agency's terms of business is usually enough; have your adviser check the final wording.

AI and automated screening
1. The Agency will tell the Client which automated or AI tools are
   used for the Client's roles, and give notice before adding any.
2. No applicant will be rejected for a Client role without a member
   of the Agency's staff reading their application.
3. Knockout questions and screening filters for Client roles will be
   agreed with the Client in advance.
4. On request, the Agency will provide the number of applications
   received and read, and an anonymised sample of rejections.
5. The Agency will not use tools that infer applicants' emotions,
   honesty or personality from video or voice.
6. The Agency will tell applicants how AI is used in its process and
   provide a route to request human review, and will inform the
   Client of any such request relating to a Client role.

These terms don't stop an agency using AI; they make sure the AI sorts and a person decides. That's also a good principle if you later bring screening in-house, which setting up an AI-assisted hiring process for a small team covers. And if the agency tells you its screening lives inside a well-known applicant tracking system, evaluating the AI features in recruitment software helps you understand what that system can and can't be configured to do.

Further reads

• AI CV Screening for Recruitment Agencies: Setup and Safeguards — How a careful agency sets up AI screening, from its side.
• AI Screening: Bolt It Onto Your ATS or Switch Systems? — The systems agencies use and where the AI sits in them.
• AI Recruiting Tools for Small Businesses: What's Worth Paying For — If you'd rather screen in-house, which tools are worth paying for.
• How to Write Job Descriptions With AI That Attract Good Hires — A clear job description makes any screening fairer.
• How to Test Job Candidates' AI Skills in an Interview — What to test once the shortlist reaches you.
• How Recruiters Use AI to Write Candidate Summaries Clients Read — Read agency candidate summaries with a critical eye.
• How Much Time Does AI Save a Recruitment Agency per Role? — Calculate recruitment hours saved per vacancy, including candidate checks, ATS corrections and the cost of setting up the process.
• AI Tools and AI Development: The Complete 2026 Guide — the AI hub, including every tutorial in the AI-for-business series.

Sources: EU AI Act Annex III point 4 (text via the AI Act Explorer); the series fact sheet for AI Act dates and the Article 5(1)(f) prohibition.