Shipping an AI prompt without testing is like shipping code without running it. It might work, but you have no idea how it handles edge cases, adversarial inputs, or the full range of real-world queries.
This guide covers practical prompt testing patterns — from simple manual checks to automated evaluation pipelines.
Quick answer
Build a test set of 20-50 representative inputs, define what 'correct' looks like for each, run the prompt against all test cases, and check the results against your criteria. Automate this so you can rerun tests whenever you change the prompt.
- You are about to deploy a prompt in a production application.
- You want to change a production prompt and need to know if the change is safe.
- You need to compare prompt versions to see which performs better.
Why prompt testing matters
A prompt that works on your three favourite examples might fail on the next ten inputs from real users. Without testing, you ship and hope — then discover failures from user complaints.
Prompt testing is not optional for production AI. It is the equivalent of unit testing for traditional code. The difference is that test assertions are softer — 'the output should contain X' rather than 'the output should equal X.'
Building a test set
Start with 20-50 test cases that cover the range of inputs your prompt will see in production. Include easy cases, hard cases, edge cases, and adversarial inputs.
For each test case, define what a correct response looks like — not the exact text, but the criteria the response should meet.
- Include 5-10 'golden' examples where you know the exact right answer
- Add 10-20 typical inputs from real or simulated users
- Add 5-10 edge cases: empty input, very long input, ambiguous input, off-topic input
- Add 3-5 adversarial inputs: prompt injection attempts, requests to ignore instructions
Evaluation criteria
Define what 'correct' means for your prompt. This typically includes: factual accuracy, format compliance, tone, completeness, and absence of harmful or off-topic content.
Some criteria can be checked automatically (format, length, required keywords). Others need human review or LLM-as-judge evaluation.
# Example: automated prompt test runner
def test_prompt(prompt_template, test_cases):
results = []
for case in test_cases:
response = call_llm(prompt_template.format(**case["input"]))
checks = {
"contains_answer": case["expected_keyword"] in response,
"within_length": len(response) < case["max_length"],
"no_hallucination": not contains_known_false(response),
"correct_format": matches_format(response, case["format"]),
}
results.append({"input": case["input"], "checks": checks})
return resultsRegression testing
When you change a prompt, run all test cases again and compare results to the previous version. This catches regressions — cases where the new prompt is worse than the old one.
Keep a version history of your prompts and their test results. This makes it easy to roll back if a change causes problems in production.
LLM-as-judge evaluation
For criteria that are hard to check programmatically (tone, helpfulness, accuracy), use another LLM to evaluate the output. This is called LLM-as-judge.
Create a judge prompt that describes your criteria clearly and asks the judge model to score each response. This is not perfect, but it is much faster than human review for large test sets.
Worked example: testing a customer support prompt
You have a prompt that generates responses to customer support tickets. You build a test set of 30 tickets covering common issues, rare issues, angry customers, and spam. Each test case defines expected criteria: should mention relevant product, should offer a specific solution, should not make promises about timelines. You run the test suite before every prompt change.
Common mistakes
- Testing only on easy examples and shipping without edge case coverage.
- Not saving test results — you cannot compare prompt versions without historical data.
- Testing manually and inconsistently instead of building a rerunnable test suite.
When to use something else
For evaluating AI outputs after they are in production, see evaluating AI outputs in real apps. For reducing errors in AI tool use, see reducing hallucinations in tool-based AI.
Frequently asked questions
Why test prompts at all?
A prompt that works on your three favourite examples can fail on the next ten real inputs. Without a test set you ship and hope, then learn about failures from user complaints.
How do I build a test set?
Collect 20-50 representative inputs covering easy, hard, edge, and adversarial cases, and define what 'correct' looks like for each. Seed it from real user inputs, not just happy paths.
What should I check each output against?
Your defined criteria: factual accuracy, format compliance, tone, completeness, and absence of harmful or off-topic content. Make as many checks machine-readable as possible.
How do I make this repeatable?
Automate the run so you can re-test the whole set whenever you change the prompt or model. Treat prompts like code with a test suite, not a one-off.
What do I do when a test fails?
Decide whether it is a prompt fix or a genuinely hard case, adjust, then re-run the whole set to confirm you did not regress other cases. Keep the failure as a permanent test.
How is this different from evaluation?
Prompt testing is the pre-ship gate on a fixed set; ongoing evaluation tracks quality in production over time. Same criteria, different stage — do both.
Related guides on this site
These guides cover output evaluation, error reduction, and quality assurance for AI applications.