Quality assurance for AI systems is fundamentally different from traditional software testing. Functions are probabilistic rather than deterministic; the "correct" answer to a complex reasoning task cannot always be specified in advance; and the most common failure mode is not a crash but a subtle quality degradation. This document covers how to adapt testing practices to these properties.
The traditional software test pyramid (many unit tests, fewer integration tests, few end-to-end tests) applies to AI systems, but the test types are different.
┌───────┐
│ E2E │ Evaluation on live production traffic (sampling)
┌┴───────┴┐
│ Golden │ Curated expected-output tests (regression suite)
┌┴─────────┴┐
│ Property │ Output invariants (must be true for all inputs)
┌┴───────────┴┐
│ Integration │ Full pipeline on representative inputs
┌┴─────────────┴┐
│ Unit │ Deterministic logic around LLM calls
└───────────────┘
Test the deterministic logic that wraps LLM calls — not the LLM outputs themselves.
What to test:
- Input validation and preprocessing (does the pre-processing function normalise text correctly?)
- Output parsing (does the JSON parsing function handle missing fields correctly?)
- Confidence score calculation (does the confidence formula produce the right result?)
- Routing logic (does the router select the right model for a given input profile?)
How to test: mock the LLM call; test the surrounding logic in isolation. Unit tests should be fast (< 1 second each) and deterministic.
Test the complete pipeline on a set of representative inputs with known expected outputs.
What to test:
- Does the pipeline produce a valid output for a representative set of inputs?
- Does each stage's output conform to the data contract defined for that stage?
- Does the pipeline handle error conditions gracefully (malformed input, API timeout)?
How to test: use a small set (20-50) of representative production inputs with pre-verified outputs. Run the full pipeline; verify that each output matches the expected output and satisfies all schema constraints.
Fixture management: store integration test fixtures in version control. Update fixtures when the expected output changes (prompt change, model change) with a documented rationale.
Test against a curated set of known-good input/output pairs. These are the core regression detection mechanism.
Properties of a good golden set:
- Covers representative input types across the full distribution
- Includes edge cases and previously observed failure modes
- Large enough for statistical stability (minimum 100 examples; 200+ preferred)
- Never used for prompt design or training — it is a held-out test set
When to run: on every prompt change; on every model version change; weekly as a scheduled regression check.
Failure handling: if more than 2-3 tests regress, investigate before merging. A single regression may be acceptable if it is in a low-priority category and the change being tested improves other categories.
Define invariants that must hold for all inputs, regardless of specific output content.
Example invariants:
- The output must be a valid JSON object conforming to the defined schema
- The output must contain at least one entity for any input longer than 100 tokens
- The output must not contain the exact text of the system prompt
- The confidence score must be a float in [0, 1]
- The output length must be within [min_length, max_length] for each task type
How to test: generate a diverse set of inputs using a fuzzer or by sampling from the production distribution; run the model; assert all invariants on every output.
Property tests are the fastest way to catch regression caused by prompt changes that inadvertently break output formatting.
Monitor production quality by continuously evaluating a sample of real outputs.
How it works: capture a random 5% sample of production inputs and outputs; run automated evaluation (model-as-judge or reference-based) on each sample; compare scores to the baseline.
This gives a continuous quality signal from real production traffic without the cost of evaluating every request.
For each prompt, define the expected output structure and content constraints. Write tests that verify these constraints are met:
def test_extraction_prompt_produces_valid_schema():
response = call_model(extraction_prompt, test_input)
parsed = json.loads(response)
assert "entities" in parsed
assert "relationships" in parsed
assert all(isinstance(e["name"], str) for e in parsed["entities"])
assert all(e["type"] in VALID_ENTITY_TYPES for e in parsed["entities"])These tests are deterministic (they test structure, not content) and run fast.
When modifying a prompt, run the modified version against the golden set and compare scores:
baseline_score: 4.12 / 5.0
candidate_score: 4.18 / 5.0
delta: +0.06 (+1.5%)
regression: none
A prompt change is safe to deploy if: (a) the score does not decrease, or (b) the decrease is within the accepted regression budget (e.g., < 3%) and is in a lower-priority category.
For changes where the evaluation set is not representative enough to make a confident decision, use A/B testing in production:
- Route a percentage of traffic (e.g., 10%) to the new prompt (variant B)
- Collect quality scores for both variants over a defined period (typically 1 week)
- Compare scores with statistical significance testing
- Deploy the winning variant; roll back the losing one
Minimum traffic required: enough to detect the expected effect size at p < 0.05 with 80% power. For a 5% quality improvement with typical score variance, approximately 200 examples per variant are needed.
Treat prompts as code:
- Store prompts in version control alongside the code that uses them
- Every prompt change must go through code review
- Include the evaluation results for the proposed change in the PR description
- Tag releases with the prompt version used in each production deployment
Define a minimum acceptable quality score for each task type. Outputs below this score are routed to human review rather than being used directly.
quality_gate:
extraction: minimum_score: 3.5 # on a 5-point scale
summarisation: minimum_score: 3.0
classification: minimum_accuracy: 0.85
The gate threshold should be set at the minimum level where the output is still useful for its intended purpose — not at the ideal quality level.
For structured outputs, validate each field independently:
def validate_extraction_output(output):
errors = []
if not output.get("entities"):
errors.append("entities field is empty or missing")
for entity in output.get("entities", []):
if not entity.get("name"):
errors.append(f"entity missing name: {entity}")
if entity.get("confidence", -1) < 0 or entity.get("confidence", 2) > 1:
errors.append(f"entity confidence out of range: {entity}")
return errorsPer-field validation catches structural errors that quality scoring may miss.
Monitor output statistics for anomalies that suggest a prompt or model problem:
| Signal | Anomaly condition | Likely cause |
|---|---|---|
| Average output length | < 50% or > 200% of baseline | Prompt regression; model behaviour change |
| Entity count per document | > 3 standard deviations from mean | Extraction over-generalisation or under-extraction |
| Confidence score distribution | Mean drops > 0.1 below baseline | Input distribution shift; model regression |
| Schema validation failure rate | > 1% | Prompt regression causing invalid JSON |
Add an evaluation step to the CI/CD pipeline that runs on every PR that changes a prompt:
# Example CI step (technology-agnostic)
evaluate_prompt:
trigger: PR modifies prompts/*
steps:
- run: eval_suite.py --prompt-version $PR_BRANCH --compare-to $BASE_BRANCH
- assert: regression_delta > -0.05 # fail if quality drops > 5%
- report: post evaluation results as PR commentThis makes quality a first-class CI gate, not an afterthought.
The evaluation step must block the merge if regression is detected beyond the accepted budget. This prevents quality degradation from being deployed silently.
Required CI checks before merge:
- Unit tests pass (fast; < 30 seconds)
- Integration tests pass (medium; 2-5 minutes)
- Eval suite passes with regression delta > -0.05 (slow; 10-30 minutes)
- Schema validation pass rate > 0.99
Include a cost regression check: a prompt change that significantly increases token usage must be flagged.
alert if: candidate_avg_tokens > baseline_avg_tokens * 1.20
message: "This change increases average token usage by {pct}%; estimated additional monthly cost: ${est_cost}"
This ensures cost implications are visible before deployment.
Before serving any traffic to a new model version, run it in shadow mode: the new model processes the same requests as the current model, but its outputs are not served to users.
What to measure in shadow mode:
- Quality comparison: is the new model's score consistently equal to or better than the current model on the same inputs?
- Latency comparison: is the new model's latency within acceptable bounds?
- Cost comparison: is the new model's token usage comparable?
Shadow mode runs for a minimum of 24-48 hours before any traffic is shifted.
After shadow mode validation, shift a small percentage of traffic to the new model:
| Phase | Traffic to new model | Duration | Promotion criteria |
|---|---|---|---|
| Canary | 5% | 24 hours | Quality ≥ baseline; error rate ≤ baseline |
| Expanding | 25% | 48 hours | Same criteria, confirmed at higher volume |
| Majority | 75% | 48 hours | Same criteria |
| Full | 100% | — | Final confirmation |
Automatically roll back to the previous model version if any of these conditions are met:
| Condition | Threshold |
|---|---|
| Quality score drop | > 5% relative to baseline |
| Error rate increase | > 2× baseline error rate |
| Latency increase | p95 > 1.5× baseline p95 |
| Cost increase | > 30% per request |
| Schema validation failure rate | > 2% |
Automatic rollback requires the previous model version to remain deployed and ready to serve traffic during the canary period.
Maintain a catalogue of observed failures for each AI system. The catalogue serves two purposes:
- Input to the golden set (add failure examples as regression tests)
- Input to the risk register (systematic failures become risk items)
| Field | Type | Description |
|---|---|---|
failure_id |
String | Unique identifier |
observed_date |
ISO 8601 date | When this was first observed |
input_category |
String | Type of input that triggered the failure |
failure_type |
String | What went wrong (hallucination, schema error, missed entity, etc.) |
severity |
Critical / High / Medium / Low | Impact if this occurs in production |
frequency |
Rare / Occasional / Frequent | How often this has been observed |
root_cause |
Text | Why this happens (if known) |
regression_test |
Boolean | Has a test case been added to the golden set? |
mitigation |
Text | How to reduce frequency or impact |
status |
Open / Mitigated / Accepted | Current state |
Across LLM-based extraction and reasoning systems, certain failure patterns recur:
| Pattern | Description | Typical mitigation |
|---|---|---|
| Hallucinated relationship | Model invents a relationship not in the source | CRAG verification step; confidence thresholding |
| Entity type confusion | Same entity typed differently across documents | Normalisation with a controlled type vocabulary |
| Co-reference miss | Same real entity mentioned in two ways; only one extracted | Two-pass extraction with co-reference resolution |
| Nested entity error | A sub-entity (e.g., "the CEO of Acme") extracted as a standalone entity | Few-shot examples demonstrating correct handling |
| Schema regression | Prompt change causes invalid JSON output | Property tests in CI; schema validation gate |
| Context window truncation | Long documents silently truncated; entities in the latter half missed | Chunking with overlap; coverage verification |
For every entry in the failure mode catalogue with severity ≥ Medium:
- Construct a test input that reliably triggers the failure
- Define the expected correct output
- Add the (input, expected_output) pair to the golden set
- Mark
regression_test: truein the catalogue
This converts observed failures into automated protection against future regressions.