All Resources
R-41
Intelligence (AI)
Harness Engineering: Making AI Quality Measurable
The discipline that turns prompt changes from guesswork into engineering — golden sets, scorer selection, regression gates in CI, and reading a result that is genuinely noisy.
PAR2 Labs
August 22, 2026
16 min
Every team building with models hits the same wall: someone changes a prompt, everyone agrees it feels better, and three weeks later a customer finds the case it broke. A harness is what replaces that with a number. It is the least glamorous part of AI engineering and the single highest-leverage thing you can build.
01
What a harness actually is
A harness is four things that run together: a fixed set of cases, a scorer that turns an output into a number, a runner that executes the set against a version of your system, and a gate that decides whether a change ships. Miss any one and it is not a harness — it is a spreadsheet someone updates occasionally.
The crucial property is that it is versioned and frozen. A test set that grows every time someone adds a case cannot tell you whether yesterday's change helped.
Fig 1 — the harvest step is what keeps a harness honest. Without it the set slowly stops resembling production and the number stops meaning anything.
02
Choosing a scorer
Scorer choice determines whether your number means anything. Reach for the cheapest scorer the task permits, and be suspicious of model-graded scoring — it is genuinely useful and it introduces a second model's biases into your measurement.
| Scorer | Use for | Cost | Watch out for |
|---|---|---|---|
| Exact match | Classification, routing, structured extraction | Free | Too brittle for anything with formatting variance |
| Schema validation | Structured output, tool arguments | Free | Valid shape with wrong values still scores as a pass |
| Fuzzy / F1 over fields | Extraction with several fields | Free | Field weighting has to reflect what actually matters |
| Deterministic assertions | 'Must cite a source', 'must not mention X' | Free | Only checks what you thought to assert |
| Model-graded rubric | Tone, helpfulness, summary quality | Per run | Position bias, verbosity bias, drift when the grader changes |
| Human review | The final word on subjective work | Expensive | Reserve for a sample, not the whole set |
Combine scorers. A case can require valid schema, a cited source, and a rubric score above a threshold — all three, checked separately.
A harness is what replaces 'it feels better' with a number you can argue about.
03
Building one
Seven steps, and steps 01 to 03 can be done in an afternoon. The value arrives the first time the harness blocks a change that everyone was sure was an improvement.
01
Collect real cases, weighted toward pain
TOOL
Production logs
USE
50–200 cases; over-sample known failures
GET
A candidate golden set
Pull real inputs from production, not invented ones. Deliberately over-sample the cases that have failed before, the edge cases, and the ones people complain about. Include a handful of straightforward cases as a sanity floor.
Why: A set of easy cases scores well and tells you nothing. The set exists to catch regressions, so it must contain the things that regress.
02
Write the expected outputs by hand
TOOL
Domain expertise, not a model
USE
The correct answer, agreed by someone who knows
GET
Ground truth you can defend
Have a person who understands the domain write or approve the expected output for every case. Record why an answer is correct where it is not obvious, because you will revisit it.
Why: Ground truth generated by a model encodes that model's mistakes into your measurement, and you will then optimise toward them without ever seeing it.
03
Freeze and version the set
TOOL
Git
USE
The set lives in the repo, versioned like code
GET
Comparable runs over time
Commit the set as data in the repository. When you add cases, bump the set version and note it — never silently extend the set you are currently comparing against.
Why: If the set changes between runs, the comparison is meaningless. Versioning is what turns a collection of examples into a measurement instrument.
04
Score per case, and record everything
TOOL
promptfoo or your own runner
USE
Per-case score, full output, tokens, latency
GET
A results artefact, not a number
Store the complete output for every case alongside its score, plus token counts and latency. Keep the last several runs so a regression can be diffed rather than re-derived.
Why: An aggregate score tells you that something changed. The per-case diff tells you what, and that is the only actionable form of the result.
05
Establish a baseline and a threshold
TOOL
Your current production version
USE
Baseline score plus an explicit tolerance
GET
A gate that can actually block
Run the current production version to set the baseline. Define what blocks a release: typically no case regressing from pass to fail, and the aggregate not dropping beyond a stated tolerance.
Why: Without a pre-agreed threshold, every regression becomes a negotiation at the moment it is least convenient to have one.
06
Run it in CI on every change to the AI path
TOOL
Your CI
USE
Triggered by prompt, model, retrieval or tool changes
GET
Regressions caught before merge
Run the harness on any change touching prompts, model versions, retrieval configuration or tool definitions. Post the diff against baseline into the pull request.
Why: A harness run manually is a harness run rarely. In CI it is the difference between catching a regression in review and hearing about it from a customer.
07
Account for non-determinism honestly
TOOL
Repeat runs
USE
Temperature 0 where possible; 3–5 repeats where not
GET
A result you can trust
Pin temperature to zero and fix seeds where the provider supports it. Where output is inherently variable, run each case several times and report the distribution rather than a single sample.
Why: A one-point difference on a single run is noise. Treating it as signal leads teams to chase changes that never existed and to dismiss real regressions as variance.
04
Reading results without fooling yourself
A harness makes it possible to be wrong precisely, which is progress — but only if the results are read with the same discipline they were produced with.
Discipline for reading a run
01
Small aggregate movements on a small set are noise. With 100 cases, a two-point change is well within variance.
02
Always read the per-case diff. An unchanged aggregate can hide five improvements and five regressions.
03
Optimising against the harness eventually overfits to it. Harvest fresh production failures regularly.
04
A model-graded scorer is a model — pin its version, and re-baseline when it changes.
05
Track cost and latency alongside quality. A change that improves scores and doubles spend is not obviously a win.
06
Never generate ground truth with the model under test. That measures agreement, not correctness.
05
Reference
Tooling here is genuinely interchangeable — the discipline is what matters. Start with something that runs a set of cases and diffs against a baseline, and grow from there.
Primary documentation
promptfoo
Declarative test cases, assertions, and side-by-side diffs between versions.
Tool use
Where structured outputs make schema-validated scoring possible.
OpenTelemetry — GenAI conventions
Recording tokens and latency alongside quality, in a standard shape.
JSON Schema
The basis for schema-validation scoring of structured output.
LangGraph
Where the system under test is a multi-step graph rather than a single call.
Key Takeaways
01
A harness is a frozen set, a scorer, a runner and a gate — miss one and it is not a harness.
02
Over-sample known failures; a set of easy cases scores well and tells you nothing.
03
Never generate ground truth with the model under test — that measures agreement, not correctness.
04
Read the per-case diff, not the aggregate; and treat small movements on small sets as noise.
PAR2 Labs · Intelligence (AI)
Work With Us