Teams spend weeks debating which model to use and then ship without any way of telling whether the answer got better.

Start with a bounded evaluation before committing to a model. A first regression set can be small, but high-quality labels and representative coverage take work. Write down the intended task, the errors that matter, and how the comparison will be scored before viewing the candidate outputs.

Leaderboards measure someone else's job

Public benchmarks are useful for what they are: broad, standardised, comparable across vendors. They are also, from your perspective, measuring a different task on different data with different success criteria.

Public benchmark results also need context: overlap with training data can affect some evaluations, and different reports select different tasks and settings. HELM’s scenario and metric framework is useful for making coverage and omissions explicit. It does not turn a public score into a measurement of your own users’ workload.

A model that tops a reasoning leaderboard may be worse than a smaller one at extracting fields from your particular malformed invoices. There is exactly one way to find out.

Keep a challenge set and a representative sample

Use two sets for different questions. A deliberately selected challenge set catches known failures and boundary cases. A separate held-out random or appropriately stratified sample of actual eligible traffic estimates performance on the workload you serve. Protect private data and record the sampling window and exclusions. Do not report the challenge-set pass rate as the production success rate.

Build it from four buckets:

The boring middle — half a dozen genuinely typical inputs. If it fails here, nothing else matters.

Known-hard cases — the ones that have already caused problems. Every team has a folder of these, usually in a support thread.

Edge cases you can articulate — empty input, enormous input, wrong language, mixed languages, adversarial phrasing, an input that should produce a refusal.

Cases with a defensible right answer — where you can say what correct looks like. Some tasks are genuinely subjective, and those need a rubric rather than an answer key.

Twenty to fifty examples can be an editorial starting point for a hand-built regression set, not a statistically established minimum. On twenty binary cases, one answer changes the score by five percentage points. Report counts, task categories, and uncertainty rather than treating a small score difference as a winner. Choose the representative sample size around the decision you need to make.

Write down what "correct" means before you look

This is the step people skip, and skipping it is how evaluation becomes rationalisation.

For each case, record the expected output or the criteria a good output must meet. Do it before running any model. Once you have seen an impressive answer, your definition of correct quietly reshapes itself around it.

For subjective tasks, write a rubric with a small number of dimensions and a scale you can apply consistently. Two people scoring the same output should land close. If they do not, the rubric is the problem.

Grading, and the trap in automating it

Exact match works for tasks with a defined canonical answer. For extraction and structured output, specify whether case, whitespace, ordering, equivalent dates, or numeric tolerances matter. Validate the schema separately from factual correctness so valid JSON does not receive credit for the wrong value.

For open-ended outputs, model graders can help, but their behaviour needs calibration. The MT-Bench/Chatbot Arena paper reports position, verbosity, and self-enhancement biases in the judges it studied. Those findings motivate checks, not an assumption that every current grader fails identically. Blind candidate names, swap answer order, and compare a subset with human ratings.

If you grade with a model, calibrate it. Hand-score a subset yourself, compare, and find out whether your grader agrees with you before you trust it across the whole set.

What it makes decidable

Once the set exists, arguments turn into measurements.

Run candidates on the same frozen cases and compare paired results, latency, and cost. Track which cases changed, not just the total. A cheaper model within one percentage point on a small set is a reason for more investigation, not evidence of equivalent quality. Repeat stochastic cases where relevant, and keep the final holdout out of prompt tuning so repeated experimentation does not quietly train your decision on the test answers.

The set is really a specification

There is a side effect worth naming, because it often turns out to be the main benefit.

Writing down what a correct output looks like, case by case, forces the team to agree on what the feature is actually supposed to do. A striking number of AI projects have never done this. "Summarise the ticket" sounds like a requirement until two people write expected outputs for the same ticket and produce different things — one a two-line abstract, the other a structured list of actions.

That disagreement was always there. The eval just made it visible early, on a Thursday afternoon, rather than after launch in a stakeholder review.

Keep it alive

An evaluation set decays. Add suitable production failures to the regression suite after removing private data, and refresh the representative sample separately. Cases selected because they failed are valuable for prevention, but their growing share should not be mistaken for the frequency of those failures in normal traffic.

Version it with your code. Run it on every meaningful change. And accept the one uncomfortable consequence of having it: sometimes it will tell you the expensive change you just made did nothing, and you will have to believe it.