Build The Eval First. Then Argue About Models

Teams spend weeks debating which model to use and then ship without any way of telling whether the answer got better.

Invert it. The evaluation set is the cheap part, it takes an afternoon, and until it exists every model comparison you run is people taking turns being impressed.

Leaderboards measure someone else's job

Public benchmarks are useful for what they are: broad, standardised, comparable across vendors. They are also, from your perspective, measuring a different task on different data with different success criteria.

Two more problems compound it. Benchmark contamination is real — widely published test sets leak into training data, and a score partly reflects familiarity rather than capability. And vendors choose which benchmarks to report, which is a selection process with an obvious direction.

A model that tops a reasoning leaderboard may be worse than a smaller one at extracting fields from your particular malformed invoices. There is exactly one way to find out.

Twenty examples, chosen badly on purpose

Do not sample randomly. Random sampling gives you mostly easy cases and tells you little.

Build it from four buckets:

The boring middle — half a dozen genuinely typical inputs. If it fails here, nothing else matters.

Known-hard cases — the ones that have already caused problems. Every team has a folder of these, usually in a support thread.

Edge cases you can articulate — empty input, enormous input, wrong language, mixed languages, adversarial phrasing, an input that should produce a refusal.

Cases with a defensible right answer — where you can say what correct looks like. Some tasks are genuinely subjective, and those need a rubric rather than an answer key.

Twenty to fifty is the useful range for a hand-built set. Small enough to curate honestly, big enough that a change of a few points means something.

Write down what "correct" means before you look

This is the step people skip, and skipping it is how evaluation becomes rationalisation.

For each case, record the expected output or the criteria a good output must meet. Do it before running any model. Once you have seen an impressive answer, your definition of correct quietly reshapes itself around it.

For subjective tasks, write a rubric with a small number of dimensions and a scale you can apply consistently. Two people scoring the same output should land close. If they do not, the rubric is the problem.

Grading, and the trap in automating it

Exact-match grading works where there is a right answer: extraction, classification, structured output.

Where there is not, most teams reach for a model to grade the outputs. This works better than expected and fails in specific ways worth knowing: model graders favour longer and more confident answers, they are lenient toward outputs that resemble their own style, and they inherit the blind spots of the model being graded when it is the same family.

If you grade with a model, calibrate it. Hand-score a subset yourself, compare, and find out whether your grader agrees with you before you trust it across the whole set.

What it makes decidable

Once the set exists, arguments turn into measurements.

Model selection stops being about vibes. Prompt changes get verified instead of assumed — and a surprising share of clever prompt engineering turns out to be neutral or negative when measured. Upgrades become checkable, which matters because a provider can change a model's behaviour underneath a stable name, and your eval is the tripwire that catches it. Cost trade-offs become concrete: if the cheaper model scores within a point on your actual task, that is a real finding.

The set is really a specification

There is a side effect worth naming, because it often turns out to be the main benefit.

Writing down what a correct output looks like, case by case, forces the team to agree on what the feature is actually supposed to do. A striking number of AI projects have never done this. "Summarise the ticket" sounds like a requirement until two people write expected outputs for the same ticket and produce different things — one a two-line abstract, the other a structured list of actions.

That disagreement was always there. The eval just made it visible early, on a Thursday afternoon, rather than after launch in a stakeholder review.

Keep it alive

An evaluation set decays. Every genuine production failure should be added as a case, which turns your bug reports into regression tests and makes the set progressively more representative of what actually goes wrong.

Version it with your code. Run it on every meaningful change. And accept the one uncomfortable consequence of having it: sometimes it will tell you the expensive change you just made did nothing, and you will have to believe it.