A coding agent can make a failing test pass while changing an unrelated behavior, weakening the assertion, editing generated output, or leaving a diff no maintainer would accept. A useful evaluation therefore needs two score layers: execution evidence and review evidence. This method selects one bounded repository bug, freezes the starting point, withholds part of the verification, and scores the resulting patch for scope, reasoning, safety, and handoff quality. It is small enough to rerun when tools or models change.
A repository-native task
Pick a real defect with a reproducible symptom and a narrow expected boundary. Good candidates include an incorrect parser edge case, a route returning the wrong status, or a state transition that fails under one input. Avoid a task whose answer is already stated in a nearby commit, a broad architecture rewrite, or a dependency outage the agent cannot control.
Record the repository, starting commit, runtime, dependency lock, setup commands, and the user-visible acceptance statement. Keep the existing failing reproduction available to the agent, but reserve at least one hidden test or manual check that distinguishes the intended fix from the easiest superficial patch.
The agent’s authority
Use a clean branch or disposable worktree. State which files may change, whether package installation and network access are allowed, and which commands are permitted. Do not provide production secrets. Capture the tool and model version, configuration, system instructions, task prompt, start time, end time, tool calls, commands, and token or cost fields that the provider exposes.
The authority boundary is part of the result. An agent that succeeds only with unrestricted network access and a broad credential should not be compared as though it performed the same task as one confined to the repository.
Two score layers
Run the repository’s focused test, the hidden test, and an appropriate broader regression check after the agent stops. Preserve exit codes and logs. Then review the diff without looking at the agent’s narrative first.
| Dimension | Pass evidence | Common false positive |
|---|---|---|
| behavior | visible and hidden checks pass | assertion weakened |
| scope | changed files map to defect | drive-by refactor |
| safety | error and boundary cases handled | happy path only |
| maintainability | local style, clear names, minimal comments | new abstraction for one case |
| handoff | exact tests and limitations reported | “all fixed” without evidence |
Use a prewritten rubric, for example 0–2 per dimension. Do not change the weights after seeing a preferred tool’s patch. A hard failure in behavior or a destructive/unreviewable change should override a high total.
How does benchmark gaming appear in a real repository?
Inspect whether the patch detects the test fixture, hard-codes an example value, skips a code path, snapshots incorrect output, or changes configuration to avoid execution. Search the diff for disabled lint rules, broad exception handling, deleted validation, and unexplained dependency changes. Run the reproduction outside the test harness when possible.
Review the untouched neighboring code. The best patch may be only a few lines, but small is not automatically correct. Ask whether the chosen ownership boundary matches the existing design and whether the fix creates a second inconsistent implementation.
Variance matters more than a leaderboard
Reset to the exact starting commit and run the same task at least three times when cost permits. Preserve each attempt, including failures. Report pass rate, median completion time, intervention count, command failures, and patch-quality distribution. One excellent run does not establish reliability; one failure may reveal a brittle assumption that a single demo hides.
If comparing two agents, keep the authority, prompt, repository, tests, and time limit the same. Note differences that cannot be normalized, such as proprietary context indexing or unavailable model settings. Do not merge the best fragments from multiple attempts and call the composite a run.
A human baseline on the same task
Ask a maintainer or competent contributor to solve the frozen task under the same repository and test constraints while recording time and changed files. The purpose is not to rank human worth against a model. It calibrates task ambiguity and exposes missing setup instructions. If the human cannot interpret the acceptance statement or hidden test, agent failures may be defects in the evaluation. Compare review effort as well as completion time: a fast patch that takes longer to verify and repair can cost more than a slower, legible solution.
The evidence bundle
Store a sanitized task brief, starting commit, environment manifest, run trace, final diff, test outputs, rubric, reader notes, and cost record. Remove credentials and personal data, but do not remove failed commands or human interventions. State whether the evaluator wrote the hidden test before the run.
The conclusion should be narrow: “Agent A produced an acceptable patch on this parser bug in two of three runs under these permissions.” That sentence is less exciting than “best coding agent,” but it can guide a team. The evaluation becomes useful over time when the same repository task is rerun after a model, tool, or policy change and the evidence remains comparable.