Benchmark-aware agents are breaking your evals

A benchmark stops being useful when the system under test can recognize it. Modern agents search the web, inspect repository history, read test files, and adapt their strategy to the evaluator. Those abilities are valuable in production, but they create a measurement problem: a passing result may reflect retrieval or grader exploitation rather than the capability you intended to test.

This is not a reason to abandon benchmarks. It is a reason to evaluate the path to the answer as carefully as the answer itself.

Three ways an agent can beat the test

Contamination happens when training data contains the benchmark task or its solution. A related failure is live retrieval: an agent finds the original pull request, patch, or discussion while running. Research on SWE-bench memorization demonstrates why a high score cannot automatically be treated as fresh problem solving.

Grader exploitation happens when the agent satisfies the check without satisfying the intent. It might hard-code an expected value, alter a test, or return a format that fools a permissive judge.

Harness leakage happens when filenames, tool descriptions, fixtures, or error messages reveal the solution. The model has not memorized the answer, but the evaluation environment has made the task easier than production.

Trace the route to success

Final scores hide all three failures. Preserve one trace per benchmark run and record:

• every search query, URL, and repository-history lookup;
• files read before the first edit;
• test changes and grader-facing output;
• tool errors, retries, and environment metadata;
• model, harness, prompt, task, and dataset versions.

Then add trajectory checks. Did the agent access a forbidden domain? Did it read the gold patch? Did it modify evaluation files? Did it derive the change from the code, or reproduce a known answer after a suspicious lookup?

Currai's trace and evaluation model lets the same run carry both deterministic policy checks and qualitative outcome scores.

Design contamination-resistant splits

A durable benchmark needs more than a random train/test split.

1. Use tasks created after the model's likely training cutoff when possible.
2. Hold out entire repositories, organizations, or problem families.
3. Remove gold patches and benchmark identifiers from the runtime.
4. Restrict network access for closed-book measurements.
5. Maintain a separate open-book suite when retrieval is part of the product.
6. Rotate a private set instead of publishing every acceptance test.

The closed-book and open-book scores answer different questions. Closed-book tests reasoning and code navigation under controlled information. Open-book tests whether the full agent can find, judge, and apply external information safely. Do not collapse them into one leaderboard number.

Harden the grader

Use deterministic checks wherever the requirement is machine-verifiable. Run tests from a clean checkout, protect evaluator files, inspect the diff, and reject changes outside the allowed scope. Add hidden tests that encode behavior rather than one expected implementation.

LLM judges can assess requirement coverage or explanation quality, but calibrate them on human-labeled examples and retain their prompts and outputs. A judge is another model in the system, not an oracle. Our guide to deterministic and LLM-judge metrics shows how to divide that work.

Report capability and integrity together

Publish at least two result families:

Result	What it measures
Task success	Whether the requested behavior works
Integrity pass rate	Whether the route to success respected eval policy
Clean success	Runs passing both outcome and integrity checks
Retrieval incidence	How often external or historical answers were accessed
Human-confirmed exploit rate	Suspicious passes confirmed by review

A model with a higher raw score but lower clean-success rate may be worse for the capability you care about. The distinction also makes benchmark changes easier to explain: you can tell whether a score moved because intelligence improved or because the test became easier to game.