Evaluating self-improving agents without training on the test

An agent receives review feedback, turns it into a rule, and avoids the same mistake next time. That loop is valuable. It also creates a familiar machine learning risk inside the product: the system can optimize directly against the examples used to judge it.

A rising eval score may mean the agent learned a general behavior—or memorized the test.

Define what is allowed to learn

List every adaptive surface:

• user and team rules;
• retrieved memories or prior trajectories;
• prompt updates and generated examples;
• tool-description changes;
• routing or retry policies;
• fine-tuning and reinforcement-learning data;
• evaluator feedback shown to the agent.

For each surface, record its scope, retention period, provenance, and promotion process. A personal preference should not silently become a global rule, and a failed evaluator output should not enter training without review.

Separate three data pools

Use distinct learning, validation, and release sets.

The learning set can provide full feedback and accepted corrections. The validation set guides iteration but must not become memory or generated rules. The release set remains sealed until a candidate is ready for promotion.

Hold out entire failure families or data sources, not just random examples. Near duplicates can leak the same answer across splits. For coding agents, hold out repositories and time windows; for support agents, hold out intents, customers, and policy variants.

Preserve provenance

Every learned rule should point back to the traces and reviewer decisions that created it. Store author, scope, creation time, examples, confidence, and version. When the rule fires, include its identifier in the trace.

This supports two essential questions:

1. Did the candidate use information derived from this eval case?
2. Which learned rule caused a production regression?

Without provenance, adaptive context becomes an uninspectable second model.

Test generalization and interference

After learning from a failure, evaluate four outcomes:

Test	Purpose
Original case	Confirm the immediate error is fixed
Paraphrased cases	Check robustness to wording changes
Same principle, new domain	Measure generalization
Counterexamples	Detect over-application and interference

A rule like "never modify migrations" may fix one unsafe run and break every legitimate migration task. Counterexamples reveal whether the agent learned the principle or a blunt prohibition.

Evaluate the learning policy itself

Score which feedback becomes a rule, how conflicts are resolved, when rules expire, and whether low-quality feedback is rejected. Inject contradictory reviewers, poisoned examples, outdated policy, and feedback from an unauthorized scope.

The agent should surface conflicts rather than arbitrarily applying the newest or longest instruction. Security-sensitive global rules require stronger review than personal style preferences.

Use production feedback carefully

Production traces are the best source of representative failures, but they can contain private data and user-specific instructions. Redact sensitive fields, retain consent and tenancy boundaries, and avoid treating every user correction as ground truth.

Our guide to turning production traces into better AI describes the broader feedback loop. For self-improving agents, add split hygiene and rule provenance before those traces influence the runtime.

Report fresh-set performance

Publish results on tasks created after the learning window or from sealed sources. Show performance with and without learned context, regression rate on counterexamples, and the number of rules activated per run. A smaller improvement on a fresh set is more credible than a dramatic gain on examples the agent has already seen.