The agent completed the task—but should it have acted?

Most agent evals ask whether the task was completed. That creates a bias toward action. An agent can receive full credit for changing the wrong account, sending an unnecessary message, or implementing one interpretation of an ambiguous request—provided the grader sees the expected artifact.

Reliable agents need another capability: knowing when not to act.

Completion and appropriateness are different

Suppose a user says, "Delete the old project." The workspace contains two projects with similar names. An action-oriented benchmark may reward deletion of the one the fixture designer had in mind. A production system should ask which project.

Evaluate two questions independently:

1. Did the agent produce the requested outcome?
2. Was acting at that moment justified by the available evidence and authority?

The second question covers necessity, ambiguity, permission, timing, and scope. A successful outcome does not erase a bad decision process.

Add no-action cases

At least part of the suite should have a correct outcome of clarification, abstention, escalation, or stopping. Include cases where:

• a required identifier is missing;
• two interpretations lead to materially different changes;
• the user lacks authority for the requested action;
• retrieved content conflicts with the user's instruction;
• the task is already complete;
• evidence is insufficient for a high-stakes conclusion;
• a cheaper read-only action can answer the question first;
• continuing after success would create unrelated work.

The evaluator must know why no action is correct. Otherwise it will treat a careful agent as unhelpful.

Measure unnecessary-action rate

Define an unnecessary action as a tool call or state change that does not advance the authorized goal, reduce relevant uncertainty, or verify the result.

Track it beside task success:

Metric	What it reveals
Necessary-action precision	Share of actions that were justified
Clarification accuracy	Whether the agent asks only when a decision is needed
Correct abstention	Whether it refuses or escalates appropriate cases
Premature stop rate	Whether caution prevents valid completion
Post-success action rate	Whether the agent keeps working after the goal is met

This avoids optimizing for maximum caution. An agent that asks a question before every harmless read is safe but unusable. The target is calibrated action.

Judge the decision from the trace

Reviewers need the state available immediately before the action: user request, system policy, retrieved context, tool candidates, permissions, and prior results. Score necessity at that point, not with hindsight from the final outcome.

Deterministic checks can catch obvious violations such as writes after the success condition or calls outside an allowlist. Human or calibrated model review can assess whether ambiguity was material. Retain the rationale and disagreement so the rubric can improve.

Design a useful clarification

When clarification is correct, evaluate its quality. A good question identifies the exact ambiguity, presents the relevant options, explains the consequence, and asks for the smallest decision needed. "Can you clarify?" is safer than guessing but still poor assistance.

For escalation, require evidence: what was attempted, what is blocked, what risk prevents continuation, and what authorization or information would unblock it.

Learn from production overrides

Collect traces where users cancel an action, deny approval, immediately undo a change, or say the agent did too much. These are strong signals for missing no-action cases. Also sample accepted actions to prevent the dataset from becoming biased toward failures.

Combine this analysis with human-in-the-loop agent evaluation and risk-weighted autonomy tests. The result is an agent evaluated not only for what it can do, but for judgment about when its capability should be used.