How to evaluate text-to-SQL and natural-language-to-SQL agents

Text-to-SQL evaluation should determine whether an agent returns the correct result safely, not whether its query text matches a reference string. Equivalent SQL can use different joins, subqueries, or aggregations. Execute candidates in a controlled database and compare normalized results while inspecting schema use, permissions, and resource limits.

Build realistic questions and databases

Include simple filters, joins, grouping, time windows, nested queries, ambiguous business terms, missing data, and requests the user is not authorized to answer. Pin the database snapshot and schema version for reproducibility.

Store each example's natural-language question, user role, expected result or properties, acceptable assumptions, and clarification requirements. Avoid leaking the reference SQL into model context.

Use layered metrics

Metric	What it catches
Execution success	Syntax and runtime errors
Result equivalence	Semantically wrong answers
Schema grounding	Invented tables or columns
Safety	Writes, broad scans, unauthorized access
Clarification	Guessing ambiguous definitions
Efficiency	Slow plans, tokens, latency, and cost

Exact SQL match is useful only as a supplementary signal. Compare rows with defined ordering, numeric tolerance, null handling, and duplicate semantics.

Trace the agent path

Record schema retrieval, table selection, generated SQL, validation, execution, error feedback, retries, and answer generation as child spans. Never place live credentials or unrestricted query results in telemetry.

When a run fails, the trace separates bad schema retrieval from bad SQL or bad result explanation. Attach database and prompt versions to the root.

Test safety before execution

Use read-only credentials, query parsing, allowlists, row limits, timeouts, and a sandbox or replica. Reject multiple statements and prohibited operations. Test prompt injection inside schema comments or cell values; retrieved database text is untrusted input.

Deterministic evaluators should enforce these rules. Do not rely on a model judge to notice a destructive statement.

Evaluate business semantics

Many failures come from definitions rather than SQL. “Active customer” or “revenue” may have an organization-specific meaning. Test whether the agent retrieves metric definitions and asks for clarification when several are valid.

Use experts to label ambiguous cases and calibrate semantic judges. Score the final explanation against the executed result so prose cannot overstate what the query proved.

Monitor production safely

Track execution errors, repeated retries, denied queries, expensive plans, user corrections, and result-quality evals. Sanitize representative failures before adding them to the offline suite.

Currai can trace retrieval, generation, validation, and execution while recording tokens, latency, and cost. That evidence turns text-to-SQL from a demo judged by query appearance into a system evaluated by correct, authorized outcomes.