Multi-agent RAG evaluation for trusted financial data retrieval
Evaluate multi-agent RAG systems for financial data by separating retrieval, routing, calculations, citations, synthesis, and policy compliance.
Multi-agent RAG evaluation measures more than whether the final financial answer sounds plausible. It tests which agent handled the request, which records were retrieved, whether dates and entities matched, how calculations were performed, and whether every material claim is supported by an authoritative source.
Financial retrieval is unforgiving because a correct number from the wrong period, filing, currency, or entity is still wrong. This article describes an engineering evaluation method, not investment advice.
Decompose the multi-agent RAG system
A typical system may include a router, filing-retrieval agent, market-data agent, calculation tool, policy agent, and synthesis agent. Evaluate each boundary.
| Component | Primary failure | Evaluation signal |
|---|---|---|
| Router | Wrong specialist selected | Routing accuracy by intent |
| Retrieval | Wrong document, entity, or period | Recall and metadata match |
| Extraction | Value detached from unit or table | Field-level correctness |
| Calculation | Incorrect formula or stale input | Deterministic recomputation |
| Synthesis | Unsupported or contradictory claim | Citation entailment |
| Policy | Advice or disclosure violation | Rule-based and human review |
The final answer can pass while one intermediate step is fragile. Trace every agent and tool as a child span so evaluators can score the component responsible, not just the prose at the end.
Build examples around financial identity
Each golden example should identify the entity, requested metric, reporting period, source class, unit, and expected result. Store acceptable source IDs and calculation steps separately from the natural-language reference answer.
Include hard cases:
- similarly named issuers and ticker changes;
- fiscal versus calendar periods;
- restated filings;
- values reported in thousands versus millions;
- multiple currencies and conversion dates;
- missing data that should trigger abstention;
- questions that cross sources or require calculation.
Use authoritative records wherever possible. In the United States, the SEC's EDGAR system is the primary source for company filings. Market prices and derived datasets require their own source, timestamp, and licensing policy.
Evaluate retrieval before generation
Retrieval evaluation asks whether the necessary evidence entered the context. Measure document recall, precision, entity match, period match, and source authority. Chunk similarity alone is insufficient: two tables may use the same words while reporting different quarters.
Attach retrieval results and metadata to the trace. When an answer is wrong, you can then separate “the model ignored good evidence” from “the right evidence was never retrieved.” The same principle appears in our guide to RAG evaluation metrics.
Test calculations deterministically
Do not ask an LLM judge whether arithmetic looks right. Capture the structured inputs and formula, recompute them in code, and compare within an explicit tolerance. The trace should show the source value, unit normalization, formula, and result.
For ratios and growth rates, evaluate sign, denominator, period alignment, and rounding independently. A correct-looking percentage can conceal a denominator from the wrong year.
Score citations and synthesis
For each material sentence, check that a cited source exists, contains the claim, and refers to the correct entity and period. An LLM judge can help classify semantic entailment, but calibrate it with analysts and keep deterministic checks for missing citation IDs or invalid sources.
Then evaluate synthesis: does the answer reconcile conflicting sources, state important uncertainty, preserve units, and abstain when evidence is incomplete? Avoid rewarding verbosity. The goal is a correct, traceable answer.
Evaluate routing and handoffs
Multi-agent systems add failure modes at handoff boundaries: dropped constraints, duplicated work, circular delegation, and bloated context. LangChain's official multi-agent router documentation distinguishes routing from supervisor-style orchestration; your dataset should cover whichever pattern you use.
Track selected agents, handoff count, step count, tokens, and completion status. Set budgets and flag loops. A route can be semantically reasonable yet operationally wasteful.
Connect offline and production evaluation
Run the full suite before release, then sample production traces by intent and risk. Send low-confidence, high-value, or policy-sensitive cases to qualified reviewers. Promote confirmed failures into the offline set with identifying data removed.
Currai keeps retrieval spans, tool inputs, model calls, citations, tokens, and evaluation metadata under one trace. That makes trusted financial RAG a property you can inspect and regress-test—not a promise based on a polished final answer.