Build a document Q&A chatbot (and evaluate it so it's actually right)
A step-by-step build for a chatbot that answers questions from your documents with RAG — plus the part most tutorials skip: how to know its answers are grounded and correct instead of confidently wrong.
TL;DR: A document Q&A chatbot is a RAG app — index your documents, retrieve the relevant chunks for each question, and have the model answer from them. The build is the easy 80%; the part that decides whether users trust it is evaluation: scoring retrieval and generation separately so a wrong answer points to whether the bot fetched the wrong context or ignored the right one.
"Chat with your PDFs" is the canonical LLM app, and most tutorials stop at "it returns an answer." That's the dangerous point to stop — a fluent wrong answer is worse than no answer, because users act on it. This walks the build and the evaluation that makes it reliable.
Step 1: Ingest and chunk
Load your documents and split them into chunks small enough to retrieve precisely but large enough to carry context — a few hundred to ~1000 tokens with some overlap is a sane default. Chunking quality quietly determines retrieval quality: chunks that split mid-thought retrieve badly no matter how good the rest of the pipeline is.
Step 2: Embed and index
Embed each chunk into a vector that captures its meaning and store it in a vector index. At query time you'll embed the question the same way and find the nearest chunks. (For the concept behind this, see what is retrieval-augmented generation.)
Step 3: Retrieve
For each user question, embed it, fetch the top-k most similar chunks, and optionally rerank them so the most relevant land first. This step is where most "the bot gave a wrong answer" failures actually originate — if the right chunk isn't retrieved, the model can't answer from it, period.
Step 4: Generate with grounding
Insert the retrieved chunks into the prompt and instruct the model to answer only from them, cite its sources, and say "I don't know" when the context doesn't contain the answer. That last instruction is what separates a trustworthy document bot from one that hallucinates when retrieval comes up empty. (See knowledge-base RAG for AI chat.)
Step 5: Evaluate — the step tutorials skip
Now the part that decides whether you can ship it. A document Q&A bot fails in two distinct places, and you score them separately (see RAG evaluation metrics):
- Retrieval quality (contextual relevancy) — did it fetch the right chunks? If not, fix chunking, embeddings, or reranking.
- Answer faithfulness — is every claim in the answer supported by the retrieved context, or did the model make something up? Score with an LLM-as-a-judge that sees both the context and the answer.
- Answer relevancy — did it actually address the question, grounded or not?
Splitting retrieval from generation is what turns "it gave a bad answer" into "the right chunk was never retrieved" or "retrieval was fine, the model ignored it" — two different fixes.
Step 6: Evaluate on real questions, not just yours
Your test questions cover what you thought to ask. Real users ask things your documents don't cover, phrase questions ambiguously, and try to trip the bot. Score production conversations continuously and feed the failures back into your test set. (See run LLM evals on production traces.) Watch performance too — retrieval and reranking add latency. (See debugging a slow RAG pipeline.)
Build vs. evaluation, side by side
| Stage | Failure it hides | Metric that catches it |
|---|---|---|
| Chunk / index | Un-retrievable context | Contextual relevancy |
| Retrieve | Wrong or missing chunks | Contextual relevancy |
| Generate | Hallucination despite good context | Faithfulness |
| Generate | Off-topic answers | Answer relevancy |
| Production | Questions you never tested | Live trace scoring |
How Currai fits
Currai traces the whole Q&A flow — the question, the chunks retrieved, and the answer that used them — so you can score retrieval and generation separately and tell exactly why an answer was wrong. It runs those scores offline and on real production questions with one rubric, ties them to the prompt and model version, and turns hallucinated or unanswered questions into test cases. Your document bot stops being a black box. See RAG evaluation metrics and traces and evals in one place, or start with Currai free.
Frequently asked questions
How does a document Q&A chatbot work?
It's a RAG app: your documents are chunked and embedded into a vector index, each question retrieves the most relevant chunks, and the model answers from those chunks — grounding its answer in your documents instead of its training memory.
Why does my document bot give wrong answers?
Almost always one of two reasons: it retrieved the wrong or no relevant chunks (a retrieval problem), or it had the right context and ignored or misread it (a generation problem). Evaluate the two separately to know which.
How do I stop it from hallucinating?
Instruct the model to answer only from the retrieved context and to say "I don't know" when the context lacks the answer — then measure it with a faithfulness metric that checks every claim against the retrieved chunks.
Do I need to evaluate it after launch?
Yes. Your test questions only cover what you anticipated; real users ask things your documents don't cover and phrase questions in ways you didn't. Score production questions continuously and fold failures back into your test set.