AI chat projects: build a knowledge base with RAG

TL;DR: An AI chat project is a persistent workspace for related conversations. It combines a document knowledge base, project-level instructions, durable memory, and grouped chats. Retrieval-augmented generation, or RAG, retrieves the small parts of uploaded files that matter for each question instead of sending every document to the model on every message.

A useful implementation needs more than file upload and vector search. It needs source citations, document freshness, permission checks, editable memory, no-answer behavior, and evals for retrieval and groundedness. This guide covers the full design.

What is an AI chat project?

Every standalone AI chat begins with a context problem. The user has to paste the same product brief, upload the same handbook, and explain the same decisions again. Long conversations eventually become expensive or lose the details that matter as older context is summarized or removed.

An AI chat project changes the unit of organization from one conversation to one body of work. A project can represent a customer, product launch, research topic, codebase, policy library, or any other context that should survive across multiple chats.

A complete project has four parts:

1. Knowledge base: uploaded documents that the assistant can retrieve when answering questions.
2. Project memory: durable facts, preferences, and decisions established during conversations.
3. Instructions: guidance that applies to every chat in the project.
4. Conversations: related chat histories grouped under the same workspace.

The model can change without changing the project. Knowledge, memory, and instructions belong to the application layer. The selected model receives the relevant context when it is asked to produce an answer.

Chat attachment vs project knowledge base

Attaching a document to one conversation is useful for a temporary task. A project knowledge base is designed for repeated use.

The difference becomes important as documents grow. Sending an entire handbook for every follow-up wastes context and gives the model more irrelevant text to sort through. RAG pays an indexing cost once, then retrieves a smaller set of passages for each question.

That does not make RAG automatically cheaper or better. Embeddings, vector search, reranking, and extra model calls also cost time and money. The benefit comes from selecting useful context reliably instead of repeatedly sending the whole corpus.

How a project knowledge base works

The system has two paths: document ingestion and question answering.

Document ingestion

When a user uploads a file, the application should:

1. Validate the file type, size, project permission, and malware policy.
2. Extract text and structural metadata.
3. Split the content into retrievable chunks.
4. Create an embedding for each chunk.
5. Store the vector with its project, document, and source metadata.
6. Mark the document ready only after the full index succeeds.

Store the original file separately from the retrieval index. The original is needed for download, reprocessing, audits, and future extraction improvements. The index is a derived representation that should be safe to rebuild.

Question answering

For each user message, the application should:

1. Confirm that the user can access the project and its documents.
2. Rewrite or enrich the query when conversation context is required.
3. Embed the retrieval query.
4. Search only vectors authorized for the project.
5. Optionally rerank the candidate chunks.
6. Assemble instructions, memory, recent conversation context, and retrieved passages.
7. Generate the answer with citation identifiers attached to the evidence.
8. Validate that every citation maps to a real source before returning it.

The core flow can be represented without tying it to a particular model or vector database:

const query = await buildRetrievalQuery({ message, recentMessages });
const queryVector = await embed(query);

const candidates = await vectorStore.search({
  projectId,
  vector: queryVector,
  limit: 20,
});

const passages = await rerank({ query, candidates, limit: 6 });

const answer = await generate({
  instructions: project.instructions,
  memory: project.memory,
  messages: recentMessages,
  context: passages.map(toCitationContext),
});

Retrieval should run for every message that could require project knowledge. Reusing only the passages from the first question makes follow-ups depend on an old retrieval decision, even when the topic changes.

Parse PDFs, spreadsheets, documents, and code carefully

Different file types need different extraction strategies. Treating every upload as plain text loses the structure that helps retrieval and citations.

PDFs: Preserve page numbers, headings, and reading order. Image-only or scanned PDFs need OCR. Tables often require a layout-aware extractor because a simple text pass can scramble rows and columns.

Spreadsheets: Extract each sheet separately and preserve the workbook, sheet, row range, headers, and formulas where relevant. Converting the entire workbook into one text block produces weak retrieval and unusable citations.

Word-processing documents: Preserve headings, lists, tables, footnotes, and page or section locations when available.

Markdown and source code: Split around headings, modules, classes, functions, or other semantic boundaries. A fixed character window can separate a function signature from its implementation or a heading from its content.

CSV and JSON: Retain field names and record boundaries. Large datasets may need query tools instead of semantic retrieval; embedding millions of rows is usually not a substitute for database access.

Logs: Preserve timestamps, severity, service, request identifiers, and multiline event boundaries. Redact secrets before indexing.

Each chunk should carry enough metadata to answer “where did this come from?” even after the source file is replaced.

Choose a chunking strategy that preserves meaning

Chunking controls the evidence available to retrieval. Chunks that are too small lose context. Chunks that are too large introduce irrelevant text and consume more model tokens.

Start with semantic boundaries:

• Sections and paragraphs for prose.
• Pages or table regions for PDFs.
• Sheets and row groups for spreadsheets.
• Functions, classes, and modules for code.
• Records or logical groups for structured data.

Add limited overlap when a concept regularly crosses boundaries. Avoid large overlap by default: it increases index size, creates near-duplicate results, and can make one document dominate the retrieved context.

Store the chunking version with every index entry. When the strategy changes, create a new index version, test it, and switch only after it performs better. Silent partial reindexing makes results inconsistent and difficult to debug.

Build source citations into the retrieval contract

A citation should come from application metadata, not from the model's memory. Each retrieved passage needs a stable identifier and a source record such as:

type CitationSource = {
  chunkId: string;
  documentId: string;
  documentVersion: string;
  fileName: string;
  page?: number;
  sheet?: string;
  rowRange?: string;
  section?: string;
};

Pass citation identifiers beside the retrieved text and instruct the model to reference only those identifiers. After generation, verify each identifier and render the human-readable filename, page, sheet, or section in the interface.

This prevents several common failures:

• Citations to files that were never retrieved.
• Real filenames paired with unsupported claims.
• References to deleted or superseded document versions.
• Citations that name a file but do not help the user find the evidence.

Citation presence is not enough. Evaluate whether the cited passage actually supports the claim.

Keep project memory separate from document knowledge

Knowledge and memory answer different questions.

The knowledge base contains what uploaded sources say. Project memory contains facts established by the user or conversation: preferred terminology, active markets, project constraints, decisions, names, or formatting preferences.

Keeping them separate provides clearer control. A user can replace a policy document without deleting the decision that answers should use concise bullet points. They can also delete a mistaken memory without touching the document index.

Useful memory controls include:

• Manual add, edit, and delete actions.
• A visible source showing whether memory was user-authored or automatically extracted.
• Deduplication and conflict detection.
• Limits on the number and size of memories.
• Project-level retention and deletion behavior.
• A rule that sensitive data is never saved automatically without consent.

Automatic memory extraction should run outside the critical response path. If it fails, the answer should still succeed. New memories should be short, specific, and reviewable rather than compressed summaries of entire chats.

Project instructions should be durable and testable

Project instructions can specify audience, tone, output format, citation rules, language, scope, and how to behave when evidence is missing.

A useful instruction set might say:

Answer from project documents when the question concerns company policy.
Cite the supporting file and location for factual claims.
If the documents do not contain an answer, say so explicitly.
Do not infer policy from project memory.

Treat instructions as versioned product configuration. A wording change can alter citation behavior, refusal rates, answer length, and how often the model uses retrieved context. Record the version that produced each answer so changes can be evaluated and rolled back.

Design no-answer behavior before launch

The most trustworthy answer is sometimes “the project files do not say.” A knowledge-base assistant should not convert weak retrieval into a confident guess.

Define behavior for:

• No chunks above the relevance threshold.
• Conflicting documents or document versions.
• A question outside the project's domain.
• Retrieved passages that mention the topic but do not answer the question.
• Sources the user can no longer access.
• Files that are uploaded but still processing.

The fallback can ask a clarifying question, offer the nearest sources, suggest which document is missing, or answer from general model knowledge with an explicit label. Do not silently mix retrieved evidence and general knowledge when the user expects a grounded answer.

Secure the knowledge base by project and user

Authorization must be part of retrieval, not a filter added after vector search. Every chunk should carry a tenant and project boundary, and the search query should enforce both before candidates are returned.

At minimum, cover:

• Tenant and project isolation.
• File-level permissions when project members have different access.
• Encryption in transit and at rest.
• Malware scanning and file validation.
• PII, credential, and secret detection before indexing.
• Audit records for upload, access, replacement, export, and deletion.
• Retention rules for originals, chunks, embeddings, chats, and memories.
• Deletion that removes both the source and all derived index entries.

Do not assume embeddings are harmless metadata. They are derived from private content and should inherit the source document's security and retention policy.

Make the knowledge base model-independent

RAG should happen before generation. That lets the application use the same authorized passages with different chat models.

Model independence is useful for:

• Routing simple questions to faster or cheaper models.
• Using deeper reasoning for complex synthesis.
• Comparing answer quality without rebuilding the index.
• Switching providers without migrating project data.
• Falling back during provider outages.

The application still needs model-specific prompt formatting, context limits, and citation testing. A model switch can change how faithfully evidence is used, even when retrieval stays identical.

Evaluate retrieval and generation separately

A final answer can fail because retrieval found the wrong passage or because the model ignored the right one. One overall score cannot distinguish those causes.

Evaluate the retrieval layer with:

• Recall: did the result set contain the evidence needed to answer?
• Relevance: how much of the retrieved context was useful?
• Ranking quality: did the strongest evidence appear near the top?
• Freshness: did retrieval use the current document version?
• Permission correctness: were all returned chunks authorized?

Evaluate the answer with:

• Groundedness: are important claims supported by retrieved evidence?
• Citation correctness: does each citation support the associated claim?
• Completeness: did the answer cover the relevant evidence?
• No-answer accuracy: did the assistant decline when evidence was missing?
• Instruction adherence: did it follow project rules and formatting?

Operational metrics matter too: upload failures, indexing time, retrieval latency, generation latency, time to first token, context tokens, embedding cost, generation cost, empty searches, reranking errors, and citation-validation failures.

Build an eval dataset from real project questions and known source passages. Include ambiguous questions, conflicting documents, deleted files, scanned PDFs, large spreadsheets, and questions that the knowledge base cannot answer.

Frequently asked questions

What is a knowledge base for AI chats?

It is a managed collection of documents that an AI chat application can search for relevant evidence before generating an answer. In a project, the same knowledge base can support multiple authorized conversations.

How is a project knowledge base different from attaching a file?

A file attachment is typically scoped to one conversation and may consume a large amount of context repeatedly. A project knowledge base indexes files once, retrieves relevant passages for each message, and makes the corpus reusable across chats.

Which files can an AI project knowledge base support?

Common formats include PDFs, Word documents, spreadsheets, Markdown, source code, CSV, JSON, HTML, XML, and text files. Scanned PDFs need OCR, and complex tables or spreadsheets need structure-aware extraction.

Does RAG reduce AI chat context cost?

It can. RAG sends selected passages instead of entire files, which often reduces generation context. The complete cost also includes parsing, embeddings, vector search, optional reranking, and retrieval infrastructure.

How should project memory work?

Project memory should store short, durable facts and preferences separately from uploaded documents. Users should be able to see, edit, and delete memories and understand whether each one was manually added or automatically extracted.

Can one project knowledge base work with different AI models?

Yes. Retrieval can run independently of the chat model, allowing the same authorized passages to be sent to different models. Each model should still be tested for groundedness, citations, context limits, and instruction adherence.

Observe your AI chat knowledge base with Currai

Once an AI chat project reaches production, the difficult question is no longer whether RAG works in a demo. It is why a specific answer was wrong, slow, expensive, or based on the wrong document.

Currai traces the full custom RAG path: the user question, query embedding, vector search, retrieved chunks, reranking, prompt, model generation, citations, latency, tokens, cost, and errors. Retrieval and generation live as separate spans, so teams can see whether the knowledge base missed the evidence or the model ignored it.

Production traces can then become eval cases for retrieval relevance, groundedness, citation correctness, and no-answer behavior. That turns real knowledge-base failures into regression tests for the next chunking strategy, embedding model, prompt version, or chat model.

Start tracing with Currai, read how to debug a slow RAG pipeline, or build hallucination evaluations for grounded answers.