Data-ingestion observability: syncing sources to vector stores reliably
Observe document ingestion from source through parsing, chunking, embedding, indexing, deletion, and vector-store freshness with traceable quality checks.
Data-ingestion observability tracks each document from its source version through parsing, chunking, embedding, and vector-store indexing. Reliability means new content arrives, updates replace old versions, deletions propagate, failures are recoverable, and retrieval can identify exactly which source produced a chunk.
Give every document lineage
Record source connector, tenant, document ID, source version or timestamp, parser, chunker, embedding model, vector index, and job ID. Propagate those identifiers through every stage. Use content hashes to detect duplicates and idempotency keys to make retries safe.
Trace the pipeline
Create a root trace for an ingestion batch and child spans for fetch, parse, normalize, chunk, embed, write, and verify. Record counts, bytes, latency, errors, and safe samples. Never place unrestricted source content or credentials in trace metadata.
Monitor quality and freshness
| Stage | Quality signal |
|---|---|
| Fetch | Expected documents and cursor progress |
| Parse | Text/table extraction completeness |
| Chunk | Size, overlap, boundary preservation |
| Embed | Model/version, dimension, failures |
| Index | Written/updated/deleted counts |
| Verify | Retrieval smoke tests and source match |
Alert on cursor stalls, parse-error spikes, missing deletions, embedding-version mixes, duplicate growth, and freshness lag. A green job that indexed zero updated documents may still be broken.
Handle updates and deletions
Use stable source IDs and versioned writes. Upsert new chunks transactionally where possible, then remove superseded chunks. Tombstones or reconciliation jobs should catch deletions missed during connector downtime.
Test replay, partial failure, and connector reset. The resulting index should be the same after a safe retry.
Connect ingestion to RAG evaluation
When retrieval quality drops, traces should reveal the parser, chunker, embedding, and index versions involved. Run a representative retrieval suite after material pipeline changes and compare recall, latency, and cost.
Currai can receive ingestion and retrieval spans through OpenTelemetry, giving the data pipeline and LLM answer a shared trace vocabulary. Combine it with RAG evaluation metrics so freshness and indexing changes are judged by downstream answer quality.