Search RAG Ingestion
Search RAG - Ingestion
Most RAG systems fail before retrieval starts, because ingestion quietly destroys the evidence.
Why this matters in production
In demos, ingestion looks like a preprocessing job. In production, it is evidence design. Every bad chunk, missing heading, stale ACL, duplicate document, or lost source reference becomes a future retrieval bug.
This is usually a mistake: treating ingestion as an ETL concern owned by a background job. Search RAG needs ingestion to preserve meaning, permissions, provenance, freshness, and update history. Otherwise the retriever only has polished garbage to choose from.
What breaks
Dirty metadata breaks filters. Duplicate files break ranking. Stale indexing breaks trust. Alias explosion makes the same concept look like several unrelated things. None of this looks like a model problem until users start receiving confident answers from the wrong evidence.
The dangerous case is a correct answer from the wrong source. It feels successful, but the retrieval trace is already telling you the system is unsafe.
What works
Treat ingestion as the first retrieval layer. Normalize sources, chunk by meaning, enrich with query-facing signals, attach permissions early, and keep source provenance attached to every derived field.
Do not ship ingestion changes without retrieval regression tests. A parser tweak can look harmless while dropping table headings, page numbers, or ACL fields that downstream systems rely on.
Practical guidance
Design the ingestion output around the retrieval trace: chunk ID, source ID, heading path, permissions, timestamp, content type, enrichment version, and parser version. The answer is not the product. The trace is the product.
What happened
A team reindexed a knowledge base after changing the PDF parser. Answers still looked fluent, but product-limit questions started citing old appendix pages instead of current policy tables.
Why retrieval failed
The parser kept body text but lost table captions and page-level metadata, so retrieval matched semantically similar fragments without knowing which evidence was authoritative.
Why it was hard to detect
The answer text was often right enough. Only citation review exposed that the sources were stale or weak.
What fixed it
The fix was parser regression tests, mandatory source metadata, table-aware chunking, and retrieval tests that checked evidence identity, not only answer text.
Practical Guidance
Keep a retrieval trace for every ingested chunk.
Test ingestion changes against real retrieval questions.
Do not let derived summaries outrank source provenance.