Search RAG Ingestion
Search RAG - Ingestion - Storage
Store chunks for how they will fail, not just how they will be searched.
Why this matters in production
Storage is where retrieval observability is either preserved or lost. If you store only text and vectors, you will not be able to explain why a chunk was retrieved, whether it was allowed, or which parser version produced it.
This is usually a mistake: choosing a vector database and calling the storage problem solved. Search RAG storage needs vectors, lexical fields, metadata, permissions, source lineage, enrichment versions, and deletion semantics.
What breaks
Document-level storage retrieves too much. Chunk-only storage can lose source context. Nested storage can be powerful but expensive to score correctly. Missing version fields make stale answers almost impossible to debug.
A storage model that cannot answer 'why was this evidence eligible and current?' is not production-ready, even if nearest-neighbor search is fast.
What works
Store chunk-level records for retrieval and document-level records for lifecycle, display, and provenance. Keep dense vectors, sparse fields, generated queries, quality scores, permissions, timestamps, and source anchors together.
Do not do this unless you can trace updates: storing enriched chunks without source version and parser version. It creates silent drift where old enrichment survives new documents.
Practical guidance
Design storage around updates and deletion. Production corpora change constantly. If removing a document does not remove its chunks, summaries, vectors, generated queries, and cached retrieval entries, the system will eventually cite ghosts.
What happened
A team deleted an outdated policy PDF, but answers kept referencing policy language for two weeks.
Why retrieval failed
The source document was removed, but its generated queries and chunk embeddings remained in a secondary index.
Why it was hard to detect
Search results did not show the deleted PDF directly. The stale evidence appeared only as answer language and weak citations.
What fixed it
The fix was source-linked deletion, index consistency checks, and storage tests that verified every derived artifact disappeared with the source.
Practical Guidance
Store source lineage for every derived field.
Test deletes as seriously as inserts.
Keep lexical, vector, metadata, and permission fields together.