Search RAG Ingestion
Search RAG - Ingestion - Chunking
Chunking is evidence design, not text splitting.
Why this matters in production
The chunk is the unit of evidence the retriever can actually select. If the chunk boundary is wrong, the model receives either a fact without context or context without the fact.
Fixed-size chunking works in demos because demo documents are polite. Production documents are not. They have tables, footnotes, slides, duplicated boilerplate, nested headings, merged cells, and paragraphs that only make sense with the title above them.
What breaks
Under-chunking turns long documents into fuzzy blobs. Over-chunking creates fragments that rank well but cannot support an answer. Both failures produce fluent responses with weak evidence.
This is usually a mistake: optimizing chunk size before deciding what a complete unit of meaning is. Token count is a constraint, not the design principle.
What works
Start with semantic boundaries: headings, sections, list items, table units, slides, speaker turns, and code blocks. Only then apply token limits and overlap.
The retrieval trace should show why a chunk stands alone. If the trace cannot explain what source section a chunk belongs to, the chunk is not ready for production retrieval.
Practical guidance
Evaluate chunks by asking whether a human could cite them. A chunk that cannot support a citation should not be treated as retrieval evidence.
Do not do heavy overlap unless you can measure the storage, ranking, and duplicate-citation cost. Overlap is a boundary repair tool, not a quality strategy.
What happened
A support RAG system answered an incident question with the right troubleshooting step but cited a chunk that only contained the step, not the warning two paragraphs above it.
Why retrieval failed
Chunking split the warning from the procedure. Retrieval selected the action but not the condition that made the action safe.
Why it was hard to detect
The answer looked helpful and the citation existed. The missing adjacent warning only surfaced during a post-incident review.
What fixed it
The fix was section-aware chunking, table/list isolation, and evaluation that scored whether citations contained enough context to justify the claim.
Practical Guidance
Chunk by semantic boundary first.
Score citation completeness, not just retrieval similarity.
Use overlap to repair edges, not mask bad segmentation.