Search RAG Evaluation
Search RAG - Layer-Specific Evaluation
End-to-end answer quality is too late to tell you what broke.
Why this matters in production
A bad answer can come from chunking, enrichment, storage, permissions, query understanding, retrieval, reranking, context design, or generation. If you only score the final answer, every team guesses.
This is usually a mistake: tuning the prompt when the real failure is missing critical chunks.
What breaks
Chunking loses headings. Retrieval misses exact terms. Reranking overweights duplicates. Generation misattributes citations. The final answer simply says 'wrong.'
Without layer-specific metrics, RAG debugging becomes folklore.
What works
Evaluate chunking with coverage and citable completeness. Retrieval with recall, precision, filters, and critical chunk rate. Reranking with lift, diversity, and latency. Generation with grounding, UCR, stability, and refusal correctness.
Do not celebrate a layer metric unless it improves downstream trace quality.
Practical guidance
Build failure routing. When an answer fails, the evaluation system should say whether the source was not ingested, not retrieved, not ranked, not cited, or not supported.
What happened
A team spent a week tuning prompts to fix missing answer details, but the relevant spreadsheet rows had never been chunked correctly.
Why retrieval failed
The ingestion layer treated a table as layout noise, so retrieval never had the evidence.
Why it was hard to detect
Because evaluation looked only at final answers, the failure was misassigned to generation.
What fixed it
The fix was layer-specific tracing: source coverage tests, table chunk validation, retrieval recall checks, and only then generation evaluation.
Practical Guidance
Route failures to layers.
Evaluate chunking before retrieval.
Verify local improvements downstream.