Search RAG Evaluation
Search RAG - Quality Evaluation
If you only evaluate answers, you are debugging blind.
Why this matters in production
RAG quality is a chain: ingestion, retrieval, reranking, generation, permissions, and feedback. A final answer score compresses that chain into one number and hides the failure location.
This works in demos but fails in production: a spreadsheet of good/bad answers with no retrieval trace, no source identity, and no version history.
What breaks
Teams tune prompts to hide retrieval failures. They improve fluency while evidence coverage drops. They ship model changes that pass sample questions but regress critical edge cases.
Without trace-level evaluation, you do not know whether the system found the right evidence, used it, cited it, or merely guessed correctly.
What works
Evaluate the trace: query transform, filters, candidates, reranking, selected chunks, citation mapping, claim support, permission decisions, and output behavior.
Do not treat evaluation as a dashboard afterthought. It is the control system for continuous change.
Practical guidance
Every release should answer: did retrieval improve, did grounding improve, did permissions remain correct, and did failures move somewhere else?
What happened
A prompt update reduced unsupported claims in one benchmark but lowered critical evidence recall in production questions.
Why retrieval failed
The evaluation focused on generated text and ignored the retrieval candidates that no longer reached the model.
Why it was hard to detect
Answer samples looked safer because they were more cautious. The missing evidence only appeared in trace-level recall metrics.
What fixed it
The fix was layer-specific evaluation, regression baselines, and release gates for critical evidence retrieval.
Practical Guidance
Evaluate traces, not only answers.
Version models, prompts, retrieval, and indexes.
Add layer-specific release gates.