Search RAG Generation
Search RAG - Generation Evaluation
If you only evaluate answer correctness, you are grading luck.
Why this matters in production
A correct answer from unsupported evidence is not reliable. A slightly incomplete answer from correct evidence may be the safer system behavior.
This is usually a mistake: asking reviewers whether the answer is good without showing the retrieval trace.
What breaks
Evaluators reward fluency. They miss unsupported claims, citation misattribution, context overuse, and instability under small retrieval changes.
Generation evaluation must inspect the relationship between claims and evidence, not only the final text.
What works
Measure unsupported claim rate, evidence coverage, context efficiency, answer stability, citation accuracy, refusal correctness, and quality per token.
Do not do model evaluation separately from retrieval evaluation. The generated answer is the visible symptom; the trace explains the disease.
Practical guidance
Keep claim-to-chunk mapping in logs. Without that, grounding evaluation becomes manual archaeology.
What happened
An answer-quality benchmark improved after adding more retrieved chunks, but production users saw longer, less stable answers.
Why retrieval failed
The benchmark rewarded final-answer completeness and ignored context efficiency and answer delta.
Why it was hard to detect
The degradation looked like model verbosity, but the trace showed redundant chunks driving inconsistent synthesis.
What fixed it
The fix was adding context efficiency ratio, answer stability tests, and over-context penalties to release gates.
Practical Guidance
Evaluate trace-level grounding.
Measure context efficiency.
Reward safe partial answers.