Search RAG Evaluation
Search RAG - Metrics and Observability
RAG metrics should tell you where the answer came from, not just whether it sounded good.
Why this matters in production
Latency and thumbs-up rates are not enough. A RAG system needs metrics for evidence usage, unsupported claims, citation support, context waste, answer stability, and permission correctness.
This is usually a mistake: building dashboards around infrastructure health while leaving retrieval health invisible.
What breaks
Teams see response time and user satisfaction but not whether the model used the retrieved chunks. They discover hallucination clusters only after users complain.
Metrics without trace IDs become vanity numbers. You need to click from a metric spike into concrete retrieval traces.
What works
Track CER, UCR, evidence coverage, misattribution rate, critical chunk recall, top-k stability, permission exclusion counts, answer delta, and cost per grounded claim.
Do not aggregate everything into one quality score unless the underlying dimensions remain inspectable.
Practical guidance
Dashboards should show trends and exemplars: not just 'UCR increased,' but which queries, chunks, prompts, and model versions caused it.
What happened
A dashboard showed stable answer ratings while unsupported claim rate climbed after a retrieval-tuning release.
Why retrieval failed
User feedback lagged behind quality drift, and the system did not expose claim-to-evidence mapping in its main dashboard.
Why it was hard to detect
The answers were still useful enough to avoid complaints. The metric spike only appeared after offline audits.
What fixed it
The fix was trace-linked dashboards, unsupported-claim alerts, and sampled examples attached to every metric breach.
Practical Guidance
Instrument claim-to-evidence mapping.
Alert on grounding metrics.
Keep exemplar traces with dashboard spikes.