Search RAG Evaluation
Search RAG - Regression Monitoring
Every RAG change is a retrieval experiment, whether you call it one or not.
Why this matters in production
Model updates, prompt edits, parser changes, embedding migrations, reranker tuning, and ACL logic can all change the answer. Without regression monitoring, improvement is a feeling.
This is usually a mistake: testing only new capabilities and not the boring old questions users depend on.
What breaks
A new model improves summaries but breaks citation format. A parser improves PDFs but drops spreadsheet headers. A reranker improves average relevance but buries rare critical chunks.
Regressions are often local. Global averages hide the workflow that broke.
What works
Version every moving part. Keep a fixed scenario suite with normal, edge, missing-evidence, conflict, permission, and high-risk cases. Compare traces, not only text.
Do not ship if critical evidence recall, permission correctness, or unsupported claim rate regresses, even if overall answer quality looks better.
Practical guidance
Use release gates for trace invariants: expected source present, restricted source absent, citations valid, answer path correct, and latency inside budget.
What happened
A reranker change improved average NDCG but made a legal workflow miss the one clause that mattered.
Why retrieval failed
The metric set rewarded general topical relevance and did not include critical-clause retrieval rate.
Why it was hard to detect
Overall dashboards improved, so the regression looked like an isolated user complaint.
What fixed it
The fix was scenario-specific baselines, critical evidence assertions, and per-workflow release gates.
Practical Guidance
Version everything.
Use scenario-specific baselines.
Gate on critical evidence, not only averages.