Evaluation method
How to test what a RAG system does when the answer is not in your documents, when the only source is out of date and when two sources disagree.
Updated 21 Aug 2026
Definition
Abstention is a defined system behaviour under insufficient evidence; it is not a universal promise that a model will know when it is wrong.
For each test, specify whether the system should decline, qualify, request clarification, show the conflict or route to a human. Scoring only for the words "I don't know" will miss most of what matters.
Build five sets of questions, each one isolating a single reason the evidence is insufficient, so that when it declines you know which one caused it.
Inspect what was retrieved before judging the response. A decline after retrieval failed and a decline because no source exists are different problems.
Model, prompt, retrieval settings and document set updates can change behaviour. Record configuration and rerun critical cases rather than treating one response as stable.
Pick a real piece of work, agree what a good answer looks like, then go through the results together.