Evaluate abstention with cases that have adequate evidence, cases that do not, and cases that need clarification. Score whether the response takes the right action, rather than rewarding a system simply for refusing more often.
An abstention is a task outcome
A policy assistant should answer a question supported by the applicable policy. It should not invent a policy when the collection has no answer. If the missing fact is the customer’s region, clarification may be more useful than a refusal. Write those expected outcomes into the rubric before running the model.
| Case | Expected behavior | Failure to watch |
|---|---|---|
| Applicable evidence exists | Answer within that evidence | Unnecessary abstention |
| No adequate evidence | Acknowledge the gap and escalate or abstain | Unsupported answer |
| Scope is unclear | Ask for the missing information | Unsupported assumption |
Use two error counts
The example contains one answerable question wrongly refused and one unsupported question answered anyway. Reporting only the total refusal count hides both problems. Keep the denominator for each class and inspect high-impact cases independently.
Check beyond the wording
- Verify that an abstaining answer does not also invent a factual claim in its explanation.
- Check that promised escalation actually creates the intended review or support action.
- Repeat tests after changing retrieval, not just after editing the refusal prompt.
- Keep an answerable holdout set so a more restrictive prompt does not appear safer merely by declining everything.
Keep the scope explicit
The fixture compares labels, not natural-language truth. Real evaluation requires reviewers to decide evidence sufficiency and whether the final answer stayed within it. A refusal detector cannot make that decision reliably for every task.
Run the worked example
Save the synthetic Node.js example and run node evaluate-abstention-without-evidence.mjs with Node.js 22 or newer. The assertions check the illustrated records and calculations; they do not test a deployed provider, SDK, or access control. No credentials or network connection are needed.
// Synthetic teaching fixture. No credentials, network calls, or production verification.
import assert from 'node:assert/strict';
const cases = [
{ expected: 'answer', actual: 'abstain' },
{ expected: 'abstain', actual: 'answer' },
{ expected: 'clarify', actual: 'clarify' },
{ expected: 'answer', actual: 'answer' },
];
const unnecessary = cases.filter(c => c.expected === 'answer' && c.actual === 'abstain').length;
const unsupported = cases.filter(c => c.expected === 'abstain' && c.actual === 'answer').length;
assert.equal(unnecessary, 1);
assert.equal(unsupported, 1);
console.log(JSON.stringify({ unnecessaryAbstentions: unnecessary, unsupportedAnswers: unsupported }));
References and further reading
Maintained by DriftRail. AI assisted the October 2026 content and template revision. Examples are illustrative; they are not measured product benchmarks. Suggest a correction.