Test a RAG answer against a correct but irrelevant citation

Test citation relevance at the claim level. A citation can resolve to a real document and still fail to support the statement next to it. Link validity, source authority, and claim support are separate checks.

Use a plausible distractor

The synthetic answer claims that personalized products can be returned. Its link points to a real passage about standard shipping times. The URL works and the document belongs to the right organization, but the passage says nothing about the return claim. A link checker should pass while a support review fails.

Review each material claim

1. Identify the claims that affect the user’s decision rather than treating the answer as one undifferentiated score.
2. Open the cited passage and verify the claim is supported for the relevant scope and period.
3. Record support, contradiction, irrelevance, and unavailable evidence separately.
4. Check whether references are attached to the right claim after formatting, streaming, or summarization.

Keep annotations grounded

The fixture uses explicit, hand-authored support labels to show the evaluation structure. It does not infer semantic entailment from keyword overlap or demonstrate a perfect automated verifier. Reviewers or a calibrated assessment process must supply those labels for real answers.

Measure the right repair

If the correct supporting passage was present but the answer chose an irrelevant reference, inspect generation and citation assembly. If it was absent, inspect retrieval. A rule requiring more citations can increase the number of links without improving the evidence behind any claim.

Run the worked example

Save the synthetic Node.js example and run node evaluate-rag-citation-relevance.mjs with Node.js 22 or newer. The assertions check the illustrated records and calculations; they do not test a deployed provider, SDK, or access control. No credentials or network connection are needed.

// Synthetic teaching fixture. No credentials, network calls, or production verification.
import assert from 'node:assert/strict';

const sources = new Map([
  ['shipping', { supports: ['delivery-window'] }],
  ['returns', { supports: ['personalized-return-exception'] }],
]);
const claim = { criterion: 'personalized-return-exception', citation: 'shipping' };
assert.equal(sources.has(claim.citation), true);
assert.equal(sources.get(claim.citation).supports.includes(claim.criterion), false);
assert.equal(sources.get('returns').supports.includes(claim.criterion), true);
console.log('Citation exists, but the annotated passage does not support this claim.');

References and further reading

• DriftRail integration documentation

Maintained by DriftRail. AI assisted the October 2026 content and template revision. Examples are illustrative; they are not measured product benchmarks. Suggest a correction.