Evaluate answers when source documents conflict

Evaluate conflicting-source answers by checking whether the system recognizes incompatible evidence and follows an explicit precedence rule. If the applicable rule cannot resolve the conflict, a supported escalation is more defensible than silently choosing the answer that sounds most convenient.

Create a conflict that changes the outcome

The fixture contains two approved policy documents for the same region and period. One allows a return within 30 days; the other says 14 days. Merely citing both documents does not resolve what the customer should do. The expected outcome must state whether a controlling source exists or the case needs review.

Build the test around source authority

1. Record document versions, effective periods, audience, and approval status.
2. Define precedence outside the model, such as a signed policy overriding an informal draft, if that rule actually applies.
3. Include an unresolved case where documents have the same authority and scope.
4. Score the selected outcome, acknowledgement of uncertainty, and supported citations separately.

Avoid averaging contradictory facts

The example returns needs-review for conflicting values and resolved for consistent values. It does not average 14 and 30 into an invented return period. It also does not use publication recency alone as a universal authority rule.

What to preserve when the test fails

Keep the conflicting passages, applicable scope, retrieval result, final answer, and reviewer decision. A retrieval system may correctly surface both documents while generation wrongly presents one as settled. Fix the stage that failed; adding more passages is not necessarily the remedy.

Run the worked example

Save the synthetic Node.js example and run node evaluate-conflicting-source-documents.mjs with Node.js 22 or newer. The assertions check the illustrated records and calculations; they do not test a deployed provider, SDK, or access control. No credentials or network connection are needed.

// Synthetic teaching fixture. No credentials, network calls, or production verification.
import assert from 'node:assert/strict';

const resolve = sources => {
  const values = new Set(sources.map(s => s.returnDays));
  return values.size === 1 ? { outcome: 'resolved', days: sources[0].returnDays } : { outcome: 'needs-review' };
};
const conflicting = [{ version: 'a', returnDays: 30 }, { version: 'b', returnDays: 14 }];
assert.equal(resolve(conflicting).outcome, 'needs-review');
assert.deepEqual(resolve([{ returnDays: 14 }, { returnDays: 14 }]), { outcome: 'resolved', days: 14 });
console.log('Equal-authority conflicting evidence requires review.');

References and further reading

• DriftRail integration documentation

Maintained by DriftRail. AI assisted the October 2026 content and template revision. Examples are illustrative; they are not measured product benchmarks. Suggest a correction.