Compare LLM releases on the same task set or with an explicitly fixed task weighting. When the traffic mix changes, an overall quality score can fall even if performance improves within every measured task group.
A misleading overall decline
Imagine release A handling 90 easy cases and 10 difficult cases. Release B handles 10 easy cases and 90 difficult cases. B improves both easy-case and difficult-case correctness, yet its unadjusted overall result is lower because it sees more difficult work. The fixture makes that reversal visible.
Keep the comparison interpretable
- Define task groups using stable rules before looking at release outcomes.
- Record successes, reviewed counts, unlabeled cases, and actual traffic share per group.
- Compare each group and calculate a fixed-mix summary only where both releases have evidence.
- Report the chosen weights alongside the real traffic-weighted result; they answer different questions.
Do not weight away a serious failure
A summary can still hide rare but consequential cases. Track those as explicit release criteria even if their traffic weight is small. Keep model, prompt, tools, and retrieval versions attached to the test so a result can be reproduced.
What the synthetic numbers establish
The example calculates actual-mix and equal-weight results from invented counts. It illustrates why denominators matter; it is not a significance test, production benchmark, or reason to choose equal weights in every product. Select weights that reflect the decision and retain uncertainty for small groups.
Run the worked example
Save the synthetic Node.js example and run node compare-releases-with-a-fixed-task-mix.mjs with Node.js 22 or newer. The assertions check the illustrated records and calculations; they do not test a deployed provider, SDK, or access control. No credentials or network connection are needed.
// Synthetic teaching fixture. No credentials, network calls, or production verification.
import assert from 'node:assert/strict';
const a = { easy: { correct: 81, n: 90 }, hard: { correct: 2, n: 10 } };
const b = { easy: { correct: 10, n: 10 }, hard: { correct: 27, n: 90 } };
const raw = x => Object.values(x).reduce((n, c) => n + c.correct, 0) /
Object.values(x).reduce((n, c) => n + c.n, 0);
const fixed = x => (x.easy.correct / x.easy.n + x.hard.correct / x.hard.n) / 2;
assert.ok(raw(b) < raw(a));
assert.ok(fixed(b) > fixed(a));
console.log(JSON.stringify({ actualMix: { a: raw(a), b: raw(b) }, fixedMix: { a: fixed(a), b: fixed(b) } }));
References and further reading
Maintained by DriftRail. AI assisted the October 2026 content and template revision. Examples are illustrative; they are not measured product benchmarks. Suggest a correction.