The DriftRail knowledge base
Understand the system.
Know what to watch.
Field notes on observing, evaluating, and protecting AI in production. Clear answers, practical checks, and the limits that matter.
The reading room / 06
Start with a question.
Invalidate cached answers after a policy update
Use policy-version dependencies to reject cached answers that still refer to a superseded policy, even when their time-to-live has not expired.
Test tenant isolation in semantic caching
Check that similar questions from different tenants cannot reuse private answers, including on high-similarity cache hits.
Review what AI telemetry retains after redaction
Inspect original payloads, stored records, exports, and error paths to distinguish display masking from data removal before storage.
Restore context after an interrupted agent task
Resume from durable action outcomes and stable operation identities rather than replaying an external action from a conversational summary.
Test prompt injection through a tool error message
Treat tool error text as untrusted content and test whether an embedded instruction can influence an unauthorized follow-up action.
Verify authorization before a model-selected tool call
Check the authenticated actor against the requested resource at the tool boundary, rather than trusting account IDs proposed by a model.
Require confirmation before an AI agent sends a report
Bind approval to the reviewed recipient and report revision, then reject changed or expired approvals before an external send.
Stop an AI agent that repeats a failing tool call
Use operation fingerprints, failure categories, and explicit step and duration budgets to terminate unproductive agent loops.
Test retrieval access filters before model context assembly
Verify that unauthorized retrieved passages never enter the model context, even when they are highly relevant to the query.
Evaluate chunk boundaries around policy exceptions
Test whether retrieval preserves the exception that changes an otherwise plausible policy answer, rather than scoring only the general rule.
Handle stale and current documents in a retrieval corpus
Select policy evidence by the period and scope the question concerns, preserving historical versions without treating every newest document as applicable.
Test a RAG answer against a correct but irrelevant citation
Distinguish an existing source link from a source that actually supports the answer’s material claim.
Choose an AI trace retention policy by record purpose
Build a purpose-based retention inventory for event metadata, prompt content, review records, and temporary diagnostic data.
Prepare evidence of an enforced AI access boundary
Assemble an access-control evidence record that includes the attempted request, authenticated identity, denial, and resulting resource state.
Record a human review decision in an AI workflow
Connect reviewer identity, decision, artifact revision, and resulting application action without treating a review click as completion of the whole task.
Document a change from AI advice to automated action
Review the new permissions, effects, confirmations, and recovery paths when an assistant moves from suggesting work to performing it.
Measure LLM cost per completed workflow
Calculate complete-workflow cost with failed attempts and retries included, and report completion criteria and unmeasured cost categories.
Evaluate exact caching before semantic caching
Compare exact and semantic cache reuse on equivalent and deceptively similar questions while checking permission and freshness dependencies.
Test a cheaper model route without hiding quality loss
Compare routing candidates by accepted outcomes, escalation, total attempt cost, and hard-case performance rather than per-call price alone.
Set different latency budgets for chat and batch extraction
Choose completion milestones and latency objectives separately for interactive answers and asynchronous document-processing jobs.