Glossary
What is AI Drift Detection?
Detecting when AI behavior changes over time.
What is AI drift?
AI drift occurs when model behavior changes over time, often degrading performance. This can happen due to changes in input data patterns, model updates by providers, or shifts in the real-world phenomena the model was trained on.
Types of Drift
- Data drift: Input distribution changes
- Concept drift: Relationship between inputs and outputs changes
- Model drift: Provider updates change behavior
- Performance drift: Quality degrades over time
Detection Methods
- Track metrics over time (hallucination rate, latency)
- Compare against historical baselines
- Statistical tests for distribution changes
- Alert on significant deviations
Why It Matters
- API models change without notice
- User behavior evolves
- World knowledge becomes outdated
- Early detection prevents user impact
How do I detect drift?
Monitor key metrics over time: hallucination rates, classification distributions, latency, and user feedback. Set baselines and alert when metrics deviate significantly. Compare current performance against historical benchmarks.