Monitoring ML Models: Data Quality and Integrity (cross-section)
Full summary: Monitoring ML Models in Production: Data Quality and Integrity
A Towards Data Science article by Evidently AI co-founders covering input data quality and integrity as the foundation of production ML monitoring, with a five-check framework: model call volume, data schema validation, missing data detection, feature value range/statistics, and per-step pipeline validation.
Run, Monitor, and Maintain Angle
This source directly addresses the data quality monitoring responsibilities of the run-monitor-and-maintain topic area — specifically the ongoing, continuous nature of input validation during production operation. Key contributions to this section:
- Liveness Monitoring via Call Volume: Tracking model requests and responses separately as a basic operational health check; request/response divergence reveals fallback activation or timeout patterns
- Schema Drift Detection in Production: Continuous validation of incoming feature schemas against training expectations — an ongoing operational check, not a one-time deployment gate; schema drift caused by upstream system changes is the most common silent failure mode
- Per-Feature Missing Data Thresholds: Configurable thresholds per feature defining when the model is paused or a fallback is invoked — a key operational decision point in production monitoring
- Fallback and Pause Policies: When data quality violations occur, the operational response options are: fix source (if immediate), pause model, or route to a rule-based fallback — these are run-time operational decisions, not evaluation decisions
Connections
- A Guide to Monitoring ML Models in Production (cross-section) — companion article providing the broader functional/operational monitoring framework; this article provides the detailed implementation of the input data monitoring layer
- Observability Concepts (LangSmith) — LangSmith’s per-run metadata tagging and feedback scoring are the LLM-application equivalent of per-step pipeline validation and feature value monitoring described here