Why every AI incident becomes a cross-team incident
Three teams own different parts of the AI pipeline. Each team's monitoring shows green. The failure exists in the interaction between components.
At 2am, the AI classification service starts returning incorrect results. The on-call SRE gets paged because error rates crossed the threshold.
The SRE checks the infrastructure. Kubernetes pods are healthy. CPU and memory are nominal. Network is fine. The service is returning 200 OK. The errors are not infrastructure errors. They are wrong answers that downstream systems are flagging.
The SRE pages the ML team. The ML team checks the model. Model version is correct. Inference latency is normal. They run a test batch against known inputs. Results look fine. The model is working.
Meanwhile, the data engineering team is asleep. Nobody has checked the data pipeline.
A schema change in a source system caused the data pipeline to silently drop a field. The model was receiving incomplete inputs and producing confident but wrong outputs. The confidence scores were high enough that no quality threshold was triggered.
This is the pattern I see in every AI incident. Three teams own different parts of the pipeline. Each team's monitoring shows green for their component. The failure exists in the interaction between components, which nobody monitors.
- A single on-call rotation that covers the full AI pipeline, not just infrastructure
- End-to-end integration tests that validate the complete path from data source to model output
- Shared dashboards that show data freshness, model input quality, and output accuracy in one view
- Incident runbooks that start with "check data freshness" before "check the model"
- Regular cross-team game days that simulate data quality failures, not just service outages
The organisation that gets AI incident response right is the one that stops treating it as three separate systems owned by three separate teams. It is one pipeline. It needs one owner.
Get the next one in your inbox
One short, opinionated field note per fortnight on platform engineering, cloud, and making AI work in production. No spam. Unsubscribe anytime.
