Your observability stack is theatre. Here are the three signals you are missing.
Most observability stacks measure what is easy, not what is real. Three signals matter. Most teams capture none of them.
Pattern·Dashboards measure activity, not outcomesWhat your dashboard shows vs what is actually breaking
23 observability reviews. Same gap every time.
all green
well within SLO
under threshold
comfortable
stable
All dashboards green
no shadow eval running
no rate-of-change alerts
not measured at all
no dependency map
no freshness metric
The actual problems
The dashboards are green because they were built to be green.
I have reviewed observability stacks at 23 organisations in the last two years. They all have dashboards. They all have alerts. They all have a six-figure annual bill for Datadog, New Relic, Honeycomb, or some self-hosted equivalent.
Almost none of them can answer the question that actually matters: is the system doing the right thing right now?
Uptime is not reliability. Latency is not user experience. Error rate is not correctness. Most observability is theatre because it measures the easy things and prints them on a dashboard so leadership feels safe.
Three signals matter. None of them are in your default observability stack.
Signal one: outcome correctness. Is the system producing the correct result, not just any result? An API that returns 200 with the wrong data is more dangerous than one that returns 500. The wrong data flows downstream and corrupts state. The 500 wakes someone up.
Signal two: silent degradation. The slow drift in metrics that does not trigger any threshold. P99 latency went from 200ms to 240ms over six weeks. Nobody noticed. Now the SLO budget is consumed and one bad deploy will breach it. The signal exists in your data. Your stack is not configured to surface it.
Signal three: blast radius. When something fails, what else fails with it? Most teams find out during the incident, not before. A real observability stack tells you the dependency graph before the failure, not after.
- Outcome correctness: shadow comparison, statistical sampling, ground-truth eval pipelines
- Silent degradation: rate-of-change alerting, anomaly detection on derivative metrics, weekly trend reviews
- Blast radius: service maps from actual traffic, dependency drift detection, chaos drills before incidents
If you have any one of these, you are ahead of 90% of teams. If you have all three, you do not need to read this. You also do not need to hire me.
The dashboards are green because they were built to be green. That is not observability. That is theatre with a six-figure subscription.
Get the next one in your inbox
One short, opinionated field note per fortnight on platform engineering, cloud, and making AI work in production. No spam. Unsubscribe anytime.
