A payment system stopped recording transactions for hours. Every dashboard was green.
I found it by querying the database by hand. Roughly 7 rows in two hours where there should have been thousands. Every dashboard on the platform was green for the entire window, because not one of them was watching the thing the business actually sells.
Pattern·Dashboards measure activity, not outcomesTwo questions, and only one was being asked
The same two hours, measured two different ways.
Every dashboard green. Nothing stale, nothing broken.
expected / 2h
thousands
actual / 2h
7 rows
Transactions were not being recorded.
Found by hand, with a database client, hours in. Nobody was paged, because no alert was watching the outcome.
Is the service running, and did my transaction go through, are different questions.
A core transaction path stopped writing records and ran that way for hours. Nobody was paged. Nobody noticed. I found it the least glamorous way available: I opened a database client and counted rows. Roughly 7 in two hours, where a normal two hours produces thousands.
Every dashboard stayed green for the entire window, and that is the uncomfortable part. They were not stale, broken or misconfigured. They were doing exactly what they had been asked to do. Nobody had asked them to answer the question the customer cared about.
Your monitoring was answering: is the service running. Your customer was asking: did my transaction go through. Those are different questions, and only one of them is the business.
Once I started looking, there was plenty wrong.
- Every insert was wrapped in an explicit exclusive table lock, serialising all writes and deadlocking under any concurrency.
- Retries after those deadlocks produced duplicate key violations against a unique constraint.
- A lookup selected full rows including a large blob column, with no limit, so an oversized batch could exhaust the heap on its own.
- The process was OOM crash looping roughly every four minutes.
- The connection pool leaked after errors until it was exhausted, at which point new work simply timed out.
One write path, eight compounding failures
Each link is the direct cause of the one below it.
Every insert wrapped in an explicit EXCLUSIVE table lock
Taken per row, not per batch. Held for the whole statement.
All writes serialise
The table admits one writer at a time, whatever the concurrency setting says.
Deadlocks under concurrency
Lock ordering collides the moment more than one writer queues behind another.
Retries fire after each deadlock
The identical payload is submitted a second time.
Duplicate key violations
The retry collides with the row the first attempt had already committed, on a unique constraint.
Connections leak after every error
The error path never returns the connection to the pool.
Pool exhausted
There is no connection left to hand out.
New work simply times out
Nothing raised, nothing thrown. The request waits, then gives up.
SELECT of full rows
Including a large blob column, on a lookup that needed neither.
No LIMIT clause
Batch size was whatever the source happened to return.
Heap exhausted
One oversized batch was enough to take the process down.
Out of memory, then crash looping
Roughly one kill every four minutes, for as long as work kept coming in.
Every link was visible in a log. Not one of them was alerted on.
Any one of those faults could have given somebody a breadcrumb, because every one is visible in a log or a metric. None of them paged anyone. More importantly, none of them was needed to detect the outage. One number would have been enough: rows written per minute, compared against the same hour last week.
The five alerts I specified, in the order I would build them.
- Write rate below baseline. This catches every failure mode listed above, and every one I have not thought of, from the customer's point of view rather than the machine's.
- A synthetic end to end transaction pushed through the real ingress on a schedule, committing a real record.
- Crash loops, OOM kills and restart counts on the service itself.
- Error fingerprints for deadlocks, duplicate keys and pool timeouts. These do not detect the outage. They explain it once you are already looking.
- Database connections approaching maximum, and sessions sitting idle in transaction, which is the signature of the pool leak.
Five alerts. Two of them find the outage.
The other three are only useful once you are already looking.
The path a write actually travels
Each alert below is scored on how much of that path it can see.
Write rate below baseline
Rows committed per minute, compared against the expected floor for the time of day.
2 of 5 stages covered
Synthetic end-to-end transaction
Pushed through the real ingress, on the real routing, committing a real record. Alerts when the record does not land.
would have caught this in minute one5 of 5 stages covered
Crash loops, OOM kills, restart counts
Tells you the process is dying and how often.
1 of 5 stages covered — one component, not the outcome
Error fingerprints
Deadlocks, duplicate keys and pool timeouts, counted by shape rather than by line.
1 of 5 stages covered — one component, not the outcome
Connections near maximum
Plus sessions sitting idle in transaction, which is where a leak shows first.
1 of 5 stages covered — one component, not the outcome
Only alert 2 crosses every boundary. It is also the only one that fails when a component is green and the result is still missing.
Everything else watches a component. Only the synthetic watches the outcome.
The synthetic belongs near the top because it tests the path a real customer depends on: ingress, routing, application, database, commit. It is easy to leave that one until later, because it feels heavier to build than a dashboard threshold. In this case it would have caught the failure in minute one. Every other check watches a component. The synthetic watches whether the work happened.
A green dashboard is not evidence that your system is working. It is evidence that the things you chose to measure are inside the ranges you chose for them. Those are not the same claim.
What I put in place that day was a holding pattern, and I said so to the team in those words. Deduplication triggers at the database, a bounded connection pool, a shorter statement timeout, the poller scaled to zero, and a restart. That stopped the bleeding. It fixed nothing. The table lock, the unbounded query and the pool leak are all code, and code is where they have to be fixed.
The finding that outlasted the incident: every other production environment runs the same application at the same memory limit with the same unbounded query. The incident was in one environment. The exposure was in all of them. That is usually true and it is almost never checked during an incident, because during an incident you are trying to make one thing work again.
Three questions worth asking your own team this week.
- Name the single number that tells you your product is doing its job. Is there an alert on it, or only a dashboard?
- If your busiest write path silently stopped right now, how would you find out, and would the first signal be a customer?
- Do you have one synthetic check that goes through the real ingress and commits a real record, rather than pinging a health endpoint?
Nobody on that platform had asked the simplest question available: are transactions being recorded. Every dashboard was green because every dashboard was measuring the machine instead of the work.
Get the next one in your inbox
One short, opinionated field note per fortnight on platform engineering, cloud, and making AI work in production. No spam. Unsubscribe anytime.
