Six services down. 21,000 cache errors. The cache was idle the whole time.

The ticket came in already wrapped in a neat story: six services down, infrastructure. Six customer environments returning 502. Events stopped integrating into downstream records. The application had logged the same error roughly 21,000 times, and the error named the cache.

Twenty-one thousand cache errors will make most of us look at the cache first, and that is reasonable. It gives you a component name, a direction of travel and a team to page. But a good-looking error message is still only a witness statement, and this one was wrong.

Before touching anything I pulled the managed cache metrics for the exact burst window.

• Engine CPU utilisation peaked under 1 percent.
• Current connections sat flat at around 80, with no drop and no spike.
• No failover, no eviction, no memory pressure, no disconnections.

The cache was idle, healthy and continuously connected for the exact window that mattered. That is the point where the ticket should change shape. If the dependency never saw the pressure, then those 21,000 cache errors are not the cache describing its own failure. They are the client telling you it could not get far enough to make the call.

The cause was one image build performing synchronous I/O on asynchronous request paths. A blocking flush on a stream, inside a request handler. The runtime is explicit about this and throws on it, but the damage is done before the exception: under load, blocking calls on async paths consume thread pool threads and do not return them.

The chain from there is mechanical.

• A blocking call on an async path takes a thread pool thread and holds it.
• The pool starves. Every outbound call now waits on a thread that is not coming.
• Cache calls time out and are logged as cache errors. Twenty-one thousand of them.
• The heartbeat endpoint times out too. nginx logs upstream prematurely closed.
• The ingress has no healthy backend. Customers get 502.

The pattern looked broad because six customer environments failed together, but the shared thing was not the cache, the database or authentication. Each namespace had its own copy of all three, and all three were healthy. The shared thing was the image. Anomaly detection then made the suspicion concrete: one build version accounted for 100 percent of the cache errors, which moved the investigation out of infrastructure and into release state.

The more useful lesson is why those six were still on that build. Image tags had changed format, from bare integers to semantic versions, and the automated image updater had silently stopped advancing anything left on the old format. It did not fail. It did not alert. It still looked present and healthy. For months, six production customer environments sat behind automation that everybody believed was doing its job.

Three questions for your own estate.

• When your application logs errors naming a dependency, does anyone check that dependency's own metrics for the same window before an infrastructure ticket gets opened?
• If your image updater stopped advancing tags tomorrow, what would tell you, and how long would it take?
• Can you answer which build version is producing a given error, for every service, right now?

That last one is not rhetorical. Several services in that estate run a floating latest tag whose digest matches no numbered release. That tag reaches the observability platform as the service version, which makes the version dimension meaningless for exactly the services running it. Which build is erroring was the question that cracked this incident. On those services it is unanswerable.