Four theories. Four measurements. The obvious fix would have destroyed a million jobs.
Redis at its memory ceiling, login down, millions of keys with no expiry. The obvious fix was to delete the keys with no TTL. I sampled them first. They were not cache. They were a million pieces of unrun work.
Pattern·Blast radius unmappedFour theories, four measurements
The work is not finding the answer. It is killing the wrong ones.
Cache leak. Delete the keys with no TTL.
The connections are consuming the memory.
It is fragmentation, not real data.
Failed jobs accumulating (the vendor README says so).
Pending jobs never expire. By design.
1,023,693 queued jobs at 3 keys each = 3,071,079 immortal keys. Measured: 3,062,717. There was no leak. There was a queue nobody was watching.
Every wrong answer died to a single number that took under a minute to read.
At first glance this looked like the kind of Redis incident everybody has seen. Redis hit its configured memory ceiling and began rejecting every write. Login was down across the platform. Inside Redis were millions of keys with no expiry set, growing steadily, and someone had already named the fix: delete the keys with no TTL. It is the standard answer, it is usually correct, and here it would have been a data loss event.
I want to frame this as four theories rather than a single root cause, because the theories are where the engineering actually happened. The answer always sounds tidy afterwards. In the room, the work is making each plausible explanation earn its place, then killing it the moment a measurement says no. Each one here took under a minute to test.
Theory one. It is a cache leak, so delete the keys with no expiry.
I sampled the key names with a bounded scan before deleting anything. They were not cache entries. They were job records belonging to the background job scheduler: a payload, a state and a history, three keys per job. Deleting every key without an expiry would have deleted roughly a million pieces of work that had been accepted from customers and not yet run.
The most expensive incident decisions are the ones that look like cleanup. Sample before you delete. It costs a minute and it is the whole difference between a fix and an incident of your own making.
Theory two: the connections are consuming the memory. That is a tempting one when Redis is full, but a single field settled it. Client memory was 5.7 megabytes against 3.13 gigabytes in use, which is 0.17 percent. Dead.
Theory three. It is fragmentation, not real data. The fragmentation ratio was 1.03. A value near 1.0 means allocated and used memory agree and there is nothing to reclaim. Dead, in one field.
Theory four: failed jobs are accumulating. This one came with some weight behind it, because the library's own documentation describes exactly that failure mode. So I treated it as a hypothesis and counted. The failed set did not exist. The failed statistic was nil. There were zero failed jobs and there had been none. The README was describing a real problem, just not this problem.
A vendor README is a hypothesis, not evidence. It can tell you what usually goes wrong with a component. It cannot tell you what went wrong with yours.
What it actually was. The scheduler attaches an expiry to a job only when that job reaches a final state, succeeded or deleted. Jobs that are enqueued, scheduled, awaiting or processing have no expiry at all, and that is deliberate and correct. Pending work must never be silently discarded because a cache decided it looked old.
At that point the arithmetic closed cleanly. There were 1,023,693 jobs in the default queue. At three keys each that is 3,071,079 keys with no expiry. The measured count of keys with no expiry was 3,062,717. That is close enough to stop talking about a mysterious leak. The apparent leak was the backlog: a queue nobody was watching.
The arithmetic that ended the leak theory
One multiplication, one measurement, and four days of hypothesis died in a minute.
Step 1 · predict
1,023,693
queued jobs
3
keys per job
a job payload
a state
a history
Step 2 · compare
Expected
3,071,079
keys that should never expire
≈
agree
Measured
3,062,717
keys with no expiry set
8,362 apart, or 0.27 percent. That is the gap you get counting a queue while it moves.
There is no leak
Every one of those keys is pending work that the framework is holding on purpose. Pending jobs do not expire, by design. The memory was not lost. It was occupied.
The tempting fix was to delete every key with no expiry. That would have destroyed roughly a million items of pending work, silently, with no way to tell anyone what had been lost.
There was no leak. There was a queue nobody was watching.
Which raises the better question: why was there a backlog. Ten workers, two pods running five each, processing 4.6 jobs per second against roughly 5.15 coming in. That is not a spike and it is not a bad afternoon. It is a permanent deficit of half a job per second, which means the queue can never drain on its own. It shrank slightly overnight when incoming work fell, and grew again the next morning.
Throughput below the incoming rate is not a backlog. It is a ratchet. Quiet periods do not claw it back, and no amount of retention tuning touches it.
Capacity permanently below demand
Ten workers across two pods. The queue has never been able to drain, and it never will.
Jobs per second
Everything the product asks the queue to do.
10 workers, 2 pods running 5 each, at roughly 0.46 jobs per second per worker.
-0.55
per second
The deficit is permanent
Roughly 33 jobs a minute, about 2,000 an hour, added to a pile that no amount of waiting will clear.
Queue depth, and the floor it never reaches
It shrank slightly overnight, when incoming work fell below capacity for a few hours. It grew again the next morning, from a higher starting point than the morning before.
This is not a spike
A spike drains when it passes. This never passes.
The queue cannot drain on its own
Draining requires capacity above the incoming rate. There is none.
Every incident adds to it forever
An hour of downtime is added permanently, not repaid.
The only two levers
Raise throughput above 5.15, or lower the incoming rate below 4.60. Restarting the workers, adding memory and clearing the alert do none of that, which is why the queue was there the next morning and the morning after.
Throughput below the incoming rate is not a backlog. It is a ratchet.
That is worth dwelling on, because we did tune retention. Shortening the completed-job expiry from 24 hours to 180 minutes freed 1.60 gigabytes overnight, against a prediction of 1.5. The change worked exactly as designed and did nothing whatsoever for the actual problem, because it shortens retention on work that has already finished. Predicting a result correctly is not the same as fixing anything.
And then the finding that explains how a job queue took down authentication. One Redis instance was serving sessions, the distributed cache, distributed locks, background job storage and the framework's data protection key ring. All on database zero, through one shared connection string, with no database index separating any of it. A backlog of background jobs consumed the memory that sessions needed, so login stopped working. No architecture diagram anywhere said that a slow batch job could take down authentication, because no diagram showed them sharing a process.
Five concerns, one process, one blast radius
Database 0. One connection string. No index, so everything lands in the same keyspace.
One connection string, shared by everything
redis://cache-host:6379
No database index on the end of it, so every caller silently defaults to database 0.
One instance · database 0
Sessions
who is logged in
Distributed cache
read-through data
Distributed locks
mutual exclusion
Background job storage
queued work, no expiry
Encryption key ring
framework secrets
Memory, by occupant
The other four share the sliver on the right. None of them grew. One of them dies anyway.
Memory ceiling reached
background job storage has taken the room
Every write rejected
the policy refuses new keys, process stays up
Sessions cannot be written
so nobody can log in
No diagram showed these five sharing a process
The architecture drawings had a box for the cache and a box for the queue, and they were different boxes. Nothing on paper said that a slow batch job could take down authentication, so nobody predicted it, and nobody could have.
Shared X is your blast radius, and it is never the X on the diagram.
Three questions this leaves for your own platform.
- For every shared datastore you run, list every distinct concern using it, then ask what happens to each one when another fills the memory.
- Is your queue depth a metric with an alert on it, or a number you can only obtain by connecting to the datastore during an incident?
- Do you know your processing rate and your incoming rate as two separate numbers? Without both you cannot tell a backlog from a ratchet.
Shared X is your blast radius. Not the X you drew on the diagram. The one that four teams are quietly using at the same time.
Get the next one in your inbox
One short, opinionated field note per fortnight on platform engineering, cloud, and making AI work in production. No spam. Unsubscribe anytime.
