We hardened Redis. That is how a self-healing crash loop became a total outage.
A month before the outage, someone gave Redis a memory ceiling and a no-eviction policy. It reads like hardening. It replaced a system that crashed and recovered with one that fails silently and permanently, behind a green health check and a Synced GitOps dashboard.
Pattern·Dashboards measure activity, not outcomesWhat the hardening change actually replaced
Same datastore. Same memory pressure. Two completely different outcomes.
Memory grows unbounded
Kernel OOM kill
In-flight work is dropped
Kubernetes restarts it
Seconds later, no human
Self-healing
Ugly in a dashboard. Recovers without you.
Memory grows to 3200mb
Every write rejected
Reads still served, process alive
Indefinitely
No crash, so no restart, so no recovery
Silent, permanent
Green everywhere. Nobody can log in.
The crash was the recovery mechanism. Removing the crash removed the recovery.
The platform looked alive, but the front door was shut. Every user sign in returned HTTP 500. The browser showed the Redis error more or less raw: OOM command not allowed when used memory is greater than maxmemory. Redis had reached its ceiling and was rejecting every write, so sessions could not be created and nobody could log in.
This is the part that should make you uneasy as a platform engineer. The pod was green. The health check was passing. The GitOps application reported Synced and Healthy throughout. Nothing alerted at any point. The first signal was a human being saying they could not log in.
The interesting part is not the memory. The interesting part is a change that went in a month earlier and looked, in review, like an improvement.
- A maxmemory ceiling was set at 3200mb.
- The eviction policy was set to noeviction.
- The container memory limit was raised from 1Gi to 4Gi.
Read that as the person approving the change. A datastore gets an explicit memory bound instead of growing unbounded, and the container gets more headroom at the same time. That looks like hardening. I would probably have approved it too, which is exactly why this one is worth sitting with.
Now look at the behaviour it replaced. Before that change Redis had no ceiling. It grew until the kernel OOM killer terminated it, then Kubernetes restarted it a few seconds later. Ugly, yes. It dropped in-flight work and left restart counts sitting in the dashboards. But it also recovered on its own, every time, without a human being getting involved.
And what it became. With a ceiling and no eviction, Redis does not die. It reaches the ceiling and refuses every write from that moment onward, indefinitely, while continuing to serve reads and pass every health check pointed at it. The process is alive. The pod is Ready. The dashboard is green. The product is completely broken.
Making a system fail quietly is not making it fail less. The crash was the recovery mechanism. Removing the crash removed the recovery and left the failure exactly where it was.
It got worse from there in a way worth describing, because it is a second instance of the same lesson. One service's liveness probe called a heartbeat endpoint that writes to Redis, with a one second timeout and a failure threshold of three. Redis refusing writes meant the heartbeat failed, which meant the kubelet killed every replica, which meant the ingress had no backend at all. The 500 became a 502. A second service on the same pattern restarted 232 times, abandoning in-flight jobs on every kill and deepening the underlying problem with each one.
How a refused write became a total outage
Six links. Every one of them behaving exactly as configured.
The only question a liveness probe asks
Should this container be killed and restarted? It was answered by writing to a shared datastore.
Datastore refuses writes
Ceiling reached, nothing evicted
Heartbeat writes a key
The health endpoint touches the datastore
Liveness fails 3 times
One second timeout, each attempt
kubelet sends SIGKILL
All 3 replicas, at the same moment
Ingress has no backend
Nothing left to route to
Users see 502
Not 500. The application never got the chance to answer.
A 500 would have meant
The service is up and something inside it is broken.
A 502 meant
There is no service left to ask. Degradation had become absence.
232
restarts
A second service carried the same pattern.
Its health check wrote to the same datastore, so it was killed and restarted 232 times. Each restart abandoned whatever work was in flight. Nothing in the platform recorded that as a failure.
A health check that writes to a dependency makes that dependency mandatory for survival.
The same change had a second effect nobody caught. Setting the configuration block replaced the chart's defaults wholesale rather than merging with them, which silently switched off append-only persistence and dropped the snapshot settings with it. Two hours later a follow-up commit put persistence back. The snapshot setting was never restored and nobody noticed for a month, which left the instance forking snapshots against a steadily growing dataset the entire time.
A configuration block that replaces defaults instead of merging with them will remove settings you never knew you had. The diff shows you what you added. It does not show you what you displaced.
Two lines added. Two lines deleted without a diff.
Setting the configuration block replaced the defaults. It never merged with them.
Whole-key replacement, not a merge. Anything the defaults held is gone.
maxmemory 3200mb
A hard ceiling on memory
maxmemory-policy noeviction
Refuse writes rather than drop keys
Two lines. Reviewed, approved, deliberate.
appendonly yes
Persistence switched off
save ""
Snapshot settings lost
Two lines. Never written down, never reviewed, never seen.
Two hours later, persistence came back.
A follow-up commit restored the append-only setting. Somebody noticed that one.
The snapshot setting was never restored.
It stayed displaced for a month, unnoticed, because no diff ever showed it leaving.
The diff shows what you added. It does not show what you displaced.
Then the estate-wide number, which is the one I would want if this were my platform. Of 93 environments running Redis, only 4 set any memory configuration at all. It is tempting to read that as 89 environments at risk and 4 protected. It is closer to the reverse. The 89 fail the old way: they grow, they get OOM killed, they restart, and nobody notices because it self heals. Two of the four carry the identical fuse that produced this outage. One of those two also lost its persistence setting the same way, and append-only is silently off in a live environment right now.
Ninety-three environments, and the four that were tuned
Configuration was not the protection. It was the exposure.
How the number reads
Only 4 environments are configured, so 89 are at risk and 4 are protected.
What it actually means
It is closer to the reverse. The 89 fail the way they always have: grow, get killed for memory, restart, and nobody notices. The 4 that were tuned are the ones that can stop serving and stay that way.
Self-healing, unnoticed
Memory grows, the kernel kills the process, the platform restarts it. Ugly, and it recovers without anybody.
Silent, and permanent
The identical ceiling and the identical refuse-writes fuse that took the platform down. Nothing crashes, so nothing recovers.
93
environments
89
fail the old way
2
carry the fuse
1
unprotected right now
One of those two lost its persistence setting the same way.
Append-only persistence is switched off in a live environment right now, displaced by the same block, recorded in no diff, and reviewed by nobody.
A default nobody chose, inherited by every environment.
What good looks like here is not complicated, it is just unglamorous.
- If you set a memory ceiling, alert on used memory as a fraction of that ceiling, well before it is reached. A ceiling without an alert is a trapdoor.
- Alert on rejected writes directly. It is a single counter and it is the exact thing your users will feel.
- Choose an eviction policy by reasoning about what is in the datastore, not by copying a value from somewhere. Here, evicting was genuinely unsafe: the encryption key ring shared the same instance and has no expiry, so evicting it would have invalidated every session cookie in the namespace.
- When a health check is the only thing between you and an outage, check what it actually asserts. A check that never touches the failing path will stay green through anything.
And three questions to take to your own estate.
- Which of your production failure modes are currently self healing, and would a hardening change convert any of them into a permanent stall?
- For every datastore with a memory bound, is there an alert before the bound, or only a dashboard after it?
- Does any health check in your estate assert on the thing that actually breaks?
The most dangerous change in an incident review is never the one that looks reckless. It is the one that looks responsible and quietly deletes the way the system used to save itself.
Get the next one in your inbox
One short, opinionated field note per fortnight on platform engineering, cloud, and making AI work in production. No spam. Unsubscribe anytime.
