Insights
Platform8 September 2026

163 restarts in 8.6 hours. The application never failed once.

Three pods restarting every few minutes. No errors in the logs, no OOM kills, no crashes. The application was healthy every single time it was killed. The probe timeout was the fault, and the fix shares a mechanism with the failure.

Pattern·Platform substrate never measured

The probe kill loop

163 restarts in 8.6 hours. The application never failed once.

CFS periods during one 2-second probe

Twenty consecutive windows in which the process may be parked.

1

CPU quota exhausted

1000m = 100ms per 100ms period

2

Process parked

Not broken. Not slow. Stopped.

3

Probe times out

2s timeout = 20 throttle windows

4

kubelet sends SIGTERM

exit 0, reason Completed

5

Cold start

JIT, DI graph, bus topology, scheduler

The cold start is the most CPU-hungry work the service ever does, and it runs under the same quota. Back to step 1.

lifetime 70slifetime 100slifetime 201sbackoff caps at 5min

The probe did not detect a failure. It caused one, then caused it again.

If you saw three pods sitting at 50, 55 and 58 restarts, you would probably reach for the crash logs first. I did too. The deployment had restarted 163 times in 8.6 hours, roughly 19 times an hour, so the obvious assumption was that the application was crashing. It was not.

The first thing to establish is how a container died, not why. The exit code narrows it faster than any log.

  • Every termination recorded exit code 0 with reason Completed.
  • 137 would mean the kernel OOM killer. It was not 137.
  • 139 would mean a segmentation fault. It was not 139.
  • Exit 0 means the process received SIGTERM and shut down cleanly on request.

What actually killed the container

Three exit codes, three completely different investigations.

137exit code

Kernel OOM killer

The container asked for more memory than its limit allowed and was terminated by the kernel.

Ruled out
139exit code

Segmentation fault

The process touched memory it did not own. A genuine crash inside the runtime.

Ruled out
0exit code

SIGTERM, clean shutdown

Reason: Completed. The process was asked to stop, and it stopped properly.

Observed

A kubelet sends SIGTERM to an otherwise healthy container for essentially one reason: a liveness probe that failed.

Supporting evidence, from the previous container

0

ERROR lines

0

FATAL lines

about 90 seconds of ordinary operation, in silence

shutting down

The service started, ran, said nothing at all, and then announced its own shutdown. That is not a program failing. That is a program being told to leave.

Establish how it died before you ask why.

After the exit code, I went to the logs from the previous container, which is the bit that gets missed once a pod is already back up. Zero ERROR lines. Zero FATAL lines. The service booted fully, ran for about 90 seconds in silence, then logged that the application was shutting down. That is not a crash signature. That is a healthy process being told to leave.

A kubelet sends SIGTERM to a healthy container for essentially one reason. The liveness probe failed.

The events gave that outside actor a name: probe failed, context deadline exceeded. That wording comes from Go's HTTP client, so it is the kubelet giving up on waiting, not the application returning an error. The distinction matters more than it looks. The application may well have been one millisecond from answering correctly, which is exactly the sort of detail that matters when a probe is allowed to kill the process.

So why was it slow to answer. A CPU limit is not a speed limit. A limit of 1000m is a CFS quota of 100 milliseconds of CPU per 100 millisecond period. When a process exhausts its slice it is parked until the next period begins. It is not broken and it is not degraded. It is stopped, then resumed, repeatedly.

This service had a liveness probe timeout of 2 seconds. The estate standard is 30. Two seconds against a 100 millisecond period is twenty consecutive throttle windows. Park the process across enough of them and the probe times out, and the kubelet kills a container that was about to answer.

From there the loop feeds itself. A kill forces a cold start under the same quota: JIT compilation, the dependency injection graph, message bus topology, scheduler initialisation. That is the service at its hungriest, running inside the constraint that just got it killed. The container lifetimes tell the story: 70 seconds, 100 seconds, 201 seconds. One slow heartbeat became a 55 restart loop, and the CrashLoopBackOff timer capping out at five minutes was the only reason it did not spin faster.

The probe did not detect a failure. The probe caused one, and then caused it again, and the backoff timer was the only brake in the system.

I found four variations of this in one estate.

  • A transaction service with a 2 second timeout against an estate standard of 30. 163 restarts in 8.6 hours.
  • A service whose heartbeat endpoint performs a Redis write, with a 1 second timeout. When Redis stopped accepting writes, every replica was killed and the ingress had no backend at all.
  • A device service on the same endpoint and the same timeout. 232 restarts, abandoning in-flight background jobs on every kill.
  • A batch exporter whose health endpoint returns 503 when a downstream dependency is unavailable. The worker was killed mid-export because something else was down.

The same misconfiguration, four different shapes

All four found across a single estate. Two of them convert a dependency failure into a total outage.

a

A transaction service

Configuration

Liveness timeout set to 2 seconds, against an estate standard of 30

Outcome

163 restarts in 8.6 hours

The application itself never failed once.

b

A service whose heartbeat endpoint performs a write

look for this first

Configuration

Every probe call writes to Redis. Timeout set to 1 second

Outcome

Every replica killed at the same moment

When Redis stopped accepting writes, the probe failed everywhere at once and the ingress had no backend at all.

c

A telemetry ingest service

Configuration

Same heartbeat endpoint, same 1 second timeout

Outcome

232 restarts

In-flight background jobs abandoned on every kill, with no drain and no resume.

d

A batch exporter

look for this first

Configuration

Health endpoint returns 503 whenever a downstream dependency is unavailable

Outcome

Worker killed part-way through an export

Restarted because something else was down. The exporter was working perfectly.

The dangerous pattern

A probe that depends on anything outside the container. When the dependency goes, every replica is killed together.

What it should test

Whether this process is still able to serve. Nothing else. Dependencies belong in readiness, not liveness.

Liveness answers one question only: should this container be restarted.

The second and fourth are the ones I would go looking for in your estate first. A liveness probe that writes to a dependency has promoted that dependency into a mandatory component of your pod's survival. A health endpoint that reports dependency status has converted a partial outage into a total one. Liveness answers exactly one question: should this container be restarted. Dependency health is a readiness concern at most, and frequently not even that.

One honest caveat about the evidence, because it matters. The first service had already stopped restarting on its own about 11 hours before the timeout patch went in, so I cannot call this a controlled before and after. The comparison I am comfortable standing behind is narrower: the same clock window on two consecutive nights showed 163 restarts, then zero.

Fixing it is less trivial than it sounds, and this is the trap. In that estate the probe timeout is a namespace-wide variable, not a per-service one. Changing it rewrites the pod spec of all 45 manifests in the namespace, so the deploy rolls every service at once.

Rolling the whole namespace to fix the probe produces exactly the mass cold start that a tight probe cannot survive. The fix and the failure share a mechanism.

So sequence it. Raise the timeout and the failure threshold in the quietest window you have, let the estate settle, and only then look at CPU limits. Do not change both at once, and do not do either at nine in the morning.

ShareLinkedIn

Get the next one in your inbox

One short, opinionated field note per fortnight on platform engineering, cloud, and making AI work in production. No spam. Unsubscribe anytime.

Senna Semakula

Senna Semakula

Founder, Atruvo

Bring your architecture diagram, cloud bill, or last incident summary.

I will tell you what is actually breaking.

30 minutes. No pitch. Ranked risks and a clear next step.