Primers › Kubernetes

Why a pod keeps restarting

A service in Kubernetes restarts every few hours. Its logs end in a normal-looking shutdown, there’s no crash in them, and the first guess is usually that it ran out of memory. That’s one possibility. There are others that look almost the same from the outside, and the fix for each is different, so it pays to know which one happened before raising any limits.

A few terms

• Pod: the smallest thing Kubernetes runs, one or more containers that are scheduled together.
• Kubelet: the agent on each node that starts a pod’s containers, watches them, and restarts them.
• Probe: a health check the kubelet runs against a container on a schedule. A failing liveness probe gets the container restarted. A failing readiness probe only stops traffic from being sent to it. A startup probe holds off the other two until the app has finished starting [1].
• Signal: a message the operating system delivers to a process. SIGTERM (signal 15) asks it to shut down, and a well-behaved app cleans up and exits. SIGKILL (signal 9) ends it immediately, with no chance to clean up [2].
• Exit code: the number a process leaves behind when it ends. When a process is ended by signal number N, the exit code is 128 + N, so SIGKILL shows up as 137 and SIGTERM as 143 [3].
• CPU limit: the most CPU time a container may use, written in thousandths of a CPU, so 500m is half a CPU [4].

Read the exit code first

kubectl describe pod shows the state of each container in the pod:

kubectl describe pod <pod>

For a container that has restarted, that part of the output looks something like this:

    State:          Running
      Started:      Tue, 06 Oct 2026 14:02:11 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    143
      Started:      Tue, 06 Oct 2026 10:15:40 +0000
      Finished:     Tue, 06 Oct 2026 14:02:09 +0000
    Restart Count:  4

• State is the run going on now.
• Last State is how the run before it ended: the reason, the exit code, and when it started and finished.
• Restart Count is how many times the container has been restarted [5].

Last State only goes back one run. Kubernetes keeps the last termination and nothing older, and it’s empty for a container that has never restarted [5]. A container that restarted four times shows why the fourth run ended, not the three before it. To see whether every restart ends the same way, read it again after each one. describe has no flag for older runs. The logs do: kubectl logs <pod> --previous prints the logs of the run before the current one, and only that one [6].

Last State narrows things down a lot:

The second row surprises people most. Nothing ran out, no error is logged, and the log ends in what looks like a normal shutdown, because SIGTERM lets the app close down cleanly. A burst of “interrupted” errors in the last few seconds is that shutdown in progress, not its cause.

For Java services there’s one more case. The heap can fill up and throw OutOfMemoryError while the container itself is well under its limit, so it may not restart at all. Java heap size and container memory covers how to tell the two kinds of out-of-memory apart.

How a failed probe turns into a restart

A liveness probe is usually a request to a health endpoint in the app. Three settings decide how much silence the kubelet tolerates [1]:

• periodSeconds: how often to probe.
• timeoutSeconds: how long to wait for each answer.
• failureThreshold: how many failures in a row count as dead.

With a 30-second period, a 10-second timeout, and a threshold of 3, an app that stops answering gets about 90 seconds before the kubelet gives up on it:

Three failed probes in a row, then a polite SIGTERM, then a forced SIGKILL only if the app doesn't stop within the grace period.

Why a healthy-looking app fails its probe

An app can go quiet for 90 seconds without anything being wrong in the usual sense:

• CPU throttling. A CPU limit is a hard ceiling. When a container has used its share for the current scheduling period, the kernel holds it back until the next one [4]. Everything in the container stalls, the health endpoint included. The kubelet never kills a container for using CPU, but a throttled one can be too slow to answer a probe. The container’s cpu.stat file counts how often it happened, in nr_throttled, and for how long, in throttled_usec [8]:

  kubectl exec <pod> -- cat /sys/fs/cgroup/cpu.stat

• A long garbage-collection pause. A collector that stops the whole application, running on a throttled CPU, can freeze it for seconds. Java services in small containers often get exactly that kind of collector, which Java heap size and container memory explains.

• Every worker busy. If all the threads that handle requests are stuck waiting on something slow, like a database, none is free to answer the probe.

• Slow startup. A liveness probe that begins before the app has finished starting can kill it partway through, every time. A startup probe fixes that, because Kubernetes runs no liveness or readiness probes until the startup probe has succeeded [1].

The Kubernetes documentation also warns that liveness probes set up wrong can cause cascading failures: containers restarted under heavy load, more failed requests, and more load on the pods that are left [1]. The usual way into that trap is a liveness check that also tests a database or another service. When that dependency gets slow, every replica fails its probe at once and restarts together. Checks on dependencies belong in the readiness probe, whose failure only takes the pod out of rotation without restarting anything [1].

An order to check things in

1. Read Last State with kubectl describe pod. OOMKilled means memory. 143, or 137 without OOMKilled, means a signal from outside, usually a probe.

2. Read the events, which include the kubelet’s “Liveness probe failed” messages. Events are deleted after an hour by default, so read them soon after a restart [9]:

   kubectl get events --field-selector involvedObject.name=<pod>

3. Compare the probe settings with how long the app takes to start and how long it can stall.

4. Check for throttling in cpu.stat.

5. For a Java service, read the live heap with jcmd, as in Java heap size and container memory. A heap far below its maximum isn’t the cause.

6. Catch it in the act. The next time it starts stalling, take a thread dump and turn on garbage-collection logging, so the evidence exists when it happens again.

Once the cause is known, the fix usually follows from it: a startup probe for a slow start, a higher failure threshold or timeout for an app that stalls briefly, a higher CPU limit for one that’s throttled, or a memory change only when the evidence points at memory. A bigger memory limit does nothing for a probe failure.

References

Every source cited across the primers is collected on one page.