A liveness probe restarts a container, a readiness probe only takes it out of the Service

kubernetes - 2026-10-25

Fifteenth entry in the RKE2/Kubernetes series. The object model entry showed a Deployment replacing a Pod that disappears. It didn't cover a container that keeps running but has stopped working: a deadlocked server, or a process stuck on a dependency. Kubernetes only knows about that if it asks, and a probe is how it asks. This entry runs the three kinds of probe on RKE2 v1.36.4 and checks what each one does when the answer is "no".

Three probes, three different consequences

A probe is a check that the kubelet, the agent on each node that starts and watches containers, runs against a container on a schedule. Per Kubernetes' probe docs, there are three kinds, and what separates them is what happens when the check fails:

Probe	Question	On failure
`livenessProbe`	Is the process still working?	The kubelet kills the container and restarts it
`readinessProbe`	Should it get traffic right now?	The Pod is taken out of its Services' endpoints; nothing restarts
`startupProbe`	Has it finished starting?	Liveness and readiness checks wait; after `failureThreshold` failures, the container is killed

Each one checks in one of four ways: an HTTP GET that must return a status from 200 to 399, a TCP connection that must open, a gRPC health check, or a command run inside the container (exec) that must exit 0. The timing fields are the same for all of them. The defaults: every periodSeconds: 10, give up on one attempt after timeoutSeconds: 1, act after failureThreshold: 3 failures in a row.

The examples use agnhost, the Kubernetes project's own test image. Its liveness mode serves /healthz with status 200 for 10 seconds, then with 500 for good, which makes it a server that breaks on cue.

Liveness: a failing check restarts the container, in place

apiVersion: v1
kind: Pod
metadata:
  name: liveness-http
  namespace: probes
spec:
  containers:
    - name: app
      image: registry.k8s.io/e2e-test-images/agnhost:2.53
      args: ["liveness"]
      livenessProbe:
        httpGet:
          path: /healthz
          port: 8080
        periodSeconds: 3
        failureThreshold: 3

75 seconds later:

kubectl -n probes get pod liveness-http -o wide
kubectl -n probes get events --field-selector involvedObject.name=liveness-http

NAME            READY   STATUS             RESTARTS     AGE   IP           NODE
liveness-http   0/1     CrashLoopBackOff   3 (3s ago)   75s   10.42.0.16   vm

REASON      COUNT   MESSAGE
Started     4       Container started
Unhealthy   12      Liveness probe failed: HTTP probe failed with statuscode: 500
Killing     4       Container app failed liveness probe, will be restarted
BackOff     2       Back-off restarting failed container app in pod liveness-http_probes(...)

What this shows:

The timing adds up. The container started at 20:10:48 and was killed at 20:11:05: 10 healthy seconds, then three failed checks 3 seconds apart.
It's the same Pod. The name, the IP and the node don't change. The kubelet restarts the container inside the existing Pod. No new Pod is scheduled, which is the difference from the Deployment replacement in the object model entry.
CrashLoopBackOff doesn't mean it crashed. The process never exited on its own. After repeated restarts, the kubelet waits longer and longer before each new start. That waiting state is called CrashLoopBackOff whatever caused the restarts. The Killing event tells a liveness kill apart from a real crash.
Four restarts in 75 seconds didn't fix anything. A restart only helps when the fault is in the process's own state. This server fails again ten seconds after every start.

Readiness: a failing check removes the Pod from the Service, nothing restarts

A Deployment of two replicas behind a Service. The readiness probe is an exec check that passes as long as /tmp/ready exists:

          command: ["sh", "-c", "touch /tmp/ready && exec /agnhost netexec --http-port=8080"]
          readinessProbe:
            exec:
              command: ["cat", "/tmp/ready"]
            periodSeconds: 2
            failureThreshold: 1

A Service doesn't send traffic to Pods directly. It sends it to the addresses listed in its EndpointSlices, which the control plane keeps up to date, with a ready condition on each address. Both Pods start ready. Deleting the file in one of them:

kubectl -n probes exec web-6dc6747479-5dvhz -- rm /tmp/ready
kubectl -n probes get pods -l app=web
kubectl -n probes get endpointslices -l kubernetes.io/service-name=web \
  -o jsonpath='{range .items[0].endpoints[*]}{.targetRef.name} ready={.conditions.ready}{"\n"}{end}'

NAME                   READY   STATUS    RESTARTS   AGE
web-6dc6747479-5dvhz   0/1     Running   0          9s
web-6dc6747479-hvl89   1/1     Running   0          9s
web-6dc6747479-hvl89 ready=true
web-6dc6747479-5dvhz ready=false

0/1, still Running, zero restarts, and marked not ready in the EndpointSlice, so the Service stops routing to it. Putting the file back brings it back to 1/1 within one check, still with no restart. Readiness is for "alive, but don't send traffic now": warming a cache, a dependency that's down, being overloaded.

Startup: protecting a slow start from the liveness probe

A container that takes 30 seconds to start listening, with a liveness probe that gives up after 15 (3 failures, 5 seconds apart):

      command: ["sh", "-c", "sleep 30 && exec /agnhost netexec --http-port=8080"]
      livenessProbe:
        httpGet: {path: /healthz, port: 8080}
        periodSeconds: 5
        failureThreshold: 3

Created twice: slow-without exactly as above, and slow-with with a startup probe added that allows up to 60 seconds (12 failures × 5 seconds):

      startupProbe:
        httpGet: {path: /healthz, port: 8080}
        periodSeconds: 5
        failureThreshold: 12

Three minutes later:

NAME           READY   STATUS    RESTARTS     AGE
slow-with      1/1     Running   0            3m2s
slow-without   1/1     Running   4 (2s ago)   3m2s

== slow-without
Unhealthy   8   Liveness probe failed: Get "http://10.42.0.19:8080/healthz": dial tcp 10.42.0.19:8080: connect: connection refused
Killing     2   Container app failed liveness probe, will be restarted
== slow-with
Unhealthy   6   Startup probe failed: Get "http://10.42.0.20:8080/healthz": dial tcp 10.42.0.20:8080: connect: connection refused

slow-without never gets to serve: liveness kills it before startup finishes, every time, forever. slow-with fails its startup probe six times (30 seconds), then passes, and only then does the liveness probe start checking. Raising initialDelaySeconds on the liveness probe would also get it through. But that delay applies to every start, and liveness can't detect anything while it runs. A startup probe stops waiting as soon as the container answers.

slow-without also shows 1/1. With no readiness probe, a container counts as ready as soon as it's running, so for the first 30 seconds of each start a Service would have sent it requests it refused.

A liveness kill waits for the grace period

The last run of slow-without ended like this:

kubectl -n probes get pod slow-without \
  -o jsonpath='{.status.containerStatuses[0].lastState.terminated.startedAt} -> {.status.containerStatuses[0].lastState.terminated.finishedAt} exit={.status.containerStatuses[0].lastState.terminated.exitCode}{"\n"}'

2026-10-24T20:13:53Z -> 2026-10-24T20:14:37Z exit=137

The probe gave up after about 15 seconds, yet the container lived 44. A liveness kill is an ordinary stop. The kubelet sends SIGTERM and waits up to the Pod's terminationGracePeriodSeconds, 30 by default, before sending SIGKILL. The process here is sh, running as PID 1 of the container, and the Linux kernel doesn't deliver signals to PID 1 unless it has installed a handler for them. So SIGTERM did nothing, and SIGKILL came 30 seconds later. Exit code 137 is 128 + 9, killed by signal 9 (SIGKILL). The first Pod, where agnhost itself was PID 1, exited at once with code 2.

The probe has its own terminationGracePeriodSeconds field, which overrides the Pod's value for liveness and startup kills only. It exists because a hung process, the thing liveness is there to catch, is the one least likely to exit cleanly on SIGTERM.

What a liveness probe should check

The restarts above follow from what each probe checks:

A liveness check that calls a database restarts every replica at once when the database goes down. None of the restarts bring the database back.
An exec probe forks a process every periodSeconds in every container. HTTP and TCP checks don't.
timeoutSeconds: 1 counts as a failure. On a loaded node, a slow response is enough to trigger a restart.

The probe docs draw the same line between the two checks: a liveness probe is for a container that can't recover without a restart, a readiness probe for one that is temporarily unable to serve.

Where this series goes next

Autoscaling with the HorizontalPodAutoscaler: adding and removing replicas based on the CPU requests that the namespaces entry made mandatory.

These commands ran on a single-node RKE2 v1.36.4 cluster inside a Docker container in a cloud sandbox, not on the Proxmox homelab the earlier entries used.

Created 2026-10-25T22:11:47+01:00 · Edit