How a Probe Checks a Container

The kubelet runs each probe itself: httpGet (from the node to the Pod's IP; 200 to 399 passes), tcpSocket, grpc (the standard health service, stable since 1.27) or exec (a command exits 0; it forks a process per check). periodSeconds (default 10), timeoutSeconds (1) and failureThreshold (3) time it. This Pod starts slowly, then "hangs" by deleting its marker file:

A startup probe, then a liveness failure and a restartShell

kubectl apply -f - <<'EOF' >/dev/null
apiVersion: v1
kind: Pod
metadata: { name: probes }
spec:
  terminationGracePeriodSeconds: 2
  containers:
  - name: app
    image: localhost:33500/booknest-api:1.4
    command: [sh, -c, 'sleep 10; touch /tmp/ok; sleep 20; rm /tmp/ok; sleep infinity']
    startupProbe: { exec: { command: [cat, /tmp/ok] }, periodSeconds: 3, failureThreshold: 10 }
    livenessProbe: { exec: { command: [cat, /tmp/ok] }, periodSeconds: 5, failureThreshold: 2 }
EOF
sleep 50
kubectl get events --field-selector involvedObject.name=probes --sort-by=.firstTimestamp \
  -o custom-columns=N:.count,REASON:.reason,MESSAGE:.message \
  | grep -v -e Scheduled -e Pulled -e Created -e '^$'
kubectl get pod probes

Output

N     REASON      MESSAGE
2     Started     Container started
4     Unhealthy   Startup probe failed: cat: /tmp/ok: No such file or directory
3     Unhealthy   Liveness probe failed: cat: /tmp/ok: No such file or directory
1     Killing     Container app failed liveness probe, will be restarted
NAME     READY   STATUS    RESTARTS     AGE
probes   0/1     Running   1 (8s ago)   50s

Startup checks failed while the container slept, liveness failed once the marker vanished, and the kubelet restarted the container, not the Pod. Because httpGet comes from the node, a server bound to 127.0.0.1 fails its probes while working fine through kubectl 5,150 exec.

MENU

How a Probe Checks a Container