Reconciliation Pattern

The Reconciliation Loop Pattern

Every controller loops: observe the actual state, compare it with the spec, act on the gap, record the result in status. The loop is level-triggered, reacting to the current state rather than to events, so a missed event is repaired on the next pass. The kubelet runs such a loop for containers; watch it with kubectl 5,150 get --watch while a container is stopped behind Kubernetes 5,150 ' back:

Watching the kubelet restart a container stopped behind its backYAML

timeout 20 kubectl get pods -l app=web --watch --output-watch-events &
sleep 2
P=$(kubectl get pods -l app=web -o jsonpath='{.items[0].metadata.name}')
docker exec l3-booknest-worker sh -c \
  "crictl stop \$(crictl ps -q --label io.kubernetes.pod.name=$P)" >/dev/null
wait

Output

EVENT      NAME                  READY   STATUS    RESTARTS   AGE
ADDED      web-bf584dfbc-gcxpb   1/1     Running   0          37s
ADDED      web-bf584dfbc-ps2ld   1/1     Running   0          37s
MODIFIED   web-bf584dfbc-gcxpb   0/1     Error     0          40s
MODIFIED   web-bf584dfbc-gcxpb   1/1     Running   1 (1s ago)   40s

Within a second the kubelet restarted the container, because the Pod still asked for it. Repeated failures back off exponentially up to five minutes: CrashLoopBackOff (CrashLoop and ImagePull).

MENU