Watches and Informers

Watches, Informers and the List-Watch Protocol

A watch is a GET with ?watch=1 that never finishes, streaming one JSON event per line. Every object carries a resourceVersion from etcd 137,357 's revision counter; a reconnecting client resumes after the last one it saw, or lists again if the API server answers 410 Gone. Here is a raw stream while one front-end Pod is deleted:

A raw watch stream from the API server while a Pod is deletedYAML

URL='/api/v1/namespaces/default/pods?watch=1&labelSelector=app%3Dweb'
timeout 15 kubectl get --raw "$URL" > watch.json &
sleep 2
P=$(kubectl get pods -l app=web -o jsonpath='{.items[0].metadata.name}')
kubectl delete pod "$P" --wait=false
wait
jq -r '[.type, .object.metadata.name, .object.metadata.resourceVersion,
  .object.status.phase] | join(" ")' watch.json

Output

pod "web-bf584dfbc-gcxpb" deleted from default namespace
ADDED web-bf584dfbc-gcxpb 4511 Running
ADDED web-bf584dfbc-ps2ld 4404 Running
MODIFIED web-bf584dfbc-gcxpb 4549 Running
ADDED web-bf584dfbc-qsztl 4552 Pending
MODIFIED web-bf584dfbc-gcxpb 4553 Running
MODIFIED web-bf584dfbc-qsztl 4557 Pending
MODIFIED web-bf584dfbc-qsztl 4562 Pending
MODIFIED web-bf584dfbc-gcxpb 4563 Succeeded
MODIFIED web-bf584dfbc-qsztl 4572 Pending
MODIFIED web-bf584dfbc-gcxpb 4574 Succeeded
DELETED web-bf584dfbc-gcxpb 4575 Succeeded
MODIFIED web-bf584dfbc-qsztl 4577 Running

With no starting resourceVersion, the stream opens with a synthetic ADDED per existing Pod. The deleted Pod then passes through MODIFIED (deletion timestamp set, Nginx 75 exiting cleanly) to DELETED, while its replacement appears and starts. Controllers use client-go's informer instead of parsing this by hand: it lists, watches, keeps a local cache and queues the keys of changed objects for the reconcile function:

An informer turns the list-watch stream into a cache and a queue of keys to reconcile

Shared informers and the API server's own watch cache keep thousands of watchers cheap.

MENU