Logs and Audit Trail

Logs, Events and the Kubernetes Audit Trail

kubectl 5,150 logs reads files that vanish with the Pod, so production ships logs off with a DaemonSet such as Fluent Bit 83,977 . Events (kept one hour) record what controllers did. Who changed something is in the audit log, off by default in kind 14,561 . Switch it on in the API server's static Pod manifest, then scale the front end by hand:

k8s/audit-policy.yaml: record writes in booknest, skip readsYAML

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: [RequestReceived]
rules:
- { level: None, verbs: [get, list, watch] }
- { level: Metadata, resources: [{ group: "", resources: [secrets] }] }   # no Secret bodies
- { level: Metadata, namespaces: [booknest] }
- { level: None }

Turning on the audit log in kind, then reading itYAML

N=l3-booknest-control-plane; M=/etc/kubernetes/manifests/kube-apiserver.yaml
export P=/etc/kubernetes/audit-policy.yaml D=/var/log/kubernetes
docker cp k8s/audit-policy.yaml $N:$P
docker exec $N cat $M > ~/apiserver.yaml
yq -i '.spec.containers[0].command += ["--audit-policy-file=" + strenv(P),
      "--audit-log-path=" + strenv(D) + "/audit.log", "--audit-log-maxbackup=2"]
  | .spec.containers[0].volumeMounts += [{"name": "audit", "mountPath": strenv(P)},
      {"name": "audit-log", "mountPath": strenv(D)}]
  | .spec.volumes += [{"name": "audit", "hostPath": {"path": strenv(P)}},
      {"name": "audit-log", "hostPath": {"path": strenv(D), "type": "DirectoryOrCreate"}}]' \
  ~/apiserver.yaml
docker exec -i $N sh -c "cat > $M" < ~/apiserver.yaml
sleep 20; until kubectl get --raw /readyz >/dev/null 2>&1; do sleep 2; done
kubectl scale deployment web --replicas=1 >/dev/null
until [ "$(kubectl get deploy web -o jsonpath='{.spec.replicas}')" = 2 ]; do sleep 2; done
docker exec $N cat $D/audit.log | jq -r 'select(.objectRef.name == "web" and .verb == "patch")
  | [.stageTimestamp[11:19], .user.username, .objectRef.resource,
     .objectRef.subresource // "-"] | join(" ")'

Output

21:15:24 kubernetes-admin deployments scale
21:15:49 system:serviceaccount:argocd:argocd-application-controller deployments -

kubernetes-admin scaled the Deployment and Argo CD 126 's controller patched it back; its events said only "Scaled down" and "Scaled up". Ship audit logs off the node, where an attacker could edit them.

MENU