Parallel Jobs and Work Queues

Setting completions and parallelism fans work out. With completionMode: Indexed (stable since 1.24) each Pod gets its own index in JOB_COMPLETION_INDEX, and the Job finishes when every index has succeeded once:

An Indexed Job with five work items, two at a timeShell

kubectl apply -f - >/dev/null <<'EOF'
apiVersion: batch/v1
kind: Job
metadata: { name: shards }
spec:
  completions: 5
  parallelism: 2
  completionMode: Indexed
  template:
    spec:
      restartPolicy: Never
      containers: [{ name: c, image: "localhost:33500/booknest-web:1.3",
                     command: [sh, -c, 'echo "shard $JOB_COMPLETION_INDEX on $(hostname)"'] }]
EOF
kubectl wait --for=condition=Complete job/shards --timeout=120s >/dev/null
kubectl logs -l job-name=shards --prefix=false | sort
kubectl delete job shards >/dev/null

Output

shard 0 on shards-0
shard 1 on shards-1
shard 2 on shards-2
shard 3 on shards-3
shard 4 on shards-4

Each Pod's hostname carries its index, which maps to a fixed slice of work such as a range of book IDs. In the work queue pattern you set parallelism without completions, and Pods pull items from Redis 2,763 , RabbitMQ 28,807 or SQS until it is empty. backoffLimitPerIndex and successPolicy (stable since 1.33) refine retries and success.

MENU