kafka-consumer-groups

kafka-consumer-groups and Offset Inspection

A group's committed offsets live in the internal __consumer_offsets topic. kafka-consumer-groups.sh subtracts them from each partition's log end offset to give the lag, the first number to check when a pipeline falls behind (Tracking Consumer Lag), and moves a stopped group's offsets.

Measuring a group's lag and rewinding it

# A consumer group reads 500 events and stops; inspect its lag, then rewind it by 100
K="docker exec -i l3-kafka /opt/kafka/bin"
B="--bootstrap-server l3-kafka:9092"
T="--topic booknest.order-events"
G="--group booknest-audit"
$K/kafka-console-consumer.sh $B $T $G --from-beginning --max-messages 500 >/dev/null 2>&1
$K/kafka-consumer-groups.sh $B $G --describe | awk 'NF {print $2, $3, $4, $5, $6}' | column -t
$K/kafka-consumer-groups.sh $B $G $T --reset-offsets --shift-by -100 --execute 2>/dev/null |
  awk '{print $2, $3, $4}' | column -t

Output

Consumer group 'booknest-audit' has no active members.
TOPIC                  PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
booknest.order-events  0          455             455             0
booknest.order-events  1          45              337             292
booknest.order-events  2          0               372             372
TOPIC                  PARTITION  NEW-OFFSET
booknest.order-events  1          0
booknest.order-events  2          0
booknest.order-events  0          355

The group is 664 records behind. Resets need a group with no active members: --to-earliest, --to-offset, --to-datetime, --shift-by and others pick the target, --dry-run previews and --execute commits; shifts stop at the log start. kafka-groups.sh --list shows this group's type as Classic (The New Protocol (KIP-848)).

MENU