MirrorMaker 2

MirrorMaker 2 (MM2, KIP-382, Kafka 2.4 129 ) is Apache Kafka's tool for copying topics between clusters, the only one since Kafka 4.0 removed the original MirrorMaker. It is three Connect source connectors (Distributed Mode): MirrorSourceConnector copies records and topic settings and writes offset syncs, matching source and target offsets; MirrorCheckpointConnector uses them to translate group offsets; MirrorHeartbeatConnector shows that a flow is alive.

MM2 runs on a Connect cluster, under Strimzi 326,081 , or as a dedicated process, connect-mirror-maker.sh. The default replication policy prefixes the source alias (old.booknest.order-events), keeping two-way flows apart; IdentityReplicationPolicy keeps the names, so a one-way migration needs no application change:

clusters = old, new
old.bootstrap.servers = l2-old:9092
new.bootstrap.servers = l2-new:9092
old->new.enabled = true
old->new.topics = booknest\\..*
old->new.groups = booknest-.*
# keep topic names: booknest.order-events, not old.booknest.order-events
replication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy
sync.group.offsets.enabled = true
sync.group.offsets.interval.seconds = 5
emit.checkpoints.interval.seconds = 5

migration/setup.sh starts the clusters (migration/clusters.yaml), creates booknest.order-events on l2-old with two non-default settings, writes 300,000 events with Producing Order Events's shop producer, and lets booknest-audit read them all and booknest-analytics 200,000.

# mirror.sh: start MirrorMaker 2, wait until new holds every record, compare the clusters
. ./lib.sh                        # k CLUSTER TOOL ARGS runs a Kafka tool; ends CLUSTER
docker compose -f clusters.yaml --profile mm2 up -d mm2 2>/dev/null
for i in $(seq 60); do [ "$(ends new)" = "$(ends old)" ] && break; sleep 5; done
sleep 20                                              # two checkpoint and offset-sync rounds
echo "end offsets: old $(ends old), new $(ends new)"
k new configs --describe --entity-type topics --entity-name booknest.order-events |
  grep -o '^  [a-z.]*=[0-9]*' | sort -u | xargs echo "new topic configs:"
for g in audit analytics; do
  for c in old new; do
    k $c consumer-groups --describe --group booknest-$g 2>/dev/null |
      awk -v c="$c $g" '$3 ~ /^[0-9]/ {s = s " " $4} END {print c " committed:" s}'
  done
done

end offsets: old 100320 100176 99504, new 100320 100176 99504
new topic configs: max.message.bytes=262144 retention.ms=1209600000
old audit committed: 100320 100176 99504
new audit committed: 100320 100176 99504
old analytics committed: 89427 65627 44946
new analytics committed: 77872 57369 27776

End offsets and topic settings match. Translated group offsets never pass the source's but can trail them: booknest-audit, at the end of the log, matched, while booknest-analytics landed 36,983 records short (9,006 and 26,378 in earlier runs). MM2 writes an offset sync only every offset.lag.max (100) records and keeps a sparse history, so an old commit maps to the nearest sync below it: it re-reads rather than skips. MM2 ran out of memory with a 256 MB heap; with 1 GB it used about 1 GiB.

Target offsets drift from the source wherever transaction markers or compaction gaps exist, so consumers must rely on translation. MM2 copies at least once; dedicated mode can copy exactly once since Kafka 3.5 (exactly.once.source.support=enabled, dedicated.mode.enable.internal.rest=true).