Debezium PostgreSQL Connector

Deploying the Debezium PostgreSQL Connector on Kafka Connect

quay.io/debezium/connect:3.7 is a Kafka Connect 129 worker (Kafka Connect) with every Debezium 317,608 connector installed, here on Kafka 4.3.1 129 in distributed mode with its configs, offsets and status in three Kafka topics. A connector is a JSON document posted to the worker's REST API:

cdc/orders-source.json (excerpt): Debezium's PostgreSQL source for the orders tableJSON

{
  "name": "booknest-orders",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "l2-pg",
    ...
    "topic.prefix": "booknest_cdc",
    "table.include.list": "public.orders",
    "plugin.name": "pgoutput",
    "publication.name": "booknest_cdc",
    "publication.autocreate.mode": "disabled",
    "slot.name": "booknest_orders",
    "snapshot.mode": "initial",
    "producer.override.compression.type": "zstd"
  }
}

cdc/c1.sh posted it, waited for the snapshot and looked at the slot:

Output of 45

{"name":"booknest-orders","type":"source","tasks":0}
{"connector":"RUNNING","task":"RUNNING"}
snapshot: 100000 events in 45 s
slot_name|plugin|slot_type|active|confirmed_flush_lsn
booknest_orders|pgoutput|logical|t|0/29151810

Topics are named <topic.prefix>.<schema>.<table> and keyed by the primary key, so all changes to one order land in one partition, in order. snapshot.mode=initial copies the table once (no_data skips that; when_needed re-snapshots if the slot is lost), and publication.autocreate.mode=disabled keeps the publication under the DBA's control, so the role needs no table ownership. In production, prefer Avro 129 or Protobuf with a schema registry (Schema Registry) to JSON that repeats its schema, and a config provider for the password.

MENU