Streaming the Orders Table

Streaming BookNest's Orders Table to a Sink

Debezium 317,608 's own JDBC sink connector, in the same Connect image, applies the events to another database: a reporting replica here, a warehouse staging schema in practice:

cdc/orders-replica-sink.json (excerpt): apply the order changes to a replicaJSON

{
  "name": "booknest-orders-replica",
  "config": {
    "connector.class": "io.debezium.connector.jdbc.JdbcSinkConnector",
    "topics": "booknest_cdc.public.orders",
    "connection.url": "jdbc:postgresql://l2-pg:5432/booknest_replica",
    ...
    "insert.mode": "upsert",
    "delete.enabled": "true",
    "primary.key.mode": "record_key",
    "schema.evolution": "basic",
    "collection.name.format": "orders",
    ...
  }
}

cdc/c2.sh registered it; schema.evolution=basic created the table from the event schemas (order_ts as timestamp with time zone, total as numeric), and the 100,000 snapshot rows arrived in 34 seconds. c3.sh then inserted, updated and deleted order 900001, first with the table's default replica identity, then with REPLICA IDENTITY FULL:

Output of 47

{'order_id': 900001}  op=c  before=None  after={'status': 'placed'}  lag=240 ms
{'order_id': 900001}  op=u  before=None  after={'status': 'shipped'}  lag=360 ms
{'order_id': 900001}  op=d  before={'status': '', 'total': 'AA=='}  after=None  lag=63 ms
{'order_id': 900001}  tombstone (lets compaction drop the key)
...
{'order_id': 900001}  op=u  before={'status': 'placed', 'total': 'B2s='}  after={'status': ...
{'order_id': 900001}  op=d  before={'status': 'shipped', 'total': 'B2s='}  after=None  ...

The replica showed placed, shipped and then no row at the script's two-second checks, and lag (event time minus commit time) stayed under half a second. With the default identity PostgreSQL 1,289 logs only the key of an updated or deleted row, so before is empty or holds placeholders ('', zero); FULL logs the whole old row, which audits and "status changed from placed to shipped" consumers need, at the cost of more WAL. The delete is followed by a tombstone, a null value that lets a compacted topic forget the key. cdc/down.sh drops both slots, the publication and the replica afterwards.

MENU