Event-Driven Architecture

From Batch Pipelines to Event-Driven Architecture

In a batch pipeline the application writes to its database, and at 1:00 a.m. a job extracts the day's rows and loads the warehouse (Analytical SQL and Data Warehouses and Batch Processing with Apache Spark). The data is a day old, and every new consumer adds another extract.

In an event-driven architecture the application announces what happened, "order 7 was placed", the moment it happens, without knowing who listens. BookNest's shop publishes each order event once to the topic booknest.order-events; a fraud check reacts within a second, a stock alert within a minute, the lakehouse (Lakehouses, Data Quality and Governance) loads every few minutes, and adding another consumer changes nothing upstream.

Data Engineering Foundations weighed the trade-off (Batch and Streaming Processing) and the Kappa architecture built on a retained log (Lambda and Kappa Architectures): streaming costs an always-on service and harder reasoning about time and failure. Use it when several consumers need the same events or minutes matter; keep the nightly batch when a daily report is all anyone reads.