Analytical SQL and Data Warehouses to Apache Kafka and Managed Cloud Kafka built BookNest's dbt 37,942 models, Spark 129 jobs and Kafka 129 topics, each run by hand. A data platform runs every night without you, in the right order, and recovers when a step fails at 3 a.m.: the job of a pipeline and its orchestrator.
You start with the rules that make pipelines safe to re-run, then run BookNest's daily pipeline in Apache Airflow 3 129 , Dagster 177,056 and Prefect 74,615 , and finish with change data capture, ingestion, alerting and security.
What you will learn
ETL versus ELT, idempotency, data contracts and safe backfills
Airflow 3's architecture, what 3.0 changed, and how to write, test and run DAGs in Docker Compose 514
Orchestrating dbt with Cosmos, and the same pipeline in Dagster and Prefect
Change data capture with Debezium 317,608 , ingestion tools, alerting and securing the orchestration layer
Sections
- Pipeline Foundations
- Airflow's Architecture
- Airflow 3.x: What Changed
- Airflow in Docker Compose
- Writing DAGs
- Sensors and Task Mapping
- Connections and Testing
- Orchestrating dbt with Cosmos
- Dagster
- Prefect 3: Python-Native Flows
- Comparing Orchestrators
- Kestra, Mage and Luigi
- CDC with Debezium 3.x
- Ingestion Tools
- Pipeline Testing and Alerts
- Managed Orchestration
- Securing Orchestration
- Test Yourself!