Airflow 129 describes work: a DAG of tasks with a schedule, where data stays outside and assets are labels that tasks claim to update. Dagster 177,056 describes data: assets and their dependencies, from which runs, lineage and staleness follow. Prefect 74,615 describes code: a Python function whose task calls are observed as they happen. The orchestration code each needed around the shared module:
| Airflow 3.3.2 | Dagster 1.13.25 | Prefect 3.8.7 | |
|---|---|---|---|
| File (lines) | booknest_daily.py (70) | definitions.py (142) | booknest_flow.py (67) |
| Graph known | At parse time | At load time | While running |
| A day of data | ds, data interval | Partition key | Flow parameter |
| dbt 37,942 | BashOperator or Cosmos | dagster-dbt assets | subprocess or prefect-dbt |
| Quality gate | check task | Blocking asset check | check task |
Dagster's file is longest because it also holds a resource, an op job, a sensor and the dbt translator, and it is the only one whose UI could answer "which days of fct_sales exist, and what feeds the mart?" without reading code. Prefect's is the shortest and the only one that is still an ordinary Python program you can run with python. Each model has its trap, and BookNest hit all three: Airflow's XComs are for metadata, not data (The TaskFlow API and XComs); Dagster orders steps only along edges inside the selection (dagster-dbt); Prefect records a dependency only where a value is passed (The Pipeline as a Flow).