A few jobs on one machine need no scheduler, metadata database or web server. Pick the lightest option that covers your failures, and keep steps idempotent and logic in plain functions (Pipeline Foundations), so moving up a row means rewriting glue, not the pipeline:
| Situation | Enough | Move up when |
|---|---|---|
| A few independent scripts | cron or a systemd 142,543 timer | You need retries or history |
| A chain of file-producing steps | Luigi 18,778 | Several workers or a UI matter |
| Pipelines owned by non-Python teams | Kestra 284,108 | Logic outgrows embedded scripts |
| Exploratory, notebook-first work | Mage | Data no longer fits in memory |
| Many teams, SLAs, backfills | Airflow 129 , Dagster 177,056 , Prefect 74,615 | (you are there) |