Backfills

Airflow 2 129 's airflow dags backfill ran a private scheduler loop in your terminal: close it and the backfill died. In Airflow 3 a backfill is a database object that the real scheduler executes, visible in the UI's Backfills tab and creatable from the UI, REST API or CLI. --reprocess-behavior decides whether dates that already have a run are skipped (none), re-run if failed (failed) or re-run anyway (completed).

29 June already had a manual run. A first backfill for 28-30 June ran 28 June, skipped 29 June ("already exists") and missed 30 June: --to-date 2026-06-30 means midnight, and that run's logical date is 02:00. A second, with --reprocess-behavior completed and a later end, started all three days at once: a backfill has its own max_active_runs (default 10) that ignores the DAG's max_active_runs=1. Three concurrent dbt 37,942 builds of the incremental fct_sales interleaved their delete-and-insert batches, duplicated 263 lines from 26-27 June, outside the three-day lookback, and every retry failed the reconciliation test until a --full-refresh repaired the table. A third backfill then re-ran the failed days one at a time:

$ airflow backfill create --dag-id booknest_daily --from-date 2026-06-28 \
    --to-date 2026-06-30T02:00:00+00:00 --reprocess-behavior failed --max-active-runs 1
...
 bf |    day     |               run_id                |  state  | started  |  ended
----+------------+-------------------------------------+---------+----------+----------
  3 | 2026-06-28 | backfill__2026-06-29T02:00:00+00:00 | success | 23:19:25 | 23:19:46
  3 | 2026-06-29 | day-2026-06-29                      | success | 23:19:46 | 23:20:04
  3 | 2026-06-30 | backfill__2026-07-01T02:00:00+00:00 | success | 23:20:04 | 23:20:25

The days ran in order, 18-21 seconds each. Idempotent steps are not enough when runs share state: cap concurrency for pipelines that rebuild shared tables, and run --dry-run first to see the logical dates.