DAG Versioning

DAG Versioning

Airflow 3 129 records a DAG version whenever a parse yields a changed DAG and ties each task instance to it. Whether a run in flight stays on its version depends on the bundle. versioning.sh starts a two-task DAG whose first task sleeps 90 seconds, adds a third task to the file mid-run, then starts a second run. From the plain dags/ folder (a LocalDagBundle):

Output of 5

 run_id | task_id |  state  | version
--------+---------+---------+---------
 run-1  | first   | success |       2
 run-1  | second  | success |       1
 run-1  | third   | success |       2
 run-2  | first   | success |       2
...

Run 1 grew a third task halfway through: a local folder is not versioned, so tasks run the latest code on disk. Committed to Git 1,932 and served by a GitDagBundle (from the apache-airflow-providers-git package in the image):

docker-compose.git.yaml (excerpt): a Git bundle next to the dags/ folderYAML

AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST: >-
  [{"name": "dags-folder",
    "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle", "kwargs": {}},
   {"name": "booknest-git",
    "classpath": "airflow.providers.git.bundles.git.GitDagBundle",
    "kwargs": {"repo_url": "file:///opt/airflow/gitrepo/booknest-dags.git",
               "tracking_ref": "main", "refresh_interval": 20}}]

Output

 run_id | task_id |  state  | version | commit
--------+---------+---------+---------+---------
 git-1  | first   | success |       1 | 5268b1a
 git-1  | second  | success |       1 | 5268b1a
 git-2  | first   | success |       2 | b8e69d0
...

Each run recorded its starting commit and workers checked out exactly that commit, so git-1 finished on version 1 although version 2 was pushed while it ran. Deploy production DAGs through a GitDagBundle (git_conn_id for private repositories) for reproducible reruns; S3 and GCS bundles, like the local folder, are not versioned.

MENU