The Metadata Database

The metadata database is Airflow 129 's single source of truth: serialized DAGs and their versions, every DAG run and task instance with its state, XComs, connections, variables, pools, assets, triggers, users and the audit log. Use PostgreSQL 1,289 or MySQL 524 in production; SQLite 4,756 suits only single-process experiments. The Compose stack gives Airflow its own PostgreSQL 16 container, separate from BookNest's l2-pg, and airflow db migrate created 71 tables:

Looking inside Airflow's metadata database

P="docker exec -i l2-airflow-postgres-1 psql -U airflow -d airflow -X -P pager=off"
$P -Atc "SELECT string_agg(tablename, ' ' ORDER BY tablename) FROM pg_tables
         WHERE schemaname = 'public'" | fold -s -w 95
$P -c "SELECT dag_id, version_number, bundle_name, created_at FROM dag_version ORDER BY dag_id"

Output

ab_group ab_group_role ab_permission ab_permission_view ab_permission_view_role
...
backfill backfill_dag_run callback connection connection_test_request dag dag_bundle
...
log log_template ... serialized_dag session slot_pool ... task_instance
task_instance_history task_instance_note task_map ... task_state_store team trigger variable
  xcom
     dag_id     | version_number | bundle_name |          created_at
----------------+----------------+-------------+-------------------------------
 arch_demo      |              1 | dags-folder | 2026-10-01 20:13:04.048151+00
 booknest_daily |              1 | dags-folder | 2026-10-01 20:13:03.604623+00

The names map the architecture: serialized_dag and dag_version come from the DAG processor, dag_run and task_instance from the scheduler, trigger from the triggerer, and ab_* from the Flask 15,439 AppBuilder auth manager. Treat the schema as private, since airflow db migrate changes it between releases: read it to diagnose, automate through the REST API. Back it up, keep it close to the scheduler, add PgBouncer as connections multiply, and trim history with airflow db clean.

MENU