Secrets Backends

Secrets Backends

Secrets Backends for Connections and Variables

A secrets backend puts an external store in front of the lookups: Airflow 129 asks the configured backend first, then environment variables, then the metadata database. Providers ship backends for HashiCorp Vault, AWS Secrets Manager 24 and Parameter Store, Google Secret Manager, Azure 6 Key Vault, Kubernetes 5,150 secrets and more; core ships LocalFilesystemBackend, which reads YAML, JSON or .env files and is the easy way to see the order at work:

docker-compose.secrets.yaml (excerpt): a file backend in front

x-secrets: &secrets
  AIRFLOW__SECRETS__BACKEND: airflow.secrets.local_filesystem.LocalFilesystemBackend
  AIRFLOW__SECRETS__BACKEND_KWARGS: >-
    {"connections_file_path": "/opt/airflow/secrets/connections.yaml",
     "variables_file_path": "/opt/airflow/secrets/variables.yaml"}

The override applies the block to all four Airflow services and mounts ./secrets, where variables.yaml holds booknest_min_orders: 150 and connections.yaml defines booknest_pg_ro. With the stack restarted on both compose files, run7/w5.sh repeated the lookups:

Output of 26

$ airflow variables get booknest_min_orders
150
$ python dags/booknest_vars.py
Done. Returned value was: 195 orders on 2026-06-30, minimum 150

The file's 150 beat the database's 100, which is still stored, and airflow connections get booknest_pg_ro found the file's connection. Configure the backend on every component that resolves secrets: in Airflow 3 that includes the API server, which answers the Task Execution API's lookups for tasks, and an optional [workers] secrets_backend that workers consult first. A file backend is for development only; Securing Orchestration sets up a real secrets store.

MENU