The Compose Stack

The Compose Stack

The Official Docker Compose Stack

Download the file for your release from https://airflow.apache.org/docs/apache-airflow/3.3.2/docker-compose.yaml 129 . A YAML anchor, x-airflow-common, holds what all Airflow 129 containers share: the image, AIRFLOW__SECTION__KEY variables that override airflow.cfg, mounts for dags, logs, config and plugins, and your user ID. A one-shot airflow-init migrates the database and creates the airflow/airflow admin first.

BookNest's copy (demos/ch07/airflow/docker-compose.yaml) switches to the LocalExecutor, turns examples off, publishes port 32880, joins a network shared with l2-pg, and adds a connection and mounts:

docker-compose.yaml (excerpt): BookNest's changes to the common blockYAML

  image: ${AIRFLOW_IMAGE_NAME:-l2/airflow-booknest:3.3.2}   # official image + dbt (Dockerfile)
  environment:
    AIRFLOW__CORE__EXECUTOR: LocalExecutor
    AIRFLOW_CONN_BOOKNEST_PG: 'postgresql://postgres:booknest@l2-pg:5432/booknest'
    PYTHONPATH: /opt/airflow/include                   # booknest_pipeline.py
  volumes:
    - ../dbt:/opt/airflow/dbt                          # Chapter 4's dbt project
    - ../source:/opt/airflow/data/source:ro            # the order service's export

The pipeline needs dbt 37,942 in the image. Rather than _PIP_ADDITIONAL_REQUIREMENTS, which reinstalls at every start, extend the image, with dbt in its own virtual environment so its pins never fight Airflow's (Orchestrating dbt with Cosmos):

Dockerfile: the official image plus dbt CoreDockerfile

FROM apache/airflow:3.3.2
RUN python -m venv /opt/airflow/dbt_venv \
 && /opt/airflow/dbt_venv/bin/pip install --no-cache-dir dbt-core==1.12.5 dbt-postgres==1.11.0

The image runs as user airflow, so a first try at /opt/dbt-venv failed with Permission denied. The result, l2/airflow-booknest:3.3.2, reports dbt Core 1.12.5 with the postgres 1.11.0 plugin on Python 3.13.

MENU