Sizing Airflow

Sizing Airflow

Sizing Airflow for a Four-CPU Host

airflow-init warns below 4 GB of memory, 2 CPUs or 10 GB of disk. CPU, not memory, is the limit here: the scheduler, the DAG processor, the triggerer and every health check (a fresh airflow CLI each time) all poll:

docker-compose.yaml (excerpt): sizing for four shared CPUsYAML

    AIRFLOW__CORE__PARALLELISM: 8                     # task slots for the whole host
    AIRFLOW__DAG_PROCESSOR__PARSING_PROCESSES: 1      # one parser is plenty for a few DAGs
...
      interval: 60s
      timeout: 30s                    # the CLI needs over 10 s to start on a busy 4-CPU host

parallelism (default 32) caps running tasks and sets the LocalExecutor's process count; 8 leaves room for dbt 37,942 's own 4 threads. The longer checks came from a real failure: with the shared host's load average at 24, the official 10-second checks timed out and Docker 514 marked a working scheduler, triggerer and DAG processor unhealthy. Pools cap tasks that share a scarce resource such as database connections, and in WSL2 6 Docker can use only what .wslconfig gives the VM.

MENU