MENU


Airflow Executors

The executor, a plug-in inside the scheduler, decides where a queued task instance runs:

Executors available to Airflow 3.3 129 (Celery, Kubernetes 5,150 , Edge and AWS 24 executors ship in providers)
Executor Where tasks run Extra infrastructure Fits
LocalExecutor Worker processes inside the scheduler container None One host, this book
CeleryExecutor Celery workers on any number of machines Redis 2,763 or RabbitMQ 28,807 broker Steady load, many hosts
KubernetesExecutor One pod per task A Kubernetes cluster Isolation, bursty load
EdgeExecutor Edge workers that poll the API server over HTTP Edge provider Remote sites, other networks
EcsExecutor, BatchExecutor AWS ECS tasks or AWS Batch jobs AWS account Serverless capacity on AWS

Airflow 3.0 removed the SequentialExecutor (use LocalExecutor) and the DebugExecutor (use dag.test(), dag.test() and Local DAG Runs). Since 2.10 you can list several executors, executor = LocalExecutor,CeleryExecutor, and route a task to one with its executor argument; the first is the default.

The LocalExecutor keeps one worker process per core.parallelism slot (8 here) inside the scheduler container. While arch_demo's single task ran, procs.py (the image has no ps) listed them:

Output of 5
  PID  PPID  ARGS
    7     1  /usr/python/bin/python3.13 /home/airflow/.local/bin/airflow scheduler
   40     7  airflow serve-logs
   42     7  airflow worker -- LocalExecutor: <idle>
...
   51     7  airflow worker -- LocalExecutor: <idle>
   52     7  airflow worker -- LocalExecutor: 01a0f95a-097d-7e27-b405-33c3078d224d
   53     7  airflow worker -- LocalExecutor: <idle>
 1487    52  airflow worker -- 01a0f95a-097d-7e27-b405-33c3078d224d

The UUID is the task instance's ID. Worker 52 runs the Task SDK's supervisor; 1487 is the forked child that ran the code (the task returned its PID, and the XCom said 1487). serve-logs lets the API server fetch task logs. All tasks share the scheduler container's CPUs, the right trade on one 4-CPU host (Choosing an Executor).