Putting the components together, here is the life of arch_demo's manual run arch-2, triggered with airflow dags trigger arch_demo --run-id arch-2:
The trigger writes a queued dag_run row. For scheduled runs the scheduler creates it when the timetable says the run is due, stamping its logical date and data interval.
The scheduler moves the run to running, finds the task instance hold with no unfinished upstream tasks, sets it to scheduled and then, inside the critical section, to queued, and gives it to the executor.
A LocalExecutor worker picks it up and starts the supervisor, which asks the API server to move the task to running and forks the child that imports the DAG file from its bundle and runs the task.
The child returns a value; the supervisor posts it as an XCom and reports success. The scheduler sees the final task state and marks the run success.
The API server's access log shows steps 3 and 4 from the outside, one heartbeat every five seconds:
20:19:00 PATCH /execution/task-instances/<id>/run 200 20:19:00 PUT /execution/task-instances/<id>/heartbeat 204 20:19:01 PUT /execution/task-instances/<id>/rtif 201 20:19:06 PUT /execution/task-instances/<id>/heartbeat 204 ... 20:19:46 POST /execution/xcoms/arch_demo/arch-2/hold/return_value 201 20:19:46 PATCH /execution/task-instances/<id>/state 204
(rtif stores the task's rendered template fields for the UI.) The metadata database keeps the timestamps:
row | state | queued | started | ended ----------------+---------+----------+----------+---------- dag_run manual | success | 20:18:59 | 20:19:00 | 20:19:46 task hold | success | 20:19:00 | 20:19:01 | 20:19:46
A task stuck in scheduled is waiting for a slot (parallelism, pools, per-DAG limits); one stuck in queued was handed over but no worker started it, so look at the executor and workers. If a worker dies, its heartbeats stop, and after task_instance_heartbeat_timeout the scheduler fails or retries the task.