Running the Pipeline

Running BookNest's Pipeline End to End

Unpausing booknest_daily (written in Writing DAGs) created no run: its end_date is 30 June, when the sample orders stop, and catchup is off. A manual run with an explicit logical date processes 29 June:

Running BookNest's pipeline for 29 June
A="docker exec l2-airflow-airflow-scheduler-1 airflow"
$A dags unpause booknest_daily
$A dags trigger booknest_daily --logical-date 2026-06-29T02:00:00+00:00 --run-id day-2026-06-29
$A tasks states-for-dag-run booknest_daily day-2026-06-29 -o plain
Output
dag_id          logical_date               task_id          state    start_date
booknest_daily  2026-06-29T02:00:00+00:00  ingest.extract   success  2026-10-01T21:37:35...
...
booknest_daily  2026-06-29T02:00:00+00:00  publish          success  2026-10-01T21:37:59...
The run in the grid view: every task of the 26-second run succeeded on its first try
The run in the grid view: every task of the 26-second run succeeded on its first try

The run took 26 seconds and its publish outlet started booknest_finance (Assets Replace Datasets). An earlier attempt failed in load with 'Connection' object has no attribute 'encoding': the Postgres 1,289 provider 7.x PostgresHook returns a psycopg 3 connection, and the code then used psycopg2's execute_values(). Plain DB-API calls fixed it, and airflow tasks clear booknest_daily -s 2026-06-29 -e 2026-06-29T23:59:59 -y re-ran the failed tasks in the same run. Two more surprises: logs printed ***_daily, because Airflow 129 masks a connection's password everywhere and this sample password is booknest; and a manual run's data interval follows the trigger time, not --logical-date, so tasks should read ds.