The Pipeline as a Flow

Running BookNest's Pipeline as a Prefect Flow

run/p1.sh resets the pipeline with pipeline/setup.sh, starts the stack, deploys and runs 28 June; run/p2.sh and run/p3.sh run 29 and 30 June and repeat 28 June. The first run's log, read through the API (runlog.py):

Output of 40

caped-flamingo  {'ds': '2026-06-28'}  Completed  0:00:11.496977
02:03:43  flow           Worker 'ProcessWorker 9ce7365f-7d49-4383-b5fb-cc474541d30d' submitting
02:03:48  flow           Beginning flow run 'caped-flamingo' for flow 'booknest-daily'
02:03:49  extract-500    extracted 186 orders for 2026-06-28
02:03:49  load-fc8       Finished in state Completed()
02:03:49  flow           {'orders': 186, 'lines': 269}
02:03:59  dbt_build-096  Finished in state Completed()
02:03:59  flow           02:03:57  Done. PASS=6 WARN=0 ERROR=0 SKIP=0 NO-OP=0 REUSED=0 TOTAL=6
02:03:59  flow           269 lines, gross 6855.25
02:04:00  flow           Finished in state Completed()

The 29 June run in the Prefect 74,615 UI: extract cached, load retried once, then dbt 37,942 , check and publish

The flow body took 11.5 seconds, 17 from the worker's pick-up, and dbt was 10 of them. The timeline draws an arrow only from extract to load, the one place where a return value is passed on; dbt_build follows load simply because Python called it next, which Prefect does not record as a dependency. The repeat of 28 June took 7.2 seconds with extract cached and left the mart unchanged: 6 genre rows a day, 260, 248 and 244 lines worth 6,629.86, 6,327.10 and 6,670.05, the same figures the Dagster 177,056 assets published. If a step must wait for one that returns nothing, pass wait_for=[future] so the dependency is explicit.

MENU