Orchestrating the Sales Mart

Orchestrating BookNest's Sales Mart Build

cosmos/up.sh restarts the stack on the Cosmos image with docker-compose.cosmos.yaml, which mounts dags_cosmos/ into the DAG folder; the plain stack and the tests of Unit Testing DAGs with pytest never import Cosmos. cosmos/c2.sh then ran booknest_daily_cosmos for 30 June:

Output of 32
            task_id             |  state  | try | started  | secs
--------------------------------+---------+-----+----------+------
 ingest.extract                 | success |   1 | 00:46:32 |  2.0
 ingest.load                    | success |   1 | 00:46:34 |  1.1
 sales_mart.stg_orders_run      | success |   1 | 00:46:37 | 10.5
 ...
 sales_mart.genre_groups_seed   | success |   1 | 00:46:37 | 10.1
 sales_mart.fct_sales.run       | success |   1 | 00:46:48 | 12.6
 sales_mart.dim_book.run        | success |   1 | 00:46:49 | 12.4
 sales_mart.fct_sales.test      | success |   1 | 00:47:01 | 13.6
 sales_mart.dim_book.test       | success |   1 | 00:47:02 | 12.6
 check                          | success |   1 | 00:47:15 |  1.5
 publish                        | success |   1 | 00:47:17 |  1.8
run success in 48 s
booknest_daily_cosmos in the graph view: Cosmos's sales_mart group between ingest and check
booknest_daily_cosmos in the graph view: Cosmos's sales_mart group between ingest and check

The run took 48 seconds where the BashOperator version took 26-31; the graph now shows what dbt 37,942 did. The first attempt had failed in ingest.load before dbt started: l2-pg had been stopped underneath the stack, and the two retries could not help. Had a test failed instead, only that model's group would need clearing, and publish would never have written a partial day. For BookNest the trade is worth it at four mart models; with hundreds, use TestBehavior.BUILD or the watcher mode, and keep the per-model view.