Cosmos with dbt Fusion

Cosmos with dbt Core and dbt Fusion

Cosmos does not care which dbt 37,942 it runs. With LOCAL it can import dbt Core and call it in-process when dbt is installed in Airflow 129 's own environment; with dbt in a separate venv, as here, it runs the executable as a subprocess. Since Cosmos 1.11.0 that executable may also be the Rust engine (dbt v2, dbt Fusion and SQLMesh), with two documented limits: only ExecutionMode.LOCAL, and no AIRFLOW_ASYNC. Its docs still say Fusion supports only a few warehouses; dbt 2.0.6 has no PostgreSQL 1,289 adapter (What Fusion Changes), so BookNest's v2 DAG uses the DuckDB 61,228 target:

dags_cosmos/booknest_dbt_fusion.py (excerpt): the same project on dbt 2.0.6Python

booknest_dbt_fusion = DbtDag(
    dag_id="booknest_dbt_fusion",
    project_config=PROJECT,
    profile_config=ProfileConfig(profile_name="booknest", target_name="duck",
                                 profiles_yml_filepath="/opt/airflow/dbt/profiles.yml"),
    execution_config=DBT_V2,
    render_config=RenderConfig(load_method=LoadMode.DBT_LS),
    max_active_tasks=1,                           # DuckDB allows one writer at a time

Two details mattered. Cosmos runs each task in a temporary copy of the project, so the profile's DuckDB path had to become absolute (BOOKNEST_DUCK_PATH in docker-compose.cosmos.yaml), or every task would have written its own empty database. And PyPI 2,431 's dbt 2.0.6 is a stub that downloads a 125 MB wheel from dbt's CDN at install time; the image installs that wheel, fetched once and checked against its published SHA-256. The run:

Output of 32

 stg_order_items_run         | success |   1 | 00:50:45 |  3.2
 ...
 fct_sales.run               | success |   1 | 00:51:10 |  3.8
 fct_sales.test              | success |   1 | 00:51:26 |  3.3
run success in 52 s
Running command: ['/opt/airflow/dbt2_venv/bin/dbt', 'run', '--select', ...
 Succeeded model main.fct_sales (incremental) [1 of 1 in 0.42s]

All 14 tasks succeeded one at a time, about 3 seconds each, against 10-14 seconds for the dbt Core tasks of Orchestrating the Sales Mart: v2's native parser removes most of the per-task overhead of 7.8.1. Cosmos parsed the v2 project with dbt ls as well, warning only that it has no converter for the project's semantic models. The resulting fct_sales held 137,944 lines. Keep dbt Core for adapters v2 lacks, and mind the licence of the dbt binary (The dbt Fusion Engine) before shipping it in an image.

MENU