Quality and Governance Gates

A gate is a task that only decides whether the run may continue. There is one at each boundary, cheapest first:

Ingest: xmllint 3,427 --schema rejects a feed that breaks XML and Its Toolchain's XSD (a negative price and a duplicate SKU both failed it when tried), and sha256sum -c rejects sample data that differs from the canonical files.
Stream: events_contract runs Anomalies, Contracts, Lineage's ODCS contract (18 checks, from types to "every paid order was placed first") on the landed events; only its namespace is changed.
Model: seven dbt 37,942 tests run as the models are built, among them event status equals order status and the mart's gross equals the source lines' gross.
Publish: pii_gate samples 200 rows of every published column through Governance's Presidio classifier, and reconcile compares the four totals. Only then does publish move the tag.

This is Branches and Tags's write-audit-publish with a tag: readers query FOR VERSION AS OF 'published'. fault_run.sh plays a producer bug, reruns the DAG with the ingestion steps marked successful, and rolls the bad commit back through the REST catalog:

fault_run.sh: one bad event, and what the gates do with itShell

# Section 8.16.4: a producer bug lands one bad event in the lake; the gates must stop it.
. "/mnt/d/Books/Data Engineering/demos/ch08/capstone/af_env.sh"
T=localhost:$REST_PORT/v1/namespaces/$NS/tables
good=$(curl -s $T/order_events | jq '.metadata."current-snapshot-id"')
$PY -c "from lakeduck import connect; connect().sql('''INSERT INTO lake.$NS.order_events
  SELECT event_id, ts, 'order_refunded', order_id, customer_id, total   -- an unknown type
  FROM lake.$NS.order_events WHERE event_id = 1''')"
airflow dags test booknest_platform --mark-success-pattern \
  '^(catalog|generate|load_lake|stream_events)$' > $CAP/fault.log 2>&1
echo "dags test exit code $?"; bash "$DEMOS/capstone/states.sh" $CAP/fault.log
grep -v "^passed" $CAP/gates/contract.txt
curl -s $T/daily_genre_sales |
  jq -r '"published tag still at snapshot \(.metadata.refs.published."snapshot-id")"'
echo -n "rollback: "; bash "$DEMOS/capstone/set_ref.sh" order_events main branch $good

Output

dags test exit code 1
success: catalog generate load_lake stream_events
failed: events_contract
upstream_failed: build_mart pii_gate reconcile publish
18 checks: {'passed': 16, 'failed': 2}
failed  event_id  Check that unique field event_id has no duplicate values
failed  type      Check that field type has invalid_count = 0
contract result: failed
published tag still at snapshot 1490863382816522832
rollback: main -> snapshot 449880410518157743 (HTTP 200)

One bad row in 390,738 stopped the run at the contract, which named both faults. The all_success trigger rule marked every later task upstream_failed, so Trino 403,499 never started and readers kept the published mart. Governance rides along: dbt-ol (Capturing Lineage) can send each run's lineage to Marquez, and Lakehouse Access Control's Trino rules decide who reads platform. The PII gate fails before a personal column becomes public.

MENU