Checking with Soda Core

Checking BookNest Data with Soda Core

This contract covers Streaming into the Lakehouse's events. lakeduck.py makes lake.booknest the connection's current schema, and soda_events.py passes it to Soda (DuckDBDataSourceImpl.from_existing_cursor()) and calls verify_contract_locally():

order_events.contract.yml: a Soda Core 4 contract for the event table

dataset: lakehouse/booknest/order_events
checks:
  - schema:
  - row_count:
      threshold: {must_be_greater_than: 390000}
columns:
  - name: event_id
    data_type: bigint
    checks: [{missing: }, {duplicate: }]
  - name: ts
    data_type: timestamp with time zone
  - name: type
    checks: [{invalid: {valid_values: [order_placed, order_paid, order_shipped,
                                       order_delivered, order_cancelled, order_returned]}}]
  - name: order_id
    checks:
      - invalid:
          valid_reference_data: {dataset: lakehouse/booknest/orders, column: order_id}
  - name: customer_id
    checks: [{missing: {filter: "type = 'order_placed'"}}]
  - name: total

Output

PASSED  schema     (table)      schema_events_count=0
PASSED  row_count  (table)      check_rows_tested=392537
PASSED  missing    event_id     missing_count=0
PASSED  duplicate  event_id     duplicate_count=0
PASSED  invalid    type         invalid_count=0
FAILED  invalid    order_id     invalid_count=1800
PASSED  missing    customer_id  missing_count=0

The reference check, computed in SQL inside DuckDB 61,228 , found the 1,800 live events of Small Commits and Compaction, whose 900 orders exist only in the stream. Defect or lag? The contract forces the decision, for example a filter that exempts events newer than the last orders load, or level: warn. The script exits 1, failing its task.

MENU