Enforcing a Contract

Enforcing a Contract on BookNest's Order Events

The order service owns the event stream, so it owns this contract. Its server is the Iceberg 129 table landed in Streaming into the Lakehouse, and one rule, the order of events, needs SQL:

order_events.odcs.yaml: the order-events contract (ODCS v3.2.0)

apiVersion: v3.2.0
kind: DataContract
id: booknest-order-events
version: 1.0.0
status: active
servers:
  - {server: lakehouse, type: iceberg, catalog: lake, catalogUrl: "http://localhost:31181",
     namespace: booknest}
schema:
  - name: order_events
    properties:
      - {name: event_id, logicalType: integer, required: true, unique: true}
      - {name: ts, logicalType: timestamp, required: true}
      - name: type
        logicalType: string
        required: true
        quality:
          - type: library
            metric: invalidValues
            arguments: {validValues: [order_placed, order_paid, order_shipped,
                                      order_delivered, order_cancelled, order_returned]}
            mustBe: 0
      - {name: order_id, logicalType: integer, physicalType: bigint, required: true}
      - {name: total, logicalType: number, physicalType: "decimal(10,2)"}
    quality:
      - {type: library, metric: rowCount, mustBeGreaterThan: 0}
      - type: sql
        description: Every paid order was placed first.
        query: |
          SELECT count(*) FROM order_events p WHERE p.type = 'order_paid' AND NOT EXISTS (
            SELECT 1 FROM order_events o WHERE o.order_id = p.order_id
              AND o.type = 'order_placed' AND o.ts <= p.ts)
        mustBe: 0
slaProperties: [{property: latency, value: 15, unit: m}]

contract_test.py calls DataContract(...).test(), which reads the table through the REST catalog with PyIceberg, registers it in DuckDB 61,228 and turns every property and rule into a check; credentials come from DATACONTRACT_S3_* variables, never from the contract:

Output of 53

18 checks: {'passed': 18}
passed  event_id  Check that unique field event_id has no duplicate values
passed  type      Check that field type has invalid_count = 0
passed  (table)   Check that model order_events has row_count > 0
passed  (table)   Every paid order was placed first.
contract result: passed

Enforcement matters most before data exists: in the producer's CI, datacontract breaking compares the contract on the main branch with a proposal. A v2 that renames total to amount and makes order_id a string fails the build (breaking.sh, exit code 1):

Output of 53

[ 2 Error ]  [ 2 Info ]
│ INFO     │ Added   │ schema.order_events.properties.amount   │
│ ERROR    │ Updated │ schema.order_events.properties.order_id │
│ ERROR    │ Removed │ schema.order_events.properties.total    │
...

The producer then keeps the old fields for a deprecation period or publishes a new major version.

MENU