DataOps

DataOps applies DevOps practices to data: version control for every pipeline and SQL model, automated tests that run before a change is merged, automated deployment, and monitoring that alerts a person when something goes wrong in production. The goal is the same as DevOps: ship changes often without breaking things, and find out about failures before your users do.

Testing data differs from testing application code in one important way: the code can be perfect and the data still wrong, because the data comes from systems you do not control. So DataOps tests both. Unit tests check transformation logic on small fixed inputs. Data tests check the real data every time it lands. The simplest data test is a query that counts rule violations and fails the run unless every count is zero. Here it is against the events from Generation:

A data test that counts rule violations in the order events

-- Each column counts rule violations; a passing run is all zeros.
WITH books AS (
  SELECT unnest(books).id AS id FROM read_json('books.json')
),
e AS (SELECT * FROM read_json('order-events.jsonl'))
SELECT count(*) FILTER (WHERE qty < 1)                         AS bad_qty,
       count(*) FILTER (WHERE unit_price <= 0)                 AS bad_price,
       count(*) FILTER (WHERE book_id NOT IN (SELECT id FROM books)) AS unknown_book,
       count(*) - count(DISTINCT order_id)                     AS duplicate_ids
FROM e;

Output

┌─────────┬───────────┬──────────────┬───────────────┐
│ bad_qty │ bad_price │ unknown_book │ duplicate_ids │
│  int64  │   int64   │    int64     │     int64     │
├─────────┼───────────┼──────────────┼───────────────┤
│       0 │         0 │            0 │             0 │
└─────────┴───────────┴──────────────┴───────────────┘

Tools such as dbt 37,942 tests, Soda Core 2,433 and Great Expectations 385,595 package this pattern with reusable rules, history and alerts (Transforming Data with dbt and Data Quality). The third leg is observability: tracking row counts, freshness and schema changes over time, so you notice when today's load is half the size of yesterday's even though every rule passed.

MENU