Column Store Layout

Column Store Layout

How a Column Store Lays Out Data on Disk

DuckDB 61,228 splits a table into row groups of up to 122,880 rows, and stores each column of a row group as its own compressed column segment in 256 kB blocks. pragma_storage_info lists them:

Where DuckDB keeps two columns of the sales martSQL

SELECT row_group_id AS rg, column_name, segment_type, count, compression,
       block_id, block_offset
FROM pragma_storage_info('sales')
WHERE column_name IN ('genre', 'gross_amount') AND segment_type <> 'VALIDITY'
ORDER BY rg, column_name;

Output

┌───────┬──────────────┬───────────────┬────────┬─────────────┬──────────┬──────────────┐
│  rg   │ column_name  │ segment_type  │ count  │ compression │ block_id │ block_offset │
│ int64 │   varchar    │    varchar    │ int64  │   varchar   │  int64   │    int64     │
├───────┼──────────────┼───────────────┼────────┼─────────────┼──────────┼──────────────┤
│     0 │ genre        │ VARCHAR       │ 122880 │ Dictionary  │       39 │        92456 │
│     0 │ gross_amount │ DECIMAL(10,2) │ 122880 │ BitPacking  │       35 │            0 │
│     1 │ genre        │ VARCHAR       │  15064 │ Dictionary  │       40 │        73520 │
│     1 │ gross_amount │ DECIMAL(10,2) │  15064 │ BitPacking  │       40 │       105928 │
└───────┴──────────────┴───────────────┴────────┴─────────────┴──────────┴──────────────┘

The first row group's 122,880 genres sit together in one segment, away from the amounts; a validity segment per column (filtered out here) records NULLs. Each segment has its own compression and statistics. Row groups are the unit of parallelism and skipping (Zone Maps and Pruning), segments the unit of reading: Parquet 129 's plan (Row Groups and Pages), plus updates, a write-ahead log and transactions.

MENU