Benchmark Task and Methodology

Three tasks cover the shapes of BookNest's nightly work, each ending in a Parquet 129 file:

T1, joins and aggregation: delivered order lines joined to the catalog and to customers, revenue and line count by UTC month, genre and country (960 rows).
T2, semi-structured input: parse the raw JSON Lines orders, explode the items arrays, revenue by month.
T3, scale: T1 over the 13.8 million lines.

Each engine runs each task in a fresh process, as a scheduled batch job would, so start-up counts; GNU time records the wall time and peak memory. Every process sets UTC (TZ=UTC, Spark 129 's session time zone, DuckDB 61,228 's TimeZone), and money is summed as DOUBLE in every engine so they compare the same arithmetic.

bench/run_bench.sh: fresh process per engine and taskShell

#!/bin/bash
# run_bench.sh [runs] [engine...]: every engine and task in a fresh process with TZ=UTC.
# Appends "engine task run wall_s max_rss_mib" plus the engine's own line to results.tsv.
export TZ=UTC JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64 SPARK_HOME=/home/dev/v7-tools/spark
export PYSPARK_PYTHON=/home/dev/v7-venv/bin/python
export PYSPARK_DRIVER_PYTHON=$PYSPARK_PYTHON
B=/home/dev/v7-l3/bench; PY=/home/dev/v7-l3/bench-venv/bin/python
mkdir -p $B/out && cp "/mnt/d/Books/Data Engineering/demos/ch05/bench/"bench_*.py $B/
RUNS=${1:-3}; shift; ENGINES=${@:-duckdb polars daft pandas dask spark}
for run in $(seq 1 $RUNS); do for e in $ENGINES; do for t in T1 T2 T3; do
  cmd="$PY $B/bench_$e.py $t"
  [ $e = spark ] && cmd="$SPARK_HOME/bin/spark-submit $B/bench_spark.py $t"
  line=$( { /usr/bin/time -f "TIME %e %M" $cmd 2> $B/err.log; } 2>&1 )
  wall_rss=$(grep -h '^TIME' $B/err.log | tail -1 | awk '{printf "%s\t%.0f", $2, $3 / 1024}')
  echo -e "$e\t$t\t$run\t$wall_rss\t$line" | tee -a $B/results.tsv
done; done; done

A separate script (verify.py) then compares every engine's output month by month, rounded to the cent:

Output of 77

T1: 6 engines, 18 months, identical per month: True, first month 2025-01 = 1,813,666.65
T2: 6 engines, 18 months, identical per month: True, first month 2025-01 = 1,813,666.65
T3: 6 engines, 18 months, identical per month: True, first month 2025-01 = 18,136,666.50

A benchmark that does not check its answers measures nothing: two engines disagreeing on time zones would have shifted revenue between months here, and the check would have caught it.

MENU