Spark Alternatives

Spark 129 is the default answer to "big data", but most data is not big. This section surveys the engines that now compete with it for batch work, then runs five of them against Spark on BookNest's order history on the same 4-CPU workstation, with identical results checked month by month.

Subsections

• When Not to Use Spark
• Polars
• DuckDB
• Dask
• Ray Data
• Apache Flink in Batch Mode
• Trino
• Engine Comparison
• Benchmark Setup
• Benchmark Task and Methodology
• Benchmark Results
• Choosing a Batch Engine