Parquet 129 and ORC are formats for disk. Apache Arrow 129 is the columnar format for memory: a language-independent specification of how a column's values are laid out in RAM, plus libraries for C++, Java, Rust, Go, Python, R and more. When pandas 16,086 , Polars 268,908 , DuckDB 61,228 , Spark 129 and a Flight server all agree on that layout, data can pass between them without being converted, which used to cost more than the analysis itself. Wes McKinney and others launched Arrow in 2016; the current release is Arrow 25.0.1 (10 August 2026, Apache 2.0, github.com/apache/arrow (https://github.com/apache/arrow 17,169 )), the version of pyarrow 129 used throughout this chapter.
MENU
Apache Arrow
Apache Arrow and In-Memory Columnar Data