A user-defined function (UDF) runs your Python code inside a Spark 129 job, and how data travels between the JVM and Python decides whether that costs a few percent or several times the runtime. The Vectorized Execution Model explained why vectorized, batch-at-a-time execution wins; this section measures it in Spark.
MENU
UDFs and Arrow
UDFs, Pandas UDFs and Apache Arrow