spark-shell and pyspark

The spark-shell and PySpark Shell

Spark 129 ships three interactive shells: pyspark (Python), spark-shell (Scala) and spark-sql (SQL against the catalog). Each starts a driver with spark and sc ready and takes spark-submit's flags.

Starting the PySpark shell in local mode

pyspark --master "local[2]"

Output

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 4.2.0
      /_/
...
Spark context Web UI available at http://10.255.255.254:33040
Spark context available as 'sc' (master = local[2], app id = local-1790856189767).
...
>>> print(sc.master, spark.version, type(spark).__name__)
local[2] 4.2.0 SparkSession
>>> spark.range(1, 1_000_001).selectExpr("sum(id) AS total").show()
+------------+
|       total|
+------------+
|500000500000|
+------------+

The shell uses whatever python3 is first on the PATH unless PYSPARK_DRIVER_PYTHON says otherwise; set it and PYSPARK_PYTHON to your virtual environment so the driver and the workers agree on library versions.

MENU