The master URL tells an application where to get executors. In local mode, used throughout this chapter, driver and executor share one JVM and the bracketed number sets the task threads.
| Master URL | Meaning |
|---|---|
| local, local[4], local[*] | One JVM with 1, 4 or one-per-core task threads |
| local[4,2] | As local[4], retrying each failed task up to 2 times |
| spark://host:7077 | A Spark Standalone master |
| yarn | A Hadoop 129 YARN cluster, found through HADOOP_CONF_DIR |
| k8s://https://host:443 | A Kubernetes 5,150 API server (Spark on Kubernetes) |
| sc://host:15002 | A Spark Connect 129 server (--remote, Spark Connect) |
The deploy mode decides where the driver runs. In client mode, the default, it runs inside spark-submit on your machine; in cluster mode the cluster manager starts it on a cluster node and your terminal can disconnect. Spark Standalone does not support cluster mode for Python applications.