Authentication and Encryption

spark.authenticate makes every RPC connection between driver, master, workers and executors prove knowledge of a shared secret. On YARN and Kubernetes 5,150 , Spark 129 generates a secret per application; on Standalone you distribute it yourself. spark.network.crypto.enabled then encrypts RPC and shuffle traffic with AES (TLS through spark.ssl.rpc.* is the alternative), and spark.io.encryption.enabled encrypts shuffle and spill files on local disk. The listing starts a one-worker Standalone cluster on the lane's ports with all three switched on and submits the same job three ways.

secure/spark-defaults.conf (the secret is appended by the listing)

spark.authenticate                  true
spark.network.crypto.enabled        true
spark.io.encryption.enabled         true
spark.ui.port                       33040
spark.master.rest.enabled           false

A Standalone cluster that rejects clients without the secretShell

mkdir -p secure && rm -f secure/spark-defaults.conf && umask 077         # owner-only files
{ cat "$DEMOS/secure/spark-defaults.conf"
  echo "spark.authenticate.secret $(openssl rand -hex 32)"; } > secure/spark-defaults.conf
export SPARK_CONF_DIR=$PWD/secure SPARK_LOG_DIR=$PWD/logs SPARK_PID_DIR=$PWD/pids
export SPARK_MASTER_HOST=localhost SPARK_MASTER_PORT=33077 SPARK_MASTER_WEBUI_PORT=33081
$SPARK_HOME/sbin/start-master.sh > /dev/null
$SPARK_HOME/sbin/start-worker.sh spark://localhost:33077 --port 33078 --webui-port 33082 \
  --cores 2 --memory 2g > /dev/null
sleep 10
echo 'from pyspark.sql import SparkSession
print("count:", SparkSession.builder.getOrCreate().range(10**6).count())' > secure/job.py
NOAUTH="--conf spark.authenticate=false --conf spark.network.crypto.enabled=false"
for who in right-secret wrong-secret no-auth; do
  case $who in
    wrong-secret) OPTS="--conf spark.authenticate.secret=guessed" ;;
    no-auth) OPTS="$NOAUTH --conf spark.io.encryption.enabled=false" ;;
    *) OPTS="" ;;
  esac
  timeout 90 spark-submit --master spark://localhost:33077 $OPTS secure/job.py \
    > logs/auth.out 2>&1
  rc=$?                                                     # the result, or why it failed
  why=$(grep -m1 -oE 'count: [0-9]+|Exception while bootstrapping client|Expected Sasl\w+' \
        logs/auth.out)
  echo "$who: exit $rc, $why"
done
$SPARK_HOME/sbin/stop-worker.sh > /dev/null; $SPARK_HOME/sbin/stop-master.sh > /dev/null

Output

right-secret: exit 0, count: 1000000
wrong-secret: exit 1, Exception while bootstrapping client
no-auth: exit 1, Expected SaslMessage

Only the client holding the secret ran; the master refused the other two during the connection handshake, before any application was registered. The secret lives in a file readable only by its owner (umask 077), not on a command line where ps would show it. Two details surfaced while building this listing: on Standalone the spark.authenticate.secret.file property was not enough for the driver (A secret key must be specified via the spark.authenticate.secret config), and the master's REST submission server opened port 6066 until spark.master.rest.enabled was set to false. The web UI is a separate matter: Spark ships JWSFilter for token checks through spark.ui.filters, and spark.acls.enable with spark.ui.view.acls and spark.admin.acls controls who may view or kill jobs.

MENU