Secrets in Spark Jobs

Secrets and Credentials in Spark Jobs

Jobs need credentials for object stores, databases and APIs. Spark 129 hides any configuration key or value matching spark.redaction.regex (default (?i)secret|password|token|access[.]?key) in the UI, the REST API and event logs, but redaction is display-only: the value is still in the configuration.

What redaction hides, and what it does notJavaScript

import json, os, urllib.request
from pyspark.sql import SparkSession
spark = (SparkSession.builder.master("local[2]").config("spark.ui.port", "33040")
         .config("spark.hadoop.fs.s3a.secret.key", "s3cr3t-from-the-code")      # don't do this
         .config("spark.booknest.db.password", os.environ.get("BOOKNEST_DB_PASSWORD", ""))
         .getOrCreate())
sc = spark.sparkContext
env = json.load(urllib.request.urlopen(
    f"{sc.uiWebUrl}/api/v1/applications/{sc.applicationId}/environment"))
for key, value in env["sparkProperties"] + env["hadoopProperties"]:
    if key.startswith(("spark.", "fs.s3a")) and ("secret" in key or "password" in key):
        print(f"UI/REST  {key} = {value}")
print("in code ", spark.conf.get("spark.hadoop.fs.s3a.secret.key"))
spark.stop()

Output

UI/REST  spark.booknest.db.password = *********(redacted)
UI/REST  spark.hadoop.fs.s3a.secret.key = *********(redacted)
UI/REST  fs.s3a.secret.key = *********(redacted)
in code  s3cr3t-from-the-code

Redaction protected the UI, but the secret written into the code is now in version control, and any code in the job can read it back. Prefer, in order: no secret at all (IAM roles for EMR, workload identity on GKE 1 and Kubernetes 5,150 service accounts, managed identities on Azure 6 ), then short-lived credentials vended by a catalog, then a secret manager (AWS Secrets Manager 24 , Google Secret Manager, Azure Key Vault, HashiCorp Vault, Kubernetes secrets mounted as files) read at start-up, as the database password here came from an environment variable. On Kubernetes (Spark on Kubernetes), Spark mounts secrets itself: spark.kubernetes.driver.secrets.booknest-db=/etc/secrets mounts the Kubernetes secret booknest-db as files in the driver pod, and spark.kubernetes.executor.secretKeyRef.DB_PASSWORD=booknest-db:password exposes one key as an environment variable in every executor. Never print configuration in logs, and add your own key patterns to spark.redaction.regex when they do not contain the default words.

MENU