Temporary and Global Views

A view stores a plan, not data, so every query re-runs it (Caching and Persistence Levels caches). A temporary view belongs to one SparkSession; a global temporary view lives in the global_temp database and is visible to every session of the application.

Session-scoped and application-scoped views

from pyspark.errors import AnalysisException
spark.sql("""CREATE OR REPLACE TEMP VIEW returned AS
             SELECT * FROM lines WHERE status = 'returned'""")
lines.where("status = 'returned'").createOrReplaceGlobalTempView("returned_g")
other = spark.newSession()                     # same application, separate SQL session
for name, s in [("spark", spark), ("other", other)]:
    views = sorted(t.name for t in s.catalog.listTables())
    shared = s.sql("SELECT count(*) FROM global_temp.returned_g").first()[0]
    print(f"{name:<6} views {views}, global_temp rows {shared:,}")
try:
    other.sql("SELECT count(*) FROM returned").show()
except AnalysisException as e:
    print("other:", e.getCondition())

Output

spark  views ['books', 'customers', 'lines', 'orders', 'returned'], global_temp rows 53,307
other  views [], global_temp rows 53,307
other: TABLE_OR_VIEW_NOT_FOUND

Sessions from newSession() share the executors but not views or SQL settings, which is how one server isolates concurrent users.

MENU