Catalog and Managed Tables

The Catalog, Databases and Managed Tables

The catalog maps names such as booknest.delivered_lines to data, schema and format; Spark 129 's built-in one is spark_catalog, with a default database (database and schema are synonyms). A saveAsTable without a path is managed: Spark keeps the files under spark.sql.warehouse.dir and deletes them on DROP TABLE. With a path it is external: dropping it leaves your files.

Managed and external tables in the session catalogJavaScript

import os
spark.sql("CREATE DATABASE IF NOT EXISTS booknest")
(lines.where("status = 'delivered'").write.mode("overwrite")
 .saveAsTable("booknest.delivered_lines"))                     # managed: Spark owns the files
(customers.write.mode("overwrite").option("path", f"{os.getcwd()}/out/ext/customers")
 .saveAsTable("booknest.customers"))                           # external: you own the files
for t in spark.catalog.listTables("booknest"):
    if not t.isTemporary:                                      # temp views are listed too
        print(t.name, t.tableType, spark.table(f"booknest.{t.name}").count())
spark.sql("DROP TABLE booknest.delivered_lines")
spark.sql("DROP TABLE booknest.customers")
print(os.path.exists("spark-warehouse/booknest.db/delivered_lines"),
      os.path.exists("out/ext/customers"))

Output

customers EXTERNAL 50000
delivered_lines MANAGED 1229452
False True

MENU