Schemas

Schemas

Schemas: Inference Versus Explicit Definition

Without a schema, the JSON and CSV readers infer one by scanning the data before you have asked for anything: a full pass unless you set samplingRatio, which risks missing a rare field. Parquet 129 keeps its schema in the file footer. A DDL string, or the equivalent StructType, skips the scan and fixes the types.

Schema inference versus an explicit DDL schemaJavaScript

import time
ORDER_SCHEMA = """order_id BIGINT, customer_id INT, order_ts TIMESTAMP, channel STRING,
  status STRING, currency STRING,
  items ARRAY<STRUCT<book_id: INT, qty: INT, unit_price: DECIMAL(6,2)>>,
  coupon STRING, discount DECIMAL(8,2), total DECIMAL(10,2)"""
RAW = "data/raw/orders.jsonl"
spark.read.parquet("data/orders.parquet").count()                  # warm up the JVM
t0 = time.perf_counter()
explicit = spark.read.schema(ORDER_SCHEMA).json(RAW)
t1 = time.perf_counter()
inferred = spark.read.json(RAW)                                    # a full pass over the file
print(f"explicit {t1 - t0:.2f}s, inferred {time.perf_counter() - t1:.2f}s")
for col in ("customer_id", "order_ts", "total"):
    print(f"{col:<12} inferred {inferred.schema[col].dataType.simpleString():<7} "
          f"explicit {explicit.schema[col].dataType.simpleString()}")
print("sum(total):", inferred.agg(F.sum("total")).first()[0],
      explicit.agg(F.sum("total")).first()[0])

Output

explicit 1.45s, inferred 8.26s
customer_id  inferred bigint  explicit int
order_ts     inferred string  explicit timestamp
total        inferred double  explicit decimal(10,2)
sum(total): 34794290.24006404 34794290.24

Inference took between five and six times as long before any query ran, and guessed wrong where it matters: timestamps stayed strings and money became double, so revenue drifted by six hundredths of a cent (Numbers and Precision's floating-point trap). Keep explicit schemas for jobs in one module. The mode option handles lines that do not fit: PERMISSIVE (default) nulls the bad fields, DROPMALFORMED drops the row, FAILFAST stops the job.

MENU