The VARIANT Type

BookNest's order events are deliberately uneven: only order_placed events carry customer_id and total. Before 4.0 you either reparsed a JSON string in every query or forced it into a struct and lost unplanned fields. VARIANT stores each value in a compact binary form that keeps its own field names and types; Spark 4.1 129 made it generally available, with shredding of frequent paths into typed Parquet 129 columns.

spark.read.text("data/raw/order_events.jsonl") \
     .selectExpr("parse_json(value) AS v").createOrReplaceTempView("events")
spark.sql("""
  SELECT v:type::string AS type, count(*) AS events,
         count(v:total) AS with_total, sum(v:total::decimal(10,2)) AS total
  FROM events GROUP BY 1 ORDER BY events DESC""").show()

+---------------+-------+----------+-----------+
|           type| events|with_total|      total|
+---------------+-------+----------+-----------+
|   order_placed|1000000|   1000000|34794290.24|
|     order_paid| 939907|         0|       NULL|
|  order_shipped| 937005|         0|       NULL|
|order_delivered| 929233|         0|       NULL|
|order_cancelled|  60059|         0|       NULL|
| order_returned|  38530|         0|       NULL|
+---------------+-------+----------+-----------+

v:path::type reads and casts one path, and a missing path is NULL rather than an error, so one query handles all six event shapes. Land raw events as VARIANT (Apache Kafka and Managed Cloud Kafka) and promote stable fields to typed columns once you trust them.