explain() and Physical Plans

Reading explain() and Physical Plans

In the session of Listing 5.9.1, explain() prints the genre query's physical plan; mode="formatted" separates a compact tree from numbered operator details, "extended" adds the logical plans (The Logical Plan), "cost" adds statistics and "codegen" the generated Java.

The genre query's formatted plan

genre.explain(mode="formatted")

Output

== Physical Plan ==
AdaptiveSparkPlan (29)
+- == Final Plan ==
   ResultQueryStage (18)
   +- * HashAggregate (17)
      +- AQEShuffleRead (16), coalesced
         +- ShuffleQueryStage (15), Statistics(sizeInBytes=1328.0 B, rowCount=22)
            +- Exchange (14)
               +- * HashAggregate (13)
                  +- * Project (12)
                     +- * BroadcastHashJoin Inner BuildRight (11)
                        :- * Project (4)
                        :  +- * Filter (3)
                        :     +- * ColumnarToRow (2)
                        :        +- Scan parquet  (1)
                        +- BroadcastQueryStage (10), Statistics(sizeInBytes=32.0 MiB,
                          rowCount=6)
...
+- == Initial Plan ==
...
(1) Scan parquet
Output [4]: [status#14, book_id#15, qty#16, unit_price#17]
PushedFilters: [IsNotNull(status), EqualTo(status,delivered), IsNotNull(book_id)]
ReadSchema: struct<status:string,book_id:int,qty:int,unit_price:decimal(6,2)>
...
(11) BroadcastHashJoin [codegen id : 2]
Left keys [1]: [cast(book_id#15 as bigint)]
Right keys [1]: [book_id#35L]
...
(14) Exchange
Arguments: hashpartitioning(genre#24, 200), ENSURE_REQUIREMENTS, [plan_id=136]
...

Because the query had already run, AQE printed the final plan above the initial one, with the real sizes it learned: 22 rows reached the exchange, and the six-row catalog became a 32 MiB broadcast hash table. Stars mark operators fused by code generation. Read the tree bottom-up, then check four things in the details. Scans: ReadSchema should list only the columns you use and PushedFilters your filters. Exchanges: each is a shuffle; count them and look at their keys. Join types: a SortMergeJoin against a small table is a missed broadcast. Casts: cast(book_id#15 as bigint) shows the two tables disagree on the key type (INT in order lines, BIGINT in the catalog), harmless here but able to defeat bucketing. Before an action you would see only the initial plan (isFinalPlan=false).

MENU