Spark Under the Hood

Spark 129 tuning makes sense once you know what happens between calling an action and getting rows back. This section follows one BookNest query, revenue by genre over 1.38 million sample order lines, through the engine, in one PySpark 129 session that the first listing creates.

Subsections

• Driver and SparkContext
• Executors and Cluster Managers
• The Logical Plan
• The DAG of Transformations
• Jobs, Stages and Shuffles
• Tasks and Task Scheduling
• Shuffles and Spilling
• Catalyst Optimizer
• Tungsten and Code Generation
• Adaptive Query Execution
• A Query Through the Engine