Tuning and Memory

Most slow or failing Spark 129 jobs come down to memory, file layout or shuffle sizing. This section provokes each problem on purpose so you recognize the symptoms; timings come from the shared 4-CPU host, so compare ratios.

Subsections

• The Executor Memory Model
• Executor and Driver Memory
• Out-of-Memory Causes
• Spill and GC
• Small Files Problem
• Shuffle Partition Tuning
• A Tuning Checklist