DataFrame.cache() means MEMORY_AND_DISK_DESER (the Storage tab's "Disk Memory Deserialized"). DataFrames are always cached as compressed column batches, so the level mostly decides where partitions go when memory runs short.
| Level | Where | When memory is short | Use for |
|---|---|---|---|
| MEMORY_AND_DISK_DESER | Memory, then disk | Spills partitions to disk | The default, most cases |
| MEMORY_ONLY | Memory | Recomputes missing partitions | Small, cheap-to-rebuild data |
| DISK_ONLY | Executor disk | Not applicable | Expensive results, tight memory |
| MEMORY_AND_DISK_2 and other _2 | Two executors | As the base level | Long jobs on unreliable nodes |
| OFF_HEAP | Off-heap memory | Needs off-heap memory configured | Large caches, less GC |
Cached data competes with execution for executor memory (The Executor Memory Model); a "Fraction Cached" below 100% in the Storage tab means partitions are being recomputed.