Spark memory management: real-world lessons and examples

Introduction

Memory management in Apache Spark is often where theory meets harsh reality: a job that works perfectly on a small dataset suddenly crashes with out-of-memory errors when you scale to production volumes. Spark memory management: real-world lessons and examples cuts through the configuration noise and shows you how executors actually use memory, why your jobs fail, and how to tune responsibly instead of guessing.

We've all been there—increasing spark.executor.memory arbitrarily, hoping it sticks. In this post, we'll walk through executor memory architecture, the unified memory model, why data skew kills more jobs than actual memory limits, and the discipline of measuring before tuning.

Executor memory and overhead

When you request 4GB of executor memory in Spark, you don't actually get 4GB of usable heap. The executor JVM itself needs overhead—space for threads, buffers, off-heap structures, and the JVM's own bookkeeping. This overhead is controlled by spark.executor.memoryOverhead, which defaults to 10% of executor memory or 384MB, whichever is larger.

Here's the real impact. If you set:

spark = SparkSession.builder \
    .config("spark.executor.memory", "4g") \
    .config("spark.executor.memoryOverhead", "512m") \
    .getOrCreate()

Your YARN or Kubernetes container requests 4.5GB total (4GB heap + 512MB overhead). The executor JVM starts with a 4GB heap, but Python workers, Netty buffers, and other off-heap allocations live in that 512MB overhead pool. If you're running PySpark, add spark.executor.pyspark.memory as well—Python processes run alongside the JVM and consume their own memory.

We learned this the hard way in a ride-hailing pipeline: eight executors with 2GB each seemed reasonable until container kills started at 5PM. Overhead was set to 384MB by default. With Python workers for ML feature engineering, we were hitting 2.384GB per executor, and the cluster only had 2.5GB available. Bumping overhead to 768MB and reducing heap to 1.5GB (still 2.268GB total) fixed the issue because we had actual breathing room.

The formula to remember:

container_size = executor_memory + memory_overhead + pyspark_memory

You don't choose container size directly; you add these three to know what you're actually requesting from your cluster.

Unified memory for execution and storage

Before Spark 1.6, memory was split rigidly: execution (shuffle buffers, sort structures) and storage (cached DataFrames) each got a fixed slice. If caching needed more space, execution starved. Today's unified memory model dynamically borrows between the two.

The total unified memory region is typically 60% of heap:

unified_memory = spark.executor.memory * 0.6

Within that pool, execution and storage negotiate. If your job caches heavily but also does large shuffles, the unified model lets execution borrow from storage's cache pool if needed, then reclaim it later. You can tune this with spark.memory.fraction and spark.memory.storageFraction, but in most cases the defaults work.

A concrete example from an e-commerce data team: they cached product catalogs (4GB) early in a job, then ran a large group-by aggregation that needed shuffle buffers. Under the old model, the shuffle would have failed. With unified memory, the 60% pool dynamically shifted—storage released when execution pressure spiked, then storage was evicted to disk (spilled) temporarily. The job completed, though slower than ideal due to disk I/O.

The lesson: don't micro-tune these fractions unless you've measured and found a real bottleneck. Enable detailed memory metrics first:

spark = SparkSession.builder \
    .config("spark.executor.memoryOverhead", "512m") \
    .config("spark.sql.execution.debug.maxToStringFields", "1000") \
    .getOrCreate()

df.cache()
df.count()

# Check executor logs for "MemoryStore updated" and spill events

OOM often means skew or huge partitions, not memory limits

Here's a counterintuitive truth: most OutOfMemoryError failures aren't actually caused by insufficient memory. They're caused by data skew or partitions so large they can't fit in an executor's allocation.

Picture this: a SaaS analytics pipeline processes user events. You join events on user_id, but 40% of events are from a demo account ID that never got cleaned. That single partition balloons to 8GB while others are 100MB. Even with 16GB executors, that partition fails when it reaches 9GB during the join operation.

The traceback says java.lang.OutOfMemoryError: Java heap space, but the real problem is skew. The quick fix—throwing more memory at the problem—masks the issue:

# Bad: throwing memory at skew
spark.executor.memory = "32g"  # wasteful, doesn't fix root cause

# Good: detect and fix skew
df_events.groupBy("user_id").count().orderBy(desc("count")).show(10)

# If one user has 40% of events, filter or salt the key:
from pyspark.sql.functions import rand

df_events_salted = df_events.withColumn(
    "user_id_salted",
    when(col("user_id") == "demo_account_id",
         concat(col("user_id"), lit("_"), (rand() * 10).cast("int")))
    .otherwise(col("user_id"))
)

In a trading platform, we had similar skew on symbol data—a few high-volume stocks created lopsided partitions. Measuring partition sizes before and after salting showed we could run the job on 2GB executors after fixing skew, versus crashing with 8GB before. That's a 4x efficiency gain just from addressing the real problem.

Always profile your data first:

1. Count total rows and measure average row size
2. Check for top N keys that dominate
3. Estimate partition size after joins or shuffles
4. Only then adjust memory if truly needed

Tune after measuring

The discipline of measuring before tuning separates efficient Spark jobs from flaky ones. You need visibility into what's actually happening in memory.

Start with Spark UI metrics. After running your job, navigate to the Executors tab and look for:

• Shuffle write/read bytes: massive values suggest expensive shuffle; consider pre-partitioning
• Spill bytes: if non-zero and large, execution and storage are fighting; may need more memory or better partitioning
• Peak memory usage per executor: tells you realistic memory needs

A structured approach:

from pyspark.sql.functions import spark_partition_id

# Log partition sizes
df.withColumn("partition_id", spark_partition_id()) \
    .groupBy("partition_id") \
    .count() \
    .agg({"count": ["min", "max", "avg", "stddev"]}) \
    .show()

# Check shuffle and spill in logs
spark.sparkContext.setLogLevel("INFO")
# Re-run and search logs for "spill" and "shuffle write"

For a SaaS CRM pipeline, measuring revealed that 90% of executors ran with under 2GB used, while 10% hit 14GB during a join. Instead of raising all executors to 16GB, we repartitioned the join key and tuned the shuffle partition count. Result: all executors stayed under 4GB, reduced GC pause times, and cut wall-clock runtime by 20%.

Concrete tuning sequence:

1. Set a baseline (e.g., 4GB executor, defaults for fractions)
2. Run a representative workload and measure peak memory and spill bytes
3. If spill is high, increase spark.sql.shuffle.partitions (start at 200 or 2× your cores)
4. If peak memory exceeds 70% of allocated, increase executor memory or reduce number of cached datasets
5. Re-measure and iterate

Common mistakes

Mistake 1: Setting executor memory without understanding overhead. A team requested 8GB executors but forgot overhead; containers were killed for exceeding cluster limits. They thought the problem was CPU.

Mistake 2: Caching everything. .cache() on intermediate DataFrames eats unified memory and spills to disk, defeating its purpose. Cache only the datasets you read multiple times.

Mistake 3: Ignoring skew and blaming memory. Doubling executor memory hides data skew; the job still takes forever because one task processes 80% of the data.

Mistake 4: Tuning without baselines. "We increased memory and it got faster" doesn't tell you if memory was the bottleneck or if you fixed something else incidentally. Use timestamps and logs to prove causation.

Key takeaways

• Executor memory = heap + overhead + (PySpark memory if applicable); know the total container size you're requesting.
• Unified memory dynamically shares between execution and storage; tune fractions only after measuring.
• OOM errors often signal data skew or huge partitions, not memory shortage—detect and fix skew before throwing memory at the problem.
• Measure partition sizes, spill bytes, and peak memory usage via Spark UI before tuning; iterate based on real data.
• Cache selectively, tune shuffle partitions for even distribution, and verify improvements with reproducible baselines.