Spark configuration essentials: best practices explained

Introduction

Most data engineering teams inherit Spark jobs that run slowly, fail mysteriously, or consume far more resources than they should—and the culprit is rarely the code itself. Spark configuration essentials: best practices explained matter because a poorly tuned cluster can waste thousands in cloud costs while leaving the same work unfinished. We'll walk through the settings that have the highest impact on performance and stability in production environments.

Configuration tuning is not a one-time activity. As data volumes grow and workload patterns shift, revisiting these settings keeps your pipelines efficient and predictable. This guide focuses on the four categories that matter most for most teams: shuffle partitions, executor allocation, dynamic scaling, and query optimization.

spark.sql.shuffle.partitions

This single setting controls how many partitions Spark creates during shuffle operations—joins, aggregations, and window functions. The default value is 200, which works fine for small datasets but becomes a bottleneck or resource hog as you scale.

When you run a group-by operation on 10 GB of data with the default 200 partitions, Spark creates 200 output partitions. If you're working with 1 TB, those same 200 partitions become oversized, forcing each executor to hold too much data in memory and potentially spilling to disk. Conversely, if your shuffle produces tiny partitions, task overhead balloons and scheduling becomes inefficient.

A practical heuristic: aim for partitions between 100 MB and 200 MB in size.

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .config("spark.sql.shuffle.partitions", "400") \
    .getOrCreate()

# For a 1 TB input, 400 partitions often yields 2.5 GB each
# Adjust based on your executor memory and data skew
df = spark.read.parquet("s3://bucket/data/")
result = df.groupBy("country").count()
result.write.parquet("s3://bucket/output/")

In a real e-commerce pipeline, we found that processing 2 TB of order events with the default 200 partitions led to 20-minute shuffle waits. Increasing to 600 partitions—accounting for our 8-core executors and available memory—cut shuffle time to 4 minutes. The trade-off is slightly higher scheduling overhead, but the memory pressure relief more than compensates.

One warning: too many partitions waste memory on metadata and increase GC pressure. We rarely go above 2000 for typical workloads.

Executor memory and cores

These two settings define the compute capacity available to your Spark application. Executor memory is how much heap each JVM process gets; executor cores control parallelism within that executor.

Setting these values requires balancing available cluster resources against your application's needs. A common pattern in SaaS analytics is 4-core executors with 16 GB of memory each. A trading firm processing real-time market data might use 8 cores with 32 GB, because they need both parallelism and large buffers for intermediate join results.

spark = SparkSession.builder \
    .master("yarn") \
    .config("spark.executor.memory", "16g") \
    .config("spark.executor.cores", "4") \
    .config("spark.driver.memory", "4g") \
    .config("spark.driver.cores", "2") \
    .getOrCreate()

Memory per executor includes both task memory and storage memory. Spark divides it roughly as:

• 60% for RDD/DataFrame caching and broadcast variables
• 40% for task execution and shuffles

If you set executor memory too low, tasks spill to disk during shuffles or joins, which kills performance. Typical latency impact: 5-10x slowdown. If you set it too high, you waste money on unused capacity and risk running out of cluster resources.

A helpful approach: start with the memory requirement of your largest single task, add 30% overhead, then round to a power of 2 (8, 16, 32, 64 GB). For cores, 4 to 8 is usually optimal; beyond 8, YARN scheduling and lock contention become problematic.

Dynamic allocation

Static resource allocation means reserving the same number of executors for the entire job, even if you only need them during the shuffle phase. Dynamic allocation adds executors when demand peaks and removes them when the workload eases, cutting idle time and costs.

For batch jobs running on-demand (e.g., daily ETL), dynamic allocation can reduce cloud spend by 40-60%. For long-running streaming applications, the benefit is smaller because executors rarely sit idle between micro-batches.

spark = SparkSession.builder \
    .config("spark.dynamicAllocation.enabled", "true") \
    .config("spark.dynamicAllocation.minExecutors", "2") \
    .config("spark.dynamicAllocation.maxExecutors", "100") \
    .config("spark.dynamicAllocation.initialExecutors", "10") \
    .config("spark.dynamicAllocation.schedulerBacklogTimeout", "1s") \
    .config("spark.dynamicAllocation.sustainedSchedulerBacklogTimeout", "30s") \
    .getOrCreate()

When dynamic allocation is on, Spark monitors the task backlog. If tasks are queued (not running) for 1 second, it requests more executors. If no new tasks arrive for 30 seconds, it scales down. This aggressive scaling works well for unpredictable workloads.

In a travel booking system processing hotel inventory updates, enabling dynamic allocation reduced average job run time by 15 minutes (from 90 to 75 minutes) while cutting executor-hours by 35%. The initial phase of the pipeline (data fetch) runs on 2-3 executors; the shuffle phase scales to 30. After the join, it drops back to 5 for the final write.

Caveats: dynamic allocation doesn't work with client-mode driver deployments (only cluster mode) and requires an external shuffle service in YARN. On Kubernetes, it's more straightforward.

Adaptive Query Execution (AQE) settings

AQE is a suite of optimizations that Spark applies mid-execution based on runtime statistics. It reorders joins, combines small partitions, and skips empty partitions—all decisions that the planner couldn't make without seeing actual data.

Enabling AQE is almost always correct. The overhead is minimal (a few hundred milliseconds), and the benefits compound across large pipelines.

spark = SparkSession.builder \
    .config("spark.sql.adaptive.enabled", "true") \
    .config("spark.sql.adaptive.coalescePartitions.enabled", "true") \
    .config("spark.sql.adaptive.skewJoin.enabled", "true") \
    .config("spark.sql.adaptive.skewJoin.skewFactor", "5.0") \
    .getOrCreate()

Let's break down each setting:

• spark.sql.adaptive.enabled: Master switch. Turns on all AQE features.
• spark.sql.adaptive.coalescePartitions.enabled: After a shuffle, combines partitions if the total size falls below a threshold (default 64 MB). Reduces task count and overhead.
• spark.sql.adaptive.skewJoin.enabled: Detects skewed partitions in join keys and splits them, preventing one task from processing 90% of the data while others idle.
• spark.sql.adaptive.skewJoin.skewFactor: A partition is considered skewed if its size exceeds the median by this factor. Default 5 is reasonable; lower it to 2.5 if your data is highly uneven.

Real example: a CRM analytics job joined users (10 million rows, highly skewed by country—50% from one country) with events (500 million rows, also skewed). Without AQE, the join took 45 minutes because one partition holder processed all the major-country data. With AQE and skewFactor=3, Spark auto-split the skewed partition into smaller chunks, and the same join finished in 7 minutes.

One nuance: AQE can't help if you've already applied a prior filter that removes the skew. Plan your transformations—filter early, join late.

Common mistakes

Leaving shuffle.partitions at 200: This is the single most common misconfiguration. Teams process terabytes with the default and wonder why executors OOM or shuffle takes hours. Audit your Spark configs on day one and set this intentionally.

Setting executor memory to the node maximum: If your cluster has 256 GB nodes and you set executor memory to 200 GB, you can run only one executor per node and waste resources. Leave 20-30% headroom for the OS and other services. A 256 GB node typically supports 4–5 executors of 48 GB each.

Enabling dynamic allocation without tuning timeouts: The default schedulerBacklogTimeout is 1 second, which can cause thrashing—adding and removing executors rapidly. For stable batch jobs, increase it to 30-60 seconds.

Ignoring broadcast size limits: By default, tables smaller than 10 MB are broadcast to all executors during joins. If your cluster has 100 executors and you broadcast a 100 MB table, that's 10 GB of memory spent. Use spark.sql.autoBroadcastJoinThreshold to tune this threshold.

Key takeaways

• spark.sql.shuffle.partitions: Target 100–200 MB partitions; start at 200 and scale with data size and executor capacity.
• Executor memory and cores: Balance available cluster resources; 4–8 cores per executor and 16–32 GB memory cover most workloads.
• Dynamic allocation: Enable for batch jobs to reduce costs and wait times; less beneficial for streaming applications.
• AQE: Enable all AQE settings (coalesce, skew join) for automatic mid-execution optimization.
• Monitor task duration, shuffle time, and executor memory usage in Spark UI; these metrics reveal which config knobs to turn next.
• Configuration tuning is iterative: measure, adjust, re-measure. One setting rarely fixes everything.