Spark performance tuning for Spark & Data Eng developers

Spark performance tuning for Spark & Data Eng developers

We've all watched a Spark job crawl across the cluster, burning through time and resources while returning results that should have been instant. The gap between a slow pipeline and a fast one often isn't about hardware—it's about how well you're using the engine. In this article, we'll walk through the core tuning levers that turn sluggish workloads into efficient data processing machines.

Why performance tuning matters in production

When you're running terabytes of data through Spark daily, small inefficiencies compound into real costs. A job that runs in 30 minutes instead of 5 consumes 6× the cluster resources, delays downstream analytics, and frustrates your stakeholders. Performance tuning isn't a nice-to-have—it's how we keep pipelines sustainable.

Read less — Partition pruning and column pruning

The fastest data to process is data you never read. This is the principle behind partition and column pruning: telling Spark exactly which data it needs, so it skips everything else.

Partition pruning works when your data is organized by date, region, or another dimension that filters naturally. When you query with a WHERE clause on that partition column, Catalyst (Spark's query optimizer) discards entire partitions without reading them.

# Good: partition pruning kicks in
df = spark.read.parquet("s3://data/events/year=2024/month=01/")
filtered = df.filter(df.day == 15)

# Better: let Catalyst prune for you
df = spark.read.parquet("s3://data/events/")
filtered = df.filter((df.year == 2024) & (df.month == 1) & (df.day == 15))
# Spark reads only the matching partitions

The second approach is better because Spark can evaluate partition columns before opening files. If your data lives under year=2024/month=01/day=15/, Spark will skip everything else.

Column pruning is equally powerful. When you select only the columns you need, Spark reads only those columns from parquet files (which store data in columnar format). Loading a 500-column dataset but needing only 10 is wasteful; column pruning prevents it.

# Reading from parquet
df = spark.read.parquet("s3://data/transactions/")
# Spark reads only user_id and amount from disk
result = df.select("user_id", "amount").groupBy("user_id").sum()

In practice, this means:

• Design your partition scheme around common filter dimensions (date, region, tenant).
• Always select specific columns early; avoid select("*") in production pipelines.
• Use Spark UI to confirm column pruning is active (you'll see fewer bytes scanned).

Join smarter — Broadcast small tables

Joins are expensive. By default, Spark uses a shuffle join: both tables get repartitioned and sent across the network. When one side is small enough to fit in memory, a broadcast join is orders of magnitude faster.

In a broadcast join, Spark ships the small table to every executor in a single pass, then performs a local lookup. No shuffle. No network shuffle shuffle overhead.

from pyspark.sql.functions import broadcast

# Small dimension table
dim_users = spark.read.parquet("s3://data/users/").filter(df.active == True)
# Large fact table
events = spark.read.parquet("s3://data/events/")

# Explicit broadcast (recommended in production)
result = events.join(
    broadcast(dim_users),
    on="user_id",
    how="inner"
)

# Spark will broadcast dim_users instead of shuffling both tables

Spark has an auto-broadcast threshold (default 10 MB), but relying on it is risky. In production, we always explicitly broadcast. Why? It makes intent clear in code review and prevents surprises if data grows.

Watch out for this gotcha: if your "small" table isn't actually small, broadcast will fail or exhaust executor memory. Check the actual size in your Spark UI before broadcasting anything larger than a few hundred MB.

Fix skew — Salting or Adaptive Query Execution skew handling

Data skew is when some partition keys have vastly more rows than others. Imagine joining events by user_id, but one user has 50% of all events. That single partition becomes a bottleneck while other executors sit idle.

We have two main remedies.

Salting: Manual skew handling

Add a random salt to skewed keys during the join, then remove it after. This artificially spreads the skewed data across multiple partitions.

from pyspark.sql.functions import rand, concat, lit, when

# Detect and salt skewed keys
skewed_user_ids = [12345, 67890]  # Known hotspots
num_salt_buckets = 10

events_salted = events.withColumn(
    "salt",
    when(events.user_id.isin(skewed_user_ids), (rand() * num_salt_buckets).cast("int"))
    .otherwise(0)
).withColumn(
    "user_id_salted", concat(events.user_id, lit("_"), events.salt)
)

users_replicated = dim_users.withColumn("salt", lit(0))
for i in range(1, num_salt_buckets):
    users_with_salt = dim_users.withColumn("salt", lit(i))
    users_replicated = users_replicated.unionByName(users_with_salt)

users_replicated = users_replicated.withColumn(
    "user_id_salted", concat(users_replicated.user_id, lit("_"), users_replicated.salt)
)

# Join on salted keys
result = events_salted.join(users_replicated, on="user_id_salted")

# Remove salt column if no longer needed
result = result.drop("salt", "user_id_salted")

Adaptive Query Execution (AQE): Automatic skew handling

Spark 3.0+ includes AQE, which detects and fixes skew automatically. Enable it:

spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")

With AQE enabled, Spark monitors partition sizes during execution. If one partition is significantly larger than others (default 5× the median), Spark splits it and redistributes the work. This happens transparently—no code changes needed.

In production, we enable AQE by default. It catches skew we didn't anticipate and adds minimal overhead. Manual salting is reserved for known, severe skew cases where AQE alone won't help.

Right-size resources — Executor memory and cores for the workload

Throw too many cores at an executor and you'll hit memory contention. Give too little memory and you'll spill to disk constantly. The right balance depends on your workload.

Start with these principles:

• Executor memory: Allocate enough for your data plus overhead. A rule of thumb: total memory ÷ number of executors, minus 20% for JVM and system overhead.
• Executor cores: Typically 4–8 cores per executor for OLAP workloads. Higher numbers increase task scheduling overhead; lower numbers waste parallelism.
• Driver memory: Allocate more if you're collecting large results or holding broadcasts in memory.

spark = SparkSession.builder \
    .appName("tuned-app") \
    .master("yarn") \
    .config("spark.executor.memory", "8g") \
    .config("spark.executor.cores", "4") \
    .config("spark.driver.memory", "4g") \
    .config("spark.dynamicAllocation.enabled", "true") \
    .config("spark.dynamicAllocation.minExecutors", "2") \
    .config("spark.dynamicAllocation.maxExecutors", "20") \
    .getOrCreate()

Dynamic allocation is powerful: Spark adds executors under load and removes them when idle, saving cost. In cloud environments (AWS, GCP, Azure), this is almost always worth enabling.

Use the Spark UI's Executors tab to see how much memory and CPU each executor actually uses. If memory is 90%+ full, increase it. If average CPU is below 50%, reduce executor cores.

Common mistakes

• Ignoring the explain plan: Call df.explain(extended=True) before running large queries. Look for unexpected full scans or shuffle operations.
• Joining without filtering first: Always filter both sides before joining. Joining 1 billion rows then filtering is slower than filtering first, then joining 100 million.
• Broadcasting a table that's too large: Broadcast failures are silent until runtime. Validate size in development.
• Over-partitioning: More partitions mean more overhead. Aim for 128–256 MB per partition as a rule of thumb.
• Tuning without measuring: Run jobs with and without changes. Spark UI is your truth—look at duration, memory, spill, and shuffle metrics.

Key takeaways

• Partition and column pruning reduce I/O. Use WHERE clauses on partition columns and select() specific columns early.
• Broadcast small tables to avoid shuffle joins. Use explicit broadcast() in production code, not auto-detection.
• Handle data skew with Adaptive Query Execution (AQE) or manual salting for severe cases.
• Right-size executor memory and cores based on actual workload usage. Monitor the Spark UI and adjust iteratively.
• Always explain and measure. The Spark UI is your debugging tool; intuition often misleads in distributed systems.