Data skew: everything engineers need to know in practice

The reality of data skew

You deploy a Spark job that's supposed to finish in five minutes. The monitoring dashboard shows 199 tasks complete in under 10 seconds. One task? Still running after 45 minutes. Welcome to data skew—one of the most common performance killers in production data pipelines. Data skew: everything engineers need to know in practice is not just theory; it's a survival skill when you're running hundreds of jobs weekly across terabytes of data.

Skew happens silently. Your cluster looks busy. CPU is high. Memory usage seems normal. But you're paying for 200 machines to wait on one. Let's walk through how to spot it, understand it, and fix it with techniques we've used to cut job times from hours to minutes.

Symptom: most tasks finish, a few run much longer

The first sign is always the same: your task duration histogram looks broken. Open the Spark UI, navigate to the Stages tab, and look at the task timeline. You'll see a wall of green bars—tasks finishing in seconds. Then a handful of orange bars that dwarf everything else.

In a typical scenario, you're processing user events from an e-commerce platform. You have 200 partitions, so Spark creates 200 tasks. After two minutes, 198 tasks are done. Two tasks are still grinding away. The job's total runtime becomes the sum of the stragglers, not the average of the fast ones.

Here's what we saw in production last quarter: a daily aggregation job grouped events by user_id. Most user IDs touched 1,000–5,000 events per day. One account—a test tenant with synthetic data—had 450 million events. That single partition took 40 minutes while the other 9,999 partitions finished in 4 minutes. The job went from 4 minutes to 40 minutes because of one key.

The metric to watch is the max task duration divided by the median task duration. If that ratio is consistently above 10x, you have skew. If it's above 50x, you're in trouble. Most healthy jobs run at 1.5–3x.

Cause: some keys have far more rows than others

Data skew is fundamentally about uneven key distribution. When you shuffle data in Spark—whether through a join, groupByKey, or aggregation—rows with the same key go to the same partition. If one key has 100 million rows and another has 100, one partition does 1,000x more work.

Why does this happen? Several reasons we've encountered:

• Logical imbalances: A "NULL" category absorbs all rows where an attribute is missing. A "default" or "unknown" value is a catch-all bucket.
• Business power laws: A few customers generate most of your traffic. One product SKU is a bestseller. A particular region concentrates most events.
• Data quality issues: Dates stored as "2099-12-31" (placeholder for missing dates). A shared account ID used for testing or system jobs.
• Temporal patterns: Event timestamps clustered around specific hours due to batch processing or geographical time zones.

The root cause is almost always: you didn't know your data distribution before writing the query. The fix begins with understanding it.

Start with a simple profile:

df.groupBy("key_column").count().orderBy("count", ascending=False).show(20)

If the top few keys have orders of magnitude more rows than the median, you have skew. Capture these metrics in a monitoring dashboard. We run this daily on key transformations and alert when the ratio shifts.

Salting: add random suffixes to hot keys

Salting is the oldest, most reliable fix for skew. The idea: temporarily explode hot keys into multiple synthetic keys, process in parallel, then merge results.

Here's a real example from a CRM pipeline. We were joining customer records with a transactions table. One customer—a B2B account—had 500 million transactions. Everyone else had thousands to hundreds of thousands. The join was bottlenecked on that single key.

import random
from pyspark.sql import functions as F

# Identify the hot key (or keys)
hot_keys = df.groupBy("customer_id").count().filter(F.col("count") > 1_000_000).select("customer_id").collect()
hot_key_set = set(row.customer_id for row in hot_keys)

# Salt: add a random suffix to hot keys
def add_salt(row):
    if row.customer_id in hot_key_set:
        salt = random.randint(0, 9)  # Distribute across 10 buckets
        return (f"{row.customer_id}_{salt}", row)
    return (row.customer_id, row)

salted_df = df.rdd.map(add_salt).toDF(["salted_key", "data"]).select("data.*", F.col("salted_key"))

# Now join on salted_key
joined = salted_df.join(other_salted_df, "salted_key")

# Remove salt suffix and aggregate if needed
result = joined.withColumn(
    "customer_id", 
    F.regexp_replace(F.col("customer_id"), "_[0-9]$", "")
).groupBy("customer_id").agg(...)

The salt parameter (10 in the example) controls how many partitions the hot key spreads across. Too few—still skewed. Too many—overhead and network traffic increase. We usually aim for 5–20 depending on the data size.

Salting works, but it's manual. You need to identify hot keys upfront and handle the desalting logic. It also bloats the key space temporarily, which can increase memory pressure. It's perfect for a join bottleneck or a known problematic aggregation, but less elegant for exploratory queries.

AQE: Adaptive query execution splits skewed partitions

Adaptive Query Execution (AQE) is Spark's answer to skew without manual intervention. Introduced in Spark 3.0 and refined since, AQE automatically detects and handles skew at runtime.

Enable it in your Spark session:

spark = SparkSession.builder \
    .config("spark.sql.adaptive.enabled", "true") \
    .config("spark.sql.adaptive.skewJoin.enabled", "true") \
    .config("spark.sql.adaptive.skewJoin.skewFactor", "5.0") \
    .getOrCreate()

With AQE enabled, Spark monitors shuffle operations. When it detects a partition significantly larger than the median (controlled by skewFactor), it splits that partition into smaller ones dynamically. Each smaller piece is sent to a different executor, parallelizing the work.

We tested AQE on a ride-hailing pipeline where trip IDs were grouped by geography. Peak hours created skewed location buckets. Without AQE, the job took 22 minutes. With AQE, 8 minutes—no code changes. That's the appeal: passive performance improvement.

The catch: AQE adds latency to the planning phase. It also requires extra memory during the split. For very large partitions (>500 GB), you may still need manual salting or pre-partitioning. AQE also struggles with skew in the map phase of a map-reduce operation; it primarily targets shuffle operations.

A balanced approach: use AQE as your baseline, monitor performance, and layer in salting for known problem keys.

Common mistakes

• Salting without desalting: We've seen jobs that salt the key but never remove the salt before writing the output. You end up with synthetic keys in your data lake. Automation and tests catch this, but it's costly.
• Assuming AQE solves everything: AQE is powerful but not magic. If one key is 1,000x the size of the median, splitting it across 10 partitions still leaves you with 100x skew on each split. Sometimes you need both AQE and domain logic.
• Not profiling after fixes: Apply a fix, assume it's done. Six months later, data distribution shifts and skew returns. Rerun your key distribution query monthly.
• Ignoring NULL handling: NULLs in keys often concentrate in one partition. Explicitly filter or bucket NULLs separately before shuffles.
• Over-salting small partitions: If 99% of keys are balanced and only one is hot, salting 100 keys when you only need to salt one wastes CPU and memory.

Key takeaways

• Data skew manifests as a few long-running tasks while others finish quickly. Monitor the ratio of max task duration to median duration; anything above 10x is worth investigating.
• Skew occurs when key distribution is uneven—a few keys hold the majority of rows. Profile your data with groupBy().count() before major transformations.
• Salting adds random suffixes to hot keys, splitting them across multiple partitions for parallel processing. It's effective but manual and requires careful desalting.
• Adaptive Query Execution (AQE) detects and splits skewed partitions automatically at runtime. Enable it by default; it requires no code changes and often yields significant improvements.
• Combine both approaches: use AQE as your safety net and apply targeted salting to known, problematic keys identified through profiling and monitoring.
• Measure before and after. Use the Spark UI task timeline and execution metrics to confirm fixes actually resolved the bottleneck.