Spark partitions and parallelism: lessons from production

Getting partitioning wrong in production is a silent killer. Your Spark job runs, but it crawls—or worse, crashes with an out-of-memory error because one executor is drowning while others sit idle. We've learned the hard way that Spark partitions and parallelism: lessons from production boils down to understanding how work is split, keeping partition size reasonable, and knowing when to shuffle versus when to compact.

Partitions — Each partition is processed by one task

A Spark partition is the smallest unit of work that a single task can process on a single executor. When you load data into a DataFrame or RDD, Spark divides it into logical chunks. Each executor in your cluster can run multiple tasks in parallel, but a task operates on exactly one partition at a time.

Think of it like an assembly line. If you have 100 partitions and 10 executors with 4 cores each, Spark can run 40 tasks in parallel. But here's the catch: if one partition has 90% of the data, one task will be working overtime while the rest twiddle their thumbs. That's a bottleneck.

When you read a CSV file from S3, the number of partitions depends on the file size and how Spark chunks it. A single 1 GB file might become just 2–3 partitions. A directory with 1,000 small files becomes 1,000 partitions. Neither is ideal in isolation.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("PartitionDemo").getOrCreate()

# Reading a CSV file
df = spark.read.option("header", "true").csv("s3://bucket/data.csv")

# Check how many partitions Spark assigned
print(f"Number of partitions: {df.rdd.getNumPartitions()}")

# Let's do a simple transformation
result = df.filter(df.age > 30).select("id", "name")

# Each partition will be filtered independently by one task
print(f"Result partitions: {result.rdd.getNumPartitions()}")

When you apply a transformation like filter or select, each task processes its own partition independently. No data moves between executors yet—it's embarrassingly parallel. The key insight is that more partitions mean more parallelism, but also more overhead. Too few partitions and you're wasting cluster resources. Too many and you're creating tiny tasks that spend more time on scheduling than actual work.

Right size — Aim for roughly 100–200 MB per partition

From our experience across e-commerce and trading platforms, the sweet spot for partition size is around 100–200 MB. This balances parallelism against task overhead and memory pressure.

Here's the reasoning: if a partition is too small (under 10 MB), the overhead of creating and scheduling the task becomes significant relative to the actual computation. JVM startup, serialization, and network I/O all add latency. If a partition is too large (over 500 MB), you risk memory pressure on executors, especially if your transformation involves joins or aggregations that blow up the data size.

Let's say you're processing a 10 GB dataset. At 100 MB per partition, you get 100 partitions and roughly 100 tasks (if you have enough executor cores). That's good. At 1 GB per partition, you get 10 partitions and 10 tasks—not enough parallelism if you have 40 cores available. At 10 MB per partition, you get 1,000 partitions and 1,000 tasks—scheduler overhead becomes real.

# Example: calculating the right number of partitions
def calculate_ideal_partitions(total_size_gb, target_partition_mb=128):
    """
    Return suggested partition count for a given dataset size.
    """
    total_size_mb = total_size_gb * 1024
    ideal_partitions = max(1, total_size_mb // target_partition_mb)
    
    # Round up to nearest multiple of your executor cores
    # If you have 40 cores total, aim for a multiple of 40
    executor_cores = 40
    final_partitions = ((ideal_partitions + executor_cores - 1) 
                        // executor_cores) * executor_cores
    
    return final_partitions

# For a 50 GB dataset
partitions = calculate_ideal_partitions(50, target_partition_mb=128)
print(f"Suggested partitions for 50 GB: {partitions}")

# Repartition your DataFrame
df_repartitioned = df.repartition(partitions)

In production, we've found that monitoring partition sizes via Spark UI or logs is crucial. If you see tasks taking 30+ seconds, check if your partitions are oversized. If you see 1,000 tiny tasks finishing in milliseconds, you might have too many partitions.

repartition — Increase or rebalance partitions with a shuffle

repartition(n) changes the number of partitions to exactly n, but it triggers a full shuffle. All data gets redistributed across the cluster. This is expensive—network I/O, disk spill, CPU. Use it when you need to fix severe imbalance or when you're about to do a costly operation that requires specific partitioning.

A common scenario: you've read 1,000 small log files from S3, so Spark created 1,000 partitions. Most are tiny. A shuffle is coming anyway—maybe a join or a groupByKey. In this case, repartitioning up-front to 200 partitions (assuming 200 GB data) can actually save time by reducing overhead, even though repartition itself shuffles.

# Before: 1,000 partitions from 1,000 small files
df = spark.read.parquet("s3://bucket/logs/")
print(f"Initial partitions: {df.rdd.getNumPartitions()}")  # 1000

# Our cluster has 40 cores, dataset is ~200 GB
# Target: 200 partitions (5 MB per partition, balanced work)
df = df.repartition(200)

# Now all 200 GB is evenly distributed across 200 partitions
# This shuffle happens once; subsequent operations are faster
result = df.groupBy("user_id").count()

result.write.parquet("s3://bucket/output/")

In a trading platform, we had real-time order streams arriving at different rates. Some days, partition 0 had 10 million rows and partition 99 had 100K rows. We repartitioned monthly to keep things balanced, absorbing the shuffle cost because downstream joins were so much faster. The trade-off made sense.

Be aware: repartition can also change the partitioning key. By default, it uses a hash-based approach, spreading rows evenly. If you have a skewed distribution (like 90% of rows have the same key), a shuffle won't help—you'll just end up with one bloated partition and 199 tiny ones. That's a data skew problem, not a partitioning problem.

coalesce — Reduce partitions without a full shuffle

coalesce(n) reduces the number of partitions to n, but it does so without shuffling. Instead, Spark merges adjacent partitions on the same executor. This is much cheaper than repartition. Use it when you want to reduce overhead—for instance, before writing output.

Imagine you've done 15 transformations on your DataFrame. Each transformation might have doubled the number of partitions (due to join explosions, nested operations, etc.). You end up with thousands of tiny partitions. Before writing to S3, coalesce to something reasonable: 50 partitions means 50 output files instead of 10,000.

# After heavy transformations, we have 5,000 partitions
# (probably because of multiple joins and group-bys)
df_processed = (df
    .join(user_df, "user_id")
    .join(order_df, "order_id")
    .groupBy("region").agg({"amount": "sum"})
)
print(f"Partitions after joins: {df_processed.rdd.getNumPartitions()}")  # 5000+

# Write to S3: we don't want 5,000 files
# Coalesce to 50 partitions (50 output files)
# This is much cheaper than repartition because it doesn't shuffle
df_processed.coalesce(50).write.parquet("s3://bucket/output/")

The caveat: coalesce doesn't rebalance data. If partition 0 is huge and partitions 1–99 are tiny, coalescing won't fix it. Coalescing just merges boundaries. You end up with fewer, uneven partitions. In that case, you need repartition, even though it's slower.

In our e-commerce pipeline, we'd use coalesce as the final step before writing to the warehouse. The transformations were balanced enough, and we just wanted to reduce file count for easier downstream processing. We saved minutes per job by coalescing instead of repartitioning.

Common mistakes

• Ignoring default partitioning from source. Reading 100 small CSV files from S3 gives you 100 partitions, but if your cluster has 200 cores, you're underutilizing it. Repartition or set spark.sql.files.maxPartitionBytes in your config.
• Repartitioning at every step. Each repartition is a full shuffle. If you read data, repartition, then join (which repartitions again internally), you've shuffled twice. Plan your partitioning strategy once, early.
• Fixing data skew with repartitioning. If 99% of your rows have key="default", repartitioning won't help. You need salting (add a random suffix) or a custom partitioner. Repartitioning just moves the problem around.
• Coalescing when you need rebalancing. If partitions are uneven, coalesce creates bigger imbalance. Use repartition with a shuffle to actually rebalance, or fix the root cause (data skew).
• Not monitoring partition size in logs or Spark UI. Set up alerts on task duration. If tasks regularly exceed 30 seconds, partitions are too large. If tasks finish in milliseconds, partitions are too small.

Key takeaways

• Partitions are the unit of parallelism. Each task processes one partition on one executor. Aim for partition sizes around 100–200 MB to balance parallelism and overhead.
• repartition(n) triggers a full shuffle to reach exactly n partitions. Use it when you need to rebalance or when a shuffle is happening anyway downstream.
• coalesce(n) reduces partitions without shuffling by merging adjacent ones. Use it for cleanup before writing output, but don't use it to fix imbalance.
• Plan partitioning early—right after reading data, before expensive operations. Multiple repartitions compound latency.
• Data skew is not a partitioning problem. If 90% of rows share the same key, repartitioning spreads them evenly but then the join or groupBy collapses them back together on one executor. Salt the key or use a custom partitioner instead.
• Monitor task durations and partition sizes via Spark UI. Adjust your strategy based on what you see in production.