Bucketing in Spark: a practical Spark & Data Eng guide

The shuffle problem that costs us millions in processing time

If you've worked with large-scale joins in Spark, you know the pain: every time you join two DataFrames, Spark shuffles data across the network, and that shuffle becomes your bottleneck. Bucketing in Spark: a practical Spark & Data Eng guide walks you through one of the most underutilized optimizations in production data pipelines. When you join the same tables repeatedly—whether in real-time feature computation, reconciliation jobs, or daily aggregations—bucketing lets you pre-organize data so joins happen without any shuffle at all.

Pre-shuffle data into buckets

Bucketing is a technique where you write a DataFrame to disk in a specific way: Spark hashes the bucketing column and distributes rows into N partitions (buckets) based on that hash. Once the data is bucketed and written, future joins on that same column skip the shuffle entirely.

Think of it like organizing a warehouse. Instead of searching randomly when someone asks for "all orders from customer 42," you've already placed all of customer 42's orders in bin 42. Next time someone needs that customer's data, you go straight to bin 42 without rearranging anything.

Here's how you write bucketed data:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("bucketing-demo").getOrCreate()

# Sample orders data
orders = spark.createDataFrame([
    (1, "2024-01-15", 150.50),
    (2, "2024-01-16", 200.00),
    (1, "2024-01-17", 75.25),
    (3, "2024-01-18", 500.00),
], ["customer_id", "date", "amount"])

# Write with bucketing on customer_id into 16 buckets
orders.write \
    .mode("overwrite") \
    .bucketBy(16, "customer_id") \
    .sortBy("customer_id") \
    .mode("overwrite") \
    .option("path", "/data/orders_bucketed") \
    .saveAsTable("orders_bucketed")

The bucketBy(16, "customer_id") call tells Spark to create 16 buckets. Spark will hash each customer_id, mod it by 16, and write all rows with the same bucket number to the same physical file. The sortBy ensures rows within each bucket are sorted—this is optional but improves performance for certain access patterns.

Behind the scenes, Spark creates a manifest that tells it: "bucket 0 contains customer IDs that hash to 0, bucket 1 contains customer IDs that hash to 1," and so on. This metadata is critical for the next step.

Join bucketed tables without shuffles

Now for the payoff. Load two bucketed tables with the same bucket count and join on their bucketing columns:

customers = spark.createDataFrame([
    (1, "Alice"),
    (2, "Bob"),
    (3, "Charlie"),
], ["customer_id", "name"])

# Also bucket the customers table on customer_id
customers.write \
    .mode("overwrite") \
    .bucketBy(16, "customer_id") \
    .sortBy("customer_id") \
    .mode("overwrite") \
    .option("path", "/data/customers_bucketed") \
    .saveAsTable("customers_bucketed")

# Read both from managed tables
orders_bucketed = spark.table("orders_bucketed")
customers_bucketed = spark.table("customers_bucketed")

# Join on the bucketed column
result = orders_bucketed.join(
    customers_bucketed,
    on="customer_id",
    how="inner"
)

result.show()

When Spark executes this join, it recognizes that both tables are bucketed on customer_id with the same bucket count. Instead of shuffling, it performs a bucket-aware join: bucket 0 from orders_bucketed joins with bucket 0 from customers_bucketed, bucket 1 with bucket 1, and so on. Each bucket pair is already co-located on the same executor (or can be read sequentially), so no data moves across the network.

In production, we've seen this reduce join times by 60–80% on large tables. The larger your tables, the more you gain, because shuffle cost grows with data volume while bucketed joins stay roughly constant.

Same bucket count on both sides

This is non-negotiable: if one table has 16 buckets and the other has 32, Spark falls back to a shuffle join. The bucket counts must match exactly.

Why? Because bucket 0 in a 16-bucket table contains different customer IDs than bucket 0 in a 32-bucket table. The hash function maps to different ranges, so the bucketing contracts no longer line up.

Always verify bucket counts before joining. We recommend standardizing on 128 or 256 buckets as a team default—large enough to avoid hotspots, small enough to avoid tiny files.

You can check a table's bucket count in the Spark UI or by reading the table metadata:

spark.sql("DESCRIBE FORMATTED orders_bucketed").show()

Look for Num Buckets in the output. If you see -1, the table is not bucketed.

A common mistake is bucketing one table but not the other. If you join a bucketed table with an unbucketed one, Spark uses a shuffle join and you lose all benefits. Both must be bucketed on the join column with matching counts.

Useful for large repeated joins

Bucketing shines in specific scenarios. It's not a universal optimization—overhead costs real money if misapplied.

Ideal use cases:

• Repeated joins in the same pipeline: If you join orders with customers three times in one day's ETL, bucketing pays dividends after the first write.
• Large reference tables: Bucketing a 10 GB customer table that joins with a 100 GB orders table eliminates the shuffle of that 100 GB table, which is the expensive part.
• Streaming feature pipelines: Real-time systems that join event streams with slowly-changing dimension tables benefit tremendously; the dimension table is bucketed once, then reused for weeks of joins.
• Nightly reconciliation jobs: If your reconciliation compares tables from multiple sources every night on the same key, bucketing the reference tables once saves orchestration time globally.

When to skip bucketing:

• One-off joins that never happen again.
• Tiny tables (under 100 MB); the overhead isn't worth it.
• Complex filter predicates that would eliminate most buckets; the cost of maintaining buckets may exceed the savings.
• Frequently changing join columns; you'd have to re-bucket constantly.

A real example from our experience: a ride-hailing platform bucketed the trips table on driver_id and the driver_ratings table on driver_id. These tables grew to billions of rows each, and the platform ran 50+ joins per day across analytics, fraud detection, and operations. After bucketing, they eliminated ~4 hours of daily shuffle time. The one-time cost of writing bucketed tables paid back in under a week.

Common mistakes

Joining on a column that isn't the bucketing column. If you bucket on customer_id but join on order_date, Spark will shuffle. Always join on the bucketed column to avoid shuffle.

Mismatched bucket counts. We've seen teams bucket one table on 256 buckets and another on 128, then wonder why joins are slow. Use the same count across all tables in a join.

Bucketing too many columns. You can bucket on multiple columns, but Spark only avoids shuffle if you join on all of them in the same order. Stick to one column per table unless your use case is truly complex.

Forgetting that bucketing is persistent. Once you bucket a table, every write to that same table location must respect the bucketing. If you overwrite with mode="overwrite" but forget bucketBy(), you lose bucketing. This often happens when teams onboard new developers who aren't aware of the existing bucketing strategy.

Bucketing unbucketed data on the fly. Some teams try to optimize joins at read time with repartition-based tricks. It doesn't work; you must write bucketed tables ahead of time.

Key takeaways

• Bucketing pre-organizes data on disk so joins avoid network shuffles—critical for large, repeated joins.
• Write DataFrames with .bucketBy(N, col).sortBy(col), where N is your bucket count and col is the join key.
• Both tables must be bucketed on the join column with the same bucket count for Spark to use a shuffle-free join.
• Use bucketing for high-volume, repeated joins on static or slowly-changing dimension tables; skip it for one-offs or tiny data.
• Standardize bucket counts across your team (128 or 256 is typical) to avoid surprises and make bucketing strategy transparent.
• Monitor actual join times before and after bucketing; real-world gains vary based on cluster size, data distribution, and frequency of reuse.