Cost optimization for Spark: what developers should know

Introduction

Big data engineering often feels like a luxury—unlimited compute, massive datasets, fire-and-forget jobs. Until the AWS bill arrives. Cost optimization for Spark is what developers should know if you're running production workloads at scale, because infrastructure spend can easily outpace engineering velocity. We've seen teams at ride-hailing and e-commerce platforms cut their Spark compute costs by 40–60% with straightforward changes to cluster sizing, instance selection, and data access patterns.

The good news: most cost savings don't require architectural rewrites. They're engineering decisions you can make today—starting with how you size your cluster and how aggressively you prune data before processing.

Right-size clusters

The most common mistake we see is provisioning clusters as if they're static resources. Teams spin up a 50-node cluster for peak load, leave it running 24/7, and wonder why the bill is shocking. In reality, most jobs don't need peak capacity all the time.

Right-sizing means matching cluster resources to actual workload requirements. This involves understanding:

• Driver memory and executor count: A typical Spark job doesn't need dozens of executors if the data fits in a few gigabytes. Measure actual memory usage under load, not guesswork.
• Executor heap and off-heap memory: Monitor GC pauses and OOM errors. If you're tuning memory up blindly, you're wasting money on unused heap.
• Core count per executor: More cores per executor increase contention. 4–8 cores per executor is usually optimal; 16+ often introduces diminishing returns.

Here's a concrete example: we worked with a CRM platform running daily ETL jobs. Their original config was 20 executors with 8 GB heap and 4 cores each. Monitoring showed that 70% of jobs never exceeded 3 GB heap and didn't need parallel cores beyond 2. By switching to 8 executors with 4 GB heap and 2 cores, they reduced per-job cost from $12 to $3 with no change in runtime.

Use Spark's built-in metrics to inform right-sizing:

# Check executor memory and core usage in Spark UI
# Or programmatically:
conf = SparkConf()
print(conf.get("spark.executor.memory"))        # Default: 1g
print(conf.get("spark.executor.cores"))         # Default: 1
print(conf.get("spark.driver.memory"))          # Default: 1g

# Set sensible defaults for your workload
conf.set("spark.executor.memory", "4g")
conf.set("spark.executor.cores", "4")
conf.set("spark.sql.shuffle.partitions", "200") # Not 200 for small jobs

Start with conservative settings, measure CPU and memory under load, then optimize downward. Most teams can safely cut executor count by 30–40% after profiling.

Spot or preemptible instances

Cloud providers offer spot/preemptible instances at 70–90% discounts compared to on-demand rates. The tradeoff: they can be reclaimed with minutes' notice. For batch Spark jobs, this is a near-zero risk if you design for failure.

The strategy is mixed-mode provisioning: run your driver and critical shuffle data on on-demand instances, executors on spot. Spark's built-in task retry logic handles executor loss gracefully. Lost tasks simply re-run on the remaining executors.

In Kubernetes environments (which many teams use for Spark), you can configure separate node pools:

# Pseudocode: Kubernetes nodeSelector configuration
spark_conf = SparkConf()

# Driver uses on-demand pool (high availability)
spark_conf.set("spark.kubernetes.driver.node.selector.workload", "batch")
spark_conf.set("spark.kubernetes.driver.node.selector.capacity", "on-demand")

# Executors use spot pool (cheap, disposable)
spark_conf.set("spark.kubernetes.executor.node.selector.workload", "batch")
spark_conf.set("spark.kubernetes.executor.node.selector.capacity", "spot")

We've seen production deployments running 80% of executor capacity on spot instances, cutting cloud infrastructure costs in half. The key is idempotent jobs—if a job reruns a few tasks mid-way, results must be identical. For streaming Spark, spot instances are riskier; batch ETL is ideal.

AWS spot interruptions average 5–15% per month, meaning a 100-executor job loses a few executors occasionally. Your job just retries those tasks. Real savings: $10k/month to $5k/month for mid-sized operations.

Read less data with pruning

Every byte you don't read is a byte you don't pay to transfer, parse, or shuffle. Data pruning—filtering rows and columns before Spark processes them—is often the biggest cost lever.

Partition pruning skips entire directories in object storage. If your data is partitioned by date and you only need the last 7 days, Spark never touches older partitions.

from pyspark.sql import SparkSession
from datetime import datetime, timedelta

spark = SparkSession.builder.appName("cost_opt").getOrCreate()

# Without partition pruning: scans entire dataset
df_bad = spark.read.parquet("s3://data/events/")
df_bad = df_bad.filter(df_bad.date >= "2025-01-01")

# With partition pruning: scans only matching partitions
df_good = spark.read.parquet("s3://data/events/")
df_good = df_good.filter(df_good.date >= "2025-01-01")  # If date is a partition column, Spark prunes

# Explicitly partition your data for pruning
# Schema: events/date=2025-01-15/hour=14/data.parquet
spark.write.partitionBy("date", "hour").mode("overwrite").parquet("s3://data/events/")

Column pruning only reads columns you actually use. Parquet and ORC formats support this naturally—only the columns in your select() call are loaded.

# Reads only user_id and amount, ignores 50 other columns
df = spark.read.parquet("s3://data/transactions/").select("user_id", "amount")

# Bad: reads entire row, then drops columns
df = spark.read.parquet("s3://data/transactions/")
df = df.drop("col1", "col2", ...)  # Too late, already loaded

We've measured 70–80% reduction in S3 bandwidth by combining date partition pruning with column selection on large datasets. That directly translates to faster job runtimes and lower egress costs.

Compact small files

Small files—fragments of a few MB each instead of larger blobs—inflate both compute and storage costs. Each file requires metadata overhead, separate S3 GetObject calls, and additional shuffling. A dataset with 100k tiny files costs more to query than the same data in 100 medium files.

The problem usually arises from frequent writes. If your pipeline writes 10 MB files every hour for a month, you'll have ~720 files for 300 GB of data. Querying that later touches 720 separate S3 objects instead of, say, 30.

Compaction is simple: rewrite data with a larger target file size.

# Before compaction: many tiny files
df.write.mode("overwrite").parquet("s3://data/events/")

# After compaction: fewer, larger files
# Coalesce reduces partitions, forcing larger output files
df_compacted = df.coalesce(50)  # Write to 50 files instead of hundreds
df_compacted.write.mode("overwrite").parquet("s3://data/events/compacted/")

# Better: use repartition with a partitioning column
df_repartitioned = df.repartition(50, "date")
df_repartitioned.write.partitionBy("date").mode("overwrite").parquet("s3://data/events/")

Run compaction as a scheduled maintenance job weekly or monthly. A 500 GB dataset with 50k files might take 10 minutes to compact into 500 files. That 10-minute cost is recouped within hours as queries run faster and S3 requests drop.

Monitor file count and size in your data lake:

# Check file size distribution
import boto3
s3 = boto3.client("s3")
paginator = s3.get_paginator("list_objects_v2")
pages = paginator.paginate(Bucket="data", Prefix="events/")

file_sizes = []
for page in pages:
    for obj in page.get("Contents", []):
        file_sizes.append(obj["Size"])

import statistics
print(f"Median file size: {statistics.median(file_sizes) / 1024 / 1024:.1f} MB")
print(f"File count: {len(file_sizes)}")
# Aim for 100–500 MB files for most workloads

Common mistakes

• Over-provisioning "just in case": Spinning up a 100-node cluster for a job that uses 20 nodes 95% of the time. Use autoscaling or dynamic allocation instead.
• Ignoring shuffle partitions: The default of 200 shuffle partitions is meant for large workloads. Small jobs create 200 tiny output files and incur huge overhead. Set spark.sql.shuffle.partitions based on data volume.
• Not monitoring actual resource usage: Guessing CPU, memory, and core requirements without checking Spark UI or CloudWatch metrics.
• Rewriting compaction logic: Don't manually coalesce and write in a Python loop. Use Spark's built-in repartition and partitionBy—they're optimized and distributed.
• Spot instances for state-heavy workloads: Streaming jobs with stateful aggregations or iterative machine learning (MLlib) don't tolerate executor loss well. Batch ETL only.

Key takeaways

• Right-size clusters by profiling real workloads under load, not assumptions. Most teams can cut executor count by 30–40% without performance loss.
• Use spot/preemptible instances for 60–80% of executor capacity in batch jobs. Spark's task retry handles instance loss automatically.
• Partition and select columns explicitly. Partition pruning and column projection can reduce S3 bandwidth by 70–80%.
• Compact small files weekly or monthly. Aim for 100–500 MB files to cut S3 request costs and improve query speed.
• Monitor infrastructure spend as a core metric. A weekly check of executor utilization and file sizes catches cost creep early.
• Cost optimization is iterative. Start with cluster right-sizing, measure, then layer in pruning and compaction strategies.