Common Spark mistakes in Apache Spark & Data Engineering

Introduction

After 75 days of working with Apache Spark, you've probably built pipelines that work—but not all of them run efficiently. We see the same errors surface repeatedly across teams: operations that trigger unexpected data movement, inefficient transformations, and architectural choices that cascade into performance nightmares. Common Spark mistakes in Apache Spark & Data Engineering aren't usually about syntax; they're about understanding how the distributed execution model actually behaves, and recognizing where intuitive code choices create invisible bottlenecks.

In this post, we'll walk through four mistakes we encounter regularly in production systems, show you what goes wrong, and demonstrate how to fix them.

Mistake 1: Using collect() on Large Data

The collect() action is seductive because it's simple: you call it, and suddenly your distributed DataFrame becomes a local Python list. This feels natural when you're exploring data or debugging. In production, it's a silent killer.

When you call collect(), Spark brings all results back to the driver node's memory. If your DataFrame has 10 million rows, that's 10 million rows crossing the network and loading into your driver's heap. Drivers typically run with modest memory allocations (4–16 GB). Executors with gigabytes of RAM can process data in parallel; the driver cannot.

Real-world scenario

You're building a data quality check. You want to validate that a critical column has no unexpected values. A junior team member writes:

# WRONG: collect() on 50 million rows
df = spark.read.parquet("s3://data/events")
all_rows = df.collect()
bad_values = [row for row in all_rows if row.user_id is None]
print(f"Found {len(bad_values)} rows with null user_id")

The driver crashes with an out-of-memory error before the validation even completes.

The correct approach

Use Spark's distributed operations to answer the question on the executors, then bring back only the answer:

# RIGHT: Use Spark aggregations
df = spark.read.parquet("s3://data/events")
bad_count = df.filter(col("user_id").isNull()).count()
print(f"Found {bad_count} rows with null user_id")

If you genuinely need sample rows for inspection, use limit() or sample() before collect():

# Safer: Collect a sample
sample_bad = df.filter(col("user_id").isNull()).limit(100).collect()
for row in sample_bad:
    print(row)

The difference is dramatic: the second version processes gigabytes on executors and returns kilobytes to the driver.

Mistake 2: Python UDFs for Simple Logic

Python User-Defined Functions are powerful, but they carry a performance cost that surprises many engineers. When Spark executes a Python UDF, it serializes rows, sends them to a Python process running on each executor, executes your function, and deserializes the results back into JVM types. This happens for every row.

The performance trap

Spark's SQL functions and DataFrame API methods are implemented in Scala and compiled to JVM bytecode. They run in-process with zero serialization overhead. A Python UDF doing the same work often runs 10–50x slower per row because of the serialization tax and process-boundary crossing.

We regularly find pipelines where 80% of runtime is spent in a 5-line Python UDF that could be a one-liner in SQL.

Example: Processing timestamps

You need to extract the hour and day of week from a timestamp column, then concatenate them:

# SLOW: Python UDF
from pyspark.sql.functions import udf
from pyspark.sql.types import StringType
from datetime import datetime

def format_time_info(ts):
    dt = datetime.fromisoformat(ts)
    hour = dt.hour
    day_of_week = dt.strftime("%A")
    return f"{day_of_week}_{hour}"

format_udf = udf(format_time_info, StringType())
result = df.withColumn("time_info", format_udf(col("event_timestamp")))

On a dataset with 100 million rows, this might take 10+ minutes.

# FAST: Native Spark functions
from pyspark.sql.functions import hour, dayofweek, concat_ws, date_format

result = df.withColumn(
    "time_info",
    concat_ws("_", date_format(col("event_timestamp"), "EEEE"), hour(col("event_timestamp")))
)

This version completes in seconds. Same logic, vastly different performance.

When Python UDFs make sense

Use them for genuinely complex logic that Spark SQL can't express: calling external APIs, running ML models, or implementing domain-specific algorithms. For data transformation, exhaust Spark's built-in functions first.

If you must use Python UDFs: vectorized

If your logic is complex and unavoidably Python, use Pandas UDFs (also called "Vectorized UDFs"). They batch rows into Pandas DataFrames, reducing serialization overhead significantly:

# Better: Vectorized UDF
import pandas as pd
from pyspark.sql.functions import pandas_udf

@pandas_udf(StringType())
def format_time_info_vec(ts: pd.Series) -> pd.Series:
    dt = pd.to_datetime(ts)
    return dt.dt.day_name() + "_" + dt.dt.hour.astype(str)

result = df.withColumn("time_info", format_time_info_vec(col("event_timestamp")))

Vectorized UDFs run 3–10x faster than row-at-a-time Python UDFs because the overhead is amortized over batches.

Mistake 3: Ignoring Data Skew

Data skew occurs when values in your join or groupBy key are distributed unevenly across partitions. One partition processes 90% of the data while others sit idle. This is invisible in development because test datasets are small and evenly distributed. In production, a few hot keys can bring your pipeline to a crawl.

Recognizing skew

In the Spark UI, look at the Tasks tab under a stage. If most tasks finish in seconds but one task runs for minutes, you have skew. Similarly, if shuffle write sizes vary wildly (one executor writing 500 MB, others writing 50 GB), that's a red flag.

Real example: Grouping by user ID

You're aggregating user events:

# At risk of skew
agg_df = df.groupBy("user_id").agg(
    count("*").alias("event_count"),
    sum("revenue").alias("total_revenue")
)

If a few users (say, internal testing accounts or bots) have millions of events, their partition becomes a bottleneck. Meanwhile, 99% of your cluster is idle waiting for that one executor.

Diagnosing and fixing skew

First, confirm the skew:

# Check distribution
df.groupBy("user_id").count().describe().show()
df.groupBy("user_id").count().rdd.map(lambda x: x[1]).histogram(10)

If you find skew on a groupBy key, use repartition() or adjust partitions:

# Increase partitions to distribute hot keys
agg_df = df.repartition(2000, "user_id").groupBy("user_id").agg(
    count("*").alias("event_count"),
    sum("revenue").alias("total_revenue")
)

For join skew, the classic fix is salting the key—adding random suffixes to hot values so they distribute across multiple partitions:

# Salt the join key
from pyspark.sql.functions import concat, lit, floor, rand

left_df = left_df.withColumn("salt", floor(rand() * 10))
left_df = left_df.withColumn("join_key", concat(col("user_id"), lit("_"), col("salt")))

right_df = right_df.crossJoin(spark.range(10).select(col("id").alias("salt")))
right_df = right_df.withColumn("join_key", concat(col("user_id"), lit("_"), col("salt")))

result = left_df.join(right_df, "join_key")

This duplicates the right side 10 times but spreads the hot left-side key across 10 partitions, balancing the load.

Mistake 4: Writing Thousands of Small Files

Spark's default behavior is to write one output file per partition. If your DataFrame has 10,000 partitions, you'll write 10,000 files. This seems convenient until your data warehouse team gets a query that reads 10,000 tiny Parquet files from S3—now they're burning through API calls and network bandwidth just listing files.

Why this hurts

Object stores like S3 throttle list operations. Reading 10,000 small files is slower than reading 100 larger files with the same total data. Tools like Athena and Redshift Spectrum charge per byte scanned, and file metadata overhead increases costs. Your data cataloging system also becomes slower.

The scenario

You run a daily ETL that processes events and writes results:

# WRONG: Many small files
df_processed = df.filter(...).groupBy(...).agg(...)  # Results in 5000 partitions
df_processed.write.mode("overwrite").parquet("s3://output/results/")

You end up with 5,000 files. Next month, queries are slow and S3 bill is higher than expected.

The fix: Coalesce before writing

Use coalesce() to reduce the number of output files:

# RIGHT: Coalesce to reasonable partition count
df_processed = df.filter(...).groupBy(...).agg(...)
df_processed.coalesce(100).write.mode("overwrite").parquet("s3://output/results/")

Now you write 100 files instead of 5,000. Each file is larger and reads faster. The trade-off is that coalescing shuffles data and runs on a single stage, so it's not free—but it's almost always worth it for final outputs.

The distinction: coalesce vs repartition

coalesce(n) reduces partitions without a shuffle; it merges existing partitions. repartition(n) reshuffles all data to create exactly n partitions. For writing final outputs, use coalesce(). For intermediate transformations where you want balanced partitions, use repartition().

A practical rule: write between 50–500 output files depending on your data volume and downstream consumer patterns. Use this heuristic:

# Target: ~100 MB per file
target_file_size_mb = 100
total_size_bytes = df.rdd.map(lambda x: len(str(x))).sum()  # Rough estimate
ideal_partitions = max(1, int(total_size_bytes / (target_file_size_mb * 1e6)))
df_processed.coalesce(ideal_partitions).write.mode("overwrite").parquet("s3://output/")

Common mistakes summary

Key takeaways

• collect() is for small results only. Use distributed aggregations and filtering to answer questions on executors, then bring back the answer, not the data.
• Prefer Spark SQL functions over Python UDFs for standard transformations. The performance difference is orders of magnitude. Reserve Python UDFs for genuinely complex logic.
• Monitor partition and task distribution in the Spark UI. Uneven task runtimes signal data skew. Fix it early with repartition or key salting.
• Coalesce before final writes. 100 large files beat 10,000 small files for downstream performance and cost. Use a target file size to compute the ideal partition count.
• Profile early, especially on production data scales. These mistakes often hide until you hit real-world volumes. Run your pipeline on a representative data sample and check the Spark UI.