Transformations vs actions: patterns, pitfalls and fixes

Introduction

Most Spark performance issues trace back to one root cause: misunderstanding when work actually happens. You write a transformation, assume it's executed immediately, then wonder why your job runs for hours instead of seconds. Understanding transformations vs actions: patterns, pitfalls and fixes is fundamental to writing efficient data pipelines. In this article, we'll walk through what triggers computation in Spark, which operations belong in each category, and how to structure your code to avoid costly mistakes.

What Are Transformations?

A transformation is a Spark operation that returns a new DataFrame or RDD without triggering computation. Transformations are lazy—they build a logical execution plan but don't process data. This design allows Spark to optimize the entire chain of operations before running anything.

Common transformations include select, filter, map, withColumn, and groupBy. When you call any of these, Spark registers the operation in its DAG (directed acyclic graph) and returns control immediately.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("transforms").getOrCreate()
df = spark.read.csv("sales.csv", header=True, inferSchema=True)

# All of these are transformations—nothing executes yet
filtered = df.filter(df["amount"] > 100)
selected = filtered.select("customer_id", "amount", "date")
with_tax = selected.withColumn("total", selected["amount"] * 1.1)

print("DAG built, no data processed")

At this point, Spark has built an execution plan but hasn't read a single row from the CSV. The data stays untouched on disk or in the distributed file system.

What Are Actions?

An action is the opposite: it triggers actual computation and returns a result to the driver or writes data to storage. Actions are eager. The moment you call an action, Spark executes the entire chain of transformations leading up to it.

Common actions include count, collect, show, write, first, and foreach. Each of these requires materializing data in memory or on disk.

# This is where execution happens
result = with_tax.count()  # Action: triggers all transformations above
print(f"Total rows: {result}")

with_tax.show(5)  # Action: executes and displays 5 rows

with_tax.write.mode("overwrite").csv("output/")  # Action: writes to disk

The moment you call count(), show(), or write(), Spark schedules tasks, distributes work across the cluster, and processes the data. This is when you see CPU and memory usage spike.

Understanding Narrow vs Wide Transformations

Not all transformations are equal. Some are narrow, meaning each output partition depends on a single input partition. Others are wide, meaning output partitions depend on multiple input partitions. This distinction matters because wide transformations trigger shuffles—expensive network operations that move data across the cluster.

Narrow Transformations

Narrow transformations process partitions independently. select, filter, map, and withColumn are all narrow. Spark can execute them in parallel on each partition without moving data between nodes.

# Narrow transformations
df_filtered = df.filter(df["status"] == "active")  # Each partition filtered independently
df_selected = df_filtered.select("id", "name", "amount")  # Each partition projected independently
df_enriched = df_selected.withColumn("year", df["date"].substr(1, 4))  # Each partition enhanced independently

Because narrow transformations don't require data movement, they're fast and scale linearly with cluster size. You can chain dozens of narrow transformations without performance degradation.

Wide Transformations

Wide transformations require data from multiple partitions to compute a single output partition. groupBy, join, and orderBy are wide. They force a shuffle: Spark redistributes all data across partitions so that rows with the same key end up on the same node.

# Wide transformations
revenue_by_customer = df.groupBy("customer_id").sum("amount")  # Shuffle: all rows with same customer_id go to same partition

df_orders = spark.read.csv("orders.csv", header=True)
merged = df.join(df_orders, "customer_id")  # Shuffle: rows matched by key across partitions

sorted_data = df.orderBy("date")  # Shuffle: entire dataset rearranged for sorting

Shuffles are expensive. They involve writing intermediate data to disk, network transfer, and reading it back. A single poorly placed shuffle can turn a job that runs in seconds into one that takes minutes.

Wide Transformations Cause Shuffles

Understanding shuffles is critical to writing performant Spark code. When Spark encounters a wide transformation, it must repartition the entire dataset. This happens in two phases: a map phase where data is written to local disk, and a reduce phase where data is fetched and merged.

# Example: this query causes a shuffle
sales = spark.read.csv("sales.csv", header=True, inferSchema=True)
customers = spark.read.csv("customers.csv", header=True, inferSchema=True)

# Join is a wide transformation
result = sales.join(customers, "customer_id")

# This is another wide transformation
summary = result.groupBy("region").agg(
    F.sum("amount").alias("total_sales"),
    F.count("*").alias("transaction_count")
)

# Action: triggers both the join shuffle and the groupBy shuffle
summary.show()

When you call show(), Spark executes the plan. It shuffles data for the join, then shuffles again for the groupBy. If you have millions of rows, this can take significant time and resources.

Minimizing Shuffle Cost

Smart engineers structure queries to minimize shuffles. Some practical patterns:

• Filter before joining: Remove unnecessary rows before the join operation to reduce shuffle volume.
• Broadcast small tables: If one table is small enough to fit in memory, broadcast it to avoid a shuffle join entirely.
• Coalesce partitions: After a shuffle, use coalesce() to reduce the number of output partitions if the data is small.
• Avoid multiple shuffles: Combine operations where possible to trigger fewer shuffles.

from pyspark.sql.functions import broadcast

# Better: filter first, then join
sales_large = sales.filter(sales["year"] == 2024)
customers_small = customers.filter(customers["is_active"] == True)

# If customers_small fits in memory, broadcast it
result = sales_large.join(broadcast(customers_small), "customer_id")

# Only one shuffle now, with less data

Common Mistakes

Calling actions repeatedly on the same DataFrame: Each action re-executes the entire transformation chain. If you need the same result multiple times, use cache() or persist() to store it in memory.

# Wrong
count1 = df.count()  # Full execution
count2 = df.filter(df["amount"] > 100).count()  # Full execution again
max_val = df.agg(F.max("amount")).collect()[0][0]  # Full execution again

# Better
df.cache()  # Materialize in memory after first action
count1 = df.count()  # Executes once, result cached
count2 = df.filter(df["amount"] > 100).count()  # Uses cached df
max_val = df.agg(F.max("amount")).collect()[0][0]  # Uses cached df

Collecting large DataFrames to the driver: collect() pulls all data into driver memory. On a terabyte-scale dataset, this causes out-of-memory errors or timeouts.

# Dangerous
all_rows = df.collect()  # Pulls entire DataFrame to driver memory
for row in all_rows:
    print(row)

# Better
df.foreach(lambda row: print(row))  # Executes on worker nodes, no data moved to driver

Misinterpreting partition count: The number of partitions affects parallelism. Too few partitions = poor resource utilization. Too many = scheduling overhead. After a wide transformation, Spark defaults to 200 partitions; this may or may not be right for your data size.

# Check partition count
print(f"Partitions: {df.rdd.getNumPartitions()}")

# Repartition if needed
df_repartitioned = df.repartition(50)  # Wide transformation, causes shuffle

Ignoring Catalyst optimization: Spark's optimizer rearranges transformations automatically. Writing convoluted filter logic doesn't help—Spark reorganizes it anyway. Write clear, readable code and trust the optimizer.

Key Takeaways

• Transformations are lazy; actions trigger execution. Build your logical plan with transformations, then call exactly one action at the end.
• Narrow transformations (select, filter, map) process partitions independently and scale well. Wide transformations (join, groupBy, orderBy) require shuffles and are expensive.
• Shuffles move data across the network. Minimize them by filtering early, broadcasting small tables, and combining operations.
• Call actions deliberately. Repeated actions re-execute the entire chain; use cache() to avoid redundant work.
• Never collect large DataFrames to the driver. Use distributed operations (foreach, write) instead.
• Understand your data size and partition count. Adjust repartitioning based on actual cluster capacity and data volume.
• Write code for clarity, not micro-optimization. Spark's Catalyst optimizer rearranges operations to find the best execution plan.