Reading the Spark UI: a practical Spark & Data Eng guide

Introduction

When a Spark job runs slowly, developers often reach for logs and dashboards without knowing where to look first. The Spark UI—available by default at localhost:4040 during local development and accessible through cluster management tools in production—holds the answer. Learning to read the Spark UI effectively transforms you from guessing at performance problems to pinpointing exactly which stage, task, or operation is consuming your time. In this guide, we'll walk through the practical skills for reading the Spark UI: a practical Spark & Data Eng guide that focuses on four critical areas where bottlenecks hide.

Jobs and Stages — See Which Stage Is Slow

The Jobs tab is your entry point. Every action you trigger (like collect(), write(), or show()) launches a job. Within each job, Spark breaks work into stages, separated by shuffle boundaries or other wide transformations. This is where you discover which stage is slow.

Consider a typical ETL pipeline:

# Read raw data, transform, and aggregate
df = spark.read.parquet("s3://raw-data/events/")
df = df.filter(df.timestamp > "2024-01-01")
df = df.select("user_id", "event_type", "amount")

# Wide transformation — shuffle happens here
result = df.groupBy("user_id").agg(F.sum("amount")).collect()

When you run this, the Spark UI shows two stages: one for the filter and select (narrow), and one for the groupBy aggregation (wide). If the second stage takes 5 minutes and the first takes 10 seconds, you know the shuffle is your problem. The UI shows the DAG (directed acyclic graph) visually, too—look for the stage boxes and their timing in the job timeline.

Key metrics to check in the Jobs tab:

• Stage Duration — Total time the stage took, including scheduling, execution, and shuffle.
• Input Size — How much data entered the stage.
• Output Size — How much data left the stage (indicator of filtering effectiveness).
• Number of Tasks — Higher task count doesn't always mean faster; 200 tasks may run slower than 50 if tasks are tiny.

Click into a slow stage to drill down into task-level detail. This is where the real detective work begins.

Tasks — Look for Skew in Task Durations

A stage can appear slow because all 200 tasks are moderately slow, or because 199 tasks finish in seconds but one takes 10 minutes. The latter is skew, and the Spark UI reveals it instantly in the Tasks tab within each stage.

Open the stage detail page and scroll to the Task Metrics section. You'll see a table of all tasks with columns like Duration, Shuffle Read Size, and GC Time. Sort by Duration to spot outliers. A healthy distribution shows tasks clustered around a median time; skew shows a few tasks far above the rest.

Here's a common scenario from our work on ride-hailing data:

# Group rides by driver_id and compute statistics
rides = spark.read.parquet("s3://rides/2024/")
by_driver = rides.groupBy("driver_id").agg(
    F.count("*").alias("ride_count"),
    F.avg("fare").alias("avg_fare"),
    F.percentile_approx("duration", 0.95).alias("p95_duration")
)
by_driver.write.mode("overwrite").parquet("s3://aggregated/by_driver/")

If the data has one mega-driver with millions of rides and most drivers have dozens, the partition holding that mega-driver's data runs much longer. The UI shows this as a few tasks at 2-3 minutes while others finish in 20 seconds. This is skew, and it's fixable with salting (adding noise to the grouping key) or adaptive query execution in Spark 3.0+.

Look also at Shuffle Read Size per task. If one task reads 5 GB and others read 50 MB, that's your smoking gun.

To confirm skew programmatically, we sometimes log task metrics directly:

# Enable detailed metrics
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")

# Run the problematic query
result = by_driver.repartition("driver_id", 500).collect()
# Spark will now adapt and re-balance partitions if skew is severe

SQL Tab — Physical Plans with Metrics

The SQL tab is where you see the actual execution plan overlaid with real performance data. Unlike explain() in your code, which shows estimated metrics, the SQL tab shows what actually happened.

Click on a SQL query and you'll see its WholeStageCodegen nodes. Each node shows:

• Actual vs. Estimated Rows — If estimated rows are way off, Spark made poor decisions. Gather statistics with ANALYZE TABLE.
• CPU Time and Peak Memory — Per operator, you see where compute and memory went.
• Spill Size — If non-zero, that operator ran out of memory and wrote to disk. That's slow.

Here's a real example. You have a join between two large tables:

users = spark.read.parquet("s3://users/")
events = spark.read.parquet("s3://events/")

result = events.join(users, "user_id").filter("user_paid = true").collect()

The SQL tab shows the join used a SortMergeJoin (expensive) instead of a BroadcastHashJoin (cheap). You check the users table size: 2 GB. Spark's broadcast threshold is 10 MB by default. A quick fix:

from pyspark.sql import functions as F

# Hint the optimizer
result = events.join(F.broadcast(users), "user_id").filter("user_paid = true").collect()

The SQL tab also highlights if a stage is adaptive. Spark 3.0+ can replan joins mid-execution if the broadcast is too large—the UI marks this clearly, and you see metrics on whether coalescing helped.

Always cross-check the physical plan tab with the jobs timeline. Sometimes the slowest stage in the SQL plan isn't the slowest in clock time due to pipelining and parallelism.

Storage — Cached Data and Memory Use

The Storage tab shows every cached DataFrame and RDD, including:

• Cache Size — Disk and memory breakdown.
• Percentage Cached — If a 100 GB table is 80% cached, 20% is still on disk.
• Cache Eviction — If frequently dropped, you may be thrashing the cache.

Caching is powerful but risky. We've seen teams cache tables and then wonder why their cluster runs out of memory. The Storage tab makes this visible instantly.

Example from a SaaS analytics pipeline:

# Expensive aggregation used multiple times
daily_summary = spark.read.parquet("s3://events/").groupBy("date", "region").agg(
    F.count("*").alias("event_count"),
    F.approx_percentile("latency", 0.5).alias("median_latency")
)

daily_summary.cache()
daily_summary.count()  # Trigger materialization

# Reuse it
q1 = daily_summary.filter("region = 'us-west'").collect()
q2 = daily_summary.filter("event_count > 1000").collect()
q3 = daily_summary.join(other_df, "date").collect()

The Storage tab shows daily_summary with its memory footprint. If it's 15 GB and your executor heap is 16 GB per node, you've little room for intermediate operations. The UI will also show if GC pressure increased after caching. If so, lower the cache fraction or use disk-based caching:

daily_summary.persist(StorageLevel.MEMORY_AND_DISK)

Check the Storage tab regularly in production jobs. Stale cached tables (ones that never get read again) waste memory.

Common Mistakes

• Ignoring the Executors tab. High GC time or memory pressure often points to executor misconfig, not query logic. Check heap size, max memory per task, and number of tasks per executor.
• Trusting estimated metrics. Always run the query and read actual metrics from the SQL tab or task table. Estimates are educated guesses.
• Chasing the longest task without context. A task that takes 30 seconds is only a problem if the stage has 50 tasks. If it has 2, that's normal variation.
• Forgetting to gather table statistics. Spark's cost-based optimizer makes much better decisions if ANALYZE TABLE ... COMPUTE STATISTICS has run recently.
• Confusing job vs. stage duration. A job with 10 stages runs stages sequentially in the DAG. If stage 5 has zero duration, it likely means it was filtered out entirely. Read the DAG to confirm the plan flow.

Key Takeaways

• Start in the Jobs tab to identify which stage is slow, then drill into its tasks.
• Look for task skew by sorting the task table by duration; a few outliers indicate data imbalance or GC pauses.
• The SQL tab shows actual (not estimated) metrics per operator—use it to spot bad joins, spilling, or wrong broadcast decisions.
• Monitor the Storage tab to prevent memory bloat from unchecked caching.
• Combine UI insights with code changes: broadcast hints, repartitioning, or statistics gathering often resolve bottlenecks identified in the UI.
• Remember that the UI is a snapshot; run the same job a few times to distinguish systematic slowness from transient cluster noise.