Spark architecture explained: what developers should know

Introduction

When you submit a Spark job, it doesn't just run. Your code travels through a carefully orchestrated system where the driver plans the work, a cluster manager allocates resources, and executors scattered across machines actually process your data. Understanding Spark architecture explained: what developers should know is the foundation for writing efficient distributed applications and debugging performance bottlenecks when things go wrong.

In this article, we'll walk through the key components that make Spark work: the driver, executors, and cluster manager. We'll use concrete examples and show you what's happening behind the scenes when you call spark.sql() or rdd.map().

How a Spark job actually runs

Before diving into individual components, let's trace the complete flow. When you submit a job to Spark, several things happen in sequence:

1. Driver plans the job. Your Python, Scala, or SQL code is parsed and compiled into a directed acyclic graph (DAG) of transformations.
2. Cluster manager allocates resources. The cluster manager (Standalone, YARN, Kubernetes, etc.) assigns executor slots and memory to your job.
3. Executors run tasks on partitions. Each executor receives a set of tasks that operate on data partitions, executing in parallel.
4. Results return to the driver or storage. Completed results are either collected back to the driver or written to external storage like S3 or a data warehouse.

This architecture enables horizontal scalability: as your data grows, you add more executors to the cluster, and Spark automatically distributes the workload across them. Let's see how each piece fits together.

The driver: planning your job

The driver is the process that runs your Spark application. It's where your SparkSession lives, where transformations are defined, and where the execution plan is born. When you write a Spark program, you're writing code that runs on the driver.

The driver's primary responsibilities are:

• Parsing your transformations and actions into a DAG
• Optimizing the DAG using Catalyst (in SQL/DataFrame) or Tungsten (memory management)
• Creating tasks from stages and sending them to executors
• Collecting results when you call collect() or show()
• Maintaining the state of the entire job

Here's a practical example. Imagine you're processing order data:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("OrderProcessor").getOrCreate()

# This code runs on the driver
orders_df = spark.read.parquet("s3://data-lake/orders")
high_value = orders_df.filter(orders_df.amount > 1000)
result = high_value.groupBy("customer_id").sum("amount")
result.show()

When you execute this code, the driver doesn't immediately push the data around. Instead:

1. It builds a logical plan: read → filter → groupBy → sum
2. Catalyst optimizer rewrites it for efficiency (pushing filters down, choosing join strategies)
3. It converts the plan into stages and tasks
4. It sends those tasks to executors
5. When show() is called, it retrieves results from executors and displays them

The driver also acts as a scheduler. It keeps track of which tasks succeeded, which failed, and where to retry them. This is why the driver is a single point of coordination—and why it needs enough memory to hold metadata about your job.

Important: The driver should never do heavy computation. If you accidentally collect a massive DataFrame to the driver (df.collect() on a billion-row dataset), you'll run out of memory. The driver's job is to orchestrate, not to crunch.

Executors: the workers that process data

Executors are the JVM processes (or Python processes, depending on your language) that actually run your tasks. There are many executors in a cluster, each holding a slice of your data in memory, and they work in parallel. This parallelism is where Spark gets its performance.

Each executor has a few key attributes:

• Memory: Space for caching data and executing tasks. Typically 2–8 GB per executor in production.
• CPU cores: Number of tasks it can run simultaneously. Usually 2–8 cores per executor.
• Local disk: Temporary shuffle space if your job needs to redistribute data between stages.

When a task runs on an executor, it typically:

1. Reads a partition of data from storage or cache
2. Applies transformations (filter, map, aggregation logic)
3. Produces output partitions (either shuffled to other executors or cached)

Let's expand our order example. Say you have 100 GB of order data split across 1,000 partitions. If you have 10 executors, each with 4 cores, Spark will run 40 tasks in parallel (10 executors × 4 cores). Each task processes roughly 100 MB of data:

# Still on the driver
orders_df = spark.read.parquet("s3://data-lake/orders")  # 1,000 partitions

# When this action runs, tasks are sent to executors
# Each executor processes its partitions in parallel
high_value = orders_df.filter(orders_df.amount > 1000)
grouped = high_value.groupBy("customer_id").sum("amount")

# This triggers the actual computation across the cluster
result = grouped.collect()

Behind the scenes, each executor is running the filter and aggregation on its assigned partitions. The groupBy operation triggers a shuffle: data is redistributed so all records for the same customer_id end up on the same executor for final aggregation.

Caching: Executors also manage cache. When you call df.cache() or df.persist(), the executor stores data in memory (or disk) after the first computation. Subsequent actions on that DataFrame reuse the cached data, avoiding recomputation.

# Cache the filtered DataFrame
high_value = orders_df.filter(orders_df.amount > 1000).cache()

# First action: computes and caches
count1 = high_value.count()  # Executors cache filtered data

# Second action: reuses cache, much faster
avg = high_value.agg({"amount": "avg"})  # Reads from cache, no recomputation

Cluster managers: resource allocation

The cluster manager is the layer between Spark and your actual hardware. It allocates machines (nodes) and resources (memory, CPU) to your jobs. Spark supports multiple cluster managers, and choosing the right one depends on your infrastructure.

Standalone: Spark's built-in cluster manager. Simple to set up, good for small teams or on-premise deployments. The driver talks directly to a standalone master process, which assigns executors to worker nodes.

YARN (Hadoop): Apache Hadoop's resource manager. Common in enterprises with existing Hadoop infrastructure. Spark runs as a YARN application, competing for resources with other YARN jobs.

Kubernetes: Modern, cloud-native option. Each executor runs in a Kubernetes pod. Popular for containerized environments on AWS EKS, GCP GKE, or on-premise clusters.

Managed platforms: Databricks, AWS EMR, Google Cloud Dataproc, or Azure Synapse handle cluster management for you. You submit jobs and they handle provisioning, scaling, and cleanup.

Here's how the cluster manager fits in. When you submit a job:

# From your local machine or CI/CD pipeline
spark-submit \
  --master yarn \
  --num-executors 10 \
  --executor-memory 4g \
  --executor-cores 4 \
  my_job.py

The Spark submit script contacts the YARN cluster manager (in this case). YARN finds 10 nodes, allocates 4 GB memory and 4 CPU cores per executor, starts the JVM processes, and reports back to Spark. Your driver then begins sending tasks to those executors.

Different cluster managers have different trade-offs:

If you're using Kubernetes, your spark-submit call might look like:

spark-submit \
  --master k8s://https://kubernetes-cluster:6443 \
  --deploy-mode cluster \
  --num-executors 10 \
  --executor-memory 4g \
  --executor-cores 4 \
  my_job.py

Spark will create Kubernetes pods for the driver and each executor, managing their lifecycle automatically.

Common mistakes

Collecting large datasets to the driver: A single collect() on a billion-row DataFrame will crash your driver. Always filter or aggregate first. If you need results, use limit() or write to storage.

Underestimating memory needs: Executors need memory for both data storage and computation overhead. A 4 GB executor can't comfortably process a 10 GB partition. Leave headroom: if you have 8 GB per executor, don't try to cache more than 5–6 GB of data.

Too many small partitions: Each partition becomes a task, and task scheduling has overhead. If you have 10,000 tiny partitions but only 4 cores, your cluster will spend more time scheduling than computing. Repartition to balance workload.

Forgetting to cache intermediate results: In exploratory work, you might reuse a filtered DataFrame multiple times. Forgetting to cache it means Spark recomputes it from scratch each time. Use .cache() strategically.

Misconfiguring executor resources: Too many executors with too little memory per executor creates contention. Too few executors means you're not utilizing the cluster. Aim for 1–4 GB per core in most cases, and tune based on your workload.

Key takeaways

• The driver is the coordinator: it parses your code, builds the execution plan, and sends tasks to executors. Never do heavy computation on the driver.
• Executors do the actual work. They run tasks in parallel on data partitions, cache data, and return results. More executors and cores mean higher parallelism.
• The cluster manager allocates physical resources. Choose based on your infrastructure: Standalone for simplicity, YARN for Hadoop, Kubernetes for cloud-native, or managed services for convenience.
• A Spark job flows through these stages: driver builds DAG → cluster manager allocates resources → executors process partitions in parallel → results return to driver or storage.
• Understanding this architecture helps you debug performance, right-size your cluster, and avoid common pitfalls like collecting massive datasets or misconfiguring memory.