What Apache Spark is: a practical Spark & Data Eng guide

Introduction

If you've worked with datasets larger than a single machine can handle, you've probably heard Apache Spark mentioned in the same breath as "big data." But Spark isn't just marketing speak—it's a real, battle-tested tool that teams at scale use every day to process petabytes of data reliably. In this guide, we'll walk through what Apache Spark is: a practical Spark & Data Eng guide that explains how it works, why it matters, and when you should reach for it instead of a traditional database or Hadoop.

Distributed Processing: Splitting Work Across Machines

At its heart, Spark is a distributed computing engine. Instead of processing data on a single server, Spark partitions your dataset and sends chunks of work to multiple machines in parallel. Each worker processes its slice of data independently, then the results are combined.

This matters because a laptop with 16GB of RAM can only handle so much data in a single operation. But if you have ten machines, you've got 160GB of combined memory. Spark orchestrates this distribution for you automatically—you write the logic once, and Spark figures out how to slice it up.

Here's a practical scenario: you've got a 100GB CSV file of transaction logs. A single-threaded program would take hours. Spark will split that file into, say, 100 partitions and process chunks in parallel across your cluster.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("LogAnalysis").getOrCreate()

# Read a large CSV file
transactions = spark.read.csv("s3://bucket/transactions.csv", header=True)

# Spark automatically partitions this across the cluster
result = transactions.filter(transactions.amount > 1000).groupBy("customer_id").count()

# Trigger the computation
result.show()

What's happening under the hood: Spark reads the CSV in parallel from S3, filters rows across all workers simultaneously, groups by customer ID (reshuffling data as needed), and counts. Each step runs on multiple machines at once.

The key insight is that Spark handles partitioning and scheduling automatically. You don't manually split files or orchestrate which machine processes what. You define the logic, and Spark's DAG (Directed Acyclic Graph) scheduler figures out the optimal execution plan.

In-Memory Speed: Keeping Data in Memory Between Steps

The second major advantage of Spark is that it keeps intermediate results in memory rather than writing them to disk between operations. This is a huge performance win compared to MapReduce, which would write to disk after every map and reduce step.

Consider a typical ETL pipeline:

1. Read raw data from a data lake
2. Filter out bad records
3. Join with dimension tables
4. Aggregate metrics
5. Write results to a warehouse

With MapReduce, each of these five steps would write its output to HDFS and read it back in the next step. With Spark, steps 1–4 all happen in memory on the workers. Only the final result is written to disk or your data warehouse. The difference in speed can be 10x or more.

from pyspark.sql.functions import col, sum as spark_sum

# Step 1: Read
orders = spark.read.parquet("s3://data-lake/orders")

# Step 2: Filter
valid_orders = orders.filter(col("status") != "cancelled")

# Step 3: Join
customers = spark.read.parquet("s3://data-lake/customers")
enriched = valid_orders.join(customers, "customer_id")

# Step 4: Aggregate
daily_revenue = enriched.groupBy("date", "region").agg(spark_sum("amount"))

# Step 5: Write
daily_revenue.write.mode("overwrite").parquet("s3://warehouse/daily_revenue")

All intermediate DataFrames (valid_orders, enriched) live in memory until the final write. In a real-world deployment, this could process millions of rows in seconds instead of minutes.

One caveat: memory isn't unlimited. If you try to shuffle more data than your cluster can hold, Spark will spill to disk, and performance degrades. Knowing when to repartition or cache strategically is part of tuning production jobs.

One Engine: Batch, Streaming, SQL, and ML

A major strength of Spark is that it's one unified engine for multiple workload types. You're not juggling Hadoop for batch, Kafka Streams for real-time, Presto for SQL, and MLlib for models. It's all Spark.

Batch processing: The example above—reading a parquet file, transforming, and writing results—is classic batch. Run it once a day, once an hour, whatever. Spark excels here.

Streaming: Spark Structured Streaming lets you write the same DataFrame transformations but apply them to continuously arriving data. A Kafka topic becomes a stream, and you process micro-batches of incoming events in real-time.

from pyspark.sql.functions import from_json, schema_of_json

# Read from Kafka as a stream
events = spark.readStream.format("kafka") \
    .option("kafka.bootstrap.servers", "localhost:9092") \
    .option("subscribe", "user_events") \
    .load()

# Parse JSON and transform
parsed = events.select(from_json(col("value").cast("string"), schema).alias("data")) \
    .select("data.*")

# Write results continuously to a warehouse
parsed.writeStream.format("parquet") \
    .option("path", "s3://warehouse/events") \
    .option("checkpointLocation", "s3://checkpoints/events") \
    .start()

SQL: Spark SQL lets you query DataFrames using standard SQL. You can load a Parquet file as a table and run SELECT, JOIN, GROUP BY just like you would in PostgreSQL. Perfect for analysts or when SQL is more natural than DataFrame API code.

orders.createOrReplaceTempView("orders")
customers.createOrReplaceTempView("customers")

result = spark.sql("""
    SELECT c.region, COUNT(*) as order_count, SUM(o.amount) as total_revenue
    FROM orders o
    JOIN customers c ON o.customer_id = c.id
    WHERE o.status = 'completed'
    GROUP BY c.region
""")

Machine Learning: Spark MLlib provides distributed algorithms for classification, regression, clustering, and recommendation systems. Train on massive datasets without sampling them down.

This flexibility means you can prototype locally with a Python script, scale to production with Spark, add real-time streams as requirements evolve, and train models on the same data pipeline—all without major rewrites.

APIs: Python, Scala, Java, SQL, and R

Spark isn't locked into one language. Your team can choose what works best for them:

• PySpark (Python): The most popular choice for data scientists and analysts. Easy to learn, integrates with pandas and numpy for local work.
• Scala: Native Spark language, best performance if you need to squeeze every millisecond. Popular in finance and trading where latency matters.
• Java: Used when Spark jobs need to integrate tightly with enterprise JVM applications.
• SQL: Not a full programming language, but for pure data transformation and analysis, sometimes the clearest option.
• R: Via SparkR, useful if your team has R specialists working with statistical models.

The DataFrame API is consistent across all of these, so code patterns translate. A PySpark job doing a join and aggregation has the same logical structure as a Scala version.

# PySpark
df.filter(col("amount") > 1000).groupBy("category").agg({"amount": "sum"})

# Scala equivalent (same logic, different syntax)
df.filter(col("amount") > 1000).groupBy("category").agg(sum("amount"))

# SQL equivalent
SELECT category, SUM(amount) FROM df WHERE amount > 1000 GROUP BY category

In practice, teams often use PySpark for data engineering and ML, Scala or Java for infrastructure-level jobs where performance is critical, and SQL for analyst queries. Spark supports this mix without friction.

Common Mistakes

Not understanding partitions: A Spark job running on a single partition is just a single machine doing the work. If your input data isn't partitioned well or you accidentally repartition to 1, Spark becomes useless. Always check your partition count with df.rdd.getNumPartitions().

Collecting large DataFrames locally: df.collect() pulls all data into the driver's memory. If your DataFrame is 50GB, this crashes. Use df.limit(1000).show() for inspection or write to storage instead.

Misunderstanding lazy evaluation: Spark doesn't compute anything until you call an action like show(), write(), or collect(). Transformations like filter() and groupBy() are lazy. This trips up newcomers who write a complex query and see no output until they add .show() at the end.

Ignoring data skew: If your data skew is severe—say, 90% of rows have the same customer ID—a groupBy will cause one worker to process vastly more data than others. The job runs at the speed of the slowest worker. Bucketing or salting keys can help, but requires planning.

Running Spark locally without enough memory: Spark driver and executor JVMs each need heap. If you're testing locally and set too little memory, jobs fail mysteriously. Allocate at least 2-4GB per executor.

Key Takeaways

• Spark is a distributed engine that splits data processing work across a cluster, making it practical to handle datasets far larger than a single machine.
• In-memory computation between steps delivers massive speed advantages over MapReduce—often 10x faster for multi-step pipelines.
• One unified engine supports batch, streaming, SQL queries, and ML training, reducing operational complexity and code rewrites.
• Multiple APIs (Python, Scala, Java, SQL, R) let teams choose the right tool without abandoning Spark, though PySpark dominates in practice.
• Partitioning, lazy evaluation, and memory management are non-negotiable concepts—misunderstanding them leads to slow or broken jobs.
• Spark is not a replacement for analytical databases or data warehouses; it's a processing engine that feeds them or sits between them and raw data sources.