100 days of Apache Spark and data engineering: the recap

100 days of Apache Spark and data engineering: the recap

We've covered a lot of ground in 100 days of Apache Spark and data engineering—from foundational concepts to production hardening strategies. If you've worked through this series, you've built the mental models and practical skills needed to ship reliable, performant data pipelines at scale. This recap distills the lessons that matter most, the ones that separate weekend projects from systems that handle real traffic and real stakes.

Understand the engine — Lazy plans, partitions and shuffles

The biggest aha moment for most engineers comes when they realize Spark doesn't execute your code the moment you write it. It builds a logical execution plan, optimizes it, and only runs when you call an action. This laziness is a feature, not a bug. It lets Spark rearrange operations, combine stages, and avoid redundant work. But it also means you need to think differently about debugging and performance.

Partitions are your primary lever for parallelism. Every DataFrame in Spark is divided into partitions, and each partition is processed independently by a task. If you have 100 million rows spread across just 4 partitions, you're leaving CPU cores idle. If you have 10,000 partitions for a small join, you're drowning in scheduling overhead. The sweet spot depends on your cluster, your data, and your operation.

Shuffles are the tax you pay when data must move between partitions. A groupBy, join, or repartition triggers a shuffle. Shuffles write to disk, serialize and deserialize data, and transfer across the network. They're not evil—they're necessary—but they're expensive. In production, a single expensive shuffle can become your bottleneck.

Here's a concrete example. Imagine you're building a user event funnel: users land on a page, click a button, and complete a purchase. You have events scattered across 500 partitions. A naive approach shuffles everything by user ID to group events, then shuffles again to filter by event type:

# Naive approach: multiple shuffles
events_by_user = events.groupBy("user_id").agg(...)
purchases = events_by_user.filter(col("event_type") == "purchase")

A smarter approach filters first, reducing the data size before the expensive shuffle:

# Better: filter reduces shuffle size
purchases = events.filter(col("event_type") == "purchase") \
    .groupBy("user_id").agg(...)

The Spark UI shows you exactly what's happening: stage times, shuffle read/write sizes, task duration. Staring at those numbers teaches you more than theory ever will. Run the same query with different partition counts, different filter orders, and watch how the numbers change.

Design idempotent pipelines — Incremental loads and medallion layers

Idempotency is non-negotiable in production. Your pipeline will fail. Networks drop packets. Disks fill up. Clusters get scaled down at the wrong moment. When your pipeline restarts, it must produce the same result whether it runs once or a hundred times. That means no side effects, no "please don't duplicate my data" comments in Slack.

Medallion architecture—bronze, silver, gold layers—gives you a clear way to structure this. Bronze is raw data, as it arrived. Silver is cleaned, deduplicated, validated data. Gold is business logic and aggregations. Each layer is independently reproducible from the layer below.

Incremental loading keeps your batch times predictable. If you reprocess your entire dataset every night, that query will slow down as your data grows. Instead, track what changed since the last run. Use _change_date or _modified_timestamp columns. Partition by date. Load only today's data, merge with yesterday's state, and move on.

Here's the pattern in practice:

# Bronze: load only new/modified records
bronze_df = spark.read.parquet("s3://data/raw/events")
new_records = bronze_df.filter(col("_load_date") == current_date())

# Silver: deduplicate and join
silver_df = new_records.dropDuplicates(["event_id"]) \
    .join(user_reference, on="user_id", how="left")

# Gold: aggregate for reporting
gold_df = silver_df.groupBy("user_id", "event_type") \
    .agg(count("*").alias("count"), max("timestamp").alias("latest"))

# Write with partition for idempotency
gold_df.write.mode("overwrite") \
    .partitionBy("event_type") \
    .parquet("s3://data/gold/funnel")

The mode("overwrite") combined with partition pruning makes this operation idempotent. Run it five times in a row with the same input, get the same output five times. That's the goal.

Quality is a feature — Checks, monitoring and ownership

A fast pipeline that produces wrong answers is worse than no pipeline at all. Quality checks aren't optional, and they're not an afterthought. They're part of the pipeline itself. Every stage should validate its output before passing it downstream.

Row counts are your first line of defense. If upstream data dropped by 50%, you should know before your aggregations go wrong. Schema changes are your second. A new nullable column or a renamed field breaks downstream code silently if you're not watching. Check row counts and schemas at each stage.

Business logic checks come next. If a user has negative total revenue, something is wrong. If a timestamp is in the future, something is wrong. These checks are domain-specific and require collaboration between engineering and analytics. Encode them anyway.

Here's a pattern we've used across multiple teams:

from datetime import datetime

def validate_gold_data(df, stage_name):
    """Assert on row count, schema, and business logic."""
    
    # Row count check
    row_count = df.count()
    assert row_count > 0, f"{stage_name}: zero rows"
    assert row_count < 1_000_000_000, f"{stage_name}: suspiciously large"
    
    # Schema check
    required_cols = ["user_id", "event_count", "latest_timestamp"]
    for col in required_cols:
        assert col in df.columns, f"{stage_name}: missing column {col}"
    
    # Business logic check
    invalid_rows = df.filter(col("event_count") < 0)
    assert invalid_rows.count() == 0, f"{stage_name}: negative event counts"
    
    invalid_timestamps = df.filter(col("latest_timestamp") > lit(datetime.utcnow()))
    assert invalid_timestamps.count() == 0, f"{stage_name}: future timestamps"
    
    print(f"{stage_name} validated: {row_count} rows")
    return df

# Use in pipeline
gold_df = validate_gold_data(gold_df, "gold_funnel")

Monitor these checks continuously. Set up alerts when row counts deviate by more than 10%. Alert on schema changes. Alert on assertion failures. And assign ownership. One person is responsible for keeping this pipeline working. Give them dashboards, logs, and a clear escalation path.

Optimize with data — Spark UI, plans and right-sized clusters

Optimization isn't guesswork. The Spark UI is your telescope into what's actually happening. Every query you run in production should have a UI captured somewhere—in logs, in dashboards, or at minimum in a local run you can inspect.

Look at the SQL tab first. Your logical and physical plans show you exactly what Spark is doing. Look for full scans where you expected partition pruning. Look for broadcasts that didn't happen. Look for shuffles that could have been avoided. The optimizer is good, but it's not magic.

Look at the stages tab. How long did each stage take? Which stage had the most shuffle read bytes? Which stage spilled to disk? Spilling means your memory was too small for the operation. Either increase executor memory or reduce the working set.

Task-level metrics tell the real story. If 1,000 tasks run in 30 seconds but one task takes 2 minutes, you have a skew problem. One partition has much more data than others, and the whole job waits for the slowest task. Repartition or use salting techniques to spread the load.

Right-sizing your cluster is where data informs decisions. Don't guess. Run your job on a small cluster and record the Spark UI. Calculate CPU utilization, memory pressure, shuffle I/O. Scale up incrementally and watch how metrics improve. Usually there's a sweet spot where doubling cluster size gets you 30% more throughput—not worth it. Optimize the pipeline instead.

Common mistakes

• Calling count() for validation: Counting a large DataFrame forces a full scan. Use approximate statistics or rely on row count from write operations instead.
• Skipping the explain plan: Before running a big query, call df.explain(True). A one-minute review saves an hour of debugging.
• Overpartitioning for "performance": Too many small partitions add scheduling overhead. Aim for tasks that run 1–5 minutes each, not milliseconds.
• Writing quality checks after the pipeline: By then your bad data is already in production. Validate before each write.
• Assuming your cluster is the bottleneck: Most Spark performance issues live in the code, not the hardware. Optimize queries before spending on more nodes.

Key takeaways

• Spark's laziness and distributed execution are features. Use the UI and logical plans to see what's really happening.
• Partitions and shuffles are your primary levers. Filter early, partition wisely, minimize shuffles.
• Idempotency is non-negotiable. Structure pipelines with medallion layers and incremental loads so failures and retries produce the same result.
• Quality checks aren't optional. Validate row counts, schemas, and business logic at every stage. Assign ownership.
• Let data drive optimization. Run on small clusters, read the Spark UI, and scale incrementally.
• 100 days of Spark and data engineering teaches patterns, not rules. Your production system will be different, but these principles apply everywhere.