Medallion architecture: a practical guide for developers

Introduction

Data lakes are powerful, but without structure they become data swamps. The medallion architecture—a layered approach that separates raw ingestion, data cleaning, and business analytics—solves this problem by giving your team a clear path from source systems to insights. In this guide, we'll walk through how medallion architecture: a practical guide for developers translates into real code and patterns you'll use when building scalable data pipelines on Spark.

What is Medallion Architecture?

Medallion architecture organizes your data lake into three layers, each with a specific purpose and quality level. The bronze layer stores raw data exactly as it arrives. The silver layer applies transformations—deduplication, schema enforcement, quality checks. The gold layer builds business-ready aggregates and dimensional models. This progression from raw to refined mirrors how we approach data quality in production systems: ingest first, validate and clean second, expose curated views last.

The pattern originated in the data community but has become standard practice in organizations using Spark, Delta Lake, and similar platforms. Its strength lies in simplicity: developers know where to look, reprocessing is straightforward, and lineage is clear.

Bronze Layer — Raw Data as Ingested

Bronze is your historical record. Every byte arrives here with minimal transformation. You might deduplicate at ingest time to avoid doubling storage, but schema changes, data quality issues, and missing fields all flow through. Bronze tables use the same schema as the source system—if a third-party API sends nested JSON, bronze stores it as-is.

In practice, bronze often lives in cloud object storage (S3, ADLS, GCS) with a Delta Lake transaction log on top. This gives you ACID properties, time travel, and versioning without forcing early transformations.

Bronze in code

Here's a typical bronze write from a Kafka stream or API polling job:

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, current_timestamp, input_file_name
from datetime import datetime

spark = SparkSession.builder.appName("bronze_ingest").getOrCreate()

# Read raw events from Kafka or file batch
df_raw = spark.readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "kafka:9092") \
    .option("subscribe", "user_events") \
    .load()

# Add ingestion metadata and write to bronze
df_bronze = df_raw \
    .select(col("value").cast("string")) \
    .withColumn("_ingested_at", current_timestamp()) \
    .withColumn("_source", lit("kafka_user_events"))

df_bronze.writeStream \
    .format("delta") \
    .mode("append") \
    .option("checkpointLocation", "s3://data-lake/bronze/user_events/_checkpoint") \
    .start("s3://data-lake/bronze/user_events")

spark.streams.awaitAnyTermination()

Notice the _ingested_at and _source columns. Every bronze table should track when and where data arrived. This matters when you need to replay or debug downstream issues. If a transformation bug affects gold layer, you can always reprocess from a known bronze snapshot.

Schema flexibility

Bronze handles schema changes gracefully. If a JSON source adds a new field, bronze stores it without failing. Silver layers then decide whether to standardize or drop it. This defensive posture keeps ingestion jobs stable even when upstream systems evolve.

Silver Layer — Cleaned, Deduplicated and Conformed

Silver is where data becomes trustworthy. You parse nested JSON, enforce types, remove duplicates, validate against business rules, and align schemas across similar sources. A silver table is the contract your organization trusts: if a row exists in silver, it has passed validation.

Most business logic lives here. Deduplication windows, handling late-arriving data, joining reference tables, masking sensitive columns—all happen at silver. You're not building aggregates yet; you're preparing the foundation.

Silver transformations in practice

Let's transform raw user events to silver:

from pyspark.sql import SparkSession
from pyspark.sql.functions import (
    col, from_json, schema_of_json, row_number, 
    when, coalesce, to_timestamp, desc
)
from pyspark.sql.window import Window
import json

spark = SparkSession.builder.appName("silver_transform").getOrCreate()

# Read bronze raw events
df_bronze = spark.read.format("delta").load("s3://data-lake/bronze/user_events")

# Define expected schema
sample_json = '{"user_id": "123", "event_type": "login", "timestamp": "2024-01-15T10:30:00Z", "metadata": {"ip": "192.0.2.1"}}'
schema = schema_of_json(sample_json)

# Parse JSON and extract fields
df_parsed = df_bronze \
    .filter(col("value").isNotNull()) \
    .select(
        from_json(col("value"), schema).alias("data"),
        col("_ingested_at")
    ) \
    .select(
        col("data.user_id"),
        col("data.event_type"),
        to_timestamp(col("data.timestamp")).alias("event_at"),
        col("data.metadata.ip"),
        col("_ingested_at")
    )

# Deduplicate: keep latest event per user per event_type per second
window_spec = Window.partitionBy("user_id", "event_type", "event_at").orderBy(desc("_ingested_at"))
df_deduped = df_parsed \
    .withColumn("rn", row_number().over(window_spec)) \
    .filter(col("rn") == 1) \
    .drop("rn")

# Validation: drop events with missing critical fields
df_silver = df_deduped \
    .filter(col("user_id").isNotNull()) \
    .filter(col("event_type").isNotNull()) \
    .filter(col("event_at").isNotNull()) \
    .withColumn("_processed_at", current_timestamp()) \
    .withColumn("_source", lit("bronze_user_events"))

# Write to silver with merge (upsert) for late arrivals
df_silver.write \
    .format("delta") \
    .mode("overwrite") \
    .option("mergeSchema", "true") \
    .save("s3://data-lake/silver/user_events")

Key points here: we parse JSON with a schema, deduplicate using a window function, filter invalid rows, and add processing metadata. The merge logic handles late-arriving events from the bronze layer.

Data quality and monitoring

Silver layers often include quality metrics. Track rejection rates, validate row counts against bronze, and flag anomalies:

df_validation = df_parsed \
    .withColumn("is_valid", 
        (col("user_id").isNotNull()) & 
        (col("event_type").isNotNull()) & 
        (col("event_at").isNotNull())
    )

valid_count = df_validation.filter(col("is_valid")).count()
total_count = df_validation.count()
rejection_rate = (total_count - valid_count) / total_count

print(f"Validation: {valid_count}/{total_count} rows valid ({rejection_rate:.2%} rejected)")

Gold Layer — Business-Level Aggregates and Models

Gold builds the views your analysts and applications consume. Aggregate events to daily active users. Join dimensions. Compute metrics. Create star schemas. Gold is opinionated—it reflects how the business wants to see data. A gold table might have one row per user per day with engagement metrics, or one row per order with customer, product, and financial details.

Gold is often smaller and more denormalized than silver because it's optimized for query patterns, not storage efficiency. You're trading disk space for query speed.

Gold example: daily user metrics

from pyspark.sql.functions import (
    col, count, countDistinct, max as spark_max, 
    date_trunc, lit, datediff, current_date
)

spark = SparkSession.builder.appName("gold_metrics").getOrCreate()

# Read silver events
df_silver = spark.read.format("delta").load("s3://data-lake/silver/user_events")

# Aggregate to daily active users with engagement metrics
df_daily = df_silver \
    .filter(col("event_type").isin(["login", "purchase", "view"])) \
    .withColumn("event_date", date_trunc("day", col("event_at"))) \
    .groupBy("user_id", "event_date") \
    .agg(
        count("*").alias("event_count"),
        countDistinct("event_type").alias("event_types"),
        spark_max("event_at").alias("last_event_at")
    ) \
    .withColumn("_loaded_at", current_timestamp())

# Enrich with user attributes from a reference table
df_users_ref = spark.read.format("delta").load("s3://data-lake/gold/dim_users")
df_enriched = df_daily \
    .join(df_users_ref, on="user_id", how="left") \
    .select(
        col("user_id"),
        col("event_date"),
        col("event_count"),
        col("event_types"),
        col("last_event_at"),
        col("user_ref.signup_date"),
        col("user_ref.country"),
        col("_loaded_at")
    )

# Write to gold
df_enriched.write \
    .format("delta") \
    .mode("overwrite") \
    .partitionBy("event_date") \
    .save("s3://data-lake/gold/fact_daily_user_activity")

Notice the partition by event_date. Gold tables are often partitioned to support incremental loads and efficient time-range queries. Once in gold, analysts query via SQL or BI tools without touching bronze or silver.

Benefits: Clear Lineage and Reprocessing

The three-layer structure gives you powerful operational benefits:

Lineage and debugging

When a gold metric is wrong, you can trace backwards. Check the gold transformation logic. If that's correct, inspect silver to see what data arrived. If silver looks good, examine bronze raw data. This staged debugging is far easier than trying to diagnose a single complex pipeline.

Reprocessing without data loss

Found a bug in your silver deduplication? Reprocess silver from bronze without re-ingesting from the source system. The bronze layer is your source of truth backup. This decoupling is crucial for production systems where re-running an API call or Kafka consumer might not be possible or desirable.

Similarly, if you need to recalculate gold metrics with a new business definition, reprocess from silver. You don't touch bronze or re-ingest anything.

Schema evolution

Because bronze accepts any schema, new fields from upstream systems land safely. Silver decides what to standardize. Gold doesn't break when a source adds columns—your transformations either use the field or ignore it. This resilience is invaluable in multi-source environments.

Performance isolation

Bronze might be tiny (one week of raw data) to save costs while keeping history for debugging. Silver is larger but still compressible. Gold is optimized for analytics queries—lots of precomputed metrics, wide tables, partitioned smartly. Each layer serves a different workload without interfering with others.

Common Mistakes

Skipping silver

Some teams jump from bronze directly to gold. This works for simple pipelines but couples your analytics to data quality issues and source schema changes. When the upstream API evolves or data quality degrades, you scramble to fix gold logic. Keep silver as a buffer.

Over-transforming bronze

Bronze should be boring. Don't apply business logic, don't aggregate, don't filter out "bad" data (mark it invalid, but keep it). Bronze is audit trail and recovery mechanism. Aggressively transforming it defeats that purpose.

No deduplication strategy

Medallion doesn't prescribe when to deduplicate, but doing it in silver (not bronze) gives you flexibility. Define your deduplication key clearly—is it (user_id, event_type, timestamp)? Include enough context in window functions to handle late arrivals and retries.

Forgetting lineage metadata

Add _ingested_at, _source, _processed_at columns to every layer. When queries fail or results look odd, these timestamps tell you exactly which pipeline version processed the data. Without them, you're debugging blind.

Gold bloat

Gold tables sometimes become kitchen sinks with every possible metric and dimension. This creates maintenance burden and slow queries. Segment gold into logical domains: fact tables for events, dimension tables for entities, aggregate tables for specific reports. Keep gold focused.

Key Takeaways

• Bronze is raw: Ingest data as-is with minimal transformation. Add ingestion metadata (_ingested_at, _source) but avoid business logic. Bronze is your recovery point.
• Silver is reliable: Parse schemas, deduplicate, validate, and conform data here. Silver is the contract your organization trusts. Build quality checks and rejection tracking into every silver pipeline.
• Gold is consumable: Build aggregates, denormalize, enrich with dimensions, and optimize for query patterns. Gold is what analysts and applications actually use.
• Reprocessing is straightforward: Because layers are independent, you can reprocess silver from bronze or gold from silver without touching earlier layers. This agility is critical when bugs or requirement changes happen.
• Lineage matters: Track timestamps and sources across all layers. When debugging, you can trace backwards from gold to understand what went wrong and at which stage.
• Resist shortcuts: Don't skip silver, don't over-transform bronze, and don't let gold become a dumping ground. Keep the structure clean and your team's productivity high.