Apache Iceberg basics in Apache Spark & Data Engineering

Introduction

After nearly a month of building data pipelines, you've probably hit the wall where your Parquet tables break when you need to add a column, or time travel queries force you to maintain brittle partition schemes. Apache Iceberg basics in Apache Spark & Data Engineering solve these problems by treating data tables as first-class versioned objects rather than static file collections. Iceberg is an open table format that brings ACID guarantees, schema evolution, and snapshot isolation to the data lake without the operational complexity of traditional data warehouses.

In this article, we'll walk through the core concepts that make Iceberg powerful in production: hidden partitioning, schema and partition evolution, multi-engine compatibility, and time travel capabilities. By the end, you'll understand why teams managing petabyte-scale data are adopting Iceberg to replace older formats like Parquet and Delta Lake.

What is Apache Iceberg?

Iceberg is a table format specification that sits between your query engine and your object storage. Instead of treating a Parquet dataset as "just files," Iceberg wraps those files with metadata that tracks schema, partitioning, snapshots, and data statistics. This metadata lives in a manifest system—essentially a tree of files that catalog which data files belong to which table version.

The key insight is that Iceberg separates the logical table definition from the physical file layout. Your query engine interacts with Iceberg metadata, not raw files, giving you atomicity and consistency guarantees that are impossible with naked Parquet.

Hidden Partitioning

Partitioning is essential for query performance at scale, but it's also a source of operational pain. With Parquet or CSV, you manually create partition schemes like s3://data/year=2024/month=01/day=15/, and any change to that scheme requires rewriting terabytes of data.

Iceberg solves this with hidden partitioning. You define partitioning rules at the table level, and Iceberg transparently applies them during writes. If you change your partitioning strategy, Iceberg's metadata is updated—the physical files don't move.

For example, instead of manually organizing by date, you tell Iceberg:

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("IcebergExample") \
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
    .config("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog") \
    .getOrCreate()

spark.sql("""
CREATE TABLE events (
    event_id STRING,
    event_timestamp TIMESTAMP,
    user_id STRING,
    event_type STRING
)
USING iceberg
PARTITIONED BY (days(event_timestamp), event_type)
""")

Here, Iceberg partitions by day granularity derived from the timestamp, and also by event type. When you insert a row with event_timestamp = '2024-01-15 14:30:00' and event_type = 'click', Iceberg automatically places it in the right partition without you having to construct the path. If you later want to partition by hour instead of day, you simply update the table definition—no data rewrite.

Hidden partitioning also means query engines like Trino and Flink understand partition pruning correctly. They read the partition metadata, not the folder structure, eliminating a common source of bugs where a partition column is renamed and downstream queries silently read incorrect data.

Schema and Partition Evolution

Real-world schemas change constantly. You add telemetry fields, deprecate old columns, rename types for clarity. With Parquet, evolving schema is either impossible or requires manual rewriting. Iceberg treats schema evolution as a first-class operation.

Each schema change is tracked with a schema ID. Data files retain a reference to the schema version they were written with. When you query the table, Iceberg reconciles schema differences transparently—old files use their original schema, new files use the new one, and the query engine handles both.

# Initial table created with three columns

# Later, add a new column
spark.sql("""
ALTER TABLE events
ADD COLUMN session_id STRING
""")

# Iceberg tracks this as a schema evolution (schema ID 1 -> 2)
# Writes after this point use the new schema
# Reads work across both schemas automatically

spark.sql("SELECT event_id, session_id FROM events").show()
# New files have session_id; old files show NULL

Partition evolution works similarly. You can add new partition columns, change partition expressions, or remove existing ones without rewriting the table. Iceberg's metadata layer handles the complexity:

# Add a new partition column
spark.sql("""
ALTER TABLE events
ADD PARTITION FIELD country
""")

# Future writes will include country in partitioning
# Historical data remains accessible without the partition

This capability is transformative for teams that have experienced the operational nightmare of Parquet schema migrations. In production, we've seen this eliminate entire categories of deployment risk.

Works with Spark, Trino, and Flink

One reason Iceberg has gained traction is that it's engine-agnostic. The same table can be read and written by Spark for ETL, Trino for SQL analytics, and Flink for stream processing. This is radically different from vendor-locked formats.

Each engine sees the same logical table definition and metadata, so you avoid the problem of maintaining separate data pipelines for different tools. In a typical setup, you might have:

• Spark: bulk writes and transformation jobs
• Trino: interactive SQL queries from BI tools
• Flink: real-time stream ingestion and processing

All three engines talk to the same Iceberg table via a shared metadata store (usually Hive Metastore or Nessie). This eliminates data duplication and synchronization issues that plague multi-engine setups.

From Spark's perspective, reading an Iceberg table is straightforward:

df = spark.read.format("iceberg").load("my_catalog.db.events")
df.show()

# Or SQL
spark.sql("SELECT * FROM my_catalog.db.events WHERE event_type = 'click'").show()

Trino would query the same table identically, and Flink would consume it for streaming aggregations. The unified interface is a major operational win in organizations running multiple data tools.

Snapshot-Based Time Travel

One of Iceberg's most powerful features is snapshot isolation. Every write operation creates a new snapshot—a complete, consistent view of the table at a point in time. You can query any snapshot without locking or performance penalties.

This enables genuine time travel: retrieve data as it existed hours, days, or months ago. Combined with audit requirements, debugging, and A/B test analysis, this is invaluable:

# Query the current state
spark.sql("SELECT COUNT(*) FROM events").show()

# Query as of 10 minutes ago
spark.sql("""
SELECT COUNT(*) FROM events
FOR SYSTEM_TIME AS OF CAST(unix_timestamp() - 600 as TIMESTAMP)
""").show()

# Or by snapshot ID
spark.sql("""
SELECT COUNT(*) FROM events
VERSION AS OF 123456789
""").show()

Under the hood, Iceberg maintains a manifest list that points to one or more manifest files. Each manifest file lists the data files and their statistics. A snapshot is simply a pointer to a manifest list at a given timestamp. Querying a past snapshot means Iceberg only scans the files that were live at that time.

The performance of time travel queries is comparable to reading the current table—no hidden performance cliff. This is possible because Iceberg avoids compaction hell (common in Parquet-based lakes where old files accumulate) through its snapshot and manifest design.

In production, we use snapshots for:

• Debugging: query the data state right before a failed transformation
• Compliance: retain immutable snapshots for regulatory audits
• Rollback: revert to a known-good snapshot if a pipeline corrupts data
• Analysis: compare user behavior across time windows for A/B tests

Common Mistakes

Mistake 1: Confusing Iceberg snapshots with data retention. Creating snapshots doesn't automatically retain old data. If you delete old files from object storage, historical snapshots become unreadable. You need a retention policy that matches your compliance and analytics needs. Iceberg provides expire_snapshots, but you must configure it explicitly.

Mistake 2: Assuming hidden partitioning eliminates the need for partition pruning logic. While Iceberg handles partitioning automatically, your query engine still needs good filter predicates to avoid full table scans. A query like SELECT * FROM events without a WHERE clause still touches all partitions. Write filters in your queries.

Mistake 3: Using default Spark settings without tuning Iceberg workers. Iceberg parallelizes manifest reads and file discovery. On clusters with many executors, you may see bottlenecks in manifest processing if you haven't tuned spark.sql.iceberg.parallelism or spark.sql.iceberg.shuffle.enabled. Benchmark your setup.

Mistake 4: Not planning for metadata storage. Iceberg metadata (manifests, schemas) lives in object storage alongside data, but metadata operations benefit from fast access. If your metadata latency is high, queries slow down. Keep your metadata close to your compute, and consider dedicated metadata tiers for large deployments.

Key Takeaways

• Iceberg is a table format that decouples logical table definitions from physical file layouts, enabling ACID guarantees and consistent snapshots.
• Hidden partitioning lets you evolve partition schemes without rewriting data, and enables correct partition pruning across engines.
• Schema and partition evolution are first-class operations, tracked with version IDs, eliminating the painful manual rewrites required by Parquet.
• Engine compatibility (Spark, Trino, Flink) means a single table can be read and written by multiple tools, reducing duplication and synchronization burden.
• Snapshot-based time travel lets you query any prior state of the table efficiently, unlocking debugging, compliance, and analytics use cases.
• Plan for metadata storage and snapshot retention policies in production; Iceberg doesn't magically handle these, but it makes them tractable.