Time travel and auditing for Spark & Data Eng developers

The problem: data integrity without the safety net

You've just realized that a Spark job ran with a bug three days ago, corrupting customer records across your production data lake. Your team is scrambling to figure out what the data looked like before that bad write, and whether you can restore it without replaying the entire pipeline. Or worse—you need to debug why a transformation produced different results last week, but your logs only capture the final output, not the intermediate state. Time travel and auditing for Spark & Data Eng developers isn't a nice-to-have feature anymore; it's a critical operational capability that Delta Lake and Apache Iceberg bring to the table.

Delta and Iceberg support versions

Both Delta Lake and Apache Iceberg maintain a complete versioning history of your data lake. Every write operation creates a new version of your table, and you can query any historical version without replaying your pipelines. This is fundamentally different from traditional data warehouses where you overwrite tables in place.

Delta Lake stores version metadata in the _delta_log directory, while Iceberg maintains metadata in a separate metadata layer. When you write data, a new snapshot is created; the old snapshots remain accessible and queryable.

In practice, this means you can query your table as it existed at any point in time:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("TimeTravel").getOrCreate()

# Query the current version
current_df = spark.read.table("customers")
print(f"Current rows: {current_df.count()}")

# Query as it was 2 days ago (Delta Lake)
past_df = spark.read.option("versionAsOf", 42).table("customers")
print(f"Rows 2 days ago: {past_df.count()}")

# Or use timestamp
timestamp_df = spark.read.option("timestampAsOf", "2024-01-15 10:30:00").table("customers")

# Query using Iceberg snapshot ID
iceberg_df = spark.read.option("snapshot-id", 1234567890).table("customers")

The key difference: Delta tracks versions by operation number, while Iceberg uses snapshot IDs and timestamps. Both approaches give you the same outcome—the ability to step back in time without data loss.

Debug pipeline changes

When a transformation produces unexpected results, time travel becomes your debugging tool. Instead of rerunning the entire pipeline with debug logging, you can compare the state of your data before and after a specific operation.

Let's say your enrichment job started producing null values for a customer segment. You suspect something changed in the last 48 hours. You can query both versions side by side:

# Get the current (broken) data
broken_df = spark.read.table("customer_enriched")

# Get the version from before the suspected change
# Assuming version 150 was the last good version
good_df = spark.read.option("versionAsOf", 150).table("customer_enriched")

# Compare schemas and sample rows
print("Current schema:")
broken_df.printSchema()
print("\nPrevious schema:")
good_df.printSchema()

# Find what changed
from pyspark.sql.functions import col
broken_sample = broken_df.filter(col("enriched_field").isNull()).limit(10)
good_sample = good_df.filter(col("customer_id").isin(
    broken_sample.select("customer_id").rdd.flatMap(lambda x: x).collect()
)).select("customer_id", "enriched_field")

broken_sample.show()
good_sample.show()

This approach lets you identify exactly which records changed, what field was affected, and when it happened. You can then correlate that timing with code deployments or upstream data changes. The audit trail in your table's transaction log tells you which job wrote each version, making the root cause analysis much faster.

Restore bad writes

Once you've identified a bad version, you need to restore the table to a known good state. Both Delta and Iceberg provide simple commands for this.

With Delta Lake, use the RESTORE command:

from delta.tables import DeltaTable

delta_table = DeltaTable.forName(spark, "customers")

# Restore to a specific version
delta_table.restoreToVersion(150)

# Or restore to a timestamp
delta_table.restoreToTimestamp("2024-01-15 09:00:00")

# Verify the restoration
spark.read.table("customers").count()

The RESTORE command is atomic—either the entire operation succeeds or it fails with no partial state. This is crucial in production; you can't afford a restore operation that partially completes.

With Iceberg, you use the ALTER TABLE syntax to set the current snapshot:

# Iceberg: set the table to a specific snapshot
spark.sql("""
    ALTER TABLE customers SET IDENTIFIER (snapshot_id = 1234567890)
""")

# Or by timestamp
spark.sql("""
    ALTER TABLE customers SET IDENTIFIER (as_of_timestamp = cast('2024-01-15 09:00:00' as timestamp))
""")

One critical point: restoration doesn't delete the bad data; it just rolls the table's current state back to that version. The bad version remains in your transaction history. This is by design—you want an immutable audit trail. If you need to permanently remove data due to compliance or storage concerns, you'll handle that separately through retention policies.

Set retention for old versions

Keeping every version forever will eventually fill your storage. Both Delta and Iceberg let you define retention policies, automatically purging old versions while keeping recent ones available.

For Delta Lake, configure retention at the table level:

spark.sql("""
    ALTER TABLE customers 
    SET TBLPROPERTIES (
        'delta.logRetentionDays' = '30',
        'delta.deletedFileRetentionDays' = '7'
    )
""")

The logRetentionDays property controls how long transaction log entries are kept. deletedFileRetentionDays controls how long actual data files from deleted versions are retained in object storage before they're cleaned up by the VACUUM command.

Run VACUUM to actually remove old files:

from delta.tables import DeltaTable

delta_table = DeltaTable.forName(spark, "customers")
delta_table.vacuum(30)  # Keep files from the last 30 days

For Iceberg, set the retention policy similarly:

spark.sql("""
    ALTER TABLE customers SET TBLPROPERTIES (
        'history.expire.min_snapshots_to_keep' = '10',
        'history.expire.max_snapshot_age_ms' = '2592000000'  -- 30 days
    )
""")

# Run expiration
spark.sql("CALL system.expire_snapshots(table => 'customers')")

Iceberg's approach is slightly more flexible: you can keep a minimum number of snapshots regardless of age, and remove snapshots older than a certain threshold. This means you always have recent versions for quick rollback, even if something goes wrong frequently.

Be conservative with retention when you're starting out. A 30-day retention window is reasonable for most production systems—it's long enough to catch most data quality issues before they cascade downstream, but short enough that storage costs remain manageable. For tables with compliance requirements (financial transactions, user data), consider longer retention windows and coordinate with your legal team.

Common mistakes

Restoring without understanding the scope: Teams often restore a table thinking it fixes all downstream issues. But if bad data has already propagated to dependent tables, the restore only fixes the source. You need to trace data lineage and decide whether to restore dependent tables too. Document your restore decisions in incident reports.

Setting retention too aggressive: Setting retention to just 3 days saves storage, but leaves almost no window for debugging or recovery. A bad write discovered on day 5 becomes permanent. Start with 30 days and adjust downward only after you're confident your monitoring catches issues within that window.

Assuming VACUUM cleaned up storage immediately: After running VACUUM, object storage files may not disappear instantly—cloud storage systems like S3 sometimes take time to remove objects, and you're charged until they're actually gone. Retention and vacuum are about making old data inaccessible and eligible for removal, not instantaneous cleanup.

Not testing restore procedures: Your restore strategy only works if you've practiced it. Regularly test restoring to past versions in a staging environment to ensure your backups and versioning system actually work when you need them.

Key takeaways

• Delta and Iceberg maintain complete versioning histories, letting you query data as it existed at any point in time without replaying pipelines.
• Use time travel to debug pipeline changes by comparing table states before and after a suspected issue, correlating changes with code deployments.
• Restore corrupted or incorrect writes atomically using RESTORE (Delta) or ALTER TABLE SET IDENTIFIER (Iceberg).
• Set retention policies to balance operational safety (keep recent versions) with storage costs (purge old versions), typically 30 days is a good starting point.
• Restoration rolls back the current state but preserves the audit trail; use VACUUM (Delta) or expire_snapshots (Iceberg) to actually remove old files.
• Test your restore procedures regularly to ensure they work in production when you need them most.