The small files problem: real-world lessons and examples

Why small files matter in production

We've all seen it happen: a data pipeline that ran smoothly for months suddenly crawls to a halt. You check the cluster logs, review the query plans, and find nothing obviously broken. Then someone checks the HDFS directory listing and discovers thousands—sometimes millions—of files under 1 MB each. The small files problem: real-world lessons and examples shows up everywhere in production data systems, from streaming ingestion to ad-hoc analytics workloads. A single NameNode can only manage so many file objects, and excessive metadata pressure becomes a silent killer in Spark clusters.

Small files aren't just a cosmetic issue. Each file requires NameNode memory, increases task overhead, and forces the scheduler to create more tasks than necessary. In one e-commerce platform we worked with, a daily ETL that processed 50 GB of data spawned 8,000 tasks because the input was fragmented across 10,000 tiny files. By consolidating those files into proper-sized chunks, the same job completed in one-third the time without touching any application logic.

How many partitions create the small files problem

Partitioning is essential for performance, but it's easy to over-partition. When you write data with too many partition columns or too many distinct values within a partition, you end up with one file per partition key combination. Consider a real scenario from a ride-hailing platform:

# Over-partitioned write
df.write \
  .partitionBy("country", "city", "date", "hour", "minute") \
  .mode("append") \
  .parquet("s3://rides/events/")

If you have 50 countries, 500 cities per country, and you're writing every minute of every day, you've just created millions of partition directories. Even if each directory gets only 100 KB of data, you've created a maintenance nightmare. The NameNode metadata grows unbounded, and queries that could use partition pruning become slow because the planner must evaluate so many paths.

The fix is to use fewer partitioning levels and let filtering happen in the data layer instead:

# Better: partition only on high-cardinality, low-frequency columns
df.write \
  .partitionBy("country", "date") \
  .mode("append") \
  .parquet("s3://rides/events/")

# Filter by city at query time; it's fast enough for analytics
spark.read.parquet("s3://rides/events/") \
  .where("country = 'US' AND city = 'San Francisco' AND date = '2024-01-15'") \
  .show()

A good rule of thumb: aim for partitions that don't exceed a few thousand distinct values in total. If you're combining multiple columns, multiply their cardinalities to estimate total directories. If that number climbs above 10,000, reconsider your partitioning strategy.

Streaming writes and incremental data

Streaming is where the small files problem hits hardest. By design, Spark Streaming writes micro-batches frequently—every few seconds to a few minutes. Without intervention, you end up with hundreds of tiny files per day, even if the total daily volume is modest.

A travel platform we worked with ran Spark Structured Streaming to ingest booking events into a data lake. They configured 30-second micro-batches and wrote directly to Parquet:

# Naive streaming write: many tiny files
query = booking_events \
  .writeStream \
  .format("parquet") \
  .mode("append") \
  .option("path", "s3://bookings/raw/") \
  .option("checkpointLocation", "s3://bookings/checkpoint/") \
  .start()

Over a 24-hour period at typical event volume (1,000 events per second), they accumulated 2,880 files (one per 30-second batch). Most files were 50–200 MB, which isn't tiny, but the sheer count caused NameNode pressure and made downstream jobs slower than expected.

The solution is to use a staging layer with frequent compaction. Write to a temporary location with longer batch intervals, or use Delta Lake's unified format to manage compaction automatically:

# Better: write to Delta and compact regularly
query = booking_events \
  .writeStream \
  .format("delta") \
  .mode("append") \
  .option("path", "s3://bookings/raw/") \
  .option("checkpointLocation", "s3://bookings/checkpoint/") \
  .option("mergeSchema", "true") \
  .start()

# Then in a separate job, run OPTIMIZE hourly
spark.sql("OPTIMIZE delta.`s3://bookings/raw/` ZORDER BY timestamp")

OPTIMIZE in Delta Lake: the built-in solution

Delta Lake provides OPTIMIZE, a command designed specifically to combat small files. It rewrites partition directories to consolidate small files into larger ones without changing the data itself. For many teams, this is the simplest path forward because it requires no application refactoring.

In a SaaS product we supported, user events were streamed into Delta Lake at high velocity. After six months of production, the table had roughly 60,000 files, most under 50 MB. Queries that should have completed in seconds were timing out:

# Check table statistics
DESCRIBE DETAIL delta.`s3://events/user_actions/`

# Run OPTIMIZE to consolidate files
OPTIMIZE delta.`s3://events/user_actions/`
ZORDER BY user_id, event_timestamp

The ZORDER BY clause does double duty: it consolidates files and sorts them by the specified columns, making subsequent range queries faster. After one OPTIMIZE run, the table went from 60,000 files to 200 files, and query latency dropped by 70%.

You should schedule OPTIMIZE as a regular maintenance job. For high-velocity tables, run it daily or even hourly. For slower-moving tables, weekly is usually sufficient:

# Scheduled job: run every 6 hours
from datetime import datetime, timedelta

def compact_table():
    spark.sql("""
        OPTIMIZE delta.`s3://events/user_actions/`
        WHERE event_date >= date_format(current_date - interval 1 day, 'yyyy-MM-dd')
        ZORDER BY user_id, event_timestamp
    """)
    print(f"Compaction completed at {datetime.now()}")

# In your orchestrator (Airflow, Databricks Jobs, etc.)
compact_table()

One caveat: OPTIMIZE rewrites files, which costs compute and time. On very large tables (terabytes), running OPTIMIZE too frequently can become expensive. Balance your compaction schedule with the cost of additional cluster time. Start with weekly or bi-weekly for most tables, then adjust based on query performance and budget.

Targeting the right file size: 128 MB to 1 GB

Once you decide to compact or rewrite files, what's the target size? Industry consensus has settled around 128 MB to 1 GB per file, with many teams targeting around 256–512 MB as a sweet spot. This range balances several trade-offs:

• NameNode memory: Fewer total files means less metadata overhead.
• Task parallelism: Files larger than 128 MB may spawn multiple tasks (depending on Spark's input split logic), but you won't have thousands of tiny tasks.
• Network I/O: Large files amortize network overhead across more data, reducing per-GB latency.
• Recomputation cost: If a task fails, recomputing a 512 MB file is still fast; recomputing a 10 GB file is painful.

A CRM platform we worked with had settled on 256 MB as their standard. They monitored file sizes in their data lake weekly and flagged any partition that fell significantly below that threshold. When they noticed a sharp drop—often due to a schema change or a new upstream dependency—they'd trigger a consolidation pass.

To check file sizes in your data lake programmatically:

import os

def check_file_sizes(path):
    """List file sizes in a Parquet or Delta table directory."""
    sizes = []
    for root, dirs, files in os.walk(path):
        for file in files:
            if file.endswith('.parquet'):
                full_path = os.path.join(root, file)
                size_mb = os.path.getsize(full_path) / (1024 * 1024)
                sizes.append((file, size_mb))
    
    sizes.sort(key=lambda x: x[1])
    for name, size_mb in sizes[:10]:
        print(f"{name}: {size_mb:.1f} MB")
    
    avg_size = sum(s[1] for s in sizes) / len(sizes)
    print(f"\nAverage file size: {avg_size:.1f} MB")
    print(f"Total files: {len(sizes)}")

# Check a table
check_file_sizes("s3://data-lake/analytics/customers/")

If you find that most files are under 128 MB, that's a signal to run OPTIMIZE or manually rewrite the data. If files are consistently over 1 GB, you may want to increase parallelism or partition more aggressively—that's a different problem, but it's good to measure both extremes.

Common mistakes

• Ignoring the small files problem until it's critical: By then, your table may have millions of files and compaction will take hours or days. Monitor file counts and sizes proactively.
• Over-partitioning to avoid data skew: If you have a skewed dimension (e.g., 90% of orders from one region), partitioning by that dimension won't help. Use salting or bucketing instead, and keep partition cardinality reasonable.
• Running OPTIMIZE on the entire table every night: On multi-terabyte tables, this is overkill and expensive. Instead, OPTIMIZE only recent partitions or on a longer cadence (weekly or bi-weekly).
• Forgetting to set file size targets for manual writes: When you do a one-off load or a data repair, explicitly coalesce or repartition to your target size. Otherwise, the number of output files depends on the input size and partition count, which is unpredictable.
• Conflating "small files" with "too many partitions": They're related but distinct. Small files inside each partition are a metadata issue. Too many partitions are a query planning issue. Solve both, but understand the difference.

Key takeaways

• Small files slow Spark jobs by forcing excess NameNode metadata overhead and creating too many scheduler tasks. Aim for 128 MB to 1 GB per file as a practical target.
• Over-partitioning is a common source of small files. Use only high-value, low-cardinality partition columns, and keep total partition count under 10,000.
• Streaming workloads produce small files by nature. Use Delta Lake's OPTIMIZE command to consolidate them on a regular schedule, with ZORDER to speed up queries.
• Monitor file sizes and file counts in your data lake proactively. When average file size drops below 128 MB, trigger compaction.
• Schedule OPTIMIZE as a maintenance job on a weekly or bi-weekly cadence for most tables. For high-velocity tables, run it daily or hourly, but balance cost against benefit.