Data lake folder conventions: what developers should know

Introduction

When you're building data platforms that process terabytes of events and serve dozens of teams, folder structure becomes infrastructure. Data lake folder conventions: what developers should know is not just about tidiness—it's about making your data discoverable, preventing accidental overwrites, and letting new team members understand what they're looking at without asking questions. After day 90, you've built pipelines; today we ensure they land data where everyone expects it.

A well-organized data lake is the difference between a system that scales gracefully and one that becomes a maze of mysterious folders. We'll walk through the patterns we've seen work in production, from layer separation through partition strategy, and show you exactly how to implement them.

Layer, domain, table, and partition paths

The most effective data lakes we've managed follow a four-level hierarchy: layer (raw, bronze, silver, gold), domain (payments, users, orders), table (transactions, events, snapshots), and partition (date, hour, tenant). This structure lets you answer "where is this data?" immediately and enforce different policies at each level.

At layer zero, you have raw ingestion. This is where data lands exactly as it comes from the source system—no transformations, no deduplication. For a ride-hailing platform, that might be GPS pings from the mobile app:

s3://data-lake/raw/location/gps_pings/year=2024/month=01/day=15/hour=14/

The next layer, often called bronze, performs basic cleaning and schema validation. Data is still close to source but organized and deduplicated:

s3://data-lake/bronze/location/gps_pings/year=2024/month=01/day=15/hour=14/

Silver or curated layer is where business logic lives. Here you join reference data, compute derived fields, and enforce data quality rules. This is what analysts query:

s3://data-lake/silver/rides/trip_events/year=2024/month=01/day=15/hour=14/

Gold layer serves specific use cases: reporting, ML feature stores, dashboards. Each gold table is purpose-built for a consumer:

s3://data-lake/gold/analytics/daily_active_riders/year=2024/month=01/day=15/

The partition strategy at the end deserves attention. Time-based partitioning (year, month, day, hour) is standard because most queries filter by recency. But for multi-tenant systems, you might partition by tenant_id first, then date. The key: partition columns must match your query patterns, not just look logical.

Partitioning is not decoration. Spark's query planner uses partitions to prune which files it reads. Bad partitioning means full table scans on multi-terabyte tables.

Consistent naming conventions

Consistency in naming prevents the subtle bugs that take hours to debug. We've seen teams mix snake_case, camelCase, and UPPERCASE in the same data lake. Six months later, nobody knows whether the payments domain table is payments_transactions, PaymentsTransactions, or payments_Transactions.

Here's what works in practice: lowercase with underscores for everything. Domain names, table names, column names, partition keys. It's readable, works across all systems (SQL databases are case-insensitive on names but case-sensitive on data), and is enforced by linters easily.

# Good
s3://data-lake/silver/payments/transaction_events/year=2024/month=01/day=15/
# Table: transaction_events
# Columns: transaction_id, user_id, amount_cents, created_at_utc

# Bad (inconsistent)
s3://data-lake/silver/Payments/TransactionEvents/year=2024/month=01/day=15/

Beyond naming style, establish clear abbreviation rules. If you abbreviate, document it. Does amt stand for amount or something else? In a trading platform with thousands of tables, ambiguity compounds. We maintain a data dictionary that lists every domain and its standard abbreviations.

Version your schemas by date if they change significantly. A table that evolved over time might have transaction_events_v1 and transaction_events_v2 in different folders, or you version at a partition boundary. This prevents downstream jobs from breaking when a column is dropped.

Separate raw and curated buckets

Mixing raw and curated data in the same bucket introduces risk. A data engineer deletes a folder thinking it's old staging data; turns out it fed three reports nobody documented. Or compliance requires different retention policies: raw data kept for 7 years, aggregated data for 2 years. You can't enforce that cleanly if they share a bucket.

We recommend at least two buckets for most platforms:

• Raw ingestion bucket: s3://data-lake-raw. Only ETL write access, read for authorized engineers. Immutable where possible. Versioning enabled.
• Curated bucket: s3://data-lake. Data science, analytics, and reporting read from here. Write access controlled per domain.

Larger organizations add a third: a sandbox or development bucket where engineers experiment without affecting production tables.

Separation also simplifies cost allocation. Your raw bucket will fill with event data at constant rate. Your curated bucket grows slowly as you add new derived tables. Different teams can own different buckets and track spend independently.

From Spark's perspective, the code changes only in configuration:

# Reading raw (immutable, append-only)
raw_df = spark.read.parquet("s3://data-lake-raw/location/gps_pings/year=2024/month=01/")

# Writing curated (potentially overwritten during recalculation)
curated_df.write.mode("overwrite").parquet("s3://data-lake/silver/rides/trip_events/year=2024/month=01/")

One practical detail: in the raw bucket, enable S3 object lock or versioning if your compliance requirements demand it. In the curated bucket, you can be more aggressive with cleanup policies since you can always recalculate from raw.

Document conventions

The best folder structure fails if new engineers don't know why it exists. We've seen teams rebuild the entire data lake because a new technical lead arrived, didn't understand the convention, and assumed it was cargo-cult design. Documentation prevents this.

At minimum, maintain a README at the root of your data lake bucket:

# Data Lake Structure

## Overview
This data lake organizes data in four layers: raw, bronze, silver, gold.

## Layers

- **raw**: Ingested data, no transformations. Immutable. Retention: 7 years.
- **bronze**: Deduplicated, schema-validated. Retention: 5 years.
- **silver**: Business logic applied, joined with references. Retention: 2 years.
- **gold**: Purpose-built tables for specific consumers. Retention: 90 days unless pinned.

## Domains

Currently: `payments`, `location`, `users`, `supply`.

New domains require approval from the data governance committee.

## Partitioning

Most tables partition by: year, month, day, hour.
Multi-tenant tables partition by: tenant_id, year, month, day.

See PARTITIONING.md for exceptions.

## Access

- raw bucket (data-lake-raw): Data ingestion team + senior engineers
- curated bucket (data-lake): All engineers read, domain teams write

Additionally, document each domain and major table. In e-commerce, the orders domain might have a TABLE_CATALOG.md listing order_events, order_items, order_fulfillment, and their freshness guarantees:

This is especially valuable when you have 50+ tables. New hires can find what they need. Data consumers know freshness guarantees. Retention policies are explicit.

Common mistakes

Partitioning on too many columns: You see .../year=2024/month=01/day=15/hour=14/tenant_id=123/source=api/version=2/. This creates thousands of small files. Hive pruning still works, but Spark struggles with too many metadata calls. Partition depth beyond 4 levels usually signals wrong design.

Mixing operational and analytical data: Trying to serve both OLTP-style point lookups and OLAP-style aggregations from the same table layout causes pain. Use separate tables: one optimized for row retrieval, one for columnar analytics.

No deletion policy: "We'll keep everything forever for compliance." Then three years later, you're paying $50k/month in storage and nobody remembers why. Define retention policies per layer upfront and automate deletion with S3 lifecycle rules.

Forgetting about file count: Writing one row per file happens surprisingly often with distributed systems. Thousands of 100-byte files bloat S3 listings and slow down Spark. Use compaction or repartitioning periodically.

Schema evolution without versioning: You add a column to a silver table. Existing Spark jobs fail because they don't expect it. Document when schema changes happen and version accordingly.

Key takeaways

• Use a four-layer hierarchy (raw, bronze/silver, gold) with consistent domain and table naming under each.
• Partition tables by query patterns, not arbitrary logic. Time-based partitioning (year/month/day) works for most streaming use cases.
• Separate raw and curated data into different buckets to enforce access control and retention policies independently.
• Document your convention in a root README and maintain a table catalog so new team members understand data ownership and freshness.
• Avoid over-partitioning, enforce lowercase snake_case naming, and automate cleanup with lifecycle policies.
• Treat folder structure as infrastructure. Changes should go through review and require documentation updates.