Data streams: concepts, trade-offs and production lessons

Data streams: concepts, trade-offs and production lessons

If you're managing logs, metrics, or any other time-series data at scale, you've probably felt the pain of index proliferation. Elasticsearch data streams solve a real problem: they abstract away the complexity of managing rolling indexes while giving you a clean, append-only interface. In this article, we'll explore data streams, their architecture, trade-offs, and lessons we've learned shipping them to production.

What are data streams?

A data stream is Elasticsearch's abstraction layer for managing append-only, time-series data. Instead of writing directly to named indexes, you write to a data stream, which automatically manages underlying hidden indexes behind the scenes. Think of it as a managed queue where Elasticsearch handles the boring operational work—index creation, rollover timing, retention policies—while you focus on ingesting data.

Data streams are purpose-built for scenarios where you're continuously appending new records and rarely (or never) modifying old ones. Logs, metrics, traces, events—these are the canonical use cases. The benefit isn't just convenience; it's consistency. When your whole team writes to the same abstraction, you eliminate configuration drift and accidental index name collisions.

Backed by hidden indexes

Here's where data streams get interesting architecturally. When you create a data stream, Elasticsearch creates a hidden backing index automatically. As your stream grows, new backing indexes are created on a schedule you define. These are actual indexes—they follow all the normal Elasticsearch rules about sharding, replication, and segment merging—but they're hidden from most API calls.

Let's say you create a data stream called logs-application-prod. Behind the scenes, Elasticsearch generates backing indexes with names like:

• .ds-logs-application-prod-2024.01.15-000001
• .ds-logs-application-prod-2024.01.16-000002
• .ds-logs-application-prod-2024.01.17-000003

The .ds- prefix marks them as data stream backing indexes. The date component and generation number let Elasticsearch manage rollover deterministically. When you query the stream, you're querying all backing indexes transparently.

This hidden-index pattern is powerful because it lets you:

• Apply different index settings (refresh intervals, replicas, retention) to older vs. newer data
• Move old indexes to cheaper storage tiers
• Delete historical data at the index level without touching the stream definition

But there's a mental model cost. You need to understand that the stream is an abstraction over multiple indexes. Tools and monitoring that enumerate indexes will show you the backing indexes, not the stream name. Dashboards, alerts, and automation scripts need to account for this.

Automatic rollover

Data streams implement automatic rollover, a managed policy for rotating from one backing index to the next. By default, rollover happens based on index age or size—whichever condition is met first. You configure this at the index lifecycle policy (ILP) level.

Here's a typical ILP for a logs data stream:

{
  "policy": "logs-policy",
  "phases": {
    "hot": {
      "min_age": "0ms",
      "actions": {
        "rollover": {
          "max_age": "1d",
          "max_size": "100gb",
          "max_docs": 10000000
        },
        "set_priority": {
          "priority": 100
        }
      }
    },
    "warm": {
      "min_age": "7d",
      "actions": {
        "set_priority": {
          "priority": 50
        },
        "forcemerge": {
          "max_num_segments": 1
        }
      }
    },
    "cold": {
      "min_age": "30d",
      "actions": {
        "set_priority": {
          "priority": 0
        },
        "searchable_snapshot": {
          "snapshot_repository": "s3-archive"
        }
      }
    },
    "delete": {
      "min_age": "90d",
      "actions": {
        "delete": {}
      }
    }
  }
}

With this policy, Elasticsearch automatically:

1. Keeps the current backing index in the hot phase, rolling over daily or at 100GB—whichever comes first
2. Moves indexes older than 7 days to warm and force-merges them (fewer, larger segments mean cheaper searches)
3. Moves indexes older than 30 days to cold, converting them to searchable snapshots (compressed, cloud-native storage)
4. Deletes everything older than 90 days

The beauty of rollover is that you set it once and forget it. No cron jobs, no manual index creation, no naming-scheme bugs. Elasticsearch handles the operational mechanics. In production, we've found this reduces operational toil significantly compared to managing indexes manually.

That said, rollover timing can be tricky. If you set max_age too aggressive, you'll create many small indexes, fragmenting your data and increasing merge overhead. If you set it too permissive, individual indexes balloon and shard recovery takes forever. We typically aim for indexes 50–100GB each and adjust based on your ingest rate and query patterns.

Great for logs and metrics

Data streams shine for logs and metrics because these workloads have predictable characteristics: high write volume, append-only access, time-range queries, and clear retention policies.

Consider a typical e-commerce platform logging checkout events. You're ingesting hundreds of thousands of events per minute, all timestamped, all written once and never modified. You want to search recent events quickly, retain medium-term data for analytics, and eventually delete very old data to save costs.

With a data stream, here's the flow:

{
  "index": "logs-checkout-prod",
  "body": {
    "@timestamp": "2024-01-17T14:23:45.123Z",
    "user_id": "usr_8f9a2c",
    "order_amount": 149.99,
    "payment_status": "success",
    "latency_ms": 342
  }
}

Your application writes to logs-checkout-prod without caring about index names. Elasticsearch routes it to the current hot backing index. As that index reaches 100GB, rollover happens automatically. A day later, it rolls to warm tier. After 30 days, it's cold. After 90 days, it's gone.

Querying is just as simple:

{
  "index": "logs-checkout-prod",
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "@timestamp": {
              "gte": "now-24h"
            }
          }
        },
        {
          "term": {
            "payment_status": "success"
          }
        }
      ]
    }
  },
  "aggs": {
    "total_revenue": {
      "sum": {
        "field": "order_amount"
      }
    }
  }
}

Elasticsearch searches all backing indexes and aggregates results transparently. No index name construction, no manual range queries across multiple index names.

Metrics are similar. You write time-series metrics (CPU, memory, latency percentiles) to a data stream, and Elasticsearch handles storage and lifecycle automatically. This pattern has become standard in observability platforms; vendors like Datadog and New Relic use similar abstractions internally.

Use @timestamp

The @timestamp field is not optional in data streams—it's mandatory and central to how Elasticsearch manages them. Every document in a data stream must have a @timestamp field (a date type), and it must reflect the actual time the event occurred or was recorded.

Elasticsearch uses @timestamp to:

• Route documents to the correct backing index
• Enforce immutability constraints (updates to old documents are rejected)
• Determine lifecycle phase transitions
• Optimize time-range queries

A few practical rules we've learned:

Set @timestamp server-side if possible. If your log sources have clock skew or are offline temporarily, ingestion systems should normalize timestamps. We usually set it to the current server time on arrival, not the client timestamp. If you need the original event time, store it in a separate field.

Use ISO 8601 format. Elasticsearch accepts multiple formats, but ISO 8601 (with or without milliseconds) is the safest:

"@timestamp": "2024-01-17T14:23:45.123Z"

Never backfill old timestamps into new data streams. Data streams enforce a rule: you can't insert a document with a @timestamp older than the stream's write index. If you need to ingest historical data, either create a separate index or disable the restriction temporarily (not recommended for production).

Understand retention and @timestamp boundaries. If your ILP deletes data older than 90 days, documents with @timestamp older than 90 days ago will be in a deleted index. Queries won't return them. This is usually what you want, but it surprises teams occasionally.

Common mistakes

Mistake 1: Ignoring index lifecycle policies. We've seen teams create data streams with the default ILP (if any), which often isn't tuned for their workload. Resulting indexes are too small (thousands of tiny shards) or too large (long recovery times). Always audit your rollover settings and adjust based on ingest rate and query requirements.

Mistake 2: Not understanding the hidden-index model. Teams write automation that enumerates indexes and gets confused by the .ds- prefixed backing indexes. Monitoring tools, retention scripts, and dashboard logic need to be aware that data streams use hidden indexes. Use the data stream name in APIs, not individual index names.

Mistake 3: Treating @timestamp as optional or "just another field." Data stream constraints are strict about @timestamp presence and ordering. Missing it, or storing client time when server time is available, breaks assumptions. Always standardize on a canonical timestamp representation.

Mistake 4: Creating too many data streams. We've seen organizations create a stream for every application, environment, and log level combination. This leads to thousands of small backing indexes and operational complexity. Instead, use naming conventions and filtering: logs-{app}-{env} with level as a field, not the index name.

Mistake 5: Not testing ILP transitions. Force-merge, snapshot, and delete phases happen automatically. If you haven't tested them in staging, production surprises will follow. Simulate load, trigger rollover, and verify that warm and cold tier actions complete without errors.

Key takeaways

• Data streams are append-only abstractions over hidden indexes, ideal for logs, metrics, and time-series data.
• Automatic rollover and lifecycle policies eliminate manual index management, reducing operational overhead.
• The hidden-index model is powerful but requires understanding; tooling and automation must account for backing-index naming conventions.
• @timestamp is mandatory, not optional; standardize its format and source to avoid clock-skew issues and lifecycle misalignment.
• Index lifecycle policies need tuning; generic defaults rarely match production workloads. Monitor rollover frequency and adjust max_age and max_size based on ingest rates.
• Name data streams at a sensible granularity (logs-{app}-{env}, not one per log level); use fields for fine-grained filtering instead.
• Test ILP transitions in staging, particularly force-merge, snapshot, and delete phases, before production deployment.