Time series collections: real-world lessons and examples

Time series collections: real-world lessons and examples

If you're collecting metrics, events, or sensor data at scale, you've probably noticed that MongoDB's standard collections become bloated and slow after a few million documents. Time series collections solve this problem by handling the storage, compression, and querying of temporal data in a way that's fundamentally different from how we think about regular document storage. This article walks through how to build and query time series collections effectively, drawing from real production scenarios we've encountered across monitoring systems, payment events, and IoT platforms.

What makes time series collections different

Time series collections in MongoDB (introduced in version 5.0) are optimized for insert-heavy workloads with minimal updates or deletes. They use a technique called automatic bucketing: MongoDB internally groups measurements into compressed documents, reducing storage overhead by 70–90% compared to storing each measurement individually. This means you can store years of fine-grained metrics without the cost of traditional collections.

The key difference is structural. A standard collection might store one document per event:

// Regular collection: one doc per measurement
db.metrics.insertOne({
  _id: ObjectId(),
  timestamp: ISODate("2024-01-15T10:30:45Z"),
  host: "server-01",
  cpu_usage: 65.2,
  memory_mb: 4096,
  inserted_at: ISODate("2024-01-15T10:30:45Z")
});

A time series collection abstracts this away. You define the shape of your measurements at creation time, and MongoDB handles the bucketing:

// Time series collection: automatic bucketing
db.createCollection("metrics", {
  timeseries: {
    timeField: "timestamp",
    metaField: "metadata",
    granularity: "minutes"
  }
});

db.metrics.insertOne({
  timestamp: ISODate("2024-01-15T10:30:45Z"),
  metadata: { host: "server-01", region: "us-east-1" },
  cpu_usage: 65.2,
  memory_mb: 4096
});

The second approach uses roughly 80% less storage for the same data.

Create — Specify timeField and metaField

When you create a time series collection, you must declare two mandatory parameters:

timeField: The field that contains the timestamp. Every document must have this field, and it must be an ISODate or numeric value representing milliseconds since epoch.

metaField: An optional field containing metadata that changes infrequently or identifies the measurement source (host, sensor ID, user ID, etc.). Metadata is stored separately and referenced by each bucket, which saves space when you have repeated metadata across many measurements.

There's also granularity, which hints to MongoDB how frequently measurements arrive. Valid options are seconds, minutes, or hours. MongoDB uses this to size buckets appropriately—granularity of "seconds" creates tighter buckets than "minutes".

Here's a practical example from an e-commerce platform tracking checkout events:

db.createCollection("checkout_events", {
  timeseries: {
    timeField: "event_time",
    metaField: "session_info",
    granularity: "seconds"
  }
});

// Insert checkout events
db.checkout_events.insertOne({
  event_time: ISODate("2024-01-15T14:22:10Z"),
  session_info: {
    session_id: "sess_789xyz",
    user_id: 5432,
    region: "eu-west-1"
  },
  cart_value: 189.99,
  items_count: 3,
  payment_method: "card"
});

For real-time monitoring systems, we typically use "minutes" granularity to balance bucket size with query flexibility. For high-frequency sensor data (IoT, stock prices), "seconds" makes sense.

Automatic bucketing — MongoDB groups measurements internally

Once you insert data, MongoDB automatically groups measurements into time-ordered buckets. Each bucket is a compressed document stored in the system buckets collection. You never interact with these directly; they're managed transparently.

The bucketing strategy depends on granularity and the volume of data arriving:

• Buckets have a maximum time span (up to 1 hour for "minutes" granularity).
• Buckets have a maximum size limit (around 125 KB after compression).
• If a bucket is full or the time window has passed, a new bucket is created.

This is where storage efficiency comes from. Instead of storing metadata with every measurement, MongoDB stores it once per bucket. For a bucket containing 10,000 sensor readings from a single host, the metadata is stored once, not 10,000 times.

Let's say we're collecting CPU metrics every 5 seconds from 100 servers. With a regular collection, we'd insert 1.44 million documents per day. With a time series collection using "seconds" granularity:

// Regular collection approach: 1.44M docs/day
for (let i = 0; i < 86400; i += 5) {
  db.metrics_regular.insertOne({
    _id: ObjectId(),
    timestamp: new Date(Date.now() - 86400000 + i * 1000),
    host_id: `host-${Math.floor(Math.random() * 100)}`,
    host_ip: "10.0.1.X",
    region: "us-east-1",
    cpu_usage: Math.random() * 100,
    memory_mb: Math.random() * 65536,
    disk_io_mb: Math.random() * 500,
    network_mbps: Math.random() * 1000
  });
}

// Time series approach: automatic bucketing
for (let i = 0; i < 86400; i += 5) {
  db.metrics_timeseries.insertOne({
    timestamp: new Date(Date.now() - 86400000 + i * 1000),
    metadata: {
      host_id: `host-${Math.floor(Math.random() * 100)}`,
      host_ip: "10.0.1.X",
      region: "us-east-1"
    },
    cpu_usage: Math.random() * 100,
    memory_mb: Math.random() * 65536,
    disk_io_mb: Math.random() * 500,
    network_mbps: Math.random() * 1000
  });
}

The time series version compresses to roughly 200–300 MB on disk, while the regular collection uses 1.2–1.5 GB. The difference scales with the number of servers and retention period.

Compression — Much smaller storage than regular collections

MongoDB applies compression at the bucket level using the configured compression algorithm (typically Snappy by default, or zstd if specified during collection creation). Compression is transparent; when you query, MongoDB decompresses automatically.

The real-world impact is significant. Consider a SaaS platform tracking API request metrics:

The storage savings come from multiple sources: metadata deduplication, compression, and MongoDB's internal optimization. Faster queries happen because fewer documents need to be scanned—a single bucket might contain what would be hundreds or thousands of regular documents.

If you're running on AWS or another cloud provider, this translates directly to cost. 2.6 GB versus 200 MB over 30 days means lower storage costs and faster, cheaper backups.

One caution: once inserted, you cannot update or delete individual measurements in a time series collection (by design). If you need to correct data, you must create a new collection and re-insert. This is a trade-off for the storage and query efficiency gains.

Query — Window functions and $dateTrunc for analysis

Querying time series collections is straightforward for simple filters but requires care for analytical queries. MongoDB's aggregation framework is your primary tool.

Basic filtering on timestamp or metadata works as expected:

// Find all metrics for a specific host in the last hour
db.metrics.find({
  timestamp: { $gte: new Date(Date.now() - 3600000) },
  "metadata.host_id": "host-42"
}).limit(100);

For analysis, use aggregation with window functions. Here's a practical example: calculate the 5-minute rolling average of CPU usage for each host:

db.metrics.aggregate([
  {
    $match: {
      timestamp: { $gte: ISODate("2024-01-15T00:00:00Z"), $lt: ISODate("2024-01-16T00:00:00Z") },
      "metadata.host_id": "host-01"
    }
  },
  {
    $group: {
      _id: {
        host_id: "$metadata.host_id",
        bucket_time: {
          $dateTrunc: {
            date: "$timestamp",
            unit: "minute",
            binSize: 5
          }
        }
      },
      avg_cpu: { $avg: "$cpu_usage" },
      max_memory: { $max: "$memory_mb" },
      count: { $sum: 1 }
    }
  },
  {
    $sort: { "_id.bucket_time": 1 }
  }
]);

The $dateTrunc operator groups timestamps into regular intervals (5 minutes in this case), making it easy to compute aggregates across time windows. This is essential for dashboards, alerting, and SLA analysis.

Another common pattern: detect anomalies using window functions. Here, we compare each measurement against the 1-hour rolling average:

db.checkout_events.aggregate([
  {
    $match: {
      event_time: { $gte: ISODate("2024-01-15T00:00:00Z"), $lt: ISODate("2024-01-16T00:00:00Z") }
    }
  },
  {
    $setWindowFields: {
      partitionBy: "$session_info.region",
      sortBy: { event_time: 1 },
      output: {
        rolling_avg_value: {
          $avg: "$cart_value",
          window: {
            range: [-3600, 0],
            unit: "second"
          }
        },
        event_count: {
          $count: {},
          window: {
            range: [-3600, 0],
            unit: "second"
          }
        }
      }
    }
  },
  {
    $addFields: {
      deviation: {
        $subtract: [
          "$cart_value",
          "$rolling_avg_value"
        ]
      }
    }
  },
  {
    $match: {
      deviation: { $gt: 100 }
    }
  }
]);

Window functions are more memory-intensive than group-based aggregations, but they're powerful for comparative analysis. For large datasets, consider filtering by a narrow time range or metadata first.

Common mistakes

Mistake 1: Not declaring metaField or declaring it poorly. If your metadata (host ID, user ID, sensor location) doesn't go into metaField, it gets compressed with every measurement, defeating the purpose. Always move high-cardinality identifiers into metaField.

Mistake 2: Trying to update individual measurements. Time series collections don't support updates by design. If you insert a value with an error, you have to delete the entire bucket or live with the error. Plan for immutability from day one.

Mistake 3: Choosing the wrong granularity. If your data arrives every 5 seconds but you set granularity to "hours," buckets will be too large and updates will cause fragmentation. If your data arrives once per hour but granularity is "seconds," buckets will be inefficiently small. Match granularity to your typical measurement frequency.

Mistake 4: Querying without proper indexes. While time series collections optimize storage, they still need indexes on frequently queried fields. Index timeField and metaField fields that appear in common queries:

db.metrics.createIndex({ "metadata.host_id": 1, timestamp: 1 });
db.metrics.createIndex({ "metadata.region": 1 });

Mistake 5: Retention without a policy. Time series collections can grow indefinitely. Implement TTL (time-to-live) indexes to auto-delete old data:

db.metrics.createIndex(
  { timestamp: 1 },
  { expireAfterSeconds: 7776000 } // 90 days
);

Key takeaways

• Time series collections are purpose-built for append-heavy, time-ordered data. Use them for metrics, events, logs, and sensor data; not for general-purpose documents.
• Define timeField and metaField at creation time. Metadata deduplication is where most of the storage savings come from.
• Automatic bucketing and compression reduce storage by 70–90% compared to regular collections. This saves money and improves query speed.
• Granularity should match your measurement frequency: "seconds" for high-frequency data, "minutes" for typical monitoring, "hours" for sparse events.
• Query with aggregation framework using $dateTrunc and $setWindowFields for analytical workloads. Combine with proper indexes and TTL policies for production reliability.
• Remember that time series collections don't support updates or deletes by measurement. Plan for immutability and consider a separate collection for corrections if needed.
• Start with a small pilot—a single metric or event type—before moving your entire metrics pipeline. The operational model is different, and your team needs to understand the constraints.