Capped collections: a MongoDB & NoSQL guide for developers

Understanding capped collections in MongoDB

When you need to store high-volume, time-series data like application logs or event streams, MongoDB's capped collections offer a pragmatic solution that automatically manages storage without manual cleanup. Unlike standard collections that grow indefinitely, capped collections enforce a fixed size limit, automatically removing the oldest documents when capacity is reached. This guide explores capped collections: a MongoDB & NoSQL guide for developers working with logs, metrics, and other append-only data patterns.

Capped collections are particularly valuable in production systems where you want predictable memory footprints and don't need to retain historical data beyond a certain window. They're designed for write-heavy scenarios where insertion speed matters and you're willing to trade historical completeness for operational simplicity. However, they come with tradeoffs that make them unsuitable for general-purpose storage.

How insertion order is preserved in capped collections

One of the defining characteristics of capped collections is that insertion order is guaranteed. Unlike regular MongoDB collections where order can be unpredictable without an explicit sort, documents in a capped collection maintain their insertion sequence. This property emerges naturally from how capped collections store data: they use a circular buffer structure where new documents are written sequentially to the next available space.

This insertion-order guarantee makes capped collections ideal for event logs and message queues where temporal ordering is semantically meaningful. You don't need a separate timestamp index to preserve causality.

Here's how to create a capped collection:

db.createCollection("application_logs", {
  capped: true,
  size: 104857600,  // 100 MB
  max: 100000       // optional: maximum number of documents
})

The size parameter specifies the collection size in bytes, while max optionally limits the document count. When either limit is reached, MongoDB removes the oldest document before inserting the new one. The order of insertion is always preserved in the collection's internal structure:

// Insert documents in order
db.application_logs.insertMany([
  { timestamp: new Date("2024-01-15T10:00:00Z"), level: "INFO", message: "Server started" },
  { timestamp: new Date("2024-01-15T10:00:05Z"), level: "INFO", message: "Database connected" },
  { timestamp: new Date("2024-01-15T10:00:10Z"), level: "ERROR", message: "Failed to load config" }
])

// Query without sort returns documents in insertion order
db.application_logs.find().limit(2)
// Returns first two documents in the order they were inserted

You can verify insertion order is maintained even after multiple insertions and deletions. This is different from a regular collection where you'd need an explicit sort to guarantee order.

Automatic removal of oldest documents

The automatic eviction of oldest documents is the core mechanism that keeps capped collections bounded. When a new document is inserted and the collection has reached either its size or document count limit, MongoDB automatically deletes the oldest document to make room. This happens atomically during the insert operation, so you never need to implement cleanup logic yourself.

This behavior is particularly valuable in high-throughput logging scenarios. Consider an e-commerce platform logging every transaction:

// Create a capped collection for transaction logs
// Keep only the last 100 MB of transactions
db.createCollection("transaction_logs", {
  capped: true,
  size: 104857600  // 100 MB, roughly 10 million typical transaction documents
})

// Continuous insertion from application
setInterval(() => {
  db.transaction_logs.insertOne({
    _id: ObjectId(),
    timestamp: new Date(),
    user_id: generateUserId(),
    amount: generateAmount(),
    status: "completed"
  })
}, 100)  // Insert every 100ms

Without capped collections, you'd need background jobs to prune old records, which introduces complexity and potential query performance degradation during deletion. With capping, insertion performance remains constant because MongoDB only removes one document per insert when at capacity.

The automatic eviction respects insertion order strictly. You cannot selectively delete documents from the middle of a capped collection—only the oldest document is removed. If you need selective deletion, capped collections are not appropriate for your use case.

Tailable cursors for streaming data

Tailable cursors are a MongoDB feature designed specifically for capped collections. Unlike normal cursors that exhaust when they reach the end of a result set, tailable cursors remain open and continue to return new documents as they're inserted into the capped collection. This enables real-time streaming patterns similar to tail -f on Unix systems.

This is valuable for scenarios like real-time monitoring dashboards, event-driven architectures, or streaming analytics pipelines:

// Create capped collection for real-time events
db.createCollection("events", {
  capped: true,
  size: 52428800  // 50 MB
})

// Server-side: cursor that streams new events
const cursor = db.events.find({}).addOption(DBQuery.Option.tailable).addOption(DBQuery.Option.awaitData)

// Client continuously reads from cursor
cursor.forEach(event => {
  console.log("New event:", event)
  // Process event in real-time
})

In Node.js with the MongoDB driver, tailable cursors work like this:

const { MongoClient } = require('mongodb')

async function watchLogs() {
  const client = new MongoClient('mongodb://localhost:27017')
  const db = client.db('myapp')
  const collection = db.collection('logs')
  
  const cursor = collection.find({}, {
    tailable: true,
    awaitData: true,
    maxAwaitTimeMS: 1000
  })
  
  for await (const doc of cursor) {
    console.log('Log entry:', doc)
    // Real-time processing
  }
}

watchLogs()

Tailable cursors with the awaitData option tell MongoDB to wait up to a timeout period (in milliseconds) for new documents rather than immediately returning an empty batch. This reduces CPU usage in polling scenarios and creates a more efficient streaming pattern.

Be aware that tailable cursors don't survive collection drops or explicit index creation. If the collection is dropped while your cursor is open, you'll need to recreate both the collection and the cursor.

Consider TTL indexes as an alternative

While capped collections are useful, they're not always the best choice for time-series data. TTL (time-to-live) indexes provide an alternative that's often more flexible, though with different tradeoffs.

TTL indexes automatically delete documents after a specified time period. Unlike capped collections, TTL indexes allow selective retention of specific documents and support multiple different retention policies on the same collection:

// Create a TTL index that removes documents 30 days after creation
db.activity_logs.createIndex(
  { createdAt: 1 },
  { expireAfterSeconds: 2592000 }  // 30 days
)

// Insert documents
db.activity_logs.insertOne({
  _id: ObjectId(),
  createdAt: new Date(),
  user_id: "user123",
  action: "login"
})

// After 30 days, MongoDB automatically removes this document

When to use capped collections vs. TTL indexes:

• Use capped collections when: you have insertion rates high enough to fill a fixed size regularly, you need guaranteed insertion order, you'll use tailable cursors for streaming, and you want the fastest possible write performance with zero cleanup lag.
• Use TTL indexes when: retention periods vary, you want to keep some documents longer than others, you need flexibility in querying by creation time, or you want simpler operational management without pre-sizing a collection.

TTL indexes use a background thread that runs every 60 seconds (configurable) to find and remove expired documents. This means there's a delay between expiration and actual removal, whereas capped collections remove documents synchronously during insertion. For most applications, TTL's slight delay is acceptable and worth the operational simplicity.

TTL indexes also support any field with a Date type, whereas capped collections enforce circular buffer semantics. This flexibility often makes TTL indexes the better default choice for new projects unless you have specific requirements for tailable cursors or guaranteed constant-size behavior.

Common mistakes

Mistake 1: Creating capped collections without a size estimate. If you set the size too small, legitimate recent data gets evicted too quickly. Set it too large, and you're wasting disk space. Measure your typical document size and insertion rate, then set a size that gives you the retention window you need. For example: if your average log entry is 500 bytes and you insert 100 entries per second, you need roughly 50 MB per minute of retention.

Mistake 2: Updating documents in capped collections. Capped collections don't support updates that increase document size (because there's no space to shuffle documents in a circular buffer). Avoid updateOne or updateMany` on capped collections altogether. If you need to modify data, consider TTL indexes or a regular collection instead.

Mistake 3: Deleting specific documents mid-collection. You can delete documents from capped collections, but this breaks the efficiency promise. MongoDB must reorganize the circular buffer, which defeats the purpose. If you need selective deletion, use a regular collection with TTL indexes or manual cleanup.

Mistake 4: Expecting tailable cursors to survive collection recreation. If you drop and recreate the capped collection, all existing tailable cursors become invalid. Plan cursor lifecycle and error handling accordingly in production applications.

Mistake 5: Using capped collections for general application data. Capped collections are not suitable for user profiles, settings, or anything where you need historical data. They're specialized for high-volume, time-series, append-only workloads. Reach for them intentionally, not as a default.

Key takeaways

• Capped collections are fixed-size circular buffers that automatically remove the oldest documents when capacity is reached, making them ideal for high-volume logs and event streams.
• Insertion order is guaranteed and preserved without needing explicit indexes, a key advantage for temporal data.
• Tailable cursors enable real-time streaming patterns on capped collections, useful for monitoring and event-driven architectures.
• TTL indexes often provide more operational flexibility than capped collections and should be your default choice unless you specifically need constant-size behavior or tailable cursors.
• Never update or selectively delete from capped collections—they're optimized for append-only workloads; if you need modifications, use a regular collection.
• Size capped collections carefully based on document size and insertion rate to ensure your desired retention window is maintained.
• Capped collections are a specialized tool; use them intentionally for their specific strengths rather than as a general-purpose collection type.