Data archiving strategies for MongoDB & NoSQL developers

Data archiving strategies for MongoDB & NoSQL developers

As your MongoDB collections grow over months and years, query performance degrades and storage costs climb. Data archiving strategies for MongoDB & NoSQL developers become essential when you're managing terabytes of data where only a fraction is actively queried. We'll walk through proven techniques to keep your active collections lean while maintaining access to historical data when you need it.

The problem with unbounded growth

Consider a ride-hailing platform storing trip records. After three years, your trips collection has 500 million documents. Recent queries pull from the last 30 days, but the full collection scan still touches every document. Index sizes bloat, working set grows beyond RAM, and your storage bill reaches six figures annually.

This is where archiving enters the picture. By moving old data out of your active collection, you reduce index overhead, improve cache hit rates, and keep query latency predictable. You're not deleting data—you're reorganizing it strategically.

Moving old data to archive collections

The simplest archiving pattern is a scheduled job that moves stale records into a separate collection or database. This works well for data you'll occasionally query but rarely access in your hot path.

The approach: Each night, a batch process finds documents older than your retention threshold and copies them to an archive collection, then deletes them from the active collection.

// Archive trips older than 12 months
const archiveOldTrips = async (client) => {
  const db = client.db('rideshare');
  const cutoffDate = new Date();
  cutoffDate.setFullYear(cutoffDate.getFullYear() - 1);

  // Copy documents to archive
  const trips = db.collection('trips');
  const archive = db.collection('trips_archive_2024');

  const oldTrips = await trips
    .find({ createdAt: { $lt: cutoffDate } })
    .toArray();

  if (oldTrips.length > 0) {
    await archive.insertMany(oldTrips);
    
    // Delete from active collection
    await trips.deleteMany({ createdAt: { $lt: cutoffDate } });
    
    console.log(`Archived ${oldTrips.length} trips`);
  }
};

In production, we've seen teams run this weekly or monthly depending on their data volume. The key is batching—don't try to move 10 million documents in one go. Process in chunks of 100,000 or 1 million at a time with progress logging.

Archive collection strategy: Name them by date range (trips_archive_2024_q1, trips_archive_2024_q2) so you know exactly when data lives. This also lets you drop entire quarters when you're ready to purge, rather than hunting through a monolithic archive.

For e-commerce platforms, you might archive orders older than 24 months to a separate collection while keeping current and recent orders indexed heavily. Shipping teams can still query full order history, but it doesn't slow down the hot checkout and inventory paths.

Using MongoDB Atlas Online Archive

If you're running on MongoDB Atlas, the Online Archive feature automates this process entirely. It's transparent—you query through your primary collection, and Atlas automatically routes older documents to S3 without code changes.

How it works: You define an archival policy based on a date field. Any document older than your threshold is moved to tiered storage. Queries still work, but archived data is accessed from cheaper S3 rather than hot MongoDB nodes.

We've used Online Archive for SaaS platforms with years of audit logs or transaction history. Finance teams rarely need to drill into Q1 data, but when they do, the query completes in seconds from S3. The monthly storage savings cover the retrieval latency.

Gotcha: Online Archive respects your indexes—if an archived document would match an indexed query, Atlas must still check S3. So if you query on a field that spans both active and archived data, you'll see a performance step. Design your queries to be conscious of the archival boundary.

TTL indexes for data you can delete

Not all old data needs to be archived. Some is safe to delete outright—logs, session tokens, temporary calculations, analytics snapshots older than 90 days.

TTL (Time-To-Live) indexes automatically delete documents after a specified duration. MongoDB runs a background thread that scans and removes expired documents, requiring almost zero application logic.

// Create a TTL index: documents expire 7 days after createdAt
db.logs.createIndex({ createdAt: 1 }, { expireAfterSeconds: 604800 });

// Or 30 days for session data
db.sessions.createIndex({ lastActivity: 1 }, { expireAfterSeconds: 2592000 });

// Insert a log—it will auto-delete in 7 days
db.logs.insertOne({
  _id: ObjectId(),
  level: 'info',
  message: 'User login',
  createdAt: new Date()
});

Real scenario: A trading platform stores order fills with microsecond precision. After 30 days, they don't need the raw fill data in MongoDB—summaries live in a data warehouse. A TTL index on the fills collection keeps MongoDB lean, and a nightly job publishes to the warehouse before expiry.

TTL best practices: The background deletion task runs every 60 seconds by default, so don't expect instant removal—documents may linger a minute or two. Also, TTL deletion doesn't free disk space immediately on the filesystem; run a compact operation if you need to reclaim storage urgently.

Query archives only when needed

Once you've archived data, your application needs to know when to look there. This is where query logic becomes critical—you don't want every search scanning both active and archive collections.

Pattern: time-aware queries

// User asks for orders from the past 2 years
const getOrderHistory = async (userId, days = 730) => {
  const db = client.db('ecommerce');
  const activeOrders = db.collection('orders');
  const archiveOrders = db.collection('orders_archive');

  const cutoffDate = new Date();
  cutoffDate.setDate(cutoffDate.getDate() - days);

  // Query active collection (should hit index)
  const recent = await activeOrders
    .find({
      userId,
      createdAt: { $gte: cutoffDate }
    })
    .toArray();

  // Only query archive if cutoff is older than your active retention
  const archiveThreshold = new Date();
  archiveThreshold.setMonth(archiveThreshold.getMonth() - 12);

  let archived = [];
  if (cutoffDate < archiveThreshold) {
    archived = await archiveOrders
      .find({
        userId,
        createdAt: { $gte: cutoffDate, $lt: archiveThreshold }
      })
      .toArray();
  }

  return [...recent, ...archived].sort((a, b) => b.createdAt - a.createdAt);
};

The application decides: is this a recent query that hits the active collection, or a historical query that needs the archive? This logic often lives in your data access layer or ORM.

CRM example: When a customer views their 5-year communication history, your API queries the active messages collection (last 6 months), then fetches from messages_archive_2024, messages_archive_2023, etc. The user sees a unified timeline, but hot queries never touch old data.

Common mistakes

Archiving without an index on the cutoff field: If you're moving documents where createdAt < cutoffDate but createdAt isn't indexed, the scan takes hours. Always index the field you'll use to identify candidates.

Forgetting to rebuild indexes after archiving: After removing millions of documents, your index might be fragmented. Run reIndex() or create fresh indexes post-archive on large collections.

Archiving without a backup: Before running your first archive job on production, take a snapshot. A logic bug can lose data quickly. Modern teams use point-in-time recovery on Atlas or filesystem snapshots on self-managed MongoDB.

Setting TTL expiry too aggressively: A developer sets expireAfterSeconds: 86400 expecting 1-day retention, but the background task runs late and data vanishes mid-day. Add buffer—if you need 24 hours, set 25-26 hours and clean up with explicit deletes during low traffic.

Querying archives in the critical path: Every request checking a slow S3-backed archive drags down response time. Archive queries should be background jobs, admin dashboards, or on-demand analysis—not your checkout flow.

Key takeaways

• Archive old data to separate collections when you need occasional access but want to keep your active collection small and fast.
• Use MongoDB Atlas Online Archive for hands-off tiering to S3 when you have budget for managed storage.
• Apply TTL indexes to data that's safe to delete automatically—logs, sessions, temporary data.
• Design queries to be archive-aware: check if you need historical data before querying slow collections.
• Always index the field you'll use to identify archival candidates, and monitor TTL deletion timing in production.
• Archiving buys you performance and cost efficiency, but requires disciplined query logic to stay effective.