Building a real-time dashboard with MongoDB: a deep dive

Building a real-time dashboard with MongoDB: a deep dive

When you need to show your team live metrics—order volume, user signups, error rates—a traditional dashboard that refreshes every 30 seconds feels stale. Building a real-time dashboard with MongoDB means leveraging change streams to push updates instantly to your frontend, eliminating polling overhead and keeping your stakeholders seeing the truth as it happens. This article walks through a production-grade approach we've used across e-commerce and SaaS platforms: store events cleanly, precompute aggregates efficiently, stream changes via WebSocket, and display updates without a page refresh.

Store events—insert events into a collection

The foundation of any real-time dashboard is reliable event capture. Rather than updating a single "metrics" document, we insert one document per event. This append-only log gives us audit trails, replay capability, and the ability to compute aggregates multiple ways.

For an e-commerce order dashboard, we might insert an event like this:

const ordersCollection = db.collection('order_events');

async function logOrderEvent(order) {
  await ordersCollection.insertOne({
    _id: ObjectId(),
    type: 'order_created',
    orderId: order._id,
    amount: order.total,
    currency: 'USD',
    customerId: order.customerId,
    region: order.shippingAddress.country,
    timestamp: new Date(),
    status: 'completed'
  });
}

// Called immediately when order is placed
await logOrderEvent(newOrder);

This design has several advantages. First, each event is immutable once written. Second, we can insert hundreds or thousands per second without contention—MongoDB will shard by _id by default, spreading the write load. Third, if we need to recompute dashboard metrics, the raw events are always there.

Index the fields you'll query frequently:

await ordersCollection.createIndex({ timestamp: 1, type: 1 });
await ordersCollection.createIndex({ region: 1, timestamp: 1 });
await ordersCollection.createIndex({ customerId: 1, timestamp: 1 });

Aggregate—precompute counters per minute

Querying millions of raw events on every dashboard refresh is expensive. Instead, we precompute rollups: counts, sums, and percentiles bucketed by time. MongoDB's aggregation pipeline is ideal for this, and we run it on a schedule—say, every minute—to build summary documents.

Here's a pipeline that groups order events by region and minute:

const metricsCollection = db.collection('order_metrics_1m');

async function rollupMetrics() {
  const result = await ordersCollection.aggregate([
    {
      $match: {
        timestamp: {
          $gte: new Date(Date.now() - 2 * 60 * 1000), // last 2 minutes
          $lte: new Date()
        }
      }
    },
    {
      $group: {
        _id: {
          region: '$region',
          minute: {
            $dateToString: { format: '%Y-%m-%d %H:%M:00Z', date: '$timestamp' }
          }
        },
        totalOrders: { $sum: 1 },
        totalRevenue: { $sum: '$amount' },
        avgOrderValue: { $avg: '$amount' }
      }
    },
    {
      $out: 'order_metrics_1m'
    }
  ]).toArray();
}

// Run every minute via a scheduled job (e.g., node-cron)
cron.schedule('* * * * *', rollupMetrics);

The $out stage writes results to order_metrics_1m, replacing the collection each time. For a dashboard that needs sub-minute granularity, you might also keep a order_metrics_raw collection with events from just the last few minutes, then query aggregates for older time windows.

Store the minute boundary in a queryable format:

// Better: store as epoch seconds for easier range queries
{
  _id: {
    region: 'US',
    bucketSeconds: 1700000160 // epoch seconds rounded to minute
  },
  totalOrders: 42,
  totalRevenue: 5280,
  avgOrderValue: 125.71
}

Stream—change streams push updates to a WebSocket server

MongoDB change streams are a game-changer for dashboards. A change stream watches a collection and emits events whenever documents are inserted, updated, or deleted. We use this to push updates to connected clients instantly, with no polling.

Here's a NestJS service that listens for metric changes and broadcasts to all connected WebSocket clients:

import { Injectable, OnModuleInit } from '@nestjs/common';
import { InjectConnection } from '@nestjs/mongoose';
import { Connection } from 'mongoose';
import { WebSocketGateway, WebSocketServer } from '@nestjs/websockets';
import { Server } from 'socket.io';

@WebSocketGateway({ cors: true })
@Injectable()
export class MetricsGateway implements OnModuleInit {
  @WebSocketServer()
  server: Server;

  constructor(@InjectConnection() private connection: Connection) {}

  async onModuleInit() {
    const metricsCollection = this.connection
      .collection('order_metrics_1m');

    const changeStream = metricsCollection.watch([
      {
        $match: {
          operationType: { $in: ['insert', 'update'] }
        }
      }
    ]);

    changeStream.on('change', (change) => {
      const metric = change.fullDocument;
      
      // Broadcast to all connected clients
      this.server.emit('metric_update', {
        region: metric._id.region,
        bucket: metric._id.bucketSeconds,
        totalOrders: metric.totalOrders,
        totalRevenue: metric.totalRevenue,
        avgOrderValue: metric.avgOrderValue,
        timestamp: new Date().toISOString()
      });

      console.log(`Metric updated: ${metric._id.region}`);
    });

    changeStream.on('error', (error) => {
      console.error('Change stream error:', error);
      // Reconnect logic here
    });
  }
}

Key points: change streams require a replica set (single-node replica sets are fine for development). The pipeline filters for inserts and updates only—we don't care about deletes. Each client receives the full metric document, so the frontend has all the context it needs to update a chart.

For high-volume scenarios, consider filtering at the stream level to only watch relevant buckets:

const changeStream = metricsCollection.watch([
  {
    $match: {
      operationType: 'insert',
      'fullDocument._id.region': 'US' // only US metrics
    }
  }
]);

Display—the frontend updates charts as messages arrive

On the frontend, we connect to the WebSocket server and update charts in real-time. Using a library like Chart.js or Recharts, we can smoothly animate new data points as they arrive.

Here's a React component with Socket.IO:

import { useEffect, useState } from 'react';
import { io } from 'socket.io-client';
import { LineChart, Line, XAxis, YAxis, CartesianGrid, Tooltip } from 'recharts';

export function MetricsDashboard() {
  const [metrics, setMetrics] = useState([]);
  const [socket, setSocket] = useState(null);

  useEffect(() => {
    const newSocket = io('http://localhost:3001', {
      reconnection: true,
      reconnectionDelay: 1000,
      reconnectionDelayMax: 5000,
      reconnectionAttempts: 5
    });

    newSocket.on('metric_update', (data) => {
      setMetrics((prev) => {
        // Keep last 60 data points (1 hour if 1-minute buckets)
        const updated = [
          ...prev.filter((m) => m.bucket !== data.bucket),
          data
        ].sort((a, b) => a.bucket - b.bucket).slice(-60);

        return updated;
      });
    });

    newSocket.on('disconnect', () => {
      console.warn('Disconnected from metrics server');
    });

    setSocket(newSocket);

    return () => {
      newSocket.disconnect();
    };
  }, []);

  return (
    <div>
      <h1>Order Metrics (Live)</h1>
      
        
         new Date(val * 1000).toLocaleTimeString()}
        />
        
         val.toLocaleString()}
          labelFormatter={(label) => new Date(label * 1000).toLocaleString()}
        />
        
      
    </div>
  );
}

The component connects once on mount, filters duplicate buckets (in case updates arrive out of order), and keeps only the last 60 points in state. Chart animations make the updates feel smooth rather than jarring.

Common mistakes

• Forgetting the replica set. Change streams won't work on standalone MongoDB instances. Enable a single-node replica set locally with rs.initiate().
• Aggregating raw events on every query. If your dashboard needs sub-second responsiveness, you can't afford to run multi-stage pipelines on millions of documents per refresh. Precompute and store results.
• Not filtering change stream output. Broadcasting every single change to every client wastes bandwidth and CPU. Use the aggregation pipeline in the change stream to emit only relevant updates.
• Treating metrics as a cache. If your metrics collection is wiped, you lose the current state. Keep the raw events and rebuild metrics if needed.
• Ignoring reconnection logic. WebSocket connections drop. Your frontend must gracefully reconnect and re-query the current state to stay in sync.
• Mixing high-cardinality dimensions in the same metric. If you group by both region and customerId in a 1-minute bucket, you might create millions of documents. Be selective about dimensions.

Key takeaways

• Append-only event logging provides auditability and replay capability; insert one document per event rather than updating a summary.
• Precompute aggregates on a schedule (e.g., every minute) and store rollups in a separate collection to keep queries fast.
• Use MongoDB change streams to watch the metrics collection and push updates via WebSocket, eliminating polling latency.
• On the frontend, dedup incoming messages by bucket, keep a bounded window of data in state, and animate chart updates smoothly.
• Change streams require a replica set; enable one even for local development with rs.initiate().
• Filter change stream output at the database level with aggregation pipelines to reduce bandwidth and client CPU usage.