Metrics with Prometheus: NestJS best practices explained

Metrics with Prometheus: NestJS best practices explained

Your application is running smoothly, but you won't know when users start experiencing slowdowns until they complain. Logs tell stories; metrics show trends. In this article, we'll walk through instrumenting a NestJS application with Prometheus to capture the signals that matter most to your business and your users.

Why metrics matter differently than logs

Logs capture discrete events—a request came in, a database query executed, an error occurred. Metrics aggregate observations over time: how many requests per second, what percentage failed, how long did they take. Prometheus scrapes these metrics at regular intervals and stores them as time-series data, making it trivial to spot trends, set alerts, and correlate incidents with user impact.

The difference is concrete: when a user reports slow response times at 3 PM, logs might show you thousands of individual requests, but a metric graph showing latency percentiles will tell you exactly when the problem started and whether it's still happening now.

The RED method: Request rate, error rate, latency

RED stands for Request rate, Error rate, and Latency. These three signals capture how users experience your system. If you instrument nothing else, instrument these.

Request rate answers "how busy is my service?" A sudden drop often signals a deployment problem or a client-side outage. Steady growth indicates either more users or a runaway client loop.

Error rate is the percentage of requests that failed. A spike in errors is usually the first warning sign before users notice. Tracking errors by type (4xx vs 5xx, or by exception class) helps you triage quickly.

Latency should be captured as percentiles, not averages. A p99 spike tells you that 1% of users are suffering, even if the mean looks fine. We typically track p50, p95, and p99.

Here's how to instrument these with prom-client in NestJS:

import { Injectable } from '@nestjs/common';
import * as promClient from 'prom-client';

@Injectable()
export class MetricsService {
  private requestCounter: promClient.Counter;
  private requestDuration: promClient.Histogram;
  private errorCounter: promClient.Counter;

  constructor() {
    this.requestCounter = new promClient.Counter({
      name: 'http_requests_total',
      help: 'Total HTTP requests',
      labelNames: ['method', 'route', 'status'],
    });

    this.requestDuration = new promClient.Histogram({
      name: 'http_request_duration_seconds',
      help: 'HTTP request duration in seconds',
      labelNames: ['method', 'route'],
      buckets: [0.01, 0.05, 0.1, 0.5, 1, 2, 5],
    });

    this.errorCounter = new promClient.Counter({
      name: 'http_errors_total',
      help: 'Total HTTP errors',
      labelNames: ['method', 'route', 'error_type'],
    });
  }

  recordRequest(method: string, route: string, status: number, duration: number) {
    this.requestCounter.inc({ method, route, status });
    this.requestDuration.observe({ method, route }, duration);
    
    if (status >= 400) {
      const errorType = status >= 500 ? '5xx' : '4xx';
      this.errorCounter.inc({ method, route, error_type: errorType });
    }
  }
}

Wire this into a middleware or interceptor so every request is tracked. An interceptor is cleaner in NestJS:

import { Injectable, NestInterceptor, ExecutionContext, CallHandler } from '@nestjs/common';
import { Observable } from 'rxjs';
import { tap } from 'rxjs/operators';

@Injectable()
export class MetricsInterceptor implements NestInterceptor {
  constructor(private metricsService: MetricsService) {}

  intercept(context: ExecutionContext, next: CallHandler): Observable {
    const request = context.switchToHttp().getRequest();
    const startTime = Date.now();

    return next.handle().pipe(
      tap(() => {
        const duration = (Date.now() - startTime) / 1000;
        const response = context.switchToHttp().getResponse();
        this.metricsService.recordRequest(
          request.method,
          request.route?.path || request.path,
          response.statusCode,
          duration,
        );
      }),
    );
  }
}

Queue depth and job failure counts

If your application uses background jobs—processing payments, sending emails, generating reports—you need metrics beyond HTTP. Queue depth tells you if jobs are piling up faster than you can process them. Job failure counts reveal systemic problems like a broken third-party API or misconfigured credentials.

In a real scenario: you deploy a change at noon, and suddenly your email queue grows from 10 pending jobs to 50,000. A metric would alert you in seconds; waiting for support tickets means customers notice first.

Here's how to instrument a Bull queue (or any job queue):

import { Injectable } from '@nestjs/common';
import { InjectQueue } from '@nestjs/bull';
import { Queue, Job } from 'bull';
import * as promClient from 'prom-client';

@Injectable()
export class JobMetricsService {
  private queueDepthGauge: promClient.Gauge;
  private jobCompletedCounter: promClient.Counter;
  private jobFailedCounter: promClient.Counter;

  constructor(@InjectQueue('email') private emailQueue: Queue) {
    this.queueDepthGauge = new promClient.Gauge({
      name: 'job_queue_depth',
      help: 'Number of pending jobs in queue',
      labelNames: ['queue_name'],
    });

    this.jobCompletedCounter = new promClient.Counter({
      name: 'jobs_completed_total',
      help: 'Total completed jobs',
      labelNames: ['queue_name'],
    });

    this.jobFailedCounter = new promClient.Counter({
      name: 'jobs_failed_total',
      help: 'Total failed jobs',
      labelNames: ['queue_name', 'error_type'],
    });

    this.setupQueueListeners();
  }

  private setupQueueListeners() {
    this.emailQueue.on('completed', (job: Job) => {
      this.jobCompletedCounter.inc({ queue_name: 'email' });
    });

    this.emailQueue.on('failed', (job: Job, err: Error) => {
      this.jobFailedCounter.inc({
        queue_name: 'email',
        error_type: err.name,
      });
    });
  }

  async updateQueueDepth() {
    const count = await this.emailQueue.count();
    this.queueDepthGauge.set({ queue_name: 'email' }, count);
  }
}

Call updateQueueDepth() periodically (e.g., every 10 seconds via a scheduled task) to keep the gauge current. Pair this with a dashboard showing queue depth over time and an alert firing when depth exceeds a threshold—say, if it stays above 5,000 jobs for more than 5 minutes.

Exposing metrics with prom-client and a /metrics endpoint

Prometheus works by scraping a metrics endpoint at regular intervals (typically every 15–30 seconds). In NestJS, this is straightforward:

import { Controller, Get } from '@nestjs/common';
import * as promClient from 'prom-client';

@Controller('metrics')
export class MetricsController {
  @Get()
  metrics() {
    return promClient.register.metrics();
  }
}

Tell your Prometheus server to scrape this endpoint. A minimal prometheus.yml looks like:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'nestjs-app'
    static_configs:
      - targets: ['localhost:3000']
    metrics_path: '/metrics'

Within minutes, Prometheus will have time-series data. Query it via the Prometheus UI or—better—visualize it in Grafana. A simple Grafana dashboard might show request rate (queries/sec), error rate as a percentage, and latency percentiles side by side. That view makes it obvious when something breaks.

Alerting on symptoms users feel

Metrics are only useful if they drive action. Set alerts that fire when users are likely hurting. Don't alert on every blip; alert on sustained problems.

Good alert rules:

• Error rate above 1% for 2 minutes: Something is wrong, even if most requests succeed.
• p99 latency above 500ms for 5 minutes: The slowest users are experiencing delays; this needs investigation.
• Queue depth above threshold for 10 minutes: Jobs are backing up; scale workers or check for a bottleneck.
• Application restart: Unexpected restarts often precede outages; alert immediately.

Here's a Prometheus alert rule (in YAML):

groups:
  - name: nestjs_alerts
    rules:
      - alert: HighErrorRate
        expr: |
          (
            sum(rate(http_errors_total[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) > 0.01
        for: 2m
        annotations:
          summary: "Error rate above 1%"

      - alert: HighP99Latency
        expr: |
          histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 0.5
        for: 5m
        annotations:
          summary: "p99 latency above 500ms"

      - alert: QueueBacklog
        expr: |
          job_queue_depth{queue_name="email"} > 5000
        for: 10m
        annotations:
          summary: "Email queue depth above 5,000 jobs"

Route these alerts to Slack, PagerDuty, or your alerting system. Make sure the on-call engineer gets paged only when users are actually affected, not for every minor fluctuation.

Common mistakes

Instrumenting too much: Every metric you add increases cardinality and storage cost. Stick to RED metrics plus a few domain-specific ones (queue depth, cache hit rate, etc.). Avoid recording raw user IDs or unbounded strings as metric labels.

High-cardinality labels: If you label every metric by user ID or request ID, Prometheus will struggle. Use only low-cardinality labels: HTTP method, route, status code, queue name.

Forgetting to initialize prom-client metrics once: Creating a new Counter or Histogram in every request causes a crash. Initialize all metrics in a service or module setup, then reuse them.

Alerts that never fire or fire constantly: Set thresholds and durations based on your SLO. An alert that cries wolf gets ignored; an alert that fires only when users notice is gold.

No baseline: Metrics are useless without context. Run a canary or load test to know what "normal" looks like for your endpoints, so you can spot when something is wrong.

Key takeaways

• Instrument request rate, error rate, and latency (RED method) as your foundation; these three metrics capture user experience.
• Use Counters for totals and events; use Histograms for durations and distributions; use Gauges for snapshots like queue depth.
• Avoid high-cardinality labels (user ID, request ID) and initialize all metrics once at startup, not per request.
• Expose metrics via a /metrics endpoint and configure Prometheus to scrape it; Grafana dashboards make trends visible.
• Instrument business-critical paths: queues, external API calls, database operations—anything that degrades user experience when it fails.
• Set alerts on symptoms users feel (high error rate, high latency), not on vanity metrics; keep alert thresholds realistic and durations long enough to avoid false alarms.
• Logs tell you what happened; metrics tell you it's happening now—use both to build confidence in your system.