Instrumenting with OpenTelemetry metrics for production

Introduction

By day 80 of a monitoring transformation, your team knows the power of observability—but scattered instrumentation strategies create blind spots and vendor lock-in. Instrumenting with OpenTelemetry metrics for production gives you one unified API that works across any backend, whether that's Prometheus scraping your endpoints or OTLP exporters pushing to distributed tracing platforms. We'll walk through building production-grade metrics that scale from a single microservice to a complex platform.

Why OpenTelemetry Metrics Matter Now

Before OpenTelemetry, teams often embedded Prometheus client libraries directly into code. That couples your application to a specific exporter format and makes switching backends expensive. OpenTelemetry decouples instrumentation from export: you write once, export anywhere. This separation of concerns is especially critical in production where you might want to ship metrics to Prometheus today, OpenTelemetry Protocol (OTLP) to a vendor tomorrow, and a custom backend next year without rewriting instrumentation.

The spec is now mature enough that major frameworks ship auto-instrumentation out of the box, meaning much of your work is already done if you configure it correctly.

Meters, Counters and Histograms

OpenTelemetry metrics are built around three core instruments. Understanding the differences is crucial for choosing the right one in production.

Counters: Monotonic Growth

Counters measure monotonically increasing values—they never decrease. Use them for request counts, errors, exceptions, or any cumulative metric. In Prometheus, a counter is scraped at intervals and the rate of change is calculated by the backend.

const meter = meterProvider.getMeter('payment-service');
const requestCounter = meter.createCounter('http_requests_total', {
  description: 'Total HTTP requests processed',
});

// In your request handler
requestCounter.add(1, {
  'http.method': 'POST',
  'http.status_code': 200,
  'service.name': 'payment-api',
});

Counters are cheap to compute and reliable. Every increment is additive, so gaps in scraping don't lose data—the total is always correct when you resume.

Histograms: Distribution and Latency

Histograms capture the distribution of values across buckets. They're essential for latency tracking, payload sizes, or database response times. OpenTelemetry histograms automatically aggregate into percentiles without additional code.

const latencyHistogram = meter.createHistogram('http_request_duration_ms', {
  description: 'HTTP request latency in milliseconds',
});

// In middleware
const start = Date.now();
// ... handle request ...
const duration = Date.now() - start;
latencyHistogram.record(duration, {
  'http.method': req.method,
  'http.status_code': res.statusCode,
  'http.route': req.route,
});

Histograms let you calculate p50, p95, and p99 latencies without polling every request. In production, set explicit bucket boundaries matching your SLOs: if your SLO is 500ms response time, include a 500ms bucket so you can easily track compliance.

Gauges: Point-in-Time Snapshots

Gauges measure values that go up and down—memory usage, active connections, or queue depth. Use them sparingly; they're harder to aggregate across instances than counters.

const activeConnections = meter.createObservableGauge(
  'db_active_connections',
  {
    description: 'Number of active database connections',
  },
  (observableResult) => {
    observableResult.observe(pool.activeCount(), {
      'db.system': 'postgresql',
      'db.pool': 'primary',
    });
  }
);

Observable gauges are registered once and called during metric collection, so they're lightweight for polling internal state.

Export to Prometheus or OTLP

OpenTelemetry provides exporters for both Prometheus pull-based scraping and OTLP push-based protocols. The choice depends on your infrastructure.

Prometheus Exporter: Pull Model

If you're already running Prometheus, the Prometheus exporter exposes metrics on an HTTP endpoint that Prometheus scrapes at regular intervals (typically 15-30 seconds).

const { PrometheusExporter } = require('@opentelemetry/exporter-prometheus');
const { MeterProvider } = require('@opentelemetry/sdk-metrics');

const exporter = new PrometheusExporter({
  port: 9464,
  endpoint: '/metrics',
}, () => {
  console.log('Prometheus metrics exposed on http://localhost:9464/metrics');
});

const meterProvider = new MeterProvider({
  readers: [exporter],
});

This endpoint exposes metrics in Prometheus text format. Your Prometheus config scrapes it like any other target. No external dependencies or network reliability issues—just HTTP polling on localhost.

OTLP Exporter: Push Model

OTLP (OpenTelemetry Protocol) pushes metrics to a collector endpoint. This is useful for managed backends like Datadog, Honeycomb, or a self-hosted OTEL Collector.

const { OTLPMetricExporter } = require('@opentelemetry/exporter-metrics-otlp-http');
const { MeterProvider, PeriodicExportingMetricReader } = require('@opentelemetry/sdk-metrics');

const exporter = new OTLPMetricExporter({
  url: 'http://otel-collector:4318/v1/metrics',
});

const meterProvider = new MeterProvider({
  readers: [new PeriodicExportingMetricReader({
    exporter,
    intervalMillis: 5000,
  })],
});

OTLP is protocol-agnostic and vendor-neutral, making it ideal for multi-backend setups or future flexibility. Trade-off: you need a collector running and network reliability between your app and the collector.

Running Multiple Exporters

In production, you often want both—Prometheus for internal dashboards and OTLP for a vendor backend. The MeterProvider accepts multiple readers.

const meterProvider = new MeterProvider({
  readers: [
    exporter,  // Prometheus exporter
    new PeriodicExportingMetricReader({ exporter: otlpExporter }),
  ],
});

Auto-instrumentation for Frameworks

Writing every metric manually is tedious and error-prone. OpenTelemetry provides auto-instrumentation packages for Express, Fastify, PostgreSQL, Redis, HTTP clients, and more. These inject metrics automatically without code changes.

Express Auto-Instrumentation

The Express instrumentation captures HTTP metrics automatically.

const { registerInstrumentations } = require('@opentelemetry/auto-instrumentations-node');

registerInstrumentations({
  instrumentationConfig: {
    '@opentelemetry/instrumentation-express': {
      enabled: true,
    },
    '@opentelemetry/instrumentation-http': {
      enabled: true,
    },
  },
});

This single setup auto-captures request count, duration, status codes, and route names—no manual middleware needed. The instrumentation respects the OpenTelemetry spec, so metrics are consistent across teams and services.

Database Instrumentation

PostgreSQL, MySQL, and MongoDB instrumentations track query count, duration, and errors.

const { PostgreSQLInstrumentation } = require('@opentelemetry/instrumentation-pg');

new PostgreSQLInstrumentation().enable();

// Your code runs unchanged
const result = await client.query('SELECT * FROM users WHERE id = $1', [userId]);
// Metrics captured automatically: query duration, error status, etc.

Auto-instrumentation reduces boilerplate and ensures consistency. In production, enable only what you need to avoid overhead; a busy service with all instrumentations enabled can add 5-15% CPU.

Consistent Attribute Names

Attributes (tags in Prometheus parlance) are key-value pairs attached to metrics. Inconsistent naming creates cardinality explosions and makes dashboards brittle. OpenTelemetry defines semantic conventions for standard attributes.

Semantic Conventions

Use standard attribute names from the OpenTelemetry spec. For HTTP endpoints, use http.method, http.status_code, http.route. For databases, use db.system, db.operation, db.statement.

// Good: semantic conventions
requestCounter.add(1, {
  'http.method': 'POST',
  'http.status_code': 200,
  'http.route': '/api/payments',
  'service.name': 'payment-api',
  'service.version': '1.2.3',
});

// Bad: inconsistent custom names
requestCounter.add(1, {
  'METHOD': 'POST',
  'statusCode': 200,
  'endpoint': '/api/payments',
});

Semantic conventions ensure your metrics are queryable across the entire org. A standard set of attribute names lets teams write consistent PromQL queries and Grafana dashboards that work across services.

High-Cardinality Traps

Attributes like user IDs, transaction IDs, or customer IDs should never be tags. They create unbounded cardinality and crash Prometheus.

// Dangerous: user_id is high-cardinality
paymentCounter.add(1, {
  'user_id': userId,  // 10 million unique values
  'status': 'success',
});

// Better: only use bounded attributes
paymentCounter.add(1, {
  'payment_status': 'success',
  'payment_method': 'card',
});
// Store the user_id in traces, not metrics

Limit attributes to bounded values: HTTP methods (6), status codes (20), regions (10), environment (3). This keeps cardinality manageable and your Prometheus database size reasonable.

Common Mistakes

Mistake 1: High-Cardinality Attributes

The easiest way to break Prometheus is attaching user IDs, request IDs, or session tokens to metrics. One endpoint with 10 million users generates 10 million metric series. Use filters in traces instead; metrics should have bounded dimensions.

Mistake 2: Not Setting Histogram Buckets

Default histogram buckets are generic. If your SLO is 200ms but default buckets go up to 1000ms, you can't easily track compliance. Always set explicit buckets matching your SLOs.

Mistake 3: Mixing Manual and Auto-Instrumentation

Enabling auto-instrumentation and then adding manual metrics for the same thing (e.g., auto HTTP metrics plus manual request counters) doubles your cardinality and confuses alerts. Use auto-instrumentation as the base and add manual metrics only for business logic.

Mistake 4: Forgetting Resource Attributes

Every metric should include service.name, service.version, and deployment.environment. Without these, you can't correlate metrics across services or deployments.

const meterProvider = new MeterProvider({
  resource: Resource.default().merge(new Resource({
    'service.name': 'payment-api',
    'service.version': process.env.APP_VERSION,
    'deployment.environment': process.env.NODE_ENV,
  })),
  readers: [exporter],
});

Key Takeaways

• Counters measure monotonic growth (request count, errors); histograms capture distributions (latency, payload size); gauges track point-in-time values (active connections).
• Export to Prometheus for pull-based scraping or OTLP for push-based collection—or both simultaneously for flexibility.
• Auto-instrumentation packages handle framework and dependency metrics automatically; enable only what you need to avoid overhead.
• Use OpenTelemetry semantic conventions for attribute names to ensure consistency across your platform.
• Keep attributes low-cardinality: use bounded values like HTTP status codes and regions, never user IDs or transaction IDs.
• Set histogram buckets explicitly to match your SLOs; generic buckets hide performance problems.
• Include resource attributes (service.name, service.version, deployment.environment) on every metric for observability across your platform.