OpenTelemetry and Prometheus: what developers should know

OpenTelemetry and Prometheus: what developers should know

You've probably heard the buzz around OpenTelemetry by now, and you might be wondering how it relates to Prometheus—the monitoring tool your team already relies on. The short answer: OpenTelemetry gives you a vendor-neutral way to collect metrics, traces, and logs from your applications, while Prometheus remains the powerful backend for scraping and storing those metrics. Together, they form a modern observability stack that scales with your infrastructure.

In this article, we'll walk through how OpenTelemetry and Prometheus fit together, why this combination matters for production systems, and what it means for the way we instrument our code.

OpenTelemetry SDKs—Instrument Once for Metrics, Traces, and Logs

The core promise of OpenTelemetry is simple: instrument your application once, and export your telemetry data anywhere. Rather than writing custom metrics code for Prometheus, traces for Jaeger, and logs for your centralized logging system, OpenTelemetry SDKs provide a unified API across all three signal types.

When we instrument with OpenTelemetry, we're creating semantic conventions—a shared language for what metrics, spans, and log events mean. This standardization means less vendor lock-in and easier migrations down the road. A metric like http.server.request.duration has the same meaning whether you're exporting to Prometheus, Datadog, or your own self-hosted Grafana stack.

Here's what instrumenting a Node.js application with OpenTelemetry looks like in practice:

import { NodeTracerProvider } from '@opentelemetry/node';
import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { PrometheusExporter } from '@opentelemetry/exporter-metrics-prometheus';
import { registerInstrumentations } from '@opentelemetry/auto-instrumentations-node';

// Set up metrics export to Prometheus
const prometheusExporter = new PrometheusExporter(
  {
    port: 8888,
    endpoint: '/metrics',
  },
  () => {
    console.log('Prometheus metrics server running on port 8888');
  }
);

const meterProvider = new MeterProvider({
  readers: [prometheusExporter],
});

// Auto-instrument common libraries (Express, HTTP, database drivers)
registerInstrumentations({
  meterProvider,
});

const meter = meterProvider.getMeter('my-app', '1.0.0');

// Custom metric: order processing duration
const orderDuration = meter.createHistogram('order.processing.duration', {
  description: 'Time to process an order in milliseconds',
  unit: 'ms',
});

// In your business logic
const startTime = Date.now();
await processOrder(order);
const duration = Date.now() - startTime;
orderDuration.record(duration, { 'order.type': order.type });

This single instrumentation gives you metrics that Prometheus can scrape immediately. No custom collectors, no vendor-specific libraries for this particular metric. The OpenTelemetry SDK handles the heavy lifting: formatting, buffering, and exposing metrics in a format Prometheus understands.

Export to Prometheus—Exporter vs. OTLP Receiver

Once you've instrumented with OpenTelemetry, you need to get the telemetry data to Prometheus. There are two main paths: direct export or via the OpenTelemetry Collector.

Direct Export with PrometheusExporter

The simplest approach is using the OpenTelemetry Prometheus exporter, which we showed above. Your application runs a small HTTP server (typically on port 8888 or 9090) that exposes metrics in Prometheus text format. Prometheus scrapes this endpoint just like it would any other target.

This works well for small deployments or when your application already has a metrics endpoint. The trade-off: your application now has direct responsibility for exposing metrics, and there's no intermediate processing layer.

OTLP Receiver via OpenTelemetry Collector

For more sophisticated deployments, your application exports telemetry to the OpenTelemetry Collector using the OTLP protocol (OpenTelemetry Protocol). The Collector acts as a middleware: it receives, processes, and routes your telemetry data to multiple backends, including Prometheus.

Here's how you'd configure an application to export to the Collector:

import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';

// Export metrics to OpenTelemetry Collector
const exporter = new OTLPMetricExporter({
  url: 'http://localhost:4318/v1/metrics',
});

const meterProvider = new MeterProvider({
  readers: [new PeriodicExportingMetricReader(exporter)],
});

And the Collector's configuration (in YAML) to receive and export to Prometheus:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheus]

Now Prometheus scrapes the Collector's /metrics endpoint instead of directly from your application. This decoupling is powerful: you can restart your app without losing metric data, and the Collector can batch, enrich, or filter metrics before Prometheus sees them.

Collector—Process and Route Telemetry

The OpenTelemetry Collector is the nervous system of a mature observability setup. While not strictly required, it becomes essential once you're running multiple services or need sophisticated data pipelines.

A real-world scenario: you're running three microservices (API, workers, database layer), and you want to send metrics to Prometheus, but also ship certain high-cardinality metrics to a dedicated analytics platform without burdening Prometheus. The Collector lets you do this elegantly.

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    send_batch_size: 1000
    timeout: 10s
  
  # Drop high-cardinality dimensions
  attributes:
    actions:
      - key: user_id
        action: delete
      - key: trace_id
        action: delete

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
  
  otlp:
    client:
      endpoint: analytics.example.com:4317
      tls:
        insecure: false

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch, attributes]
      exporters: [prometheus, otlp]

In this configuration, the Collector:

• Receives telemetry from all your services via OTLP
• Batches metrics to reduce network overhead
• Strips sensitive attributes like user IDs and trace IDs
• Exports the cleaned metrics to both Prometheus and your analytics backend

Processors also handle sampling (useful for high-volume trace data), enrichment (adding environment or version tags), and filtering. This is where you apply operational policies to your telemetry pipeline.

One Standard—Avoid Vendor Lock-in

The deeper value of OpenTelemetry isn't just the current Prometheus integration; it's the insurance against being locked into a single vendor or tool. Over the life of a production system, your observability needs will evolve. Maybe Prometheus scales to your needs today, but in two years you're evaluating managed solutions like Datadog, New Relic, or Grafana Cloud.

With OpenTelemetry instrumentation baked into your codebase, switching exporters or adding new backends is often a configuration change, not a code refactor. Your team has already written the metrics collection logic once, using a standard API, and it remains portable.

This matters because:

• Cost and scaling: Self-hosted Prometheus has limits on cardinality and query complexity. A managed service might offer better scaling at a price.
• Team preferences: If your company acquires another team with different observability infrastructure, OpenTelemetry lets you bridge both systems.
• Compliance: Some deployments need data to stay in-region or on-premise. OTLP exporters can route to multiple backends simultaneously.

The semantic conventions are also crucial here. A metric like http.server.request.duration is interpreted consistently across tools, whereas custom metrics often need translation or remapping.

Common Mistakes

Over-instrumenting without sampling: Developers sometimes add OpenTelemetry instrumentation to every function call without considering cardinality or volume. High-cardinality dimensions (like request IDs or user IDs in metric labels) will overwhelm Prometheus. Use sampling strategically, especially for traces.

Ignoring the semantic conventions: OpenTelemetry provides a standard set of metric and span attribute names. Ignoring them defeats much of the portability benefit. When you need custom metrics, namespace them clearly and document their meaning.

Deploying without a Collector in production: While the Prometheus exporter works for development, the Collector adds resilience and processing capabilities that matter at scale. Plan for it early.

Forgetting to configure retention and cleanup: Metrics are smaller than logs, but they still accumulate. Set up retention policies and be mindful of label explosion. A single poorly-named dimension can grow your Prometheus database unexpectedly.

Key Takeaways

• OpenTelemetry SDKs provide a unified API for metrics, traces, and logs, instrumenting your application once and exporting to any backend.
• Prometheus remains an excellent backend for metrics; OpenTelemetry simply gives you a standardized, portable way to feed data into it.
• For simple deployments, the Prometheus exporter is sufficient; for production complexity, deploy the OpenTelemetry Collector as a middleware layer for processing and routing.
• Semantic conventions ensure your metrics are interpreted consistently across tools, protecting you from vendor lock-in.
• Start with auto-instrumentation to capture framework-level metrics, then add custom metrics with care—monitor cardinality and use sampling where appropriate.