Prometheus architecture explained: a step-by-step guide

Introduction

If you've deployed Prometheus in production, you've probably found yourself staring at scrape configs and wondering how all the pieces fit together. Prometheus architecture explained: a step-by-step guide walks through the core mechanics that make Prometheus one of the most widely adopted monitoring systems in cloud-native environments. Understanding how Prometheus collects, stores, and queries metrics isn't just academic—it directly affects your alerting reliability, retention policies, and query performance in production.

We've seen teams spend weeks debugging high cardinality problems or missing metrics because they didn't grasp how the pull model actually works. Let's build that mental model today.

Components: The Five-Step Data Pipeline

Prometheus monitoring follows a predictable flow from your applications all the way through to your dashboards and alerts. Understanding each step helps you debug issues faster and design better monitoring strategies.

Step 1: Targets expose /metrics

Every service monitored by Prometheus must expose an HTTP endpoint (almost always /metrics) that returns metrics in OpenMetrics or Prometheus text format. This endpoint is the contract between your application and Prometheus. A Node Exporter running on a server, your application's custom instrumentation, or a third-party exporter all speak the same language here.

Here's what a simple metrics endpoint looks like:

# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total{instance="localhost:8080"} 42.31
# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{handler="/api/users",method="POST",status="200"} 1523
http_requests_total{handler="/api/users",method="GET",status="200"} 5201
http_requests_total{handler="/api/users",method="GET",status="404"} 87

The format is deliberate: human-readable, parseable, and descriptive via the HELP and TYPE comments. Every metric can have labels (the key-value pairs in curly braces), which become crucial for aggregation and alerting.

Step 2: Prometheus scrapes on an interval

Rather than waiting for services to push metrics, Prometheus pulls data from each target on a regular schedule. By default, this happens every 15 seconds, controlled by the scrape_interval setting. This is the defining characteristic of the Prometheus architecture.

A typical scrape config looks like this:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'api-server'
    static_configs:
      - targets: ['localhost:8080', 'localhost:8081']
  
  - job_name: 'database'
    scrape_interval: 30s
    static_configs:
      - targets: ['db.internal:9100']

At each scrape, Prometheus makes an HTTP GET to http://target/metrics, collects all the samples (individual metric + labels + value + timestamp), and moves them to the next stage. If a target is down or slow, Prometheus records that fact and continues; it doesn't block or crash.

Step 3: Samples stored in the time-series database

Every scraped sample—metric name, labels, value, and timestamp—gets written to Prometheus's embedded time-series database (TSDB). This is where retention and storage performance matter. Prometheus stores data locally on disk by default, organized into 2-hour blocks. It's fast, indexed well for PromQL queries, and designed for high write throughput.

A typical storage layout on disk looks like:

prometheus-data/
├── wal/                  # Write-ahead log (recent samples)
├── 01ARZ3NHYvjYjN/        # 2-hour block
│   ├── chunks/
│   ├── index
│   └── meta.json
├── 01ARZ3NHYvjYjO/
└── ...

The time-series database is local by default. This is a strength for performance but a limitation for long-term retention and cross-datacenter visibility. We'll cover remote storage later in this series, but it's worth noting now that many teams use Prometheus's built-in retention (default 15 days) plus a remote storage backend for long-term analytics.

Step 4: PromQL queries the data

Once data is stored, Prometheus's query engine (powered by PromQL) allows you to slice, aggregate, and transform it. PromQL is powerful but requires learning its syntax. A simple query like rate(http_requests_total[5m]) calculates the per-second request rate over the last 5 minutes. More complex queries can join metrics across labels, apply functions, and alert on thresholds.

This is where dashboards and Grafana connect: they send PromQL queries to Prometheus's HTTP API, get back time-series data, and render it visually.

Step 5: Alertmanager routes alerts

Prometheus evaluates alert rules at a regular interval (often the same as evaluation_interval). When an alert condition fires, it sends it to Alertmanager, which handles deduplication, grouping, and routing to Slack, PagerDuty, or email. Alertmanager is a separate component that decouples alert logic from Prometheus itself, making it more resilient and flexible.

The Pull Model: Why Prometheus Doesn't Wait

This design choice is central to understanding Prometheus. Unlike push-based systems where services send metrics to a central collector, Prometheus pulls metrics on a schedule. Each scrape is a predictable, time-stamped event.

Advantages of the pull model

• Service discovery is simpler. Prometheus maintains a list of targets and scrapes them all. Services don't need to know about Prometheus or authenticate to send data.
• Overload protection. A misbehaving client can't flood Prometheus with data. The scrape interval is a natural rate limit.
• Easier testing. You can curl /metrics locally and debug what Prometheus will see.
• Multi-consumer friendly. Multiple Prometheus instances can scrape the same targets for redundancy or sharding.

Trade-offs to be aware of

Pulling means Prometheus may miss very short-lived metrics or events that occur between scrapes. If a process crashes and restarts in 5 seconds, and your scrape interval is 15 seconds, you might miss it entirely. Some teams use push-based collection (via Pushgateway) for short-lived batch jobs to work around this.

Service Discovery: Finding Your Targets

Manually listing every target in static_configs doesn't scale. Prometheus supports several service discovery mechanisms to automatically detect targets as they come and go.

Kubernetes service discovery

If you're running on Kubernetes, Prometheus can query the API to find pods and services automatically.

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)

This configuration discovers all pods with prometheus.io/scrape: "true" annotations. Relabel rules transform metadata into scrape parameters. This is how most cloud-native teams run Prometheus today.

EC2 and file-based discovery

For VMs on AWS, the ec2_sd_config queries the EC2 API. For static deployments or hybrid clouds, file_sd_config reads target lists from a JSON or YAML file that you update via config management tools.

scrape_configs:
  - job_name: 'ec2-instances'
    ec2_sd_configs:
      - region: us-east-1
        port: 9100

Local Storage: Speed and Retention Trade-offs

Prometheus's local TSDB is optimized for high throughput and query performance. But "local" means it lives on the Prometheus pod or server, and retention is limited by disk space.

Planning retention

The storage footprint depends on:

• Cardinality: Total unique metric + label combinations. High cardinality (millions of time series) kills performance.
• Scrape interval: Shorter intervals = more samples per second = more disk usage.
• Number of targets: More targets = more metrics scraped per round.

A rough estimate: 1 million time series with 15-second scrape intervals uses about 10-50 GB per day, depending on the sample density and compression. The Prometheus documentation has a helpful formula.

Retention settings

Configure retention in the Prometheus startup flags:

prometheus --storage.tsdb.retention.time=30d --storage.tsdb.retention.size=50GB

This keeps data for 30 days or until the database reaches 50 GB, whichever comes first. Both limits work together.

Local storage in production

Local storage is fast but ephemeral. If your Prometheus pod restarts, data is lost unless you use persistent volumes (PVCs in Kubernetes). For critical monitoring, plan for remote storage or multiple Prometheus instances feeding into a long-term store. We'll cover remote storage, Thanos, and federation in later articles in this series.

Common Mistakes

Assuming "real-time" metrics. Prometheus is typically 15–30 seconds behind live events. If you need sub-second alerting, consider a push-based system or shorter scrape intervals (with the CPU cost that entails).

Ignoring cardinality. A single metric like http_request_duration_seconds with high-cardinality labels (e.g., customer ID, request ID) can become millions of time series. Your TSDB chokes, queries slow down, and memory usage explodes. Always validate label cardinality before adding new dimensions.

Configuring scrape_interval too short. Every second of scrape interval costs 10x the samples ingested per minute. A 1-second interval on 1,000 targets is 1 million samples per minute. Start at 15–30 seconds and optimize from there based on actual alerting needs.

Not planning disk space. Prometheus grows silently. Without retention limits, it fills the disk and stops accepting new samples. Monitor the prometheus_tsdb_symbol_table_size_bytes metric or check disk usage routinely.

Key Takeaways

• Prometheus architecture explained centers on a pull-based model: targets expose /metrics, Prometheus scrapes on a schedule, samples go to the TSDB, PromQL queries them, and alerts route to Alertmanager.
• Pull-based design simplifies service discovery and prevents client-side overload, but means you might miss very short-lived events between scrapes.
• Service discovery (Kubernetes, EC2, file-based) automates target discovery and scales with your infrastructure; static configs work for small, stable deployments.
• Local storage is fast but ephemeral and limited by disk. Plan retention, monitor cardinality, and consider remote storage for long-term analytics.
• Cardinality discipline is non-negotiable. High-cardinality labels (IDs, timestamps, customer data) will degrade Prometheus performance. Define label schemas carefully.
• Scrape interval tuning balances freshness against cost. Start at 15–30 seconds and measure impact before going shorter.

Tomorrow, we'll dive into writing effective Prometheus queries with PromQL and building dashboards in Grafana. Understanding these architectural foundations will make those topics much clearer.