Pushgateway: when to use it — what developers should know

You've built a cron job that runs daily, scrapes data, transforms it, and finishes in seconds. But Prometheus never sees the metrics because it scraped one minute too early. Pushgateway solves this problem—but it's a tool with a very specific purpose, and using it outside that window introduces operational complexity. Understanding when Pushgateway belongs in your stack, and when it doesn't, separates teams that keep their monitoring simple from those that inherit headaches.

The core problem: short-lived jobs and the scrape interval gap

Prometheus works by scraping targets at regular intervals—typically every 15 to 30 seconds. This architecture works beautifully for long-running services: a web server, a database connection pool, a message queue consumer. The metrics are always there when Prometheus arrives.

But batch jobs are different. A Lambda function that runs for 3 seconds and then stops. A Kubernetes Job that completes in 10 seconds. A scheduled database maintenance task that runs once per hour. If that job finishes between Prometheus scrapes, the metrics evaporate. You never recorded that the job ran, how long it took, or whether it succeeded.

That's where Pushgateway comes in. Instead of waiting to be scraped, the job pushes its metrics to a central intermediary. Prometheus then scrapes that intermediary at its normal cadence. The metrics persist long enough for Prometheus to find them.

Pushgateway: when to use it — what developers should know starts with recognizing that this is a narrow, specific use case. It's not a general solution for sending metrics from anywhere. It's a targeted tool for capturing data from jobs that live and die in the gaps between scrape intervals.

For batch jobs that finish before scraping

The textbook example: you have a Kubernetes CronJob that runs a Python script every 2 hours. The script downloads a file, processes it, and exits. The entire job takes 45 seconds. Your Prometheus scrape interval is 30 seconds. Without intervention, you have a 50/50 shot of capturing the metrics—and that's if Prometheus scrapes exactly during the job's execution window.

With Pushgateway, the script pushes its metrics the moment it finishes:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: data-processor
spec:
  schedule: "0 */2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: processor
            image: myregistry/processor:latest
            env:
            - name: PUSHGATEWAY_URL
              value: "http://pushgateway.monitoring:9091"
          restartPolicy: Never

Inside the script (Python example using the Prometheus Python client):

#!/usr/bin/env python3
import time
from prometheus_client import CollectorRegistry, Counter, Gauge, push_to_gateway

registry = CollectorRegistry()
job_duration = Gauge('data_processor_duration_seconds', 'Time taken to process', registry=registry)
records_processed = Counter('data_processor_records_total', 'Number of records', registry=registry)

start = time.time()
try:
    # Your actual job logic
    records = process_data()
    records_processed.inc(len(records))
    job_duration.set(time.time() - start)
    push_to_gateway('pushgateway.monitoring:9091', job='data_processor', registry=registry)
except Exception as e:
    push_to_gateway('pushgateway.monitoring:9091', job='data_processor', registry=registry)
    raise

Now, regardless of when Prometheus scrapes, it will find the metrics sitting in Pushgateway. You capture every job execution, latency, and outcome. This is where Pushgateway earns its place in your stack.

Real-world scenarios where this works well:

• Database backup verification jobs that run for a few minutes
• Report generation services triggered by webhooks or schedules
• Data migration jobs that run overnight
• Cleanup tasks that remove old logs or cache entries
• Lambda functions and serverless workloads where you have no persistent agent

Not for long-running services

Here's where teams go wrong: they treat Pushgateway as a general-purpose metrics sink and start pushing metrics from web servers, workers, and message queue consumers. This is an architectural mistake.

Long-running services should use the Prometheus scrape model directly. The reasons are practical and important:

Connection overhead: Every push to Pushgateway requires an HTTP request. For a service that emits metrics hundreds or thousands of times per second, pushing becomes a bottleneck. Scraping pulls all metrics in one request.

Loss during crashes: If your service crashes hard, a push queue may not flush. Prometheus scraping doesn't have this problem—when the service recovers and Prometheus scrapes again, the metrics are there in their current state. With Pushgateway, you might lose data from the critical moment before the crash.

Deduplication: If your service pushes metrics and then crashes before marking them as obsolete, Pushgateway continues to report stale metrics from a dead instance. Scraping naturally handles this: if the endpoint stops responding, Prometheus knows the instance is gone.

Operational complexity: Pushgateway is another stateful component to deploy, version, monitor, and troubleshoot. Long-running services don't need it.

If you're instrumenting a Go HTTP server, a Node.js application, or a Java microservice, use the native Prometheus client libraries and rely on Prometheus scraping. If you have a fleet of services, use service discovery (Kubernetes service discovery, Consul, file-based, etc.) to let Prometheus find them automatically.

Delete stale metrics

Pushgateway keeps metrics until you explicitly delete them. This is both a feature and a trap.

The feature: metrics persist even if you stop running your jobs for a while. Helpful for dashboards that expect to see historical data.

The trap: if a job fails silently or never pushes metrics for some reason, Pushgateway continues serving old metrics as if the job succeeded. You see a 2-hour-old metric and assume the job ran successfully 2 hours ago. It didn't. The job crashed and never ran again.

The antidote is deliberate cleanup. After pushing your metrics, set a time-to-live or explicitly delete them when appropriate:

#!/usr/bin/env python3
import time
from prometheus_client import CollectorRegistry, Counter, Gauge, push_to_gateway
import requests

registry = CollectorRegistry()
job_duration = Gauge('data_processor_duration_seconds', 'Time taken to process', registry=registry)
records_processed = Counter('data_processor_records_total', 'Number of records', registry=registry)

start = time.time()
try:
    records = process_data()
    records_processed.inc(len(records))
    job_duration.set(time.time() - start)
    push_to_gateway('pushgateway.monitoring:9091', job='data_processor', registry=registry)
except Exception as e:
    # Job failed; push a failure marker
    push_to_gateway('pushgateway.monitoring:9091', job='data_processor', registry=registry)
    raise
finally:
    # Clean up after 6 hours if job doesn't run again
    # (Better: set up a separate cleanup job or use Pushgateway's grouping wisely)
    pass

Pushgateway supports deletion by HTTP DELETE request. Some teams run a cleanup job that deletes metrics older than N hours. Others assign a job grouping and delete the entire group when the job completes:

# Push with grouping
curl -X POST http://pushgateway:9091/metrics/job/data_processor/instance/server1

# Later, delete that instance's metrics
curl -X DELETE http://pushgateway:9091/metrics/job/data_processor/instance/server1

For critical jobs, instrument your monitoring to alert if metrics aren't pushed within an expected window. For example, if your backup job should push metrics every 24 hours, create an alert that fires if no metric is updated within 30 hours. This catches both job failures and Pushgateway issues.

Prefer remote write or OpenTelemetry for others

If you have metrics from systems that aren't long-lived services and aren't short-lived batch jobs, Pushgateway often isn't the right answer.

Remote write: If you're instrumenting a client-side application, an IoT device, or a third-party system that can't be scraped, consider Prometheus remote write. This allows services to push metrics directly to a remote storage backend (or to a write-ahead buffer that ingests into Prometheus). Tools like Thanos, Cortex, and Mimir all support remote write and handle multi-tenancy, retention, and querying at scale. Remote write is stateless on the client side—you send metrics and move on, and the server handles durability and storage.

OpenTelemetry: If you're building new instrumentation, especially in distributed systems, OpenTelemetry is the better choice. OpenTelemetry is a vendor-neutral standard that supports traces, logs, and metrics. You instrument your code once, and OpenTelemetry's collector agents (running as sidecars or gateways) route telemetry to Prometheus, Datadog, Honeycomb, or any backend you choose. This avoids lock-in and gives you flexibility as your monitoring needs evolve.

OpenTelemetry is particularly useful because you can collect metrics, traces, and logs from the same source and correlate them. Pushgateway only handles metrics and is purpose-built for Prometheus.

Common mistakes

• Using Pushgateway for every metric: Teams install Pushgateway and then redirect all metrics into it, treating it as a central hub. This defeats the simplicity of Prometheus's pull model and creates a bottleneck.
• Ignoring stale metrics: Forgetting that pushed metrics persist means old data gets served as current. Always plan for cleanup or deletion strategies.
• No deduplication across Pushgateway replicas: If you run multiple Pushgateway instances for redundancy, there's no built-in way to deduplicate metrics if both receive a push. You need to run Pushgateway behind a load balancer with session stickiness or use a single instance (losing high availability).
• Pushing from fast-failing jobs without error handling: If a job fails and never pushes metrics, Prometheus will still report old metrics as current. Wrap your push logic in error handlers and consider pushing a "failed" marker if the job aborts.
• Treating Pushgateway as a metrics buffer: It's not designed to be a durable queue. If Pushgateway crashes between pushes, you lose in-flight metrics. Use it for metrics you can afford to lose or pair it with persistent push-queue architecture for critical scenarios.

Key takeaways

• Pushgateway solves a specific problem: capturing metrics from jobs that finish before Prometheus scrapes them.
• Use it for batch jobs, cron tasks, Lambda functions, and short-lived Kubernetes Jobs. Don't use it for long-running services.
• Always plan for deletion or cleanup of stale metrics to avoid false positives in alerting and dashboards.
• For client-side or third-party metrics, prefer remote write (to systems like Mimir or Cortex) or OpenTelemetry, which are more flexible and often simpler at scale.
• Keep Pushgateway simple: treat it as a transient metrics store for short-lived jobs, not as a general-purpose metrics sink for your entire infrastructure.