Monitoring Spring Boot with Prometheus and Grafana explained

Monitoring Spring Boot with Prometheus and Grafana explained

Your Spring Boot application is running smoothly in production—until it isn't, and you're scrambling through logs with no visibility into what went wrong. Monitoring Spring Boot with Prometheus and Grafana explained is the difference between reactive firefighting and proactive health management. We'll walk through building dashboards that surface the metrics that matter, so you catch problems before your users do.

Why metrics matter more than logs

Logs tell you what happened; metrics tell you what's happening right now. When a request takes 5 seconds, a log entry might capture it. But metrics let you see that 95% of requests complete in 200ms, the 99th percentile hit 4.8 seconds at 3 PM, and that's when your database connection pool was exhausted. Logs are event-based noise. Metrics are signal.

Spring Boot doesn't ship with built-in observability for production environments. That's where Micrometer enters. Micrometer is an abstraction layer that instruments your application, and Prometheus is the time-series database that stores and queries those metrics. Grafana sits on top, turning raw numbers into dashboards your team actually reads.

Micrometer-registry-prometheus exposes metrics

Start by adding Micrometer's Prometheus registry to your Spring Boot project. This dependency exposes a metrics endpoint that Prometheus scrapes at regular intervals.

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
    <version>1.12.0</version>
</dependency>

Once added, your application exposes metrics at /actuator/prometheus. Hit that endpoint and you'll see Prometheus text format: metric names, labels, and values. Spring Boot auto-instruments servlet requests, database connections, JVM memory, and garbage collection out of the box.

Configure Prometheus to scrape your application by editing its configuration file:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'spring-boot-app'
    static_configs:
      - targets: ['localhost:8080']
    metrics_path: '/actuator/prometheus'

Every 15 seconds, Prometheus pulls metrics from your app and stores them with timestamps. This creates a historical record you can query and graph. The real power emerges when you stop thinking about individual requests and start thinking about distributions, trends, and thresholds.

Track latency percentiles, error rate and saturation

Three metrics matter most in production: latency (how fast?), errors (how often?), and saturation (how full?). These are your RED metrics—Request rate, Errors, Duration.

Micrometer automatically tracks HTTP request duration. You'll see histograms like http_server_requests_seconds with labels for method, URI, and status. Query the 95th percentile in Grafana:

histogram_quantile(0.95, http_server_requests_seconds_bucket)

This tells you: 95% of requests complete within X milliseconds. That's actionable. When that number climbs from 150ms to 800ms at 9 AM every day, you've got a clue about slow batch jobs or scaling issues.

Error rates come from counting requests with status 5xx:

rate(http_server_requests_seconds_count{status=~"5.."}[5m])

This shows errors per second over the last 5 minutes. A spike from 0 to 10 errors/sec is your first alert signal. Correlate it with logs, and you'll often find a specific dependency, misconfiguration, or traffic pattern.

Saturation means "how much capacity is in use?" Database connection pools are classic here. If your pool has 20 connections and you're actively using 19, you're saturated and vulnerable to traffic spikes. Micrometer tracks this:

hikaricp_connections_active / hikaricp_connections_max

Aim to keep saturation below 70%. Beyond that, response times degrade and tail latencies explode. It's the invisible wall you only hit when it's too late.

JVM heap, GC pause and thread metrics

Your application runs inside a JVM. Understanding JVM behavior separates senior engineers from those who restart the app when it gets slow. Micrometer exposes garbage collection and memory metrics by default.

Watch heap memory usage:

jvm_memory_used{area="heap"}

A steady increase means a memory leak. Your heap grows, GC runs more frequently, pause times spike. Users see timeouts. Classic scenario: you're caching something without eviction, holding references to objects that should be garbage-collected. A graph over 24 hours reveals the pattern immediately.

Track GC pause time:

jvm_gc_pause_seconds

This histogram shows how long garbage collection stops your application. During a pause, requests queue up. If pauses hit 200ms frequently, users feel it—that's a perceptible delay. Pauses over 500ms are in territory where you're losing transactions or timing out calls to downstream services.

Monitor thread count:

jvm_threads_live

Thread leaks are rare but catastrophic. A thread pool configured for 200 threads that climbs to 1,000 over weeks means you're not releasing threads after requests complete—usually in async code or thread-local variables. The JVM will exhaust heap or OS limits, and the application becomes unresponsive.

Put these on a dashboard together. When latency spikes, check GC pause times first. If pauses are high and heap is growing, you've got memory pressure. If heap is stable but threads are climbing, dig into async code. These three metrics form a diagnostic trinity.

Alert on symptoms, not causes

Alerts are where most teams fail. You can have beautiful dashboards nobody looks at. The moment you alert, every team member gets notified, and if you alert on noise, they stop listening.

Alert on observable symptoms of problems, not internal states. Don't alert on "heap above 80%"—that's a cause. Alert on "95th percentile latency above 500ms"—that's what users experience.

Here's a practical alerting strategy for Spring Boot:

• Error rate spike: Alert if errors per second jump above 5 for 2 consecutive minutes. Human review is fast; false positives are lower.
• Latency degradation: Alert if the 95th percentile latency exceeds 1 second for 5 minutes. This catches cascading failures in dependencies.
• Saturation threshold: Alert if database connection pool usage stays above 75% for 10 minutes. This gives you time to scale or investigate without false alarms from transient spikes.
• Garbage collection pressure: Alert if average GC pause duration exceeds 100ms over 5 minutes. High frequency is the real problem, not individual pauses.

In Prometheus, rules look like this:

groups:
  - name: spring_boot_alerts
    rules:
      - alert: HighErrorRate
        expr: rate(http_server_requests_seconds_count{status=~"5.."}[5m]) > 5
        for: 2m
        annotations:
          summary: "High error rate detected"
          
      - alert: HighLatency
        expr: histogram_quantile(0.95, http_server_requests_seconds_bucket) > 1
        for: 5m
        annotations:
          summary: "95th percentile latency above 1 second"

The for clause is crucial. It prevents alerting on every small fluctuation. You're saying: "This metric has to be bad for 5 minutes straight before I wake someone up." That discipline separates production-ready monitoring from noise.

Common mistakes

• Scrape interval too short: Setting Prometheus to scrape every 5 seconds means cardinality explosion if you're not careful with labels. Every unique label combination is a separate time series. Start with 15-30 second intervals; optimize if you need finer granularity.
• Alerting on thresholds without context: "CPU above 70%" means nothing. 70% on a 2-core laptop is different from 70% on a 32-core server. Alert on user-visible impact: latency, errors, saturation. Those scale with your infrastructure.
• Forgetting to set up dashboards iteratively: Don't try to build the perfect dashboard day one. Start with RED metrics (request rate, error rate, duration), get those working, then layer in JVM diagnostics. Dashboards evolve with your understanding of the system.
• Retention too short: Prometheus' default retention is 15 days. If you're investigating a problem that happened two weeks ago, you'll have no data. Plan retention based on your incident response cycle—at least 30 days for most teams.
• No correlation across systems: Your Spring Boot app isn't alone. Track your database response times, message queue depth, and external API latencies on the same Grafana dashboard. Latency spikes are usually external, not internal.

Key takeaways

• Add micrometer-registry-prometheus to expose metrics at /actuator/prometheus; Prometheus scrapes this endpoint every 15-30 seconds and stores historical data.
• Monitor RED metrics: request rate, error rate (5xx responses), and latency percentiles (95th, 99th). These are user-visible and actionable.
• Track JVM health separately: heap memory trends reveal leaks, GC pause times predict latency degradation, and thread count identifies resource leaks.
• Alert on symptoms (high latency, error spikes, saturation thresholds) rather than internal states; use the for clause to avoid noise.
• Build dashboards incrementally, starting with RED metrics, then adding JVM diagnostics and dependency health over time.