Skip to content
ExponaBLOG
PROMETHEUS, GRAFANA & MONITORING · DAY 001 / 100

Why monitoring matters: a practical guide for developers

You cannot fix what you cannot see.

Expona Admin7 min read

You ship code every day, but how do you know if it's actually working in production? Most developers discover problems the hard way—through a Slack message, a support ticket, or worse, a 3 AM page. Why monitoring matters: a practical guide for developers isn't just about keeping systems running; it's about building confidence in the code you deploy and gaining visibility into problems before they cascade into customer-impacting incidents.

Monitoring is the bridge between what you build and what your users experience. Without it, you're flying blind. Let's walk through why it matters and how to start thinking about it from day one.

Catch problems before users report them

The difference between a five-minute outage you catch and a thirty-minute outage your customer discovers is visibility. When you have proper monitoring in place, you detect anomalies as they happen, not after the damage is done.

Consider a typical e-commerce scenario. Your checkout service handles payment processing. One afternoon, the payment gateway starts returning timeouts on 2% of requests. Without monitoring, your checkout service will retry those requests, some will eventually succeed, but others fail silently. Users refresh, get angry, abandon carts. Meanwhile, you're in standup wondering why conversion dropped. By the time support tickets pile up and someone checks logs, you've lost revenue and customer trust.

With proper monitoring, you would have seen the latency spike in your payment service dependencies within seconds. An alert fires. You investigate. You discover the payment gateway is degraded, you temporarily route traffic to a backup provider or increase retry delays, and the incident is contained before it hits your users at scale.

The key metrics to watch:

  • Error rates—Catch 4xx and 5xx responses early. A sudden spike in 502s means your backend is struggling.
  • Latency percentiles—P95 and P99 latency matter more than averages. If 1% of your API calls take 10 seconds, real users feel pain.
  • Dependency health—Database connection pool exhaustion, Redis evictions, external API timeouts—these propagate fast.
  • Business metrics—For e-commerce, track checkout completion rate. A code change might not break the API but could silently tank conversions.

Start by instrumenting your critical paths. In a NestJS or Node.js application, you can use middleware to capture request timing and error rates:

yaml
import { Injectable, NestMiddleware } from '@nestjs/common';
import { Request, Response, NextFunction } from 'express';

@Injectable()
export class MonitoringMiddleware implements NestMiddleware {
  use(req: Request, res: Response, next: NextFunction) {
    const start = Date.now();
    res.on('finish', () => {
      const duration = Date.now() - start;
      console.log({
        method: req.method,
        path: req.path,
        status: res.statusCode,
        duration_ms: duration,
        timestamp: new Date().toISOString(),
      });
    });
    next();
  }
}

This simple middleware gives you baseline visibility into every request. From here, you feed these signals into Prometheus and set alerts on error rate and latency thresholds.

Know if a deploy made things worse

Deployments are high-risk events. You change code, and suddenly nothing works the way it did five minutes ago. The question every team asks: did our deploy break something?

Without monitoring, you rely on manual testing, staged rollouts, and hope. With monitoring, you have a dashboard that tells you immediately whether a deploy improved or degraded production performance.

Imagine your team deployed a new feature to optimize database queries. The code looks solid in review. Tests pass. You deploy to production with a 10% canary rollout. For the first few minutes, everything seems fine. Then you glance at your Grafana dashboard and see error rates jump from 0.1% to 2% in the canary group. The new code is hitting an edge case in your data that staging didn't catch. You roll back in seconds. The deploy never reaches 100% of traffic. This is monitoring working as intended.

To make this work:

  • Tag your metrics with deployment info—Include the git commit hash or semantic version in your metrics so you can correlate spikes with specific deployments.
  • Set up dashboards per service—Error rate, latency, and custom business metrics, all visible at a glance before and after a deploy.
  • Define success criteria before you deploy—Error rate should stay below 0.5%, P99 latency under 500ms. Have these thresholds agreed on before you ship.

In Prometheus, labels let you tag metrics by version:

yaml
http_request_duration_seconds_bucket{le="0.5",service="checkout",version="v2.3.1"} 4523
http_request_duration_seconds_bucket{le="1.0",service="checkout",version="v2.3.1"} 4890
http_requests_total{status="500",service="checkout",version="v2.3.1"} 12

With this tagging strategy, Grafana queries can filter by version and you instantly see the blast radius of a bad deploy. Rollback becomes a data-driven decision, not a gut feeling.

Plan capacity with real data

How do you know if you need to scale? Asking your CTO or looking at the bill isn't data-driven. Asking your monitoring system is.

Many teams scale reactively—they get paged at 2 AM because CPU hit 95%, they scramble to add capacity, and the incident resolves. A month later, it happens again. With monitoring and trending, you scale proactively. You see that every Monday traffic grows by 30%. You provision capacity Friday before Monday arrives.

Real-world example: A SaaS platform with a ride-hailing feature noticed that database connections spiked every evening between 6 PM and 8 PM when commuters used the app. Without monitoring, they'd either over-provision permanently (wasteful) or handle the spike with degraded performance (bad UX). With monitoring, they plotted connection pool usage over time, identified the pattern, and configured auto-scaling policies that match demand.

The metrics that guide capacity planning:

  • CPU and memory usage—Track trends, not just current state. A gradual climb over weeks tells you growth is real.
  • Queue depth—If you have background jobs or message queues, how fast can you process? If depth grows, you need more workers.
  • Resource utilization by service—One microservice might hit 80% CPU while others sit at 20%. Scale what's actually constrained.
  • Cost per transaction—As you optimize, track whether scaling choices improve efficiency. More instances doesn't always mean better cost.

Document these trends in a capacity planning dashboard. Once every quarter, review it with the team. The data answers "do we need bigger instances?" or "do we need to optimize the code?" far better than guesses.

Shorter incidents with better signals

Incidents happen. The question is how quickly you resolve them. Poor monitoring turns a five-minute problem into a thirty-minute firefight because nobody knows where to look.

Consider a distributed system: API gateway, three microservices, a PostgreSQL database, and Redis. A customer reports "the app is slow." Where do you start? Without monitoring, you SSH into boxes, tail logs, check database slow query log, all while the customer is frustrated. With good monitoring, your dashboard immediately shows you which service is the bottleneck. Was the slow query in the user service? Did Redis eviction spike? Is the gateway hitting rate limits on an external API call?

Great observability follows the three pillars:

  • Metrics—Aggregated, queryable data (Prometheus). "What's the 99th percentile response time for the payment service?"
  • Logs—Raw event data (ELK, CloudWatch). "Show me every error from the checkout service in the last hour."
  • Traces—Request flow across services (Jaeger, Datadog). "Trace request ID abc123 through all services and show me where it slowed down."

During an incident, you need all three. Metrics tell you there's a problem. Logs and traces tell you why. The faster you traverse from "something is wrong" to "here's the root cause," the faster you resolve.

Set up alert routing so the right team gets paged immediately. A database alert goes to the data team, an API gateway alert goes to platform. Wasting time on alert fatigue or waiting for the right person to join a call adds minutes to MTTR (mean time to resolution).

Common mistakes

Monitoring everything without a plan. Teams sometimes instrument every variable, every function, every query, and end up with so much noise that real signals disappear. Start small. Pick the critical paths and business metrics that matter. Add more as you identify gaps.

Alerts without actionable context. An alert that says "CPU at 85%" doesn't tell you what to do. Is that bad? Are you auto-scaling? Better alerts say "checkout service CPU at 85% and auto-scaling is maxed out; investigate query performance." The team needs to know what action to take.

Ignoring observability in development. Some teams ship code to production and only then add monitoring. By then, you've lost weeks of understanding how the system actually behaves. Instrument from day one, even in development. Your local Prometheus setup doesn't need to be production-grade, but it should exist.

Monitoring in silos. Platform teams own Prometheus, frontend teams own RUM (real user monitoring), data teams own their own dashboards. These should be connected. A frontend performance degradation might be caused by backend latency. Silos hide these relationships.

Key takeaways

  • Monitoring gives you visibility into production behavior. Without it, you discover problems from users, not from data.
  • Correlate deploys with metric changes so you know if a code change improved or degraded performance.
  • Use real trends to plan capacity proactively instead of reacting to 3 AM pages.
  • Combine metrics, logs, and traces so incident response is fast and surgical, not blind guessing.
  • Start with critical paths and business metrics, not everything. Better to have a few reliable signals than noise.
  • Invest in observability early. It's cheaper to understand a system while building it than to retrofit monitoring after it breaks in production.
ShareXLinkedInReddit

Found this useful? Support the writing.

Buy me a coffee
Written by
Expona Admin
Site administrator

Comments

Loading comments…

Leave a comment

Comments are reviewed before they appear. Be kind and stay on topic.

Related articles