Grafana dashboard tutorial: a practical Monitoring guide

Introduction

By day 18 of your monitoring journey, you've likely got Prometheus scraping metrics and alerting rules in place. But raw metrics in a terminal are useless if your team can't see them at a glance. This Grafana dashboard tutorial: a practical Monitoring guide walks you through building a dashboard that transforms raw time-series data into actionable insights. We'll move from configuring your first data source through organizing panels into logical sections—everything you need to equip your team with visibility into system behavior.

Add a data source — Point Grafana at Prometheus

Every Grafana dashboard needs to know where its data lives. Before you create a single panel, you must configure Prometheus as a data source. This is the foundation: without it, your visualizations have nothing to display.

Open Grafana and navigate to Configuration → Data Sources (or use the gear icon in the sidebar). Click Add data source and select Prometheus. You'll see a form asking for the URL where Prometheus listens. In a typical development or staging environment, that's http://localhost:9090. In production, it's usually an internal service name like http://prometheus.monitoring:9090.

# Example docker-compose snippet for local development
services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin

Fill in the URL and set the scrape interval if you've customized it (usually 15s). Test the connection by clicking Save & Test. You should see "Data source is working." If you don't, check that Prometheus is running and reachable from the Grafana container.

Pro tip: In a Kubernetes cluster, use the DNS name of the Prometheus service rather than an IP. DNS is stable across pod restarts.

Create panels — Time series for rates, stat panels for current values

With your data source connected, it's time to build panels. Panels are the building blocks of a dashboard—each one visualizes a query against Prometheus. Grafana offers many panel types; for monitoring, you'll rely on two: time series (line charts over time) and stat panels (large numbers showing current values).

Time series panels are ideal for rates and trends. Let's say you want to track HTTP request rates. Create a new dashboard, click Add panel, and paste this query into the Metrics field:

rate(http_requests_total[5m])

This query uses Prometheus's rate() function to calculate requests per second over a five-minute window. The panel will draw a line for each unique label combination—one line per endpoint, status code, or method. Use the Legend options to control what labels appear in the legend. For example, set it to {{job}} - {{method}} to show the service name and HTTP method.

Customize the Y-axis label (e.g., "Requests/sec") and give the panel a meaningful title like "HTTP Request Rate." You can also add thresholds: if the rate drops below, say, 10 req/s, the background turns yellow. This helps teams spot abnormalities.

Stat panels show a single current value, perfect for dashboards. They're excellent for displaying things like error count, memory usage percentage, or the total number of active connections. Create another panel and choose the Stat panel type. Use a query like:

increase(http_requests_total{status=~"5.."}[5m])

This gives you the total 5xx errors in the last five minutes. Set the unit to "short" or "ops" as appropriate, and enable color thresholds: green for 0–10 errors, yellow for 10–50, red for 50+. Your team glances at the number and immediately understands the severity.

Real-world example: We built a dashboard showing HTTP rates by endpoint. Within a week, the team noticed that one API endpoint's rate was cut in half, leading us to discover a bug in the client library that had gone unnoticed.

Variables — Dropdowns for service and environment

Hardcoding service names or environment labels into queries limits reusability. Instead, use Grafana variables—dropdowns that dynamically populate your queries. A dashboard with variables scales across your entire platform with a single template.

Navigate to the dashboard's Settings (gear icon) and select Variables. Click New variable and set:

• Name: service
• Type: Query
• Data source: Prometheus
• Query: label_values(up, job)

This query fetches all unique job labels from the up metric—effectively, all your services. Set Multi-select to allow users to pick multiple services at once. Save the variable.

Now, in any panel, replace the hardcoded job name with $service:

rate(http_requests_total{job=~"$service"}[5m])

If $service is "api-gateway", the query expands to job=~"api-gateway". If it's "api-gateway|auth-service", it becomes job=~"api-gateway|auth-service". The regex pattern adapts automatically.

Create a second variable for environment the same way:

• Name: environment
• Query: label_values(up{job=~"$service"}, env)

Notice the nested reference: $service inside the query. This cascades—once you pick a service, the environment dropdown only shows environments for that service. This is powerful: it keeps your dashboard responsive and prevents meaningless combinations.

Organize — Rows for overview, errors, latency and resources

A 20-panel dashboard overwhelming your team defeats the purpose. Use rows to group related panels into sections. Each row collapses, so viewers find what they need without scrolling endlessly.

Create your dashboard with this structure:

1. Overview row: Service status (up/down), request rate, error rate at a glance.
2. Errors row: 5xx count, error rate percentage, top error types.
3. Latency row: p50, p95, p99 response times; outlier detection.
4. Resources row: CPU and memory usage; disk I/O; database connection pool.

To add a row in a modern Grafana version, click Add panel, then choose Row instead of a visualization type. Title it "Overview". Add your stat panels beneath it: one for "Up" count (using up metric), one for request rate, one for current error count.

For the Errors row, include a time-series panel showing error rate as a percentage:

(
  increase(http_requests_total{status=~"5.."}[5m])
  /
  increase(http_requests_total[5m])
) * 100

This shows errors as a percentage of all requests. Add a second panel with a heatmap or bar chart breaking down errors by type if you instrument that:

increase(http_requests_total{status=~"5..", error_type!=""}[5m]) by (error_type)

For the Latency row, use histogram quantiles. If your application exports http_request_duration_seconds as a histogram, calculate percentiles:

histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

Create three stat panels for p50, p95, and p99 (replace 0.95 with 0.5 and 0.99). Set the unit to seconds and format to two decimals. This gives your team a clear picture of user experience.

In the Resources row, graph CPU usage from your host exporter:

100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

And memory usage:

(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

Each panel shows a clear percentage. Use thresholds: green below 50%, yellow 50–80%, red above 80%.

Common mistakes

• Queries that are too narrow or broad: A query for a single host won't scale; a query for all services at once becomes unreadable. Use variables to strike the balance.
• Forgetting to add units: "15" is meaningless; "15 req/s" or "15 ms" tells the story. Always set panel units.
• No thresholds or color coding: A dashboard without visual cues requires reading every number. Use color to signal health at a glance.
• Over-cramming one dashboard: If you need to scroll horizontally or vertically for more than a few seconds, split into multiple dashboards. One for overview, one for deep dives per service.
• Ignoring stale data: A metric that hasn't updated in 10 minutes is stale. Set panel options to gray out or hide data older than your scrape interval plus a buffer.

Key takeaways

• Configure Prometheus as a Grafana data source first; test the connection before building panels.
• Use time-series panels for trends and rates; stat panels for current single values.
• Variables enable reusable, scalable dashboards that adapt to service and environment selection without duplicating queries.
• Organize panels into logical rows (Overview, Errors, Latency, Resources) to prevent cognitive overload.
• Apply units, thresholds, and color coding so the dashboard communicates health status instantly.
• Iterate: your first dashboard won't be perfect. Watch your team use it, gather feedback, and refine.