Designing dashboards people use: lessons from production

Designing dashboards people use: lessons from production

We've all stared at a dashboard that looks impressive but tells us nothing about whether our system is actually working. After years of oncall rotations and post-mortems, we've learned that the difference between a dashboard people actually check and one that collects dust is discipline in design. Clarity over decoration isn't just a principle—it's the difference between catching a problem at 2 AM and discovering it in production during business hours.

Start with the four golden signals

Before you open Grafana and start building panels, anchor yourself to the four golden signals: latency, traffic, errors, and saturation. These four metrics tell you almost everything about service health. If you're tempted to add CPU usage, memory, disk I/O, or custom business metrics to your core dashboard, stop. Those belong in secondary dashboards.

Latency answers "how fast?" Traffic answers "how much?" Errors answers "what's broken?" Saturation answers "how close to the limit?" If all four are green, your service is healthy. If any one is red, you have a problem to investigate.

In practice, latency usually means p99 or p95 response time. Traffic is requests per second or transactions per minute. Errors is the error rate—either absolute count or percentage. Saturation is how full your most constrained resource is: connection pool exhaustion, database query queue depth, or compute utilization.

We built a service that processes payment events. The first dashboard we created had seventeen panels tracking everything from JVM garbage collection frequency to cache hit ratios. It looked thorough. It was useless. When an incident happened, we couldn't find the signal in the noise. We rebuilt it with four panels: p99 latency, events per second, error rate, and queue depth. Oncall engineers started actually using it.

The golden signals give you a contract: if you're looking at these metrics and they're nominal, the service is working. Everything else is context.

One dashboard per service

A natural instinct is to create a master dashboard that shows your entire system. Resist it. We've learned that teams need a focused view of their own service, not a god's-eye view of everything.

One service should have one dashboard. If your service has database, cache, and API layers, they get one dashboard. You don't create separate dashboards for "database health" and "API health"—you create one dashboard that shows the health of the service as a whole. The distinction matters because oncall engineers need to triage in seconds, not navigate through a dashboard hierarchy.

Your dashboard should answer: "Is this service healthy right now?" Everything on it should serve that question. If you find yourself adding panels that only matter during specific operational procedures, those belong on a separate "troubleshooting" or "deep dive" dashboard linked from the main one.

A common pattern we've used successfully is a two-tier structure. The service dashboard shows the four golden signals plus context about dependencies: "is the database responding normally? is the cache available? are we getting rate-limited by downstream APIs?" A separate dashboard, linked from the main one, contains deep operational metrics: query performance histograms, connection pool utilization, disk I/O patterns, deployment status. Oncall uses the first one constantly. The second one gets visited when something is actually broken.

Naming matters too. Call it what people call the service. If your team says "the order service," the dashboard should be titled "Order Service" not "OrderServiceV2Prod" or "Backend Health - Orders."

Consistent units and colors

When you have multiple panels on the same dashboard, every design choice compounds. Inconsistent units force your brain to context-switch. Inconsistent colors make patterns harder to spot.

Establish conventions for your team. If you're measuring duration, always use milliseconds or always use seconds—not both. If you're measuring throughput, always use "per second"—not sometimes per second and sometimes per minute. If error rate is expressed as a percentage on one panel, it should be a percentage everywhere.

Color conventions matter for rapid scanning. Use the same color palette as other dashboards in your organization. A common approach: green for healthy (usually >70% utilization or <2% error rate), yellow for warning (usually 50-70% or 2-5%), red for critical (usually >70% or >5%). Define thresholds in your Grafana alert rules and apply them consistently.

We use ISO 8601 for all timestamps in logs and exported data, but dashboards show relative time ("5 minutes ago") for recent events and absolute time ("Jan 19 14:32 UTC") for historical context. This convention applies across all our dashboards.

For a payment service, we standardized on milliseconds for latency (never seconds, never microseconds), events per second for throughput, percentage for error rate, and dollar amounts for transaction sizes. Every panel uses these units. Every threshold is defined in these units. When a new engineer joins the team, they can read any panel without translation.

Graphs should use consistent axes too. If two panels show response time, both should span 0-1000ms even if one service typically runs at 50ms and the other at 300ms. The whitespace tells you something: it shows you how much headroom you have.

Link to logs and traces

A dashboard is triage. Once you've identified the problem using the golden signals, you need to investigate. The fastest way to investigate is to jump directly from the dashboard to logs or traces relevant to that time window.

In Grafana, this means using annotation links. When you see an error spike in the error rate panel, clicking it should take you to your logging system filtered to errors in that time window. When you see latency spike in the latency panel, clicking it should take you to your tracing system with traces from that period.

Most Grafana installations support this out of the box through data source linking. You can configure a panel to link to Loki (for logs) or Tempo (for traces) with appropriate filters:

panels:
  - title: "Error Rate"
    datasource: Prometheus
    links:
      - title: "View logs"
        url: "loki/explore?orgId=1&left=%5B%22logs%22,%7B%22expr%22:%22{service=%22order-service%22,%20level=%22error%22}%22%7D%5D"
        targetBlank: true

We set up Grafana links so that clicking the error rate panel jumps to our logging system showing errors from the past hour. Clicking the latency panel jumps to our tracing system filtered to the 99th percentile slowest traces. This turns investigation from "I see a problem" to "I see a problem and here's why" in seconds instead of minutes.

The linking also forces you to think about which metrics matter for which investigations. If you can't write a sensible log query based on your metric spike, that metric probably doesn't belong on your main dashboard.

Common mistakes

• Too many panels: If your dashboard doesn't fit on a laptop screen without scrolling, it's too big. Aim for six to eight panels maximum on your golden signals dashboard.
• Mixing time series and gauges without purpose: Time series (line graphs) show trends. Gauges show current state. Use gauges sparingly—they're often less useful than you think.
• Stale data without warning: If a metric stops reporting, your dashboard should show that visually. Configure "no data" behavior to make silence visible rather than hiding it.
• Thresholds that don't match reality: If your alert threshold is 80% CPU and your dashboard threshold is 75%, oncall gets confused. Keep alert rules and dashboard thresholds synchronized.
• Dashboards that only show the last hour: Configure your default time range based on your incident response window. For most services, 6 hours is a good default. 1 hour is usually too short to spot patterns.
• Missing context about dependencies: If your service depends on an external API, your dashboard should show whether that API is responding. Include dependency health as context, not as separate panels.

Key takeaways

• Build dashboards around the four golden signals: latency, traffic, errors, and saturation. If these are green, your service is healthy.
• Create one focused dashboard per service. Resist the urge to build master dashboards that show your entire system at once.
• Use consistent units and color conventions across all dashboards. This reduces cognitive load during incident response and makes patterns visible.
• Link every metric to logs or traces so that triage leads directly to investigation. The dashboard identifies the problem; the logs and traces explain why.
• Aim for simplicity. A dashboard people actually use is better than a dashboard that's technically thorough but never opened.