Dashboards as code: Monitoring best practices explained

Why treating dashboards as code matters

Your monitoring infrastructure grows fast. One week you have five dashboards; three months later you have fifty, and half of them are stale or inconsistent. Dashboards as code: monitoring best practices explained is the difference between a fragile, manually-managed mess and a reproducible, auditable system that scales with your team. When your dashboards live in version control alongside your infrastructure, you gain visibility into every change, enable code review for monitoring decisions, and eliminate the painful process of manually recreating dashboards after infrastructure incidents.

Without dashboards as code, you're stuck clicking through the Grafana UI, manually configuring panels, and hoping that the dashboard you created last sprint still makes sense to everyone else. With it, you have a single source of truth, Git history that explains why a metric matters, and the ability to spin up identical environments in seconds. Let's walk through how to build this capability into your monitoring workflow.

Provisioning: Load dashboards from files at startup

Grafana's provisioning system lets you define dashboards in YAML or JSON files that Grafana loads automatically when it starts. This is your foundation. Instead of manually creating dashboards in the UI, you define them once in a file, commit it to Git, and every Grafana instance in your environment loads the same configuration.

Here's how provisioning works in practice. You create a provisioning directory (typically /etc/grafana/provisioning/dashboards) and tell Grafana where to find it. Grafana watches that directory and periodically syncs its state to match the files. If you add a new dashboard file, Grafana provisions it. If you delete a file, Grafana removes the dashboard. This closed-loop approach means your filesystem and Grafana stay in sync.

Let's set up a simple example. First, configure Grafana to use provisioning:

# /etc/grafana/provisioning/dashboards/dashboards.yaml
apiVersion: 1

providers:
  - name: 'Default'
    orgId: 1
    folder: 'Team Dashboards'
    type: file
    disableDeletion: false
    editable: true
    options:
      path: /var/lib/grafana/dashboards

This tells Grafana to load all dashboard files from /var/lib/grafana/dashboards into the "Team Dashboards" folder. The editable: true flag lets you edit dashboards in the UI (though you should version-control changes). The disableDeletion: false ensures that deleting a file removes the dashboard from Grafana.

In your Dockerfile or deployment manifest, mount your dashboard files into the container:

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboards
data:
  api-performance.json: |
    {
      "dashboard": {
        "title": "API Performance",
        "panels": [...]
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
spec:
  template:
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:latest
        volumeMounts:
        - name: dashboards
          mountPath: /var/lib/grafana/dashboards
      volumes:
      - name: dashboards
        configMap:
          name: grafana-dashboards

Now when Grafana starts, it automatically loads your dashboards. No manual UI clicks needed. If you need to add a dashboard, you update the ConfigMap, redeploy, and Grafana picks it up on restart.

JSON in Git: Review dashboard changes in pull requests

Once your dashboards are in provisioning files, the next logical step is storing them in Git. This is where you gain enormous value: every dashboard change becomes reviewable, auditable, and reversible.

When you export a dashboard from Grafana, you get a JSON blob. It's verbose—hundreds of lines—but it's complete. Every panel, every query, every color setting is encoded. Store this JSON in your Git repository, ideally alongside your application code or in a dedicated monitoring repository.

.
├── monitoring/
│   ├── dashboards/
│   │   ├── api-performance.json
│   │   ├── infrastructure.json
│   │   └── business-metrics.json
│   ├── alerts/
│   │   └── production-alerts.yaml
│   └── provisioning/
│       └── dashboards.yaml
├── application/
│   └── src/...
└── README.md

This structure keeps monitoring configuration close to the code that generates the metrics. When a developer adds a new endpoint, they can add corresponding dashboard panels in the same pull request. The team reviews the dashboard changes just like they review code changes: checking that the query logic is sound, that panel titles are clear, and that the visualization type matches the data.

A typical pull request review flow looks like this:

1. Developer adds a new query to track cache hit rates and commits the updated dashboard JSON to Git.
2. In the pull request, reviewers can diff the JSON—though it's dense, they can spot the new query and verify it's querying the right metric.
3. A CI pipeline validates the JSON schema and optionally runs it against a test Prometheus to confirm the query returns data.
4. Once approved and merged, the updated dashboard is deployed via Kubernetes or your deployment tool, and Grafana provisions it automatically.

The key advantage: if a dashboard change causes problems, you revert the Git commit and redeploy. You have a full audit trail of who changed what and when.

Grafonnet or Terraform: Generate dashboards programmatically

Manually editing 500-line JSON dashboards is error-prone and tedious. This is where programmatic generation comes in. Tools like Grafonnet (a Jsonnet library for Grafana) or Terraform (with the grafana provider) let you write dashboards in a higher-level language, reducing boilerplate and enabling reuse.

Grafonnet is a Jsonnet library that wraps Grafana's JSON API with a cleaner syntax. Instead of writing raw JSON, you write Jsonnet code that generates JSON. Here's a simple example:

// dashboards/api-performance.jsonnet
local grafana = import 'grafonnet/grafana.libsonnet';
local dashboard = grafana.dashboard;
local row = grafana.row;
local panel = grafana.panel;
local target = grafana.target;

dashboard.new(
  title='API Performance',
  schemaVersion=16,
  timezone='browser',
  panels=[
    row.new(title='Request Metrics'),
    panel.graph(
      title='Request Duration (p99)',
      dataSource='Prometheus',
      targets=[
        target.prometheus(
          expr='histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))',
          legendFormat='{{endpoint}}'
        ),
      ],
    ),
    panel.singlestat(
      title='Error Rate',
      dataSource='Prometheus',
      targets=[
        target.prometheus(
          expr='rate(http_requests_total{status=~"5.."}[5m])',
        ),
      ],
    ),
  ],
)

Run jsonnet dashboards/api-performance.jsonnet -o dashboards/api-performance.json, and you get a valid Grafana dashboard JSON file. The Jsonnet code is much shorter and more maintainable than raw JSON. You can define helper functions, import shared libraries, and reuse panel definitions across dashboards.

Alternatively, use Terraform to manage dashboards as infrastructure:

resource "grafana_dashboard" "api_performance" {
  config_json = jsonencode({
    dashboard = {
      title       = "API Performance"
      description = "Metrics for production API endpoints"
      panels = [
        {
          title = "Request Duration (p99)"
          type  = "graph"
          targets = [
            {
              expr = "histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))"
            }
          ]
        }
      ]
    }
  })
}

Terraform integrates dashboards into your infrastructure-as-code pipeline. You can manage Grafana dashboards, data sources, alert rules, and notification channels all in the same Terraform config. The advantage is unified version control and a single deployment workflow.

For most teams, Grafonnet offers a good balance: it's language-agnostic (any tool can run Jsonnet), it produces valid Grafana JSON, and it scales well as your monitoring grows.

Avoid drift: Edit in code, not only in the UI

Here's the trap many teams fall into: they set up provisioning and Git, but continue editing dashboards in the Grafana UI for "quick fixes." Over time, the UI state diverges from Git. You have a dashboard that works in production but doesn't match your repository. When you redeploy, it reverts to the old version. This is drift, and it defeats the purpose of dashboards as code.

Prevent drift with two strategies: policy and tooling. On the policy side, establish a team norm: all dashboard changes go through Git. If you need to tweak a dashboard, you export the updated JSON from Grafana, commit it, open a pull request, and deploy it. This takes discipline but pays dividends.

On the tooling side, make the UI changes harder and the code path easier. Set editable: false in your provisioning config to disable UI editing entirely:

providers:
  - name: 'Default'
    orgId: 1
    folder: 'Team Dashboards'
    type: file
    disableDeletion: false
    editable: false  # UI editing disabled
    options:
      path: /var/lib/grafana/dashboards

With editable: false, Grafana prevents UI changes. Anyone who wants to modify a dashboard must export it, commit the JSON, update Git, and redeploy. It's slightly more friction, but it guarantees that your repository is always the source of truth.

Another approach: use a dashboard syncer that runs periodically and compares the running Grafana state to your Git repository. If someone edits a dashboard in the UI, the syncer detects the drift and either alerts the team or automatically reverts to the Git version. This acts as a guardrail.

For data sources, apply the same principle. Store data source definitions in provisioning files or Terraform, not as manual UI entries. This keeps your entire monitoring stack reproducible:

# /etc/grafana/provisioning/datasources/datasources.yaml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100

Common mistakes

Storing credentials in dashboard JSON: Dashboard JSON often ends up in Git. Never hardcode passwords or API keys. Use Grafana's secret management or environment variables. Reference secrets by name, not value.

Ignoring schema drift: Grafana updates often change the dashboard JSON schema. If you commit a dashboard from Grafana 9 and later deploy Grafana 10, schema mismatches can break provisioning. Always validate your JSON against the target Grafana version's schema before committing.

Over-provisioning dashboards: Provisioning is great for stable dashboards that your team actively maintains. But don't provision every one-off dashboard someone created during an incident. Use provisioning for core dashboards; let ad-hoc dashboards live in Grafana without version control. Document which dashboards should be provisioned.

Weak review practices: Code review is only as good as the reviewers. Train your team to review dashboard changes with the same rigor as application code. Check that queries are efficient, titles are clear, and visualizations match the data type. A bad dashboard can waste everyone's time.

Key takeaways

• Provisioning loads dashboards from files at Grafana startup, making them reproducible and deployable.
• Storing dashboards as JSON in Git gives you audit trails, pull request reviews, and easy rollbacks.
• Grafonnet and Terraform enable programmatic dashboard generation, reducing boilerplate and improving maintainability.
• Lock dashboards as read-only in Grafana (editable: false) to prevent UI drift and enforce code-based changes.
• Apply the same version-control discipline to data sources, alert rules, and notification channels—treat all monitoring configuration as code.
• Train your team to review dashboard changes during pull request review, just as they would application code.