Securing Prometheus and Grafana: lessons from production

The Silent Risk in Your Monitoring Stack

Your Prometheus and Grafana setup is probably the most sensitive system in your infrastructure—yet it often receives less security attention than the applications it monitors. We've seen teams pour resources into securing databases and APIs while leaving their metrics endpoint exposed to anyone on the network, or allowing every new hire read access to dashboards that reveal customer behavior, system capacity, and business metrics. Securing Prometheus and Grafana: lessons from production is not optional; it's foundational to protecting your operational visibility and your business intelligence.

In this article, we'll walk through the practical security measures we've deployed across e-commerce platforms, trading systems, and SaaS services. These aren't theoretical best practices—they're patterns born from incidents, audits, and lessons learned when things went wrong. Let's start with the most critical layer: authentication.

Authentication: The First Gate

Prometheus has no built-in authentication. By default, any service or person with network access to port 9090 can scrape metrics, query the API, and read sensitive data: database connection strings in labels, customer counts, request latencies that hint at business volume, or infrastructure topology embedded in metric names and job tags.

The straightforward solution is to never expose Prometheus directly to untrusted networks. Here's how we've handled it in production:

Private Network Deployment

In most setups, Prometheus should live on a private subnet, accessible only to services that need to query it: Grafana, alerting systems, and internal dashboards. On AWS, that means a private subnet with no public IP. On Kubernetes, a service in the cluster network without external LoadBalancer. Your on-call engineer accesses it via VPN or bastion host, never over the public internet.

apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitoring
spec:
  type: ClusterIP
  selector:
    app: prometheus
  ports:
    - port: 9090
      targetPort: 9090

This service is internal-only. No external traffic reaches it. Team members who need to query Prometheus do so through Grafana, which sits behind its own authentication layer.

Reverse Proxy with Basic Auth

If you must expose Prometheus (for integration with third-party monitoring tools, for example), put it behind a reverse proxy that enforces HTTP Basic Authentication or OAuth. Nginx is straightforward:

server {
    listen 443 ssl;
    server_name prometheus.internal.example.com;

    ssl_certificate /etc/nginx/certs/prometheus.crt;
    ssl_certificate_key /etc/nginx/certs/prometheus.key;

    auth_basic "Prometheus";
    auth_basic_user_file /etc/nginx/.htpasswd;

    location / {
        proxy_pass http://localhost:9090;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Generate the password file with htpasswd -c /etc/nginx/.htpasswd username. Rotate credentials regularly and use strong passwords (or better yet, use OAuth via a proxy like OAuth2-Proxy).

Grafana SSO: Authentication at Scale

Grafana supports multiple authentication backends: OAuth 2.0 (GitHub, Google, Generic), SAML 2.0, LDAP, and basic auth. In production environments with dozens or hundreds of team members, SSO is non-negotiable. It keeps credentials centralized, allows you to revoke access instantly when someone leaves, and integrates with existing identity infrastructure.

OAuth 2.0 with GitHub

For many engineering teams, GitHub is already the source of truth for access control. Grafana can authenticate users via GitHub OAuth:

auth.github:
  enabled: true
  allow_sign_up: true
  client_id: YOUR_CLIENT_ID
  client_secret: YOUR_CLIENT_SECRET
  scopes: user:email,read:org
  auth_url: https://github.com/login/oauth/authorize
  token_url: https://github.com/login/oauth/access_token
  api_url: https://api.github.com/user
  team_ids: 1234567
  allowed_organizations: your-company

The team_ids field restricts access to members of specific GitHub teams. New hires added to the team gain access automatically; removed members lose it on their next session refresh.

SAML for Enterprise

Larger organizations often use SAML through Okta, Azure AD, or similar providers. SAML integrates with your organization's user directory and enforces corporate policies like multi-factor authentication:

auth.saml:
  enabled: true
  certificate_path: /etc/grafana/saml.crt
  private_key_path: /etc/grafana/saml.key
  metadata_url: https://your-idp.okta.com/app/YOUR_APP_ID/sso/saml/metadata
  assertion_attribute_login: email
  assertion_attribute_name: name
  assertion_attribute_groups: groups

One significant advantage: SAML can push group membership to Grafana automatically, so your Okta groups map directly to Grafana teams with preset permissions.

TLS: Encrypting Data in Transit

Metrics contain operational secrets: slow query details, failed authentication attempts, customer IDs in labels, and capacity metrics. All traffic between Prometheus and scrape targets, between Prometheus and Grafana, and between users and Grafana must be encrypted. This is non-negotiable in production.

Prometheus Scrape Configuration

Enable TLS for all scrape targets, not just external ones. Internal traffic is often overlooked but is just as important if you're in a shared environment or multi-tenant infrastructure:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'application'
    scheme: https
    tls_config:
      ca_file: /etc/prometheus/ca.crt
      cert_file: /etc/prometheus/client.crt
      key_file: /etc/prometheus/client.key
      server_name: app.internal.example.com
    static_configs:
      - targets: ['app.internal.example.com:9090']

  - job_name: 'database'
    scheme: https
    tls_config:
      insecure_skip_verify: false
    static_configs:
      - targets: ['db.internal.example.com:9187']

Notice insecure_skip_verify: false. Never set this to true in production unless you have no other choice. Always validate certificates.

Grafana Behind TLS

Grafana should always run behind TLS. In Kubernetes, use an ingress controller with a valid certificate:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana
spec:
  tls:
    - hosts:
        - grafana.example.com
      secretName: grafana-tls
  rules:
    - host: grafana.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: grafana
                port:
                  number: 3000

Use Let's Encrypt or your organization's certificate authority. Renew certificates automatically using cert-manager or similar tooling.

Least Privilege: Viewer Roles by Default

Grafana's permission model is powerful but requires intentional configuration. By default, all users are Admins if you're not careful, or they can see all dashboards if you don't set folder-level permissions. We follow a simple rule: start with Viewer role, grant Admin or Editor permissions only to the team that owns monitoring.

User Provisioning with Roles

When new users authenticate via OAuth or SAML, Grafana assigns them a default role. Set it to Viewer:

auth.oauth:
  auto_login: false
  role_attribute_path: contains(groups[*], 'ops-team') && 'Editor' || 'Viewer'
  role_attribute_strict: true

This expression says: if the user is in the 'ops-team' group, assign Editor; otherwise, Viewer. Viewers can read dashboards and alerts but cannot modify them or access admin settings.

Dashboard and Folder Permissions

Even with roles, use folder-level permissions to compartmentalize access. Finance dashboards should not be visible to engineers; database internals should not be visible to frontend teams:

API Tokens with Expiration

If external services need to query Grafana (alerting systems, automation scripts), use API tokens instead of user credentials. Tokens should be scoped to a specific role and expire periodically:

curl -X POST https://grafana.example.com/api/auth/keys \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{
    "name": "alerting-service",
    "role": "Editor",
    "secondsToLive": 2592000
  }'

This creates a token valid for 30 days (2,592,000 seconds). Rotate it before expiration. Store it in your secrets management system (HashiCorp Vault, AWS Secrets Manager, etc.).

Common Mistakes

• Exposing Prometheus without auth. We've seen teams leave Prometheus open to the entire cloud network or even the public internet. This is typically discovered during a security audit or incident response, not proactively. Always assume Prometheus is a high-value target and protect it accordingly.
• Using HTTP instead of HTTPS. TLS adds complexity but is not optional. Self-signed certificates are acceptable for internal services if certificate validation is still enforced.
• Making all users Admins in Grafana. It's tempting to skip the permission model early on, but it scales badly. Implement least privilege from day one, even if your team is small.
• Storing secrets in metric labels. Prometheus labels are queryable and appear in dashboards, so they're public (within your organization). Never put API keys, passwords, or PII in labels. Use structured logging or a separate audit system instead.
• Ignoring OAuth/SAML configuration. Basic auth works for a single user or a handful of test accounts, but it doesn't scale and makes offboarding error-prone. SSO is worth the setup cost.

Key Takeaways

• Keep Prometheus on a private network. Expose it only through authenticated reverse proxies if external access is required.
• Use OAuth 2.0 or SAML for Grafana authentication. Integrate with your identity provider to ensure access follows the same lifecycle as employee onboarding and offboarding.
• Encrypt all traffic with TLS. Validate certificates and rotate them automatically. This applies to scrape targets, Grafana UI, and any inter-service communication.
• Apply least privilege by default. Start users as Viewers, use folder-level permissions to compartmentalize data, and rotate API tokens regularly.
• Audit access regularly. Monitor who's connecting to Prometheus and Grafana, and review permission assignments quarterly or after organizational changes.