Monitoring Docker hosts: real-world lessons and examples

Introduction

Most teams running Docker on plain servers—whether on-premise or cloud instances—discover the same painful truth: container metrics alone don't paint the full picture. You need monitoring Docker hosts: real-world lessons and examples that show how to instrument both the containers and their underlying infrastructure together, because a healthy container can fail silently when the host runs out of disk space or exhausts file descriptor limits.

We've worked through enough production incidents across e-commerce and SaaS deployments to know where monitoring gaps typically hide. This guide walks through the concrete setup that catches the problems before they cascade: cAdvisor for container visibility, Node Exporter for host metrics, and the specific alerts that actually matter.

cAdvisor for container metrics

cAdvisor (container Advisor) runs as a daemon and exposes fine-grained container resource usage through a Prometheus endpoint. Unlike querying the Docker API directly, cAdvisor aggregates CPU, memory, network, and I/O metrics in a format Prometheus scrapes cleanly.

In practice, you'll run cAdvisor as a container itself. Here's a typical setup:

version: '3.8'
services:
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    ports:
      - "8080:8080"
    privileged: true
    devices:
      - /dev/kmsg
    restart: unless-stopped

The key insight here: cAdvisor needs access to the root filesystem and Docker socket to observe container behavior. It exposes metrics like container_cpu_usage_seconds_total, container_memory_usage_bytes, and container_network_transmit_bytes.

Once running, add cAdvisor to your Prometheus scrape configuration:

scrape_configs:
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['localhost:8080']
    relabel_configs:
      - source_labels: [__address__]
        regex: '([^:]+)(?::\d+)?'
        replacement: '${1}'
        target_label: instance

A real-world scenario: a Flask microservice starts consuming more memory than allocated. With cAdvisor, you see container_memory_usage_bytes climbing in real time. Without it, you only notice when the container OOM-kills and restarts. The difference is the ability to debug before production impact.

One common issue teams face: cAdvisor runs on the host's Docker daemon, so if the daemon itself crashes or becomes unresponsive, cAdvisor stops reporting. This is why you also need Node Exporter monitoring the host layer independently.

Node Exporter for the host

Node Exporter collects system-level metrics: CPU, memory, disk, filesystem, network interfaces, and kernel stats. It's essential because container metrics don't tell you if the host kernel is throttling processes, or if the node is running out of inodes.

Deploy Node Exporter as a privileged container binding to the host network:

version: '3.8'
services:
  node-exporter:
    image: prom/node-exporter:latest
    network_mode: host
    pid: host
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
      - '--collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$$'
    ports:
      - "9100:9100"
    restart: unless-stopped

The command-line flags here matter. By default, Node Exporter tries to mount every filesystem it finds, including things like /sys/kernel/debug that can hang. The exclude patterns prevent that friction.

Add Node Exporter to Prometheus:

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

In a real deployment, one team running a Redis cache noticed their queries slowing down mysteriously. The containers appeared healthy, but Node Exporter revealed node_context_switches_total spiking: the host was CPU-bound from unrelated batch processing. The container metrics would never have shown that. Combining both layers—container and host—revealed the root cause within an hour instead of days.

Restart counts

A container restart is often the symptom you're trying to detect. Prometheus exposes this through cAdvisor, but the metric is subtle: container_last_seen jumps backward in time when a container restarts.

A more reliable approach is to scrape Docker stats directly or query the Docker API. However, a practical alternative is to track container_start_time_seconds from cAdvisor. Each time a container starts fresh, this timestamp resets. You can alert when it's younger than it should be:

groups:
  - name: container_restarts
    interval: 30s
    rules:
      - alert: ContainerRestartingFrequently
        expr: |
          rate(container_last_seen{container_label_com_docker_compose_service!=""}[5m]) > 0
        for: 2m
        annotations:
          summary: "Container {{ $labels.container_label_com_docker_compose_service }} on {{ $labels.instance }} restarted"
          description: "Container has restarted in the last 5 minutes. Check logs immediately."

The reality: restart counts are noisy in dev environments with rolling updates, but critical in production. Set your alert thresholds conservatively—we typically fire on two restarts in 10 minutes for stateful services, and one restart in 15 minutes for stateless API servers. Each team's SLA is different, but the principle is the same: repeated restarts indicate a deeper problem (memory leak, segfault, misconfiguration).

Disk usage for images and volumes

Docker image and volume disk usage is a classic blind spot. Containers run fine, but the host filesystem fills up—and suddenly every write operation fails. Node Exporter tracks filesystem usage via node_filesystem_avail_bytes and node_filesystem_size_bytes, but doesn't break down Docker-specific usage.

To get granular visibility, query Docker directly or use a dedicated exporter. A practical middle ground is to add a script that runs periodically and exports Docker disk usage as custom metrics:

#!/bin/bash
# Export Docker disk metrics
cat /var/lib/node_exporter/textfile_collector/docker_disk.prom

docker system df --format "{{.Type}}\t{{.Size}}\t{{.Reclaimable}}" | while read type size reclaimable; do
  echo "docker_disk_usage_bytes{type=\"$type\"} $size"
  echo "docker_disk_reclaimable_bytes{type=\"$type\"} $reclaimable"
done > /var/lib/node_exporter/textfile_collector/docker_disk.prom.tmp
mv /var/lib/node_exporter/textfile_collector/docker_disk.prom.tmp /var/lib/node_exporter/textfile_collector/docker_disk.prom

Enable the textfile collector in Node Exporter:

command:
  - '--collector.textfile.directory=/var/lib/node_exporter/textfile_collector'

Then create a Prometheus alert for disk pressure:

- alert: DockerDiskSpaceRunningOut
  expr: |
    (node_filesystem_avail_bytes{mountpoint="/var/lib/docker"} / node_filesystem_size_bytes{mountpoint="/var/lib/docker"}) < 0.15
  for: 5m
  annotations:
    summary: "Docker filesystem on {{ $labels.instance }} below 15% free"
    description: "Run 'docker system prune' to reclaim space from unused images and containers."

A real scenario from an e-commerce platform: build pipelines were pushing new images to a server daily, but old images weren't being cleaned up. After six months, the 100 GB partition was 94% full. Prometheus showed the general filesystem was full, but didn't indicate the source was Docker images. With explicit Docker disk metrics, you know exactly where to look and what action to take.

Common mistakes

• Running cAdvisor and Node Exporter without persistence backups — Both exporters hold in-memory state. If they restart, historical metrics gap. Use a long scrape interval (15–30s) and increase Prometheus retention to 15+ days to smooth over brief restarts.
• Alerting on raw disk usage instead of free space ratio — A 100 GB disk and a 10 TB disk have very different tolerance for 20 GB used. Always alert on percentage free or as a ratio: (available_bytes / total_bytes) < threshold.
• Ignoring container restart patterns — A container restarting once a week might be scheduled maintenance; restarting three times per hour is a production incident. Baseline your expected restart frequency before alerting.
• Forgetting to label containers appropriately — If cAdvisor reports 50 containers and you can't identify which service each one belongs to, your alerting becomes useless. Enforce consistent Docker labels in your compose files or Kubernetes manifests.
• Not monitoring the monitoring tools themselves — If cAdvisor or Node Exporter becomes unresponsive, your visibility vanishes. Add health checks and scrape timeout settings: scrape_timeout: 10s catches hung exporters quickly.

Key takeaways

• cAdvisor gives you per-container CPU, memory, and I/O metrics; Node Exporter provides the host-level context you need to identify kernel-level constraints.
• Container restart tracking requires either parsing cAdvisor timestamps or querying the Docker API directly; raw restart counts are noisy, so baseline your thresholds per service.
• Docker disk usage (images, layers, volumes) is tracked separately from general filesystem usage; use a textfile collector script or dedicated Docker exporter to avoid blind spots.
• Always alert on ratios (percentage free, time-since-last-restart trends) rather than absolute values, and validate your alert thresholds in dev before production deployment.
• Combine container and host metrics in a single Grafana dashboard to correlate issues: when CPU climbs at the container level, cross-reference with node context switches and load average to pinpoint the root cause.