Node exporter: what every Monitoring developer should know

By day 30 of any serious monitoring initiative, you'll have seen Prometheus scrape targets, Grafana dashboards light up, and alerts fire. But the foundation of everything starts at the host level. Node exporter is what every monitoring developer should know because it turns raw Linux server metrics into queryable time-series data. Without it, you're flying blind on CPU, memory, disk and network—the four pillars of host health.

In production, we've learned that Node exporter isn't just another exporter. It's the bridge between your infrastructure and observability. Getting it right early saves debugging hours later.

CPU, memory, disk and network: the core metrics

When you deploy Node exporter on a Linux host, it immediately exposes four metric categories that answer the most common operational questions:

• CPU: user, system, idle, iowait, steal time across all cores
• Memory: total, free, available, buffers, cached, swap
• Disk: read/write bytes and operations, queue depth, I/O time
• Network: bytes and packets in/out per interface, errors, dropped frames

Let's say your team gets a report that a batch processing job runs slower on Tuesday mornings. You log into the Prometheus UI and query node_cpu_seconds_total segmented by instance and mode. If you see iowait spiking while idle drops, the culprit isn't CPU contention—the disk can't keep up. Memory pressure (checked via node_memory_MemAvailable_bytes) tells you whether the kernel is swapping, which tanks performance on I/O-heavy workloads.

Here's a real scenario: a microservice team noticed their API response times degrading at 10 AM every weekday. They checked Node exporter metrics and found node_disk_io_time_seconds_total climbing steadily. Turned out a daily backup process on the same host was contending for disk I/O. They moved the backup to a dedicated machine, and the problem vanished. Without Node exporter collecting these metrics continuously, the diagnosis would have taken days.

Network metrics like node_network_receive_bytes_total and node_network_transmit_errors_total catch issues that cost real money—misconfigured load balancers sending traffic to the wrong interface, or a flaky switch dropping packets. We've seen teams waste hours on application-layer debugging before realizing the network card had 0.5% packet loss.

Filesystem usage and inodes

Disk space isn't just about gigabytes. The filesystem layer exposes two critical metrics: used capacity and inode exhaustion. Node exporter splits these cleanly, and you need both.

The metric node_filesystem_avail_bytes shows how much space is actually available to applications. Reserve blocks on ext4 and similar filesystems mean the raw free space (from df) differs from what apps can write. Prometheus scrapes this correctly; many monitoring setups don't.

Inodes are the hidden killer. A system can have 50 GB free but zero inodes and fail to create files. We've shipped Node exporter dashboards that graph node_filesystem_files_free per mountpoint. Watch the trend. When it drops below 10–15% on a production box, page someone—not at 2 AM when it hits zero, but proactively.

Here's a concrete Prometheus query that catches this before it's a disaster:

alert: InodeUsageHigh
expr: (node_filesystem_files - node_filesystem_files_free) / node_filesystem_files > 0.85
for: 5m
annotations:
  summary: "Inode usage > 85% on {{ $labels.instance }} {{ $labels.mountpoint }}"

This fires when inodes are 85% consumed, giving you a five-minute buffer before applications start failing mysteriously. The difference between this and an unmonitored filesystem running out of inodes is the difference between a Slack notification and a production incident.

Filesystem type matters too. node_filesystem_fstype lets you segment metrics by ext4, btrfs, NFS, or tmpfs. We've caught cases where temporary data accumulating in /tmp (often a tmpfs) filled up faster than anyone expected. Alerting specifically on node_filesystem_avail_bytes{mountpoint="/tmp"} with a lower threshold than / caught it in staging first.

Run on every host

Node exporter is lightweight—typically 10–30 MB memory, negligible CPU—and should run as a systemd service or container on every Linux machine in your estate. This includes load balancers, databases, caches, and API servers. In a heterogeneous environment, it's the one constant.

Here's a production-ready systemd unit file:

[Unit]
Description=Node Exporter
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter \
  --collector.netdev.device-exclude=^(veth|docker|br).*
  --collector.diskstats.ignored-devices=^(ram|loop|fd).*
  --web.listen-address=:9100
Restart=always
RestartSec=5s
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Note the --collector.netdev.device-exclude flag. In containerized environments, Docker and Kubernetes create ephemeral virtual network interfaces. Without filtering, your metrics cardinality explodes and Prometheus starts struggling. We've seen single hosts expose 500+ unique network interface metrics when unfiltered. With exclusions, it's 2–3. Same goes for disk stats: exclude virtual devices or your cardinality skyrockets.

Deployment via infrastructure-as-code (Ansible, Terraform, Docker) means every host gets it automatically. No special handling, no exceptions. One team we worked with had Node exporter running on 95% of their fleet but missed a database server in the corner. When that server hit 99% CPU and caused cascading failures upstream, they had zero visibility. The missing 5% cost them four hours of blind debugging.

Version consistency matters. Pin Node exporter to the same version across all hosts. Collector behavior can change between releases—what worked on 1.5.0 might emit slightly different metrics on 1.6.0. Drift causes silent metric inconsistencies that show up as data quality issues months later.

Import a community dashboard to start

Rather than build dashboards from scratch, the Grafana community publishes battle-tested Node exporter dashboards. The most popular is dashboard 1860 ("Node Exporter for Prometheus"). It covers CPU, memory, disk, network and filesystem in a single pane of glass.

To import it:

1. Open Grafana, go to Dashboards → Import
2. Enter dashboard ID 1860
3. Select your Prometheus data source
4. Click Import

You'll immediately see charts for each host's CPU breakdown (user, system, idle), memory usage including buffers and cache, disk I/O operations per second, and network throughput. Most teams run this for a week out of the box before customizing. It answers 80% of the questions your ops team will ask.

However, don't just import and forget. Customize it for your environment. Remove panels you don't care about (maybe network interfaces you filtered out). Add panels for metrics your team cares about—swap usage if you've disabled it, specific mount points if you have unusual storage layouts. Add drill-down links so clicking a CPU panel takes you to a per-process breakdown dashboard.

We've also seen teams create a first-page dashboard with just the top-line metrics: CPU utilization, memory usage, disk usage, and network errors. Green means the host is healthy. Red or orange means investigate. This sits in a NOC or ops channel, updated every 30 seconds. Simple, glanceable, effective.

Common mistakes

Forgetting to exclude virtual interfaces: Docker, Kubernetes, and virtual machines create hundreds of ephemeral network interfaces. Collect them all and your Prometheus server melts under cardinality load. Use the --collector.netdev.device-exclude and --collector.diskstats.ignored-devices flags from day one.

Not alerting on filesystem space: Teams often assume their alerting system will catch disk full. It won't if the metric itself can't be scraped because the exporter can't write temp files. Alert when filesystem reaches 80–85%, not 100%.

Mixing Node exporter versions: Upgrading one host to Node exporter 1.6 while others run 1.5 introduces metric inconsistencies. Stick to one version per environment and upgrade all hosts in a maintenance window.

Ignoring the textfile collector: Node exporter's --collector.textfile.directory flag lets you drop custom metrics into a directory and have them scraped automatically. Scripts can write host-specific metadata (deployment version, environment tags) as text files. Most teams overlook this and miss out on context.

Over-relying on dashboard defaults: The community dashboard is a starting point. Customize it. Your team's alert thresholds, your specific storage topology, your network segmentation—these differ from the generic dashboard. Invest in making it yours.

Key takeaways

• Node exporter exposes CPU, memory, disk I/O and network metrics for every Linux host. These four categories answer 90% of host health questions.
• Filesystem metrics include both capacity and inode usage; alert on inodes before they hit zero.
• Deploy Node exporter consistently across all hosts using infrastructure-as-code; use device exclusions to control metric cardinality.
• Import dashboard 1860 to get started, then customize it for your environment, specific thresholds and team workflows.
• Exclude virtual interfaces, alert early on disk space, keep versions consistent, and treat the community dashboard as a template, not gospel.