Monitoring cron jobs and batch work: a step-by-step guide

The silent killer of batch systems

Cron jobs and batch processes are the workhorses of most backend systems—they handle data imports, generate reports, clean up stale records, and sync external APIs. Yet they're often the last thing you monitor, only discovered to be broken when someone asks "why is this report three days old?" or "why haven't those old records been deleted?" Monitoring cron jobs and batch work: a step-by-step guide walks you through building alerts that catch failures before your team hears about them from customers.

Why cron jobs fly under the radar

Unlike HTTP services that fail visibly (error rates spike, latency shoots up), cron jobs often run silently. A scheduled task that runs once a day can be broken for 24 hours before anyone notices. If the process exits with code 1 or gets stuck in an infinite loop, your monitoring system won't see it—unless you explicitly instrument it. The job won't emit an error log you're searching for; it will simply not produce the results you expect.

That's where explicit monitoring comes in. We need to treat cron jobs as first-class citizens in our observability infrastructure, just like our API servers.

Push last success timestamp

The foundation of cron job monitoring is knowing when the job last completed successfully. Rather than waiting for the job to fail and hoping you catch it, we push a timestamp to our monitoring backend every time the job finishes without error.

Why push, not pull? Your monitoring system can't easily pull state from a cron job that only runs once a day. By contrast, the job itself knows when it succeeded and can report that fact immediately. This is a "push" model: the application sends metrics to Prometheus, rather than Prometheus scraping them.

Here's a concrete example: an invoice finalizer that runs every morning at 2 AM.

// Go example: invoice finalizer with last success push
package main

import (
	"fmt"
	"time"
	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promauto"
	"github.com/prometheus/client_golang/prometheus/push"
)

var (
	lastSuccessTimestamp = promauto.NewGauge(prometheus.GaugeOpts{
		Name: "invoice_finalizer_last_success_unix",
		Help: "Unix timestamp of the last successful invoice finalization",
	})
	invoicesProcessed = promauto.NewCounter(prometheus.CounterOpts{
		Name: "invoice_finalizer_invoices_total",
		Help: "Total invoices finalized",
	})
	durationSeconds = promauto.NewHistogram(prometheus.HistogramOpts{
		Name: "invoice_finalizer_duration_seconds",
		Help: "Time taken to finalize invoices",
		Buckets: []float64{1, 5, 10, 30, 60, 300},
	})
)

func finalizeInvoices() error {
	start := time.Now()
	
	// Your business logic here
	count, err := processInvoices()
	if err != nil {
		return fmt.Errorf("invoice finalization failed: %w", err)
	}
	
	// Record success
	lastSuccessTimestamp.Set(float64(time.Now().Unix()))
	invoicesProcessed.Add(float64(count))
	durationSeconds.Observe(time.Since(start).Seconds())
	
	// Push to Prometheus Pushgateway
	pusher := push.New("pushgateway:9091", "invoice-finalizer")
	pusher.Collector(lastSuccessTimestamp)
	pusher.Collector(invoicesProcessed)
	pusher.Collector(durationSeconds)
	
	return pusher.Push()
}

func main() {
	if err := finalizeInvoices(); err != nil {
		fmt.Printf("Error: %v\n", err)
		// Don't push success metrics on failure
		// The old timestamp will remain, triggering alerts
		panic(err)
	}
}

Notice that we only push metrics on success. If the job fails or panics, nothing is pushed, and the old timestamp remains in Prometheus. This is crucial: a missing update is itself a signal.

Alert when it is too old

Once we're pushing a last success timestamp, we set up an alert in Prometheus that fires when that timestamp hasn't been updated recently enough.

groups:
  - name: batch_jobs
    interval: 1m
    rules:
      - alert: InvoiceFinalizerStale
        expr: |
          time() - invoice_finalizer_last_success_unix > 90000
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Invoice finalizer last ran more than 25 hours ago"
          description: |
            The invoice finalizer should run daily around 2 AM.
            Last success was {{ ($value | humanizeDuration) }} ago.
            Check logs: kubectl logs -l job=invoice-finalizer --tail=200

The threshold of 90,000 seconds (25 hours) gives you a 1-hour buffer beyond the expected 24-hour cycle. The for: 5m clause prevents false positives from brief delays. If the job runs at 2:00 AM and finishes by 2:05 AM, you'll get an alert by 3:05 AM if something goes wrong.

Practical tip: In production, we've seen developers set the threshold too tight (e.g., 24 hours exactly) and get woken at 2:01 AM by false positives because the job ran 61 minutes into the window. Add 1–2 extra hours to your expected runtime.

Track duration and records processed

Knowing the job ran is half the story. You also want to know if it's running efficiently and processing the right volume of data.

In the Go example above, we collected two additional metrics:

• invoice_finalizer_invoices_total: cumulative count of invoices processed
• invoice_finalizer_duration_seconds: histogram of execution time

These let you spot performance regressions and data anomalies. Here are Prometheus alerts to catch them:

- alert: InvoiceFinalizerTooSlow
  expr: |
    increase(invoice_finalizer_duration_seconds_bucket{le="300"}[1d]) 
    / 
    increase(invoice_finalizer_duration_seconds_count[1d]) < 0.95
  for: 1d
  labels:
    severity: warning
  annotations:
    summary: "Invoice finalizer is taking longer than usual"
    description: "Less than 95% of recent runs finished within 5 minutes"

- alert: InvoiceFinalizerLowVolume
  expr: |
    increase(invoice_finalizer_invoices_total[1d]) < 100
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Invoice finalizer processed suspiciously few invoices today"
    description: "Only {{ $value }} invoices today; typical is 1000+"

The first alert uses a histogram to detect when the job is consistently slower than before. The second catches the scenario where the job runs and succeeds but processes far fewer records than expected—a sign that your filter logic is broken or your data pipeline upstream has a problem.

Healthchecks-style dead man switches

For truly critical jobs, consider implementing a "dead man switch" pattern inspired by services like Healthchecks.io. The idea is simple: the job sends a heartbeat to an external service, and if that heartbeat doesn't arrive within the expected window, an alert fires.

This catches scenarios that prometheus-based monitoring might miss:

• The server itself is down or network unreachable
• The Prometheus Pushgateway is unreachable
• The job crashes before calling push.Push()

You can implement this with a dedicated microservice or even with Grafana's Webhook notification channel:

// Minimal Go HTTP call to a dead man switch
import "net/http"

func recordHeartbeat() error {
	// POST to your monitoring system, e.g., Sentry or a custom endpoint
	url := "https://monitoring.internal/heartbeat/invoice-finalizer"
	resp, err := http.Post(url, "application/json", nil)
	if err != nil {
		return err
	}
	defer resp.Body.Close()
	
	if resp.StatusCode != 200 {
		return fmt.Errorf("heartbeat returned %d", resp.StatusCode)
	}
	return nil
}

// In finalizeInvoices():
if err := finalizeInvoices(); err != nil {
	panic(err)
}
// Only reached on success
recordHeartbeat()

Alternatively, Sentry's issue creation has a built-in dead man's switch feature. You can post a check_in with a timeout, and Sentry fires an alert if the next check_in doesn't arrive in time.

Common mistakes

• Pushing metrics on failure. If your job fails, don't push a "success" metric with a timestamp. The absence of an update is your signal. Only push on genuine success.
• Setting alert thresholds too tight. A 24-hour job with an alert threshold of 24 hours will false-positive when the job is slightly delayed. Add padding: aim for 1.5–2x the expected interval.
• Forgetting timezone differences. If your cron is set for "2 AM" but you're viewing dashboards in a different timezone, confusion abounds. Store and alert on Unix timestamps, not local times.
• Ignoring partial failures. A job that processes 10,000 records but fails on record 9,500 will often exit silently. Track both success and the volume of records processed, so you catch "successful" runs that actually failed partway through.
• Not logging job output. Even with metrics, you need logs. Redirect stdout and stderr to a file or log aggregator so you can debug when an alert fires.

Key takeaways

• Treat cron jobs as first-class monitoring subjects: push a last success timestamp every time the job completes without error.
• Alert on staleness: fire when the timestamp hasn't been updated within 1.5–2x the expected interval.
• Track duration and volume (records processed, bytes transferred, etc.) to spot performance regressions and data anomalies.
• For critical jobs, add a dead man's switch—an external heartbeat that alerts if it doesn't arrive in time.
• Never push success metrics on failure. The absence of an update is itself a signal.
• Combine metrics with structured logging so you can correlate alerts with root cause logs.