Monitoring CI/CD pipelines: patterns, pitfalls and fixes

Introduction

Your pipelines run green on your laptop but fail in CI. Your team merges three times a day, but deploys happen once a week. Monitoring CI/CD pipelines: patterns, pitfalls and fixes starts not with alerting on every build failure, but with understanding the metrics that reveal what's actually slowing you down. The difference between teams shipping confidence and teams shipping chaos often comes down to whether they can answer simple questions: How long does a typical build take? Why did this one take twice as long? Which test made us rollback production last Tuesday?

We've worked through these problems across e-commerce, SaaS and ride-hailing systems. This guide covers the delivery metrics that matter and how to instrument them in Prometheus and Grafana.

Build duration — Track trends per pipeline

Build duration is deceptively simple. It's not just one number. A build that takes 45 seconds on your main branch but 8 minutes on a feature branch tells a story: flaky tests, parallelization issues, or environment drift.

Start by instrumenting each stage of your pipeline separately. In a typical Node.js or .NET build, you have checkout, install dependencies, lint, unit tests, integration tests, and artifact creation. Each stage matters because each can become a bottleneck.

# Example: GitLab CI job with Prometheus metrics output
stages:
  - build
  - test
  - deploy

build_app:
  stage: build
  script:
    - echo "START_TIME=$(date +%s)" > build.env
    - npm ci
    - npm run build
    - echo "END_TIME=$(date +%s)" >> build.env
  artifacts:
    reports:
      dotenv: build.env
  after_script:
    - |
      DURATION=$(($(date +%s) - $(grep START_TIME build.env | cut -d= -f2)))
      curl -X POST http://prometheus-pushgateway:9091/metrics/job/ci_build \
        -d "ci_build_duration_seconds{branch=\"$CI_COMMIT_BRANCH\",stage=\"build\"} $DURATION"

What we're doing here: pushing stage-level durations to a Prometheus Pushgateway instead of relying solely on CI platform logs. This lets you create a dashboard showing build duration trends per branch, per project.

In Grafana, query rate(ci_build_duration_seconds[5m]) to spot when builds start creeping slower. A 15% increase over two weeks usually means a new dependency, more test coverage (good), or test flakiness (bad). The graph tells you which pipeline and which stage.

Real scenario: One team we worked with noticed builds on main drifting from 3 minutes to 5 minutes. The dashboard showed unit tests were the culprit. They'd added comprehensive integration tests but hadn't parallelized them. Fixing the stage config cut 90 seconds off.

Failure rate — Flaky tests and broken builds

Not all failures are created equal. A build failure because you introduced a syntax error is honest feedback. A build failure because npm install timed out once every ten runs is a time sink.

Track two things: build success rate and test flakiness.

# Instrument your test runner to emit flakiness metrics
# Example: Jest with Prometheus client

const register = new prometheus.Registry();
const testDuration = new prometheus.Histogram({
  name: 'test_duration_seconds',
  help: 'Test execution duration',
  labelNames: ['test_file', 'status'],
  registers: [register]
});

const testRetries = new prometheus.Counter({
  name: 'test_retries_total',
  help: 'Tests that required retries',
  labelNames: ['test_file'],
  registers: [register]
});

// After test run, push to Pushgateway
const pushgateway = new prometheus.PushGateway('http://localhost:9091', [], register);
pushgateway.push({ jobName: 'jest_tests' });

In Grafana, calculate 1 - (sum(rate(ci_build_success_total[24h])) / sum(rate(ci_build_total[24h]))) to see failure rate over 24 hours. Then break it down by stage. If unit tests pass 99% of the time but integration tests pass 85%, you have a clear target for flakiness fixes.

For detecting flaky tests specifically, look for patterns: tests that fail sometimes but not always, usually in specific branches or under load. Export a metric flaky_test_count that you update after each run. Most teams find that 2–3 tests cause 50% of CI failures. Finding and quarantining them (or fixing the test, not the code) saves hours per week.

Real scenario: An e-commerce platform had a 92% CI pass rate. They thought the problem was "the infrastructure is unstable." Investigation showed three database-dependent tests were timing out randomly. Fixing test isolation and adding retry logic with exponential backoff improved pass rate to 98% without touching production code.

Deploy frequency — DORA metrics for delivery health

Deploy frequency is your first signal of team velocity and confidence. Teams deploying multiple times a day typically have better code quality and faster incident response than teams doing weekly releases. DORA metrics (from Accelerate) put a data-driven frame around this.

Track deployments as a counter with labels for environment, service, and outcome:

# Push deployment events from your CD system
deploy_count_total{service="checkout-service", environment="production", status="success"} 47
deploy_count_total{service="checkout-service", environment="production", status="failure"} 2
deploy_count_total{service="checkout-service", environment="staging", status="success"} 89

In Grafana, query increase(deploy_count_total{status="success", environment="production"}[7d]) to see how many production deployments you did in the last week. Divide by the number of days to get deploys per day. Compare that with business metrics: revenue, customer satisfaction, incident volume. Teams deploying 10+ times per day to production tend to resolve incidents faster and ship features users actually want.

Watch for the inverse signal too: if deploy frequency drops, ask why. Is the team waiting for reviews? Are tests too slow? Is deployment manual and error-prone? Low frequency is usually a symptom, and the dashboard helps you surface the root cause to leadership.

Real scenario: A SaaS team was deploying roughly twice a week. After adopting trunk-based development and fixing a flaky integration test, frequency jumped to 8–12 times daily. Customer-reported bugs dropped 40% because fixes could ship immediately instead of waiting for the next release window.

Lead time — From commit to production

Lead time measures the time from when you commit code to when it runs in production. It's the strongest predictor of deployment stability because shorter lead times mean faster feedback loops and smaller blast radius per deploy.

Measure it by correlating commit timestamps with deployment timestamps:

# In your CD pipeline, when a deployment succeeds:
ci_lead_time_seconds{service="api", branch="main"} 450
# This build took 7.5 minutes from commit to production ready

# Also track by percentile
ci_lead_time_p50{service="api"} 420
ci_lead_time_p95{service="api"} 1200
ci_lead_time_p99{service="api"} 2400

The median (p50) tells you the typical experience. The p95 and p99 tell you about outliers. If your p95 is 10 minutes but p99 is 45 minutes, something specific is blocking 1% of deployments. Find and fix it; the ROI is high.

In Grafana, create a dashboard showing lead time trends. A gradual increase usually signals growing codebase complexity or test suite bloat. A sudden spike points to a specific bottleneck: maybe you added a security scan, or a dependency started taking longer to download.

Real scenario: A trading platform had a median lead time of 8 minutes but p99 was 90 minutes. Investigation revealed one service's Docker image build was non-deterministic—sometimes it pulled fresh dependencies, sometimes not. Fixing the Dockerfile cut p99 to 12 minutes and reduced anxiety around deployments.

Common mistakes

• Alerting on every build failure. Not useful. Alert on trends: "failure rate exceeded 10% for 30 minutes" is actionable; "build 4527 failed" is noise.
• Ignoring environment differences. Build duration on CI is meaningless if your developers' machines are 10x faster. Track per-environment metrics separately.
• Not correlating with business metrics. A 20% faster build is nice, but does it ship more features or reduce bugs? Connect delivery metrics to outcomes.
• Forgetting about flaky tests. Teams often disable or skip flaky tests instead of fixing them. This hides real problems. Track and quarantine them explicitly.
• Measuring without acting. A beautiful Grafana dashboard that nobody looks at is theater. Build the metrics, then hold retrospectives when something is trending wrong.

Key takeaways

• Instrument build duration by stage to find bottlenecks; a 15% increase over two weeks signals a real problem worth investigating.
• Separate build success from flakiness; a test that fails randomly is different from broken code and requires different fixes.
• Deploy frequency is a leading indicator of confidence and team health; teams doing 5+ deploys daily typically have fewer production incidents.
• Lead time from commit to production reveals friction in your deployment process; track percentiles, not just averages.
• Connect metrics to outcomes: faster builds and higher deploy frequency should correlate with better reliability and faster incident response, or something else is broken.