Measuring test suite health: what developers should know

What matters when your test suite grows

A test suite that passes tells you almost nothing. You need to know if your tests are fast enough to run frequently, if they break for real reasons, and whether the team actually trusts them enough to use them for deployment decisions. Measuring test suite health: what developers should know is the difference between tests that protect your product and tests that waste your time.

We've built and maintained test suites across e-commerce platforms, trading systems, and SaaS applications. The teams that scaled fastest weren't the ones with the most tests—they were the ones who tracked the right metrics and acted on them. This article covers the four metrics that tell you whether your tests are helping or hurting.

Track duration and flakiness

Test speed is not a nice-to-have. When your end-to-end test suite takes forty-five minutes to run, developers stop waiting for it. They push code without validation. You lose the entire point of automation.

Duration is straightforward to measure. Run your test suite multiple times and capture the execution time for each test and the suite as a whole. But duration alone is incomplete. A test that passes ninety-nine times and fails once randomly is a flaky test, and it's worse than useless—it erodes trust.

Track flakiness by recording the pass rate of each test over multiple runs. Here's how to instrument this in a Playwright test suite:

import { test, expect } from '@playwright/test';
import * as fs from 'fs';

interface TestMetric {
  testName: string;
  duration: number;
  passed: boolean;
  timestamp: string;
}

const metricsFile = './test-metrics.jsonl';

test.afterEach(async ({ _testFinished }, testInfo) => {
  const metric: TestMetric = {
    testName: testInfo.title,
    duration: testInfo.duration,
    passed: testInfo.status === 'passed',
    timestamp: new Date().toISOString()
  };
  
  fs.appendFileSync(metricsFile, JSON.stringify(metric) + '\n');
});

test('checkout flow completes', async ({ page }) => {
  const startTime = Date.now();
  
  await page.goto('https://app.example.com/checkout');
  await page.fill('[data-testid="email"]', 'user@example.com');
  await page.click('[data-testid="submit"]');
  
  await expect(page).toHaveURL(/.*\/confirmation/);
  
  const duration = Date.now() - startTime;
  console.log(`Test completed in ${duration}ms`);
});

After collecting metrics across fifty test runs, you can identify which tests are unreliable. A test with an 85% pass rate should be quarantined and investigated before it's allowed to block deployments. Common culprits are: hardcoded waits instead of proper element conditions, timing assumptions about network requests, or flaky third-party services like email or SMS providers.

Set a target: aim for test suites where the slowest tests complete in under two minutes and the entire suite runs in under fifteen minutes for a typical project. Slower than that, and you'll see developers skip the suite.

Failure rate by test

Not all test failures are equal. A test that fails once a quarter is a different problem than a test that fails multiple times per week.

Calculate the failure rate for each test by counting failures over a fixed period. If a test has run 200 times in the last month and failed 8 times, that's a 4% failure rate. A 4% failure rate on a critical path test is alarming. A 4% failure rate on a low-priority test that rarely changes might be acceptable—but only if you've consciously made that choice.

Set up a dashboard that groups tests by failure rate. We typically see three buckets:

• Green (0-2% failure): These tests are trustworthy. They can block deployments.
• Yellow (2-10% failure): These tests need investigation. They're not reliable enough for automated gating.
• Red (10%+ failure): These tests are actively misleading your team. Disable them until they're fixed.

Here's a script to calculate failure rates from the metrics we collected earlier:

import * as fs from 'fs';

interface TestMetric {
  testName: string;
  duration: number;
  passed: boolean;
  timestamp: string;
}

interface FailureStats {
  testName: string;
  totalRuns: number;
  failures: number;
  failureRate: number;
  avgDuration: number;
}

function analyzeMetrics(metricsFile: string): FailureStats[] {
  const lines = fs.readFileSync(metricsFile, 'utf-8').split('\n').filter(l => l);
  const metrics: TestMetric[] = lines.map(line => JSON.parse(line));
  
  const grouped = new Map();
  metrics.forEach(m => {
    if (!grouped.has(m.testName)) grouped.set(m.testName, []);
    grouped.get(m.testName)!.push(m);
  });
  
  const stats: FailureStats[] = [];
  
  grouped.forEach((testMetrics, testName) => {
    const failures = testMetrics.filter(m => !m.passed).length;
    const avgDuration = testMetrics.reduce((sum, m) => sum + m.duration, 0) / testMetrics.length;
    
    stats.push({
      testName,
      totalRuns: testMetrics.length,
      failures,
      failureRate: failures / testMetrics.length,
      avgDuration
    });
  });
  
  return stats.sort((a, b) => b.failureRate - a.failureRate);
}

const stats = analyzeMetrics('./test-metrics.jsonl');

console.log('\n=== Test Failure Report ===\n');
stats.forEach(stat => {
  const rate = (stat.failureRate * 100).toFixed(1);
  const status = stat.failureRate > 0.1 ? '🔴 RED' : stat.failureRate > 0.02 ? '🟡 YELLOW' : '🟢 GREEN';
  console.log(`${status} ${stat.testName}`);
  console.log(`  Runs: ${stat.totalRuns}, Failures: ${stat.failures}, Rate: ${rate}%`);
  console.log(`  Avg duration: ${stat.avgDuration.toFixed(0)}ms\n`);
});

When you find tests with high failure rates, you have one of three problems: the test is flaky (timing, order dependency), the code under test is broken (and you should fix it immediately), or the test is testing something that doesn't matter (and you should delete it). Figure out which one before you spend another cycle chasing ghosts.

Time to fix broken tests

When a test breaks in CI, how long does it sit broken? If a test fails on Tuesday and doesn't get fixed until Friday, that's two days of your team ignoring CI signals. That's how you lose the entire value of automated testing.

Track this metric by logging when a test first fails and when it's fixed (when it passes again consistently). Calculate the mean time to recovery (MTTR) for each test. In high-performing teams, most tests that fail should be fixed within one business day. Tests that consistently stay broken for a week should be removed.

A high MTTR for a particular test often indicates:

1. Ownership is unclear. Nobody owns it, so nobody fixes it.
2. The test is not understood. It passed for months, the failure message is cryptic, and debugging it is slow.
3. The failure is intermittent. The team can't reproduce it locally, so it sits broken.
4. The test is testing deprecated functionality. Nothing has changed in the code, the test is just wrong now.

If a test has a broken status that lasts longer than three days, treat it like a production incident. Stop, investigate, and either fix it or delete it. Don't let broken tests accumulate.

Delete tests nobody trusts

The most valuable action you can take with your test suite is to delete tests that nobody uses for decision-making.

A test that fails frequently but the team deploys anyway is doing harm. It's training your team to ignore test failures. It's adding noise to your CI/CD pipeline. It's making it harder to spot real problems when they occur.

Ask yourself: would I deploy code that fails this test? If the answer is yes, delete the test. If the answer is no, fix the code or the test immediately—don't leave it in a failing state.

Here's a hard rule we use: if a test has been failing for more than a week, it gets deleted unless someone is actively working to fix it. This sounds harsh, but it works. It creates urgency. It prevents zombie tests from accumulating. It keeps your test suite tight.

Similarly, delete tests that are redundant. If you have five tests that all verify the same checkout flow under slightly different conditions, keep one. The others are maintenance burden with no additional value.

A smaller, faster, more focused test suite that your team trusts is always better than a large suite full of tests that pass but nobody believes in.

Common mistakes

• Measuring only total pass/fail. A green CI run with a 3% failure rate on a critical test is still a problem. Track individual test health, not just the suite.
• Setting unrealistic speed targets. Not all tests can be fast. Integration tests with real databases are slower than unit tests. Set reasonable targets per test category.
• Ignoring flakiness patterns. Some tests are only flaky when run at 9 AM on Tuesdays (bad timing with infrastructure). Look for patterns in when tests fail, not just how often.
• Fixing flaky tests by adding waits. Adding `await page.waitForTimeout(5000)` makes tests slow and still flaky. Use proper wait conditions with `waitForSelector`, `waitForNavigation`, or explicit assertions.
• Keeping tests that test the test framework. If a test is primarily verifying that Playwright works, not that your code works, delete it.

Key takeaways

• Measure test duration and pass rate for every test, not just the suite as a whole. Tests are only valuable if they're fast enough and reliable enough to influence deployment decisions.
• Identify flaky tests early and quarantine them. A test with an 85% pass rate is worse than no test—it trains your team to ignore failures.
• Track failure rate by test and separate tests into green (trustworthy), yellow (investigate), and red (disable) buckets.
• Monitor mean time to recovery when tests break. If a test stays broken for more than a week, delete it or it will rot your test culture.
• Delete redundant tests and tests nobody trusts. A smaller suite of reliable tests beats a large suite of tests your team ignores.
• Create a dashboard that surfaces these metrics weekly. Invisible problems don't get fixed.