Retries: use carefully — real-world lessons and examples

Introduction

Retries hide flakiness if you let them. We've all been there: a test passes locally but fails inconsistently in CI, so we add a retry policy and move on. Six months later, your suite takes twice as long to run, developers ignore failures because "they usually pass on retry," and the actual root cause remains unfixed. Retries: use carefully — real-world lessons and examples is more than a cautionary phrase; it's a principle that separates teams with reliable test suites from those constantly fighting flaky tests.

In this article, we'll walk through how to implement retries responsibly in Playwright, when to use them, and—more importantly—when to investigate instead. We'll cover the strategies that work in production environments and the mistakes that cost us time.

Why retries feel like a solution (but often aren't)

Retries appeal to us because they're fast to implement. A flaky test that passes on the second run looks solved. But retries are a band-aid. They mask underlying issues: timing problems, race conditions, external service instability, or poor test isolation. The longer you lean on retries without investigating, the more your test suite becomes a confidence killer rather than a safety net.

We learned this the hard way. A payment processing suite had retries enabled globally. Tests would pass on average after two attempts. Developers stopped reading failure logs. When a real bug slipped through, nobody noticed because the failures blended into the noise. Retries had turned a diagnostic tool into a liability.

Enable retries in CI only

Your local environment and CI should have different retry policies. Locally, you want every failure to surface immediately so you can debug. In CI, where environment variability is higher and temporary network hiccups happen, retries serve a legitimate purpose—but only as a safety valve, not a crutch.

Configure Playwright's retry mechanism to be active in CI pipelines only:

// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  timeout: 30000,
  expect: {
    timeout: 5000,
  },
  webServer: {
    command: 'npm run dev',
    url: 'http://localhost:3000',
    reuseExistingServer: !process.env.CI,
  },
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
    },
  ],
});

This setup ensures retries kick in only when CI is true. Locally, any failure stops the test immediately, forcing investigation. In CI, transient failures get a second (or third) chance without bloating local development cycles.

The number two is deliberate. One retry catches most environmental hiccups. A second retry is the ceiling. If a test needs three attempts to pass reliably, it's not flaky—it's broken and needs fixing.

Track flaky tests in reports

Without visibility into which tests are actually failing and retrying, you're flying blind. Every retry should be logged, categorized, and surfaced to the team. This is your early warning system.

Use Playwright's built-in reporter capabilities combined with custom logic to flag flaky tests:

// playwright-reporter.ts
import { Reporter, TestCase, TestResult } from '@playwright/test/reporter';
import * as fs from 'fs';

interface FlakyTest {
  title: string;
  file: string;
  retries: number;
  lastRun: string;
}

class FlakyTestReporter implements Reporter {
  private flakyTests: FlakyTest[] = [];

  onTestEnd(test: TestCase, result: TestResult) {
    // result.retry is 0 for first attempt, 1 for first retry, etc.
    if (result.retry > 0 && result.status === 'passed') {
      this.flakyTests.push({
        title: test.title,
        file: test.file,
        retries: result.retry,
        lastRun: new Date().toISOString(),
      });
    }
  }

  onExit() {
    if (this.flakyTests.length > 0) {
      const report = {
        flakyCount: this.flakyTests.length,
        tests: this.flakyTests,
      };
      fs.writeFileSync(
        'flaky-tests-report.json',
        JSON.stringify(report, null, 2)
      );
      console.log(
        `⚠️  ${this.flakyTests.length} flaky tests detected. Review flaky-tests-report.json`
      );
    }
  }
}

export default FlakyTestReporter;

Add this reporter to your config:

// playwright.config.ts
export default defineConfig({
  reporter: [
    ['html'],
    ['./playwright-reporter.ts'],
  ],
  // ... rest of config
});

Now every flaky test is logged. These reports should go to Slack, your CI dashboard, or a project management tool. Your team sees which tests are unreliable and can prioritize fixes accordingly.

Fix root causes

Once you've identified a flaky test, don't defer the investigation. Most flakiness falls into a few categories: timing issues, state leakage between tests, or external service instability.

Timing issues are the most common. A button click happens, but the page hasn't visually updated yet. Your assertion fails, but a retry passes because the second attempt gives the DOM time to settle:

// ❌ Flaky: assumes click and response happen instantly
test('update user profile', async ({ page }) => {
  await page.click('button[data-testid="save-btn"]');
  await expect(page.locator('text=Profile saved')).toBeVisible();
});

// ✅ Reliable: wait for the expected state, not time
test('update user profile', async ({ page }) => {
  await page.click('button[data-testid="save-btn"]');
  // Wait for API response via network idle or specific request
  await page.waitForLoadState('networkidle');
  await expect(page.locator('text=Profile saved')).toBeVisible();
});

State leakage happens when one test leaves data that affects the next. In a shopping cart test suite, if one test doesn't clear the cart, the next test might start with unexpected items:

test.beforeEach(async ({ page, context }) => {
  // Clear all cookies and storage to ensure clean state
  await context.clearCookies();
  await page.evaluate(() => {
    localStorage.clear();
    sessionStorage.clear();
  });
  await page.goto('http://localhost:3000');
});

test('add item to cart', async ({ page }) => {
  // Now guaranteed to start with an empty cart
  await page.click('button[data-testid="add-to-cart"]');
  await expect(page.locator('[data-testid="cart-count"]')).toContainText('1');
});

External service instability requires mocking or using test doubles. If your payment gateway occasionally times out, don't retry the test—mock the gateway:

test('process payment', async ({ page }) => {
  // Intercept payment API and return consistent response
  await page.route('**/api/payment/**', (route) => {
    route.abort('failed'); // or route.continue() with a mock response
  });

  // Or always succeed for testing
  await page.route('**/api/payment/**', (route) => {
    route.abort('succeeded');
  });

  await page.click('button[data-testid="pay"]');
  await expect(page.locator('text=Payment successful')).toBeVisible();
});

Each fix removes the need for retries. A test that waits for network idle doesn't need retries. A test with proper setup/teardown doesn't need retries. The goal is a suite that passes consistently on the first run.

Never retry forever

Some teams create a culture of "retry until it passes." That's a recipe for false confidence. A test that fails 30% of the time and succeeds 70% will eventually pass if you retry enough, but it's still a bad test.

Set hard limits and enforce them:

// playwright.config.ts
export default defineConfig({
  retries: process.env.CI ? 2 : 0, // absolute maximum
  timeout: 30000, // per test timeout
  expect: {
    timeout: 5000, // per assertion timeout
  },
});

Beyond these limits, a test failure should block CI. Create a policy: if a test fails even after two retries, it fails the build. No exceptions. This forces teams to either fix the test or remove it. It prevents the slow slide into a suite of half-broken tests.

Some teams further restrict retries to certain test tags:

// playwright.config.ts
export default defineConfig({
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
      retries: 0, // strict by default
    },
    {
      name: 'chromium-with-retries',
      use: { ...devices['Desktop Chrome'] },
      retries: 2,
      grep: /@flaky/, // only for tests marked @flaky
    },
  ],
});

Tests marked @flaky can retry. All others fail immediately. This makes the team explicitly opt in to retries and creates accountability.

Common mistakes

• Global retries without investigation: Enabling retries everywhere and assuming the problem is solved. You've just delayed discovering the real issue.
• Same retries in local and CI: Local developers waste time waiting for retries. Retries should be a CI-only safety mechanism.
• Retrying without tracking: You don't know which tests are flaky or how often they fail. You're flying blind.
• Ignoring the flaky report: You generate a flaky test report and nobody reads it. Make it visible, actionable, and part of your definition of done.
• High timeout values hiding slow tests: A test with a 60-second timeout isn't flaky; it's slow. Retrying won't help; investigating will.
• Retrying as an alternative to proper setup/teardown: If a test fails because previous tests leave state behind, retrying masks a deeper test design problem.

Key takeaways

• Retries are a safety valve for CI, not a solution for flaky tests. Use them sparingly and only in CI environments.
• Track every retry and flaky test with a custom reporter. Visibility is the first step to fixing the problem.
• Investigate and fix root causes: timing issues, state leakage, and external instability. A robust test doesn't need retries.
• Set hard retry limits (two maximum) and enforce them. Tests that fail after retries should block the build.
• Make flaky tests visible to the team. Surface them in reports, Slack notifications, or dashboards so they stay on the priority list.
• Remember: retries hide problems. Your goal is tests that pass consistently on the first run, every time.