Sharding Playwright tests in CI: lessons from production

Sharding Playwright tests in CI: lessons from production

Running a comprehensive test suite across thousands of test cases in CI can easily become a bottleneck. We've seen projects where the full Playwright suite takes 45 minutes or more to complete, blocking deployments and slowing down feedback loops. Sharding Playwright tests in CI is the practical solution we've learned works best: distribute your test workload across multiple machines in parallel, then merge the results back into a single report.

The problem with single-machine test runs

When your Playwright suite grows beyond a few hundred tests, running everything sequentially on one CI machine becomes inefficient. A typical scenario: you have 2,000 tests that take 45 minutes to run. Even if your CI runners are beefy machines with 16 cores, Playwright's default behavior still runs tests in a queue, only parallelizing within a single worker pool.

The real cost isn't just time. It's developer experience. A 45-minute feedback loop means developers context-switch, lose focus, or merge code without seeing test results. In high-velocity teams, that's unacceptable.

Sharding distributes your test suite across independent CI jobs. If you split 2,000 tests into 4 shards, each shard runs on its own machine with only 500 tests, potentially finishing in 12–15 minutes instead of 45. That's a 3x speedup with minimal complexity.

Using --shard=1/4 per job

Playwright's built-in sharding feature is straightforward. The --shard flag tells Playwright which shard to run and how many total shards exist. The syntax is --shard=index/total.

Here's a practical CI setup. Let's say you're using GitHub Actions with 4 parallel jobs:

name: Playwright Tests
on: [push, pull_request]

jobs:
  playwright:
    name: Playwright Shard ${{ matrix.shard }}
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - run: npx playwright install --with-deps
      - name: Run Playwright tests (Shard ${{ matrix.shard }}/4)
        run: npx playwright test --shard=${{ matrix.shard }}/4
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: blob-report-${{ matrix.shard }}
          path: blob-report
          retention-days: 1

Each job runs with its own shard index. Job 1 runs --shard=1/4, job 2 runs --shard=2/4, and so on. Playwright automatically determines which tests belong to each shard by hashing the test file path and name, ensuring deterministic, balanced distribution.

The key insight: you don't manage which tests go where. Playwright handles it. You just specify the shard index and total count. This means adding a fifth shard is as simple as adding another job to your matrix—no test reorganization required.

Merging reports with merge-reports

Sharding splits the work, but you still need a single, consolidated test report. Playwright provides the merge-reports command for exactly this.

Each shard generates its own report artifact (typically in the blob-report directory). After all shards complete, a final merge job downloads all artifacts and combines them:

  merge-reports:
    name: Merge Test Reports
    if: always()
    needs: playwright
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - name: Download all blob reports
        uses: actions/download-artifact@v4
        with:
          path: all-blob-reports
          pattern: blob-report-*
      - name: Merge reports
        run: npx playwright merge-reports ./all-blob-reports --reporter html
      - uses: actions/upload-artifact@v4
        with:
          name: html-report
          path: playwright-report

The merge-reports command takes a directory containing all shard reports and produces a single, unified HTML report. This report shows test counts, pass/fail status, traces, screenshots—everything as if the tests had run on a single machine.

The merge job uses if: always() to run even if some shards fail, ensuring you always get a complete picture of results. It also doesn't depend on shard success, which matters when debugging failures.

Balancing by test duration

Playwright's default sharding is deterministic but not always optimally balanced. If shard 1 gets 400 fast unit tests and shard 2 gets 100 heavy integration tests, shard 2 will still be running after shard 1 finishes. Your total time is limited by the slowest shard.

The solution is to collect test duration data and use it to inform your shard count or test organization. After a few CI runs, you'll have baseline durations for each test. Tools like Playwright's built-in test reporting and custom scripts can extract this data.

Here's a practical approach: run your full suite once, capture the JSON report, and analyze it:

import fs from 'fs';

const report = JSON.parse(fs.readFileSync('playwright-report/index.json', 'utf-8'));

const testDurations = report.suites
  .flatMap(suite => suite.tests || [])
  .map(test => ({
    title: test.title,
    file: test.file,
    duration: test.duration || 0,
  }))
  .sort((a, b) => b.duration - a.duration);

console.log('Slowest 10 tests:');
testDurations.slice(0, 10).forEach(t => {
  console.log(`${t.title} (${t.duration}ms)`);
});

Once you know which tests are slow, you have options. You could use a custom shard script that distributes tests by duration, ensuring each shard has roughly equal total time. Or you could tag slow tests and run them in a separate, smaller shard. Or you could increase your shard count from 4 to 6 if the slowest shard is consistently the bottleneck.

We've found that 4–8 shards is the sweet spot for most suites. Beyond 8, the overhead of spinning up machines and merging reports starts to outweigh parallelization gains.

Great for large suites

Sharding really shines when your test suite is large. By "large," we mean 500+ tests or 20+ minutes of total runtime. Below that threshold, the gains are marginal, and the added CI complexity might not be worth it.

Consider a real-world example: an e-commerce platform with 3,000 Playwright tests covering checkout flows, product pages, account management, and API integrations. Without sharding, the suite takes 60 minutes. With 5 shards, it takes 14 minutes. That's a 4x improvement.

The business impact is significant. Developers get feedback in 14 minutes instead of 60. A deploy that would have waited an hour now deploys in 20 minutes (including shard startup time and report merging). Over a sprint with 20 commits, that's 10+ hours of developer time reclaimed.

Sharding also helps with flaky tests. When a test fails, you know exactly which shard it ran on and can re-run just that shard to verify flakiness, rather than re-running the entire 60-minute suite.

Common mistakes

Mistake 1: Forgetting to set fail-fast to false. By default, GitHub Actions stops all matrix jobs if one fails. With sharding, you want all shards to complete so you get a full report. Always set fail-fast: false in your matrix strategy.

Mistake 2: Not cleaning up old artifacts. Blob reports add up quickly. Set retention-days: 1 on shard artifacts and clean them up after the merge job completes. Otherwise, your CI storage will balloon.

Mistake 3: Running the merge job without waiting for shards. If your merge job doesn't explicitly depend on all shard jobs finishing, it might start before all artifacts are available. Use needs: [shard-1, shard-2, ...] or a similar mechanism to enforce ordering.

Mistake 4: Assuming equal distribution is optimal. Playwright shards tests deterministically, but not by duration. If your shards finish at very different times (20% variance is normal; 50%+ suggests imbalance), investigate which tests are slow and consider rebalancing or increasing shard count.

Mistake 5: Ignoring network and startup overhead. Spinning up 10 machines has its own cost. We've seen diminishing returns after 8 shards. Test your specific setup to find the optimal count.

Key takeaways

• Sharding distributes your Playwright test suite across multiple CI machines using the --shard=index/total flag.
• Each shard is independent; Playwright handles test distribution automatically based on file hashing.
• Use merge-reports to combine shard results into a single HTML report after all jobs complete.
• Monitor test durations to ensure shards are roughly balanced; aim for all shards to finish within 10–20% of each other.
• Sharding is most valuable for suites with 500+ tests or 20+ minutes of runtime; smaller suites may not see enough benefit to justify the complexity.
• Set fail-fast: false and manage artifact retention to avoid common pitfalls.
• Start with 4 shards; increase to 6–8 if you have the test volume and see uneven distribution.