Testing preview deployments: what developers should know

Testing preview deployments: what developers should know

Every pull request you merge could break production, but catching that damage hours or days later costs far more than catching it in preview. Testing preview deployments gives your team a real URL to validate against before code reaches main—and when you automate it properly, you get instant feedback without manual work. This is day 49 of our Playwright & Browser Automation series, and we're diving into the workflow that catches environment-specific bugs, validates integrations, and keeps your confidence high.

Why preview deployments matter for testing

Preview deployments—temporary environments spun up for each pull request—are where production reality meets development speed. Unlike local development or staging, a preview environment runs your exact branch code against real infrastructure: live APIs, production databases (or their replicas), actual CDNs, and real TLS certificates.

Many teams deploy to preview but never run tests against it. They review code, merge, and hope. That gap is where environment-specific issues hide: service-to-service auth failures, misconfigured environment variables, third-party API changes, or database migration issues that only surface under real load.

Automating browser tests against preview deployments closes that gap. You run Playwright test suites the moment the URL is live, get results within minutes, and leave evidence on the PR itself. The cost is one CI job; the benefit is catching 60–70% of what would otherwise reach staging or production.

Wait for the preview URL in CI

The first challenge: your preview deployment completes asynchronously. When your CI pipeline runs, the deployment service is still spinning up containers, running migrations, and warming caches. Running tests immediately fails with "connection refused."

You need a wait mechanism that polls the preview URL until it's healthy, with exponential backoff and a timeout. This keeps your CI clean and your tests reliable.

import fetch from 'node-fetch';

async function waitForPreviewUrl(
  url: string,
  maxRetries: number = 30,
  baseDelayMs: number = 2000
): Promise<void> {
  let lastError: Error | null = null;

  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      const response = await fetch(url, { timeout: 5000 });
      if (response.ok || response.status === 200) {
        console.log(`Preview URL ${url} is ready after ${attempt + 1} attempts`);
        return;
      }
    } catch (error) {
      lastError = error as Error;
    }

    const delayMs = baseDelayMs * Math.pow(1.5, attempt);
    console.log(`Attempt ${attempt + 1}/${maxRetries} failed. Retrying in ${delayMs}ms...`);
    await new Promise((resolve) => setTimeout(resolve, delayMs));
  }

  throw new Error(
    `Preview URL ${url} did not become ready after ${maxRetries} attempts. Last error: ${lastError?.message}`
  );
}

This pattern handles the common case where a preview environment needs 30–60 seconds to boot. The exponential backoff avoids hammering the service early on, and you get clear logs for debugging.

In your CI config (GitHub Actions, GitLab CI, etc.), call this before running your test suite:

const previewUrl = process.env.PREVIEW_URL;
if (!previewUrl) {
  throw new Error('PREVIEW_URL environment variable is required');
}

await waitForPreviewUrl(previewUrl);
console.log(`Running tests against ${previewUrl}`);

Run smoke tests against it

Once the preview URL is live, you want fast, targeted tests—not your full 2-hour suite. Smoke tests are your answer: a small set of high-value journeys that verify core functionality end-to-end.

For an e-commerce platform, smoke tests might be:

• Homepage loads and displays products
• Search works and returns results
• Add to cart flow succeeds
• Checkout modal opens and validates input
• User login/logout works

These run in 2–5 minutes and catch 80% of showstoppers: missing environment variables, broken API endpoints, database connectivity issues, TLS cert problems, and CSS/layout regressions.

Tag your smoke tests explicitly so CI can run only them:

// smoke.spec.ts
import { test, expect } from '@playwright/test';

test('smoke: homepage loads and displays product grid', async ({ page }) => {
  const baseUrl = process.env.PREVIEW_URL || 'http://localhost:3000';
  await page.goto(`${baseUrl}/`);

  await expect(page.locator('[data-testid="product-grid"]')).toBeVisible();
  const productCount = await page.locator('[data-testid="product-card"]').count();
  expect(productCount).toBeGreaterThan(0);
});

test('smoke: search returns results', async ({ page }) => {
  const baseUrl = process.env.PREVIEW_URL || 'http://localhost:3000';
  await page.goto(`${baseUrl}/`);

  await page.fill('[data-testid="search-input"]', 'laptop');
  await page.click('[data-testid="search-button"]');

  await expect(page.locator('[data-testid="search-results"]')).toBeVisible();
  await expect(page.locator('[data-testid="product-card"]')).toHaveCount(5);
});

test('smoke: add to cart works', async ({ page }) => {
  const baseUrl = process.env.PREVIEW_URL || 'http://localhost:3000';
  await page.goto(`${baseUrl}/product/laptop-pro`);

  await page.click('[data-testid="add-to-cart-button"]');
  
  const cartBadge = page.locator('[data-testid="cart-badge"]');
  await expect(cartBadge).toHaveText('1');
});

Mark them with @smoke tag and run only those in your preview CI job. Full regression tests can run on a schedule or on merge to main, where you have time to wait.

Comment results on the PR

Test results locked in CI logs are useful for you, but your team reviews PRs on GitHub or GitLab. Posting results there—pass/fail, failure screenshots, error logs—makes feedback immediate and visible to everyone reviewing the code.

Use Playwright's JSON report and a CI action to post comments:

// ci.yml example (GitHub Actions)
- name: Run smoke tests against preview
  id: smoke
  continue-on-error: true
  env:
    PREVIEW_URL: https://pr-${{ github.event.pull_request.number }}.preview.example.com
  run: |
    npm run test:smoke -- --reporter=json > test-results.json 2>&1

- name: Comment test results on PR
  if: always()
  uses: daun/playwright-report-comment@v3
  with:
    report-path: test-results.json
    github-token: ${{ secrets.GITHUB_TOKEN }}

Or write a custom comment script:

import { readFileSync } from 'fs';
import { Octokit } from '@octokit/action';

const report = JSON.parse(readFileSync('test-results.json', 'utf-8'));
const passed = report.stats.expected;
const failed = report.stats.unexpected;
const status = failed === 0 ? '✅' : '❌';

const body = `
${status} **Smoke tests against preview deployment**

- Passed: ${passed}
- Failed: ${failed}
- Duration: ${(report.stats.duration / 1000).toFixed(1)}s

[View full report](${process.env.REPORT_URL})
`;

const octokit = new Octokit();
await octokit.rest.issues.createComment({
  owner: context.repo.owner,
  repo: context.repo.repo,
  issue_number: context.issue.number,
  body,
});

This gives reviewers immediate confidence: they see the green checkmark, know the preview environment is healthy, and can merge with less anxiety.

Catch environment-specific issues

Preview deployments are where many real-world problems surface for the first time. Your code works locally, passes unit tests, but fails in preview because of infrastructure or configuration differences.

Common catches:

• Missing environment variables: A feature flag or API key isn't set in the preview environment, so a feature silently fails. A smoke test that checks for that feature catches it immediately.
• Database state: A migration runs during preview deployment; if it's broken or incomplete, your tests fail and alert the team before production.
• Third-party API changes: An external service changed its response format or rate limits. Your preview environment hits real APIs; local mocks don't catch this.
• Authentication/CORS: Service-to-service auth, JWT token validation, or CORS policies work locally (with relaxed rules) but fail in production-like preview. Tests expose the gap.
• Asset CDN paths: Images, fonts, or scripts live on a CDN in preview but are local files in development. A visual regression test catches broken asset paths.

To maximize catches, structure your smoke tests to validate these layers:

test('smoke: environment is healthy', async ({ page, request }) => {
  const baseUrl = process.env.PREVIEW_URL;
  
  // Check API health
  const apiHealth = await request.get(`${baseUrl}/api/health`);
  expect(apiHealth.ok).toBe(true);

  // Check critical assets load (CDN, etc.)
  await page.goto(`${baseUrl}/`);
  const images = await page.locator('img').all();
  for (const img of images.slice(0, 3)) {
    const src = await img.getAttribute('src');
    const response = await request.head(src!);
    expect(response.status).toBe(200);
  }

  // Check auth flow
  const loginResponse = await request.post(`${baseUrl}/api/auth/login`, {
    data: { email: 'test@example.com', password: 'test123' },
  });
  expect([200, 401]).toContain(loginResponse.status);
});

Common mistakes

Testing against stale preview URLs: A PR is closed, the preview environment torn down, but your CI still tries to test it. Always fetch the current preview URL from your deployment service, don't hardcode or cache it.

Tests that are too brittle: Smoke tests should be resilient. Don't assert exact product counts or prices if they can change; assert that elements exist and are interactive. Use data-testid attributes instead of fragile selectors.

Ignoring flakiness: A test passes 90% of the time. Don't ignore it; find the root cause (timing issue, race condition, external dependency). Flaky tests erode trust in your entire suite and waste time investigating false failures.

Not cleaning up preview state: If your smoke tests create database records or sessions, clean them up. Each test run should start fresh, or you'll see cascading failures and false negatives.

Running the full suite in CI: Your 45-minute test suite on every preview wastes CI resources and delays feedback. Smoke tests in CI, full regression on schedule or on merge. Different tools for different jobs.

Key takeaways

• Preview deployments are real environments; test them before main to catch infrastructure, integration, and configuration issues early.
• Wait for the preview URL to be healthy before running tests, using exponential backoff polling to handle variable boot times.
• Run a focused smoke test suite (2–5 minutes) on every preview to validate critical user flows and catch showstoppers.
• Post test results as PR comments so reviewers have immediate confidence and visibility into environment health.
• Use preview testing to surface environment-specific bugs—missing config, API changes, auth failures, CDN issues—that local testing misses.
• Keep smoke tests simple and data-agnostic; they verify behavior, not exact output. Use data-testid selectors for stability.
• Treat flaky tests as bugs, not noise; each failure erodes trust in the entire suite.