Smoke tests in production: practical tips and trade-offs

Most production issues don't announce themselves—they slip in quietly and compound until customers notice. Smoke tests in production: practical tips and trade-offs gives you a way to catch these problems within minutes of deployment, using lightweight automated checks against your live systems. In this article, we'll walk through how to build reliable synthetic monitoring without the operational burden that often makes teams abandon the practice.

Why production smoke tests matter

By day 50 of working with Playwright and browser automation, you've likely built solid test suites for your feature branches. But the gap between "tests pass locally" and "the site works for real users" remains real. A misconfigured CDN, a database connection pool exhaustion, or a third-party API change can all slip past unit and integration tests.

Production smoke tests run against your live environment on a schedule, catching issues before the support queue fills up. They're not replacing your CI/CD test suites—they're supplementing them with real-world validation. The trade-off is straightforward: you gain confidence in actual system behavior but accept the complexity of maintaining test environments that mirror production closely enough to be meaningful.

Read-only checks — Load key pages, search, view products

The safest smoke tests perform read-only operations: they load pages, fill search boxes, and view data without modifying anything. This approach eliminates the risk of corrupting production state while still validating the most critical user journeys.

For an e-commerce site, a minimal read-only smoke test might check that the homepage loads, the product catalog renders, and search returns results for a common query. A SaaS platform might verify that the dashboard loads, the API responds to auth requests, and key charts render without JavaScript errors.

Here's a practical example using Playwright:

import { test, expect } from '@playwright/test';

test('smoke: homepage loads and responds to interaction', async ({ page }) => {
  const response = await page.goto('https://api.example.com', {
    waitUntil: 'networkidle',
    timeout: 15000
  });
  expect(response?.status()).toBeLessThan(400);
  
  // Verify hero section renders
  await expect(page.locator('[data-testid="hero"]')).toBeVisible();
  
  // Verify navigation works
  const navLink = page.locator('a:has-text("Products")').first();
  await expect(navLink).toBeVisible();
});

test('smoke: search returns results', async ({ page }) => {
  await page.goto('https://api.example.com/search');
  
  const searchInput = page.locator('input[placeholder="Search products"]');
  await searchInput.fill('laptop');
  await page.keyboard.press('Enter');
  
  // Wait for results to load
  const results = page.locator('[data-testid="product-card"]');
  await expect(results.first()).toBeVisible({ timeout: 5000 });
  
  const count = await results.count();
  expect(count).toBeGreaterThan(0);
});

test('smoke: product detail page loads', async ({ page }) => {
  // Use a known product ID that always exists
  await page.goto('https://api.example.com/products/123456');
  
  const productName = page.locator('[data-testid="product-name"]');
  const price = page.locator('[data-testid="product-price"]');
  
  await expect(productName).toBeVisible();
  await expect(price).toBeVisible();
  
  // Verify price is a valid number
  const priceText = await price.textContent();
  expect(parseFloat(priceText?.replace(/[^0-9.]/g, '') || '0')).toBeGreaterThan(0);
});

Notice the deliberate use of known IDs and common search terms—we're not testing functionality, we're confirming that the system is breathing. The 15-second timeout on the initial navigation is generous; in production, you want to catch slow-loading pages before users do.

Read-only checks scale well because they carry no cleanup burden. You can run them frequently without worrying about orphaned test data or state pollution. This makes them ideal for the continuous monitoring loop we'll discuss later.

Synthetic users — Dedicated test accounts with no side effects

Sometimes read-only checks aren't enough. You need to validate that checkout works, that form submission succeeds, or that user invitations process correctly. That's where synthetic user accounts come in: dedicated test users that your monitoring scripts use, with data that's easy to clean up or reset.

The key constraint is isolation. Your synthetic user's activity must not pollute metrics, analytics, or customer-facing reports. It must not trigger real notifications or affect billing. Most teams accomplish this by:

• Creating test accounts with obvious names (like smoke-test-user-001@example.com) that your analytics code filters out
• Using a dedicated test organization or tenant that doesn't affect your live dashboard
• Tagging all synthetic transactions with a special flag in your database
• Implementing easy cleanup routines to roll back test data between runs

Here's how you might implement a synthetic user checkout flow:

import { test, expect } from '@playwright/test';

const TEST_USER = {
  email: 'smoke-test-user@example.com',
  password: process.env.SMOKE_TEST_PASSWORD,
  firstName: 'Smoke',
  lastName: 'Test'
};

const TEST_PRODUCT_ID = '123456'; // A product that always exists in inventory

test('smoke: end-to-end checkout flow', async ({ page }) => {
  // Log in with synthetic user
  await page.goto('https://example.com/login');
  await page.locator('input[type="email"]').fill(TEST_USER.email);
  await page.locator('input[type="password"]').fill(TEST_USER.password);
  await page.locator('button:has-text("Sign in")').click();
  
  // Wait for dashboard
  await page.waitForURL('**/dashboard');
  
  // Navigate to product
  await page.goto(`https://example.com/products/${TEST_PRODUCT_ID}`);
  
  // Add to cart
  await page.locator('button:has-text("Add to cart")').click();
  await expect(page.locator('[data-testid="cart-count"]')).toContainText('1');
  
  // Go to checkout
  await page.locator('a:has-text("Checkout")').click();
  await page.waitForURL('**/checkout');
  
  // Verify cart summary loaded
  await expect(page.locator('[data-testid="order-summary"]')).toBeVisible();
  
  // Verify payment form is present but don't submit
  // (submitting real charges would be inappropriate)
  await expect(page.locator('[data-testid="payment-form"]')).toBeVisible();
  
  console.log('Checkout flow validated successfully');
});

Notice we validate the checkout up to payment submission but don't actually charge a card. The goal is to confirm that every step of the flow works—form validation, inventory checks, calculations—without creating real transactions. You can test actual payment processing separately with your payment provider's sandbox.

The synthetic user approach also lets you monitor state-dependent features. A test admin account can verify that moderation tools work. A test team lead can confirm that approval workflows function. By maintaining a small roster of synthetic users with different roles, you catch permission bugs and role-based feature regressions that read-only checks would miss.

Run on schedule — Every few minutes after deploys

Smoke tests lose their value if they run sporadically. The window between a broken deployment and detection should be measured in seconds or minutes, not hours. That requires a scheduled execution strategy that treats production monitoring as a first-class concern.

Most teams use one of two patterns:

• Post-deploy triggers: Run smoke tests automatically whenever a deployment completes, before handing control back to the team.
• Periodic polling: Run every 2–5 minutes regardless of deployment, catching issues that emerge gradually (database slowdowns, cache misses, third-party failures).

A robust setup uses both. Here's how you might configure it with GitHub Actions:

// .github/workflows/smoke-tests.yml (simplified example)

name: Production Smoke Tests

on:
  # Run on deployment completion
  deployment_status:
  # Run every 5 minutes
  schedule:
    - cron: '*/5 * * * *'
  # Allow manual trigger
  workflow_dispatch:

jobs:
  smoke:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-node@v4
        with:
          node-version: '18'
      
      - run: npm ci
      
      - name: Run smoke tests
        run: npx playwright test --project=smoke
        env:
          BASE_URL: https://api.example.com
          SMOKE_TEST_PASSWORD: ${{ secrets.SMOKE_TEST_PASSWORD }}
      
      - name: Upload traces on failure
        if: failure()
        uses: actions/upload-artifact@v3
        with:
          name: smoke-traces
          path: test-results/
          retention-days: 7

Running every 5 minutes strikes a balance: frequent enough to catch regressions within minutes, infrequent enough that the cost remains negligible. Many teams adjust this cadence based on deployment frequency (more frequent deploys justify more frequent tests) and risk tolerance (customer-critical features might warrant 2-minute intervals).

One critical consideration: ensure your smoke tests don't create test data that compounds over time. If each checkout test creates an order, you'll fill your database with test junk. Either clean up aggressively (delete test orders after each run) or use separate test environments that you reset nightly.

Alert — Notify on failures with traces attached

A smoke test that fails silently is worse than no smoke test at all. You need immediate, reliable alerting that reaches the right people with enough context to act quickly.

The essentials of a good alert are:

• It arrives in your incident channel (Slack, PagerDuty, etc.) within seconds
• It includes the test name, failure reason, and relevant error messages
• It attaches a Playwright trace or screenshot for debugging
• It avoids alert fatigue by grouping related failures and preventing duplicate notifications within a window (e.g., suppress duplicate alerts for 5 minutes)

Here's a practical alerting function you might integrate into your test suite:

import { test, expect } from '@playwright/test';
import axios from 'axios';
import fs from 'fs';
import path from 'path';

async function sendAlert(testName: string, error: string, tracePath?: string) {
  const slackWebhook = process.env.SLACK_WEBHOOK_URL;
  if (!slackWebhook) return;
  
  let attachments: object[] = [];
  
  // Attach trace file if available
  if (tracePath && fs.existsSync(tracePath)) {
    const traceBuffer = fs.readFileSync(tracePath);
    // In practice, you'd upload to S3/CDN and include a link
    attachments.push({
      title: 'Playwright Trace',
      value: `Trace file attached (${(traceBuffer.length / 1024).toFixed(2)} KB)`
    });
  }
  
  const payload = {
    text: `🚨 Production Smoke Test Failed`,
    blocks: [
      {
        type: 'section',
        text: {
          type: 'mrkdwn',
          text: `*Smoke Test Failure*\nTest: \`${testName}\`\nError: \`${error}\``
        }
      },
      {
        type: 'context',
        elements: [
          {
            type: 'mrkdwn',
            text: `Timestamp: `
          }
        ]
      }
    ]
  };
  
  try {
    await axios.post(slackWebhook, payload, { timeout: 5000 });
  } catch (err) {
    console.error('Failed to send alert:', err);
  }
}

test('smoke: checkout with alert on failure', async ({ page }) => {
  try {
    await page.goto('https://example.com/checkout');
    await expect(page.locator('[data-testid="order-form"]')).toBeVisible({ timeout: 5000 });
  } catch (error) {
    const errorMsg = error instanceof Error ? error.message : 'Unknown error';
    await sendAlert('checkout', errorMsg);
    throw error;
  }
});

In a production setup, you'd integrate with your monitoring platform (Datadog, New Relic, Grafana) which handles deduplication, escalation, and on-call routing. The key is ensuring that failures surface immediately and that the team can see exactly what broke without having to dig through logs.

Common mistakes

Testing too much: Smoke tests should validate critical paths only. If your test suite takes 10 minutes to run, it defeats the purpose of quick feedback. Focus on the 5–10 journeys that represent 80% of production traffic.

Ignoring flakiness: A flaky smoke test that fails randomly in 5% of runs will be ignored or disabled. Invest in stabilizing your tests: use explicit waits instead of sleeps, handle network variability with retries, and isolate external dependencies. A flaky smoke test is worse than none at all.

Forgetting to clean up: Test data left behind in production databases corrupts metrics and confuses future debugging. Implement cleanup routines that run after each test cycle, or use a separate test database altogether.

Sending alerts to the wrong channel: If your smoke test alerts flood a team Slack channel that half the team has muted, you'll miss real issues. Verify that alerts reach an actively monitored channel and that the on-call engineer knows to check it.

Not evolving tests with the product: Smoke tests become stale quickly. When you deprecate a feature or redesign a page, update the corresponding smoke tests. Stale tests that pass when the system is broken are dangerous.

Key takeaways

• Start with read-only checks: they're safe, reliable, and easy to run frequently without side effects.
• Use synthetic user accounts for flows that require state mutation, but isolate their data so they don't pollute production metrics.
• Schedule smoke tests to run both post-deployment and periodically (every 2–5 minutes), catching both regression and gradual degradation.
• Wire up immediate alerting with trace attachments so your team can debug failures in seconds, not hours.
• Keep smoke tests focused and stable: flaky tests and excessive coverage undermine the entire approach.