Monitoring payment systems: patterns, pitfalls and fixes

Introduction

Payment systems are the heart of any revenue-generating product, yet they remain among the hardest systems to monitor effectively. By day 57 of our payment engineering journey, you've likely built integrations with Stripe, handled webhooks, and processed real transactions. Now comes the harder part: knowing when money stops flowing. Monitoring payment systems—patterns, pitfalls and fixes—requires a different mindset than monitoring application health. A 500 error on your API affects features; a payment success rate drop affects your bottom line.

In this article, we'll walk through the patterns that matter in production payment systems: success rate drops, webhook failures, disputes and refunds, and revenue dashboards. We'll also cover the pitfalls we've seen sink teams—silent webhook losses, lag-blind monitoring, and dashboards that look healthy while customers lose money.

Alert on payment success rate drops

Payment success rate is the single most important metric you can monitor. A drop from 95% to 90% success might sound small, but on $100K daily volume, that's $5K in lost revenue per day.

The trap most teams fall into is monitoring raw success counts instead of rates. If you process $1M one day and $500K the next, your absolute failures might be lower on the second day, yet your success rate could be worse. Always calculate success rate as successful charges divided by total attempts.

Set up alerts on success rate thresholds, not absolute counts. A good baseline is to alert if success rate drops more than 2–3 percentage points from your rolling 7-day average. For teams processing high volume, even a 1-point drop deserves investigation.

// Example: Monitor success rate with rolling average
import { createClient } from "@supabase/supabase-js";

const supabase = createClient(SUPABASE_URL, SUPABASE_KEY);

async function checkSuccessRateAnomaly() {
  const now = new Date();
  const last24h = new Date(now.getTime() - 24 * 60 * 60 * 1000);
  const last7d = new Date(now.getTime() - 7 * 24 * 60 * 60 * 1000);

  // Last 24 hours
  const { data: today } = await supabase
    .from("charges")
    .select("id, status")
    .gte("created_at", last24h.toISOString());

  const todaySuccess =
    today?.filter((c) => c.status === "succeeded").length || 0;
  const todayRate = todaySuccess / (today?.length || 1);

  // Rolling 7-day average
  const { data: week } = await supabase
    .from("charges")
    .select("id, status")
    .gte("created_at", last7d.toISOString());

  const weekSuccess = week?.filter((c) => c.status === "succeeded").length || 0;
  const weekRate = weekSuccess / (week?.length || 1);
  const dailyAvgRate = weekRate;

  // Alert if today's rate dropped 3+ points from 7-day average
  if (todayRate < dailyAvgRate - 0.03) {
    await sendAlert(
      `Success rate dropped to ${(todayRate * 100).toFixed(1)}% (7-day avg: ${(dailyAvgRate * 100).toFixed(1)}%)`
    );
  }
}

checkSuccessRateAnomaly().catch(console.error);

Beyond the alert, investigate the why. Stripe returns rich failure reasons: card declined, insufficient funds, lost card, expired card, processing error, authentication required. Log the failure reason with every charge, then slice your success rate by failure type. If declines spike, it might be your risk rules or a fraud wave. If processing errors spike, there's likely a Stripe API issue or your retry logic is broken.

Track webhook failures

Webhooks are the asynchronous glue holding payment systems together. Stripe sends charge succeeded, charge failed, charge refunded, and dispute events. Your code listens, updates database records, sends emails, triggers fulfillment. Webhook failures are silent killers: Stripe thinks the message delivered, your system never heard it, and the order sits in limbo.

Most monitoring approaches miss this because they focus on inbound webhook success rates, not on whether you're actually processing events. The hard truth: even a 99% webhook delivery rate will cause failures.

Store every webhook received with a processing status. This gives you a complete audit trail and lets you detect gaps.

// NestJS example: Store webhooks before processing
import {
  Controller,
  Post,
  Body,
  RawBodyRequest,
  Req,
} from "@nestjs/common";
import Stripe from "stripe";

@Controller("webhooks")
export class StripeWebhookController {
  constructor(
    private stripe: Stripe,
    private db: Database
  ) {}

  @Post("stripe")
  async handleStripeEvent(
    @Req() req: RawBodyRequest,
    @Body() body: any
  ) {
    const signature = req.headers["stripe-signature"] as string;
    const rawBody = req.rawBody;

    // Verify signature
    let event: Stripe.Event;
    try {
      event = this.stripe.webhooks.constructEvent(
        rawBody,
        signature,
        process.env.STRIPE_WEBHOOK_SECRET
      );
    } catch (err) {
      return { received: false };
    }

    // Store webhook record immediately
    const webhookRecord = await this.db.webhooks.create({
      event_id: event.id,
      event_type: event.type,
      payload: event.data,
      received_at: new Date(),
      processed_at: null,
      processed_status: null,
      error: null,
    });

    // Process asynchronously and update record
    this.processWebhookEvent(event, webhookRecord.id).catch((err) => {
      this.db.webhooks.update(webhookRecord.id, {
        processed_status: "failed",
        error: err.message,
        processed_at: new Date(),
      });
    });

    // Return 200 immediately so Stripe stops retrying
    return { received: true };
  }

  private async processWebhookEvent(
    event: Stripe.Event,
    webhookRecordId: string
  ) {
    switch (event.type) {
      case "charge.succeeded":
        await this.handleChargeSucceeded(event.data.object as Stripe.Charge);
        break;
      case "charge.failed":
        await this.handleChargeFailed(event.data.object as Stripe.Charge);
        break;
      // ... other event types
    }

    // Mark as processed
    await this.db.webhooks.update(webhookRecordId, {
      processed_at: new Date(),
      processed_status: "success",
    });
  }
}

Now query for unprocessed or failed webhooks regularly. A simple SQL query tells you if there's a backlog:

// Query for webhook gaps
async function checkWebhookHealth() {
  const { data: unprocessed } = await supabase
    .from("webhooks")
    .select("id, event_id, event_type, received_at")
    .is("processed_at", null)
    .gt("received_at", new Date(Date.now() - 5 * 60 * 1000).toISOString()); // Last 5 minutes

  if (unprocessed && unprocessed.length > 0) {
    await sendAlert(
      `${unprocessed.length} webhooks unprocessed (oldest: ${unprocessed[0].event_id})`
    );
  }

  const { data: failed } = await supabase
    .from("webhooks")
    .select("id, event_id, error")
    .eq("processed_status", "failed")
    .gt("processed_at", new Date(Date.now() - 1 * 60 * 60 * 1000).toISOString()); // Last hour

  if (failed && failed.length > 0) {
    await sendAlert(
      `${failed.length} webhooks failed processing in last hour`
    );
  }
}

Monitor dispute and refund rates

Disputes (chargebacks) and refunds are inverse indicators of health. A spike means either customer dissatisfaction, fraud, or a product issue. Refunds are usually intentional; disputes mean the customer went to their bank instead of asking for help.

Set baselines for both. Most SaaS platforms see 0.1–0.5% dispute rates and 1–3% refund rates. If your dispute rate hits 1%, it's a red flag. Refunds should trend with customer churn—if they spike without churn increasing, something broke.

Track these by volume, not just count. A single disputed transaction is noise; 10 disputes on 1000 transactions (1%) is a pattern.

// Monitor dispute and refund anomalies
async function checkDisputeAndRefundRates() {
  const last24h = new Date(Date.now() - 24 * 60 * 60 * 1000);
  const last7d = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000);

  // Today's metrics
  const { data: today } = await supabase
    .from("charges")
    .select("id, status")
    .gte("created_at", last24h.toISOString());

  const todayTotal = today?.length || 0;
  const todayDisputes = today?.filter((c) => c.status === "disputed").length || 0;
  const todayRefunds = today?.filter((c) => c.status === "refunded").length || 0;
  const todayDisputeRate = todayTotal > 0 ? todayDisputes / todayTotal : 0;
  const todayRefundRate = todayTotal > 0 ? todayRefunds / todayTotal : 0;

  // 7-day average
  const { data: week } = await supabase
    .from("charges")
    .select("id, status")
    .gte("created_at", last7d.toISOString());

  const weekTotal = week?.length || 0;
  const weekDisputes = week?.filter((c) => c.status === "disputed").length || 0;
  const weekRefunds = week?.filter((c) => c.status === "refunded").length || 0;
  const weekDisputeRate = weekTotal > 0 ? weekDisputes / weekTotal : 0;
  const weekRefundRate = weekTotal > 0 ? weekRefunds / weekTotal : 0;

  // Alert on 2x increase from baseline
  if (todayDisputeRate > weekDisputeRate * 2) {
    await sendAlert(
      `Dispute rate ${(todayDisputeRate * 100).toFixed(2)}% (baseline: ${(weekDisputeRate * 100).toFixed(2)}%)`
    );
  }

  if (todayRefundRate > weekRefundRate * 2) {
    await sendAlert(
      `Refund rate ${(todayRefundRate * 100).toFixed(2)}% (baseline: ${(weekRefundRate * 100).toFixed(2)}%)`
    );
  }
}

Use Stripe's Radar rules or webhooks to track dispute reasons. Common patterns: customer claims fraud (they didn't authorize the charge), customer says it's duplicate, customer says the product wasn't delivered. Each tells a different story about your operation.

Dashboards for daily revenue

The most important dashboard isn't technical—it's financial. Your team should check one number each morning: did we collect what we expected yesterday?

Build a simple dashboard that shows:

• Daily revenue (gross charges)
• Payment success rate
• Refunds and disputes (count and amount)
• Failed charges by reason
• Pending charges (captured but not settled)

Stripe has the data; use their Reports API or query your own event logs. For visual clarity, include a 30-day trend line. A flat or declining line with no alerting is a failure mode.

Integrate this dashboard into your communication loop: weekly revenue emails, Slack channels that post daily summaries, and a pagerduty equivalent for hard stops (success rate below 80%, zero revenue collected, webhook queue stalled).

Common mistakes

• Alerting on raw counts instead of rates. Fifty failed charges sounds bad until you learn you processed 5000 charges that day (99% success). Always normalize by volume.
• No webhook audit trail. You can't fix what you can't see. Store every webhook before processing it. This adds 50ms and saves 50 hours of debugging.
• Ignoring lag in monitoring. If you check success rates hourly but batch process charges every 6 hours, your alerts are blind during the batch window. Know your data latency.
• Not linking technical metrics to business impact. "Success rate dropped 2%" is noise. "$50K in daily revenue at risk" is an incident. Translate every alert into dollars.
• Building dashboards no one reads. Pretty Grafana panels don't matter if your team checks them once a quarter. Tie monitoring to a ritual: daily Slack summary, weekly email, monthly review.

Key takeaways

• Monitor success rate as a percentage with a rolling average baseline; alert on 2–3 point drops, not raw failure counts.
• Store every webhook before processing it, then track processing status; unprocessed or failed webhooks are revenue at risk.
• Slice success rates by failure reason to diagnose root causes (card declined, fraud, processing error, etc.).
• Set baseline expectations for dispute and refund rates, then alert on 2x increases; both indicate operational or product issues.
• Build a simple daily revenue dashboard that your team checks every morning; include success rate, refunds, disputes, and failed charge reasons.
• Translate technical metrics into business impact (dollars at risk); this drives urgency and prioritization.
• Set up automatic replay or dead-letter processing for failed webhooks; don't rely on manual fixes.