Payment engineering is its own discipline for production

You've shipped dozens of features, scaled databases, optimized API response times, and debugged race conditions in production. None of it matters if your payment system loses money. Payment engineering is its own discipline for production—it demands different practices, different thinking, and different standards of verification than any other code you write.

The difference is immediate and measurable: every bug in your checkout flow, reconciliation logic, or webhook handler costs your business real money. Not reputation points or delayed features—actual currency. This article walks through what separates payment systems from the rest of your stack and why treating them as a specialized discipline is not optional.

Every bug has a direct cost

In most codebases, a bug means a user can't see their profile picture or a report takes ten seconds longer to load. You fix it, deploy, move on. Payment bugs are different. A logic error that causes a double charge doesn't just upset one customer—it generates chargeback fees, refund overhead, support tickets, and potential regulatory scrutiny. A rounding error in discount calculation might go unnoticed for weeks and silently drain margin across thousands of orders.

Consider a real scenario: your team builds a loyalty discount feature that applies a percentage reduction. The calculation looks reasonable:

const discountedPrice = Math.round(basePrice * (1 - discountPercent / 100));

It works for most cases. But across 50,000 orders, rounding inconsistencies compound. Some customers pay $9.99, others pay $10.00, when both should pay the same. After a month, you discover the system undercollected $847. Finding the root cause takes days. The fix takes minutes. The cost is real.

Every payment feature must be built with the assumption that bugs will happen—and that their impact will be financial. This assumption changes how you architect, test, and operate the system. It raises the bar for code review rigor, test coverage, and monitoring. It means you don't ship payment features on Friday afternoon. It means you have a plan to detect and reverse erroneous transactions before customers and regulators discover them.

Webhooks, retries and idempotency are mandatory

Stripe sends your server a webhook notification when a payment succeeds, fails, or is disputed. You rely on that webhook to update your database, trigger fulfillment, send a receipt email, and record the transaction. But webhooks are not guaranteed delivery. Networks fail. Your server restarts. A bug crashes your webhook handler. Stripe retries—but what if your second processing is a duplicate?

This is where idempotency becomes non-negotiable. Every payment operation must be designed to produce the same outcome no matter how many times it's executed. Not "mostly the same" or "usually safe"—mathematically identical.

Here's a webhook handler that is not idempotent:

app.post('/webhook', async (req, res) => {
  const event = req.body;
  
  if (event.type === 'charge.succeeded') {
    const charge = event.data.object;
    
    // BAD: No idempotency check
    await db.transactions.create({
      stripeChargeId: charge.id,
      amount: charge.amount,
      status: 'completed'
    });
    
    await db.orders.update(
      { id: charge.metadata.orderId },
      { status: 'paid' }
    );
    
    res.json({ received: true });
  }
});

If this webhook fires twice (which will happen), you insert the transaction twice, update the order twice, and create confusion in your records. Money gets tracked incorrectly. Reconciliation fails.

Here's the corrected version:

app.post('/webhook', async (req, res) => {
  const event = req.body;
  
  if (event.type === 'charge.succeeded') {
    const charge = event.data.object;
    
    // GOOD: Idempotency via upsert on unique identifier
    const transaction = await db.transactions.upsert(
      { stripeChargeId: charge.id },
      {
        create: {
          stripeChargeId: charge.id,
          amount: charge.amount,
          status: 'completed',
          processedAt: new Date()
        },
        update: {} // Do nothing if already exists
      }
    );
    
    // Only update order if transaction was newly created
    if (transaction.createdAt.getTime() === new Date().getTime()) {
      await db.orders.update(
        { id: charge.metadata.orderId },
        { status: 'paid' }
      );
    }
    
    res.json({ received: true });
  }
});

The key principle: use a unique identifier (Stripe charge ID, idempotency key, or equivalent) to ensure the operation is truly idempotent. Process the same event ten times and your database state remains unchanged. Stripe itself uses idempotency keys for API requests—you must do the same for webhook processing.

Retries are equally critical. Your webhook handler should fail gracefully and allow Stripe to retry. Return a 5xx status code or throw an error if processing fails, so Stripe knows to try again. We've seen production incidents where teams swallowed exceptions and returned 200 OK even when the payment wasn't recorded—Stripe assumed success and never retried, leaving the transaction lost.

Reconciliation proves the numbers

You cannot trust your payment system just because it doesn't crash. You must verify it regularly and systematically. This is reconciliation: comparing your internal records against Stripe's records and proving they match.

Every morning (or at least weekly), you should run a reconciliation job that:

1. Pulls all charges from Stripe for the period using the Charges API
2. Compares them against your local transaction records
3. Flags discrepancies: charges in Stripe but not in your database, or vice versa
4. Alerts your team if the totals don't match

Here's a simplified example:

async function reconcileCharges(startDate: Date, endDate: Date) {
  // Fetch all charges from Stripe for the period
  const stripeCharges = await stripe.charges.list({
    created: {
      gte: Math.floor(startDate.getTime() / 1000),
      lte: Math.floor(endDate.getTime() / 1000)
    },
    limit: 100 // Paginate in production
  });
  
  // Fetch all transactions from our database
  const localTransactions = await db.transactions.findMany({
    processedAt: {
      gte: startDate,
      lte: endDate
    }
  });
  
  const stripeIds = new Set(stripeCharges.data.map(c => c.id));
  const localIds = new Set(localTransactions.map(t => t.stripeChargeId));
  
  // Find missing transactions
  const missingInLocal = Array.from(stripeIds).filter(id => !localIds.has(id));
  const missingInStripe = Array.from(localIds).filter(id => !stripeIds.has(id));
  
  if (missingInLocal.length > 0 || missingInStripe.length > 0) {
    await sendAlert({
      missingInLocal,
      missingInStripe,
      message: 'Payment reconciliation failed'
    });
  }
  
  // Verify amounts
  const stripeTotalCents = stripeCharges.data.reduce((sum, c) => sum + c.amount, 0);
  const localTotalCents = localTransactions.reduce((sum, t) => sum + t.amountCents, 0);
  
  if (stripeTotalCents !== localTotalCents) {
    await sendAlert({
      stripeTotalCents,
      localTotalCents,
      message: 'Payment amount mismatch'
    });
  }
}

Reconciliation is your safety net. It catches bugs early. It proves to auditors and regulators that you know your numbers. More importantly, it gives you confidence that your payment system is working correctly. No amount of unit tests replaces regular reconciliation against your actual payment processor.

Security and compliance shape the design

You cannot store credit card data. Full stop. This isn't a nice-to-have—it's legally required. PCI DSS (Payment Card Industry Data Security Standard) prohibits storing raw card details. Violating it can result in fines, card brand restrictions, and mandatory audits.

Stripe eliminates this problem: you never touch the card data. Instead, you collect it via Stripe Elements, Stripe Hosted Checkout, or Payment Method API, and Stripe returns a token. Your application stores the token, not the card.

// WRONG: Never do this
app.post('/charge', async (req, res) => {
  const { cardNumber, exp, cvc, amount } = req.body;
  
  // You are now liable for PCI compliance
  await db.payments.create({
    cardNumber, exp, cvc, amount // DO NOT DO THIS
  });
});

// RIGHT: Use Stripe tokenization
app.post('/charge', async (req, res) => {
  const { paymentMethodId, amount } = req.body; // Token from frontend
  
  const charge = await stripe.charges.create({
    amount,
    payment_method: paymentMethodId,
    confirm: true
  });
  
  // You never see the card details
  await db.transactions.create({
    stripeChargeId: charge.id,
    amount
  });
});

Beyond PCI, consider:

• HTTPS everywhere: All payment endpoints require TLS 1.2+. No exceptions.
• Webhook signature verification: Stripe signs every webhook with a secret key. Verify the signature before processing. This proves the webhook came from Stripe, not an attacker.
• Rate limiting: Prevent abuse of payment endpoints with aggressive rate limiting.
• Audit logging: Log all payment operations (charges, refunds, disputes) with timestamps and user information. Immutable logs help during investigations.
• Access control: Restrict who can access payment data and APIs. Use role-based access control (RBAC) and least privilege principles.

These aren't optional enhancements—they're foundational requirements that must be designed in from day one.

Common mistakes

• Treating webhooks as guaranteed delivery: They are not. Always design for retry and idempotency.
• Skipping reconciliation: "It works in staging" is not proof. Reconciliation against your payment processor is the only proof.
• Rounding errors in financial calculations: Always work in the smallest unit (cents, not dollars). Use integer math. Never use floating-point for currency.
• Not verifying webhook signatures: A simple check that takes seconds prevents entire categories of fraud and injection attacks.
• Storing Stripe data locally instead of querying live: Your source of truth is Stripe (or your payment processor). Keep your cache in sync via webhooks, but always reconcile.
• Deploying payment changes on Friday: Payment bugs are time-sensitive. Deploy early in the week when your team can respond quickly to issues.

Key takeaways

• Payment bugs have direct financial cost. Treat them as your highest-priority bugs.
• Idempotency is mandatory for all payment operations. Process the same event 100 times and get identical results.
• Webhooks require explicit retry and idempotency logic. Never assume a webhook fires exactly once.
• Reconciliation is your source of truth. Compare your records against Stripe daily or weekly, every single time.
• Security and compliance are part of the design, not an afterthought. Never store card data. Verify webhook signatures. Use HTTPS everywhere.
• Payment engineering is a discipline with its own patterns and practices. Invest in getting it right.