100 days of payment engineering and Stripe: the recap

We've reached day 100 of payment engineering and Stripe: the recap—a journey from rookie mistakes to production-grade payment systems. The lessons we've learned aren't academic; they're burned into our systems through incidents, reconciliation gaps, and the kinds of bugs that wake you at 3 a.m. with a Slack alert.

Webhooks are the truth — Fulfill from verified events, never redirects

The biggest mental shift in payment engineering is understanding that webhooks are your source of truth, not redirects. A user completes a Stripe Checkout session and gets redirected to your success page. That redirect is a courtesy to the frontend. It proves nothing about payment state.

The webhook—a signed charge.succeeded or payment_intent.succeeded event—is what you act on. If a webhook is slow, dropped, or retried, your idempotent handler will deal with it. If a redirect is interrupted by a network failure, a browser crash, or a Cloudflare timeout, the user won't be refunded, but you also won't fulfill their order. That asymmetry is dangerous.

Here's a pattern we've seen fail repeatedly: developers trigger order fulfillment on the redirect, then rely on a periodic sync to catch mismatches. When syncs run nightly, you accumulate hours of unfulfilled orders and frustrated customers. Instead, fulfill only when the webhook arrives and is verified.

// Webhook handler: the source of truth
app.post('/webhooks/stripe', express.raw({type: 'application/json'}), async (req, res) => {
  const sig = req.headers['stripe-signature'] as string;
  let event: Stripe.Event;

  try {
    event = stripe.webhooks.constructEvent(
      req.body,
      sig,
      process.env.STRIPE_WEBHOOK_SECRET!
    );
  } catch (err) {
    console.error('Webhook signature verification failed:', err);
    return res.status(400).send(`Webhook Error: ${err}`);
  }

  // Only act on verified events
  if (event.type === 'payment_intent.succeeded') {
    const paymentIntent = event.data.object as Stripe.PaymentIntent;
    const orderId = paymentIntent.metadata.order_id;

    // Idempotent fulfillment
    await fulfillOrder(orderId);
    console.log(`Order ${orderId} fulfilled via webhook`);
  }

  res.json({received: true});
});

// Redirect: informational only
app.get('/success', async (req, res) => {
  const sessionId = req.query.session_id as string;
  const session = await stripe.checkout.sessions.retrieve(sessionId);
  
  if (session.payment_status === 'paid') {
    // Order fulfillment already happened via webhook
    res.render('success', {orderId: session.metadata.order_id});
  } else {
    res.redirect('/checkout');
  }
});

The redirect becomes a status check, not a trigger. Your frontend can poll for order status if the user closes the browser mid-redirect, but fulfillment is asynchronous, driven by webhooks, and verified against Stripe's signed events.

Idempotency everywhere — Retries are normal; double charges are not

Payment networks are unreliable. Webhooks can be delayed, retried, or delivered out of order. Your database can fail mid-transaction. The answer is idempotency: designing every operation so it can be safely retried without side effects.

Idempotency has two layers. First, use Stripe's idempotency keys when creating charges, refunds, or payment intents. Stripe stores the result keyed by your idempotency key; retry with the same key, get the same response, no double charge.

Second, store the idempotency key and operation result in your own database. When a webhook retry arrives, check: have we already processed this event? If yes, return the cached result. Never re-execute.

// Idempotency pattern
async function fulfillOrder(orderId: string, eventId: string): Promise {
  // Check: have we processed this webhook event before?
  const processed = await db.query(
    'SELECT result FROM webhook_events WHERE event_id = $1',
    [eventId]
  );

  if (processed.rows.length > 0) {
    console.log(`Event ${eventId} already processed. Skipping.`);
    return;
  }

  try {
    // Fulfill: inventory, send email, create shipment record
    await db.query('BEGIN');
    await inventory.reserve(orderId);
    await email.sendConfirmation(orderId);
    await shipments.create(orderId);
    
    // Log success before commit
    await db.query(
      'INSERT INTO webhook_events (event_id, result) VALUES ($1, $2)',
      [eventId, 'success']
    );
    await db.query('COMMIT');
  } catch (err) {
    await db.query('ROLLBACK');
    throw err;
  }
}

When creating charges on behalf of customers, always pass an idempotency key:

const charge = await stripe.charges.create(
  {
    amount: 2000,
    currency: 'usd',
    customer: customerId,
  },
  {
    idempotencyKey: `order-${orderId}-charge`,
  }
);

If the request fails, retry with the same key. Stripe will recognize it and return the result of the first attempt. Combined with your webhook event deduplication, you're protected against retries at every level.

Model money carefully — Integers, currencies, state machines and ledgers

The single most damaging mistake in payment systems is treating money as a float. Use integers (cents, not dollars). Currency is always explicit, never implicit. And every state transition belongs in an immutable ledger.

Here's a real example: a SaaS platform stored invoice amounts as floats in PostgreSQL. After a few thousand invoices, rounding errors accumulated. Reconciliation showed $47 in discrepancies. They had to write off the difference and rebuild the financial reporting module.

Integer storage removes ambiguity:

// Schema: money as integers
type Payment = {
  id: string;
  amount_cents: number;      // Always integers
  currency: 'USD' | 'EUR' | 'GBP';  // Always explicit
  status: 'pending' | 'succeeded' | 'failed' | 'refunded';
  created_at: Date;
};

// Always convert from Stripe (also in smallest units)
const payment: Payment = {
  id: charge.id,
  amount_cents: charge.amount,  // Stripe gives cents
  currency: charge.currency.toUpperCase() as Currency,
  status: charge.status,
  created_at: new Date(charge.created * 1000),
};

State machines prevent invalid transitions. A payment can't go from succeeded to pending. A refund amount can't exceed the original charge. Encode these rules:

class PaymentStateMachine {
  static canRefund(payment: Payment, refundAmount: number): boolean {
    if (payment.status !== 'succeeded') return false;
    if (refundAmount <= 0 || refundAmount > payment.amount_cents) return false;
    return true;
  }

  static canCancel(payment: Payment): boolean {
    return payment.status === 'pending';
  }
}

// Before issuing refund
if (!PaymentStateMachine.canRefund(payment, refundAmountCents)) {
  throw new Error('Invalid refund operation');
}

Build an immutable ledger of every money movement:

type LedgerEntry = {
  id: string;
  payment_id: string;
  type: 'charge' | 'refund' | 'adjustment';
  amount_cents: number;
  currency: string;
  reason: string;
  created_at: Date;
};

async function recordLedgerEntry(
  paymentId: string,
  type: LedgerEntry['type'],
  amountCents: number,
  currency: string,
  reason: string
): Promise {
  await db.query(
    `INSERT INTO ledger (payment_id, type, amount_cents, currency, reason, created_at)
     VALUES ($1, $2, $3, $4, $5, NOW())`,
    [paymentId, type, amountCents, currency, reason]
  );
}

The ledger is append-only. You can sum it to verify balances, trace every money movement, and debug discrepancies. This is your audit trail.

Reconcile and monitor — Prove the numbers every day

You cannot know if your payment system is working until you reconcile it against Stripe's records. Reconciliation is not optional; it's your baseline health check.

Daily reconciliation answers a simple question: Do our payment records match Stripe's? Run a script that fetches all charges from Stripe for the past 24 hours, compares amounts and statuses to your database, and flags mismatches. A single mismatch might be a legitimate refund you're unaware of; ten mismatches mean a bug in your webhook handler.

async function dailyReconciliation(): Promise {
  const yesterday = new Date();
  yesterday.setDate(yesterday.getDate() - 1);

  // Fetch Stripe charges from yesterday
  const stripeCharges = await stripe.charges.list({
    created: {
      gte: Math.floor(yesterday.getTime() / 1000),
      lt: Math.floor(new Date().getTime() / 1000),
    },
  });

  // Compare to our database
  for (const stripeCharge of stripeCharges.data) {
    const ourRecord = await db.query(
      'SELECT * FROM payments WHERE stripe_id = $1',
      [stripeCharge.id]
    );

    if (ourRecord.rows.length === 0) {
      console.warn(`MISMATCH: Stripe charge ${stripeCharge.id} not in our database`);
      await alerts.slack(`Reconciliation mismatch: ${stripeCharge.id}`);
    } else {
      const payment = ourRecord.rows[0];
      if (payment.amount_cents !== stripeCharge.amount) {
        console.warn(`MISMATCH: Amount mismatch for ${stripeCharge.id}`);
      }
    }
  }

  console.log('Reconciliation complete');
}

// Run daily
cron.schedule('0 2 * * *', dailyReconciliation);

Beyond reconciliation, monitor these metrics in production:

• Webhook latency: How long between Stripe event creation and your handler processing it. Spikes here indicate either Stripe delays or your handler bottlenecks.
• Webhook retry rate: If more than 5% of webhooks are retried, investigate your handler for errors.
• Payment success rate: What percentage of payment attempts succeed? A drop from 98% to 95% signals a problem.
• Refund processing time: How long before a refund shows in the customer's account? Track this separately by processor.
• Ledger balance accuracy: Sum your ledger entries and compare to Stripe balance. Should be zero delta daily.

Common mistakes

• Treating redirects as fulfillment triggers. The success page redirect can fail without the payment failing. Always wait for the webhook.
• Not deduplicating webhook events. Processing the same webhook twice leads to double orders, double charges, or ledger corruption.
• Using floats for money. 0.1 + 0.2 ≠ 0.3 in floating-point arithmetic. Use integers.
• Skipping reconciliation. You won't discover bugs in webhook handling until reconciliation fails. Run it daily.
• Allowing refunds larger than the original charge. State machines prevent this; lack of validation guarantees it.
• Storing PII in logs. Never log card numbers, customer names, or email addresses. Log IDs and amounts only.
• Missing currency context. A charge of 1000 could be $10 USD or €100 EUR. Always store currency explicitly.

Key takeaways

• Webhooks are the source of truth for payment events. Redirects are informational only; fulfill from verified webhooks.
• Design every operation to be idempotent. Retries are normal; double charges are catastrophic. Use Stripe's idempotency keys and your own event deduplication.
• Store money as integers in the smallest currency unit. Currencies are explicit, not implicit. Model state transitions as a state machine.
• Reconcile against Stripe daily. Monitor webhook latency, retry rates, and success rates. Build an immutable ledger of every money movement.
• Test your webhook handler's retry logic before going live. Simulate failures, delays, and out-of-order delivery.
• Never log or store PII. Always use Stripe's tokenization; never handle raw payment data.
• When in doubt, add a webhook handler. It's the most reliable pattern payment engineering has.