Introduction
By day 82 of our payment engineering journey, we've covered the theory: webhooks, retry logic, reconciliation. Now comes the harder part: learning from the bugs that slip through. Lessons from real payment bugs: best practices explained aren't just nice-to-have patterns—they're the difference between a system that works in QA and one that doesn't leak money or customer trust in production.
We've collected four categories of failures we've seen across production systems handling millions in transaction volume. Each one hurt someone, and each one could have been prevented. Let's walk through them.
Double charges from missing idempotency
The most common payment bug we see: a customer gets charged twice because the same request was processed twice. This usually happens when a timeout or network error causes your client to retry, but your backend doesn't recognize it as a replay.
Here's a realistic scenario: a customer submits an order, your frontend hits the Stripe API, the network stutters, the request times out on the client side. The customer refreshes or clicks "Pay" again. Your server receives the same request again—and if you're not careful, you create two charges.
Stripe's charge object has a built-in defense: the idempotency_key. When you send the same key, Stripe returns the same result without re-processing. But you have to use it. And you have to store it.
// Without idempotency key—WRONG
async function chargeCustomer(customerId: string, amountCents: number) {
const charge = await stripe.charges.create({
amount: amountCents,
customer: customerId,
currency: 'usd',
});
return charge;
}
// With idempotency key—CORRECT
async function chargeCustomer(customerId: string, amountCents: number, orderId: string) {
const idempotencyKey = `order-${orderId}`;
const charge = await stripe.charges.create(
{
amount: amountCents,
customer: customerId,
currency: 'usd',
description: `Order ${orderId}`,
},
{
idempotencyKey,
}
);
return charge;
}
The key insight: the idempotency key must be derived from something stable and unique on your side, not something random. Order ID, user ID plus timestamp, invoice number—anything that will be the same if the same request comes in again.
But here's where teams stumble: you also need to make sure your database transaction is atomic. Charge first, then record the charge in your database. If you record it first and the charge fails mid-flight, you've logged a charge that never happened. If the charge succeeds but your database write fails, the idempotency key won't save you on the next retry because the charge is already created.
// Better: charge and log atomically
async function chargeAndLog(customerId: string, amountCents: number, orderId: string, db: Database) {
const idempotencyKey = `order-${orderId}`;
try {
const charge = await stripe.charges.create(
{
amount: amountCents,
customer: customerId,
currency: 'usd',
metadata: { orderId },
},
{ idempotencyKey }
);
// Write to database in the same transaction
await db.transaction(async (trx) => {
await trx('charges').insert({
order_id: orderId,
charge_id: charge.id,
amount: amountCents,
created_at: new Date(),
});
});
return charge;
} catch (error) {
// Even if we throw, Stripe has the charge on file
// Next retry with same idempotencyKey will return the existing charge
throw error;
}
}
Lost orders from missed webhooks
Webhooks are unreliable by design. Stripe sends them, but your server might be down. Your code might crash. The internet might drop the packet. And if you don't handle that properly, an order sits in limbo forever—the payment succeeded on Stripe's side, but your system never knew about it.
We've watched entire days of orders go missing because a production deployment took down the webhook endpoint for three minutes and the retry window closed. Or because a database connection pool was exhausted and webhook handlers started timing out and failing silently.
The solution has three parts:
- Idempotent webhook handling: Use the Stripe event ID to deduplicate. If you've already processed event
evt_1234, don't process it again. - Exponential backoff on your side: Don't just fail once and move on. Queue the webhook for retry with backoff.
- Reconciliation as a safety net: Periodically query Stripe's API to find charges that exist in Stripe but not in your system.
// Idempotent webhook handler
async function handleStripeWebhook(event: Stripe.Event, db: Database) {
// Check if we've already processed this event
const existing = await db('webhook_events')
.where('event_id', event.id)
.first();
if (existing) {
console.log(`Event ${event.id} already processed at ${existing.processed_at}`);
return { received: true };
}
try {
if (event.type === 'charge.succeeded') {
const charge = event.data.object as Stripe.Charge;
await db.transaction(async (trx) => {
// Only insert if order doesn't exist
const existingOrder = await trx('orders')
.where('stripe_charge_id', charge.id)
.first();
if (!existingOrder) {
await trx('orders').insert({
stripe_charge_id: charge.id,
amount: charge.amount,
customer_email: charge.receipt_email,
status: 'paid',
created_at: new Date(),
});
}
// Record this event as processed
await trx('webhook_events').insert({
event_id: event.id,
event_type: event.type,
processed_at: new Date(),
});
});
}
} catch (error) {
console.error(`Failed to process event ${event.id}:`, error);
// Queue for retry instead of throwing
await db('webhook_queue').insert({
event_id: event.id,
event_data: JSON.stringify(event),
retry_count: 0,
next_retry_at: new Date(Date.now() + 5000), // Retry in 5 seconds
});
throw error; // Return 500 to Stripe so it retries
}
return { received: true };
}
The reconciliation job runs daily or on-demand:
// Reconciliation: find payments in Stripe but not in our database
async function reconcileStripeCharges(db: Database, stripe: Stripe) {
const lastReconcile = await db('reconciliation_log')
.orderBy('checked_until', 'desc')
.first();
const since = lastReconcile?.checked_until || new Date(Date.now() - 30 * 24 * 60 * 60 * 1000);
let hasMore = true;
let startingAfter: string | undefined;
while (hasMore) {
const charges = await stripe.charges.list({
limit: 100,
starting_after: startingAfter,
created: { gte: Math.floor(since.getTime() / 1000) },
});
for (const charge of charges.data) {
const existing = await db('orders')
.where('stripe_charge_id', charge.id)
.first();
if (!existing && charge.paid) {
console.warn(`Missing order for charge ${charge.id}, creating now`);
await db('orders').insert({
stripe_charge_id: charge.id,
amount: charge.amount,
customer_email: charge.receipt_email,
status: 'paid',
created_at: new Date(charge.created * 1000),
reconciled: true,
});
}
}
hasMore = charges.has_more;
if (charges.data.length > 0) {
startingAfter = charges.data[charges.data.length - 1].id;
}
}
await db('reconciliation_log').insert({
checked_until: new Date(),
checked_at: new Date(),
});
}
Wrong totals from float math
Money is not a floating-point number. Ever. We've seen orders with a total of $9.999999999, refunds that were off by a penny, and invoice line items that didn't add up because someone used JavaScript division.
Always work in the smallest unit of currency your system supports. For USD, that's cents. For Bitcoin, satoshis. Do all math in integers.
// WRONG—floats will bite you
function calculateOrderTotal(lineItems: LineItem[]) {
return lineItems.reduce((sum, item) => sum + item.price * item.quantity, 0);
}
// CORRECT—work in cents, use integers
function calculateOrderTotalCents(lineItems: LineItem[]): number {
let totalCents = 0;
for (const item of lineItems) {
const itemTotalCents = item.priceCents * item.quantity;
totalCents += itemTotalCents;
}
return totalCents;
}
// When sending to Stripe, always use cents
const chargeAmount = calculateOrderTotalCents(lineItems);
await stripe.charges.create({
amount: chargeAmount, // already in cents, no conversion needed
customer: customerId,
currency: 'usd',
});
Apply this to taxes, discounts, and refunds too. Calculate everything in cents, store everything in cents, and only convert to decimal when displaying to a user (and use proper formatting, not string concatenation).
Access granted after failed payments
A subtle one: a payment fails, but your code continues anyway. The customer never paid, but you unlock their premium features, process their order, or grant them access. This usually happens because error handling is incomplete.
Real example: a SaaS subscription system checks if a payment succeeded by looking for a charge.succeeded webhook, but the handler crashes before it finishes. The payment went through on Stripe's side, but the user never gets access. Or worse: the payment failed on Stripe, but your code assumes it succeeded because there was no explicit error thrown.
// WRONG—doesn't check if charge actually succeeded
async function processSubscription(userId: string, planId: string, stripe: Stripe) {
const customer = await stripe.customers.create({
metadata: { userId },
});
const subscription = await stripe.subscriptions.create({
customer: customer.id,
items: [{ price: planId }],
});
// Assume it worked
await db('subscriptions').insert({
user_id: userId,
plan_id: planId,
stripe_subscription_id: subscription.id,
status: 'active',
});
return subscription;
}
// CORRECT—verify payment status before granting access
async function processSubscription(userId: string, planId: string, stripe: Stripe, db: Database) {
try {
const customer = await stripe.customers.create({
metadata: { userId },
});
const subscription = await stripe.subscriptions.create({
customer: customer.id,
items: [{ price: planId }],
});
// Check if the subscription is actually active (not pending payment, not past due)
if (subscription.status !== 'active' && subscription.status !== 'trialing') {
throw new Error(`Subscription created but status is ${subscription.status}, not provisioning access`);
}
await db('subscriptions').insert({
user_id: userId,
plan_id: planId,
stripe_subscription_id: subscription.id,
status: subscription.status,
});
return subscription;
} catch (error) {
console.error(`Failed to provision subscription for user ${userId}:`, error);
// Re-throw, don't grant access
throw error;
}
}
The pattern: every payment operation must have an explicit success state. Check the Stripe object's status field. Verify the webhook fired and was processed. Don't assume silence means success.
Common mistakes
- Not storing idempotency keys: Using a random UUID or no key at all means retries create duplicate charges.
- Forgetting to handle webhook failures: A single timeout or database error loses the order forever if there's no retry mechanism.
- Using floats for money: Results in rounding errors and mismatched totals that trigger chargebacks.
- Assuming "no error" means "success": Silent failures or incomplete error handling grant access or fulfillment after failed payments.
- Skipping reconciliation: Webhooks fail. Reconsolidate with Stripe weekly to catch the gap.
- Not logging transaction details: When a payment bug surfaces, you need audit trails. Log the Stripe charge ID, idempotency key, and timestamp with every payment operation.
Key takeaways
- Use idempotency keys derived from stable data (order ID, not random UUIDs) and store them with your charge records.
- Treat webhooks as unreliable: build retry queues, deduplication, and weekly reconciliation into your pipeline.
- Do all money math in the smallest currency unit (cents for USD) and use integers, never floats.
- Verify payment status explicitly before granting access or fulfilling orders; don't assume silence means success.
- Log everything: charge IDs, idempotency keys, webhook event IDs, and timestamps for post-mortem investigation when things go wrong.
Comments
Loading comments…