Building Resilience Into Your API
When an external service goes down for 30 seconds, or a database connection pool becomes temporarily exhausted, a naive request will fail immediately. In production, that single failure cascades: users see errors, your error rate spikes, and your team gets paged. Retries and backoff done right is a retry strategy that is part of your API design, not an afterthought bolted on when things break. Over 35 days into building with NestJS, this is the difference between a system that bends under load and one that fails gracefully.
We'll walk through how to implement exponential backoff with jitter, cap your attempts sensibly, route persistent failures to a dead-letter queue, distinguish transient errors from real problems, and ensure every job is idempotent. This is how mature teams handle failure.
Exponential Backoff with Jitter
The naive retry—fire the same request immediately—hammers a struggling service and often makes things worse. Exponential backoff spaces retries further apart: wait 1 second, then 2, then 4, then 8. The service gets breathing room to recover.
But if every client retries on the same schedule, you get thundering herd: the moment a service comes back online, it gets slammed by thousands of synchronized requests. Jitter—a random delay added to each backoff—spreads those retries out.
Here's a helper function that calculates backoff with jitter:
export function calculateBackoff(
attemptNumber: number,
baseDelayMs: number = 100,
maxDelayMs: number = 30000,
): number {
const exponentialDelay = baseDelayMs * Math.pow(2, attemptNumber - 1);
const cappedDelay = Math.min(exponentialDelay, maxDelayMs);
const jitter = Math.random() * cappedDelay * 0.1; // 10% jitter
return cappedDelay + jitter;
}
For attempt 1, you wait ~100ms plus jitter. For attempt 2, ~200ms plus jitter. By attempt 5, you're at the 30-second ceiling. The jitter ensures clients don't all retry in lockstep.
A practical decorator in NestJS might look like this:
export function Retry(options: {
maxAttempts?: number;
baseDelayMs?: number;
maxDelayMs?: number;
onlyTransient?: boolean;
} = {}) {
const {
maxAttempts = 3,
baseDelayMs = 100,
maxDelayMs = 30000,
onlyTransient = true,
} = options;
return (target: any, key: string, descriptor: PropertyDescriptor) => {
const originalMethod = descriptor.value;
descriptor.value = async function (...args: any[]) {
let lastError: Error | null = null;
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
try {
return await originalMethod.apply(this, args);
} catch (error) {
lastError = error as Error;
// Only retry transient errors if enabled
if (onlyTransient && !isTransientError(error)) {
throw error;
}
if (attempt < maxAttempts) {
const delayMs = calculateBackoff(
attempt,
baseDelayMs,
maxDelayMs,
);
await new Promise((resolve) => setTimeout(resolve, delayMs));
}
}
}
throw lastError;
};
return descriptor;
};
}
Use it on any method:
@Injectable()
export class PaymentService {
@Retry({ maxAttempts: 4, baseDelayMs: 200 })
async chargeCard(cardToken: string, amount: number) {
// Call external payment provider
}
}
Cap Attempts and Move Failures to a Dead-Letter Queue
After you've retried a few times and still failed, you need a place to send that failed job so a human or a separate process can investigate. A dead-letter queue (DLQ) is your safety net.
When using a queue library like Bull or RabbitMQ, configuring a DLQ looks like this:
@Module({
imports: [
BullModule.registerQueue({
name: 'payments',
defaultJobOptions: {
attempts: 3,
backoff: {
type: 'exponential',
delay: 2000,
},
removeOnComplete: true,
},
}),
],
})
export class PaymentQueueModule {}
When a job exhausts its attempts, Bull can move it to a companion queue (the DLQ). You then subscribe to that queue:
@Processor('payments')
export class PaymentProcessor {
constructor(
@InjectQueue('payments') private paymentQueue: Queue,
private logger: Logger,
) {}
@Process()
async handlePayment(job: Job) {
try {
await this.chargeCard(job.data);
} catch (error) {
this.logger.warn(
`Job ${job.id} failed: ${error.message}. Attempt ${job.attemptsMade} of ${job.opts.attempts}`,
);
throw error; // Re-throw to trigger Bull's retry mechanism
}
}
@OnQueueFailed()
async handleFailed(job: Job, error: Error) {
this.logger.error(
`Job ${job.id} permanently failed after ${job.opts.attempts} attempts`,
);
// Move to DLQ or alert ops
await this.paymentQueue.add(
'dlq-payload',
{ jobId: job.id, originalData: job.data, error: error.message },
{ queue: 'payments-dlq' },
);
}
}
A separate worker monitors the DLQ, logs issues, and triggers alerts. This keeps your main pipeline clean and gives you audit trail of what failed and why.
Only Retry Transient Errors
Not every error deserves a retry. If a user sends a malformed request (400 Bad Request), retrying won't fix it. If your code has a bug that throws an exception, retrying five times won't change the outcome. But if a database is temporarily offline, or a third-party API is rate-limiting, a retry makes sense.
Transient errors are temporary, recoverable faults. Permanent errors are not. Here's how to distinguish:
export function isTransientError(error: any): boolean {
// Network timeouts and connection errors
if (
error.code === 'ETIMEDOUT' ||
error.code === 'ECONNREFUSED' ||
error.code === 'ECONNRESET'
) {
return true;
}
// HTTP status codes that indicate transient issues
if (error.status) {
const transientStatusCodes = [408, 429, 500, 502, 503, 504];
return transientStatusCodes.includes(error.status);
}
// Database connection errors
if (error.message?.includes('ENOTFOUND')) {
return true;
}
return false;
}
Then use this function in your retry logic:
async chargeCard(cardToken: string): Promise {
try {
await this.externalPaymentApi.charge(cardToken);
} catch (error) {
if (!isTransientError(error)) {
this.logger.error(`Non-transient error: ${error.message}`);
throw error; // Fail immediately, don't retry
}
// If transient, the decorator will handle retries
throw error;
}
}
This prevents wasting retry budget on errors that won't change.
Make Every Job Idempotent
Here's the catch: if you retry a charge operation, you might charge the customer twice. Retries only work safely if your operations are idempotent—running them once or ten times produces the same result.
The standard pattern is to use an idempotency key: a unique identifier for the intent, not the execution.
@Injectable()
export class PaymentService {
constructor(
private db: Database,
private externalApi: ExternalPaymentApi,
) {}
async chargeWithIdempotency(
cardToken: string,
amount: number,
idempotencyKey: string,
): Promise<{ transactionId: string }> {
// Check if we've already processed this intent
const existing = await this.db.transaction.findOne({
idempotencyKey,
});
if (existing) {
return { transactionId: existing.id };
}
// Process for the first time
const result = await this.externalApi.charge(cardToken, amount);
// Store with the idempotency key
await this.db.transaction.create({
idempotencyKey,
amount,
externalTransactionId: result.id,
});
return { transactionId: result.id };
}
}
The client generates a UUID for each logical operation and passes it with every request. Your service checks if that key already exists in the database; if it does, you return the cached result instead of processing again. Now retries are safe.
Common Mistakes
- Retrying without checking the error type. Retrying a validation error or permission denied error won't help. Always inspect the error first.
- Using the same delay for all retries. Unneeded sleep increases latency for the client. Use backoff, and cap it. After 30 seconds, stop waiting.
- Forgetting the jitter. Without jitter, synchronized retries hammer your service and can cause cascading failures.
- No idempotency key. Once you retry, you must track what you've already done. A missing idempotency layer means duplicate charges, duplicate emails, or corrupted data.
- Ignoring dead-letter queues. If a job fails permanently, log it somewhere you can inspect it later. Don't just drop it silently.
Key Takeaways
- Exponential backoff with jitter spreads retries across time and reduces thundering herd risk.
- Cap your retry attempts (3–5 is typical) and move persistent failures to a dead-letter queue for investigation.
- Only retry transient errors (timeouts, 503, connection failures). Let permanent errors fail fast.
- Implement idempotency keys and check them before processing. This makes retries safe and prevents duplicate side effects.
- Retries are part of your API contract. Design them in from the start, not as an emergency patch.
Comments
Loading comments…