Graceful degradation in practice

I've watched production incidents go from "one service is slow" to "entire platform unusable" more times than I'd like to admit. The difference between a controlled outage and a cascading failure often comes down to one decision made long before things break: graceful degradation in practice. When something fails, you need a plan to break a little — not everything.

Graceful degradation means your system continues delivering value when a dependency becomes unavailable, slow, or unreliable. It's not about pretending nothing happened. It's about being deliberate: this feature is critical, that one can wait, and this other one can show a sensible fallback. Over 5+ years building platforms at scale, I've learned that the teams with the calmest incident responses aren't the ones with perfect infrastructure — they're the ones who decided their fallbacks before things went wrong.

Recommendations down? Show popular items

Your machine learning recommendation service is a heavyweight. It's usually worth it, but when it's slow or broken, your e-commerce platform shouldn't go silent. I've seen product listing pages that show nothing but spinners because the recommendations endpoint timed out.

A better approach: when recommendations fail, fall back to pre-computed popular items. These are cheap to calculate and cache. You lose personalization, but you keep the browsing experience alive.

Here's a practical pattern using NestJS and Redis:

typescript

@Injectable()
export class ProductService {
  constructor(
    private readonly http: HttpClient,
    private readonly cache: CacheService,
    private readonly logger: Logger,
  ) {}

  async getProductsForUser(userId: string): Promise {
    try {
      // Try personalized recommendations first
      const recommendations = await this.http
        .get(`/ml-service/recommendations/${userId}`, {
          timeout: 2000, // Hard timeout
        })
        .toPromise();
      
      return recommendations;
    } catch (error) {
      this.logger.warn(`Recommendations failed for user ${userId}`, error);
      
      // Fallback: return popular items
      const popular = await this.cache.get('popular-products');
      if (!popular) {
        // If cache is also cold, return a curated default list
        return this.getDefaultFeaturedProducts();
      }
      
      return popular;
    }
  }

  private async getDefaultFeaturedProducts(): Promise {
    // Hard-coded or database-backed featured products
    // This should rarely execute, but when it does, the page still loads
    return [
      { id: 1, name: 'Best Seller 1', price: 29.99 },
      { id: 2, name: 'Best Seller 2', price: 49.99 },
    ];
  }
}

The key details: a hard timeout (2 seconds, not 30), explicit error handling, and layered fallbacks. When the ML service is down, users still see products. That's graceful degradation.

Search down? Show categories

Full-text search is powerful, but it's also complex. Elasticsearch, Meilisearch, or Algolia can be single points of failure if you haven't built defensively. I've seen platforms where search outages meant customers couldn't browse product categories at all because everything funneled through the search API.

The fix: your category navigation should work independently. When search is unavailable, users navigate by category. They lose the ability to search for specific terms, but they can still find products.

Structure it like this:

typescript

@Controller('products')
export class ProductController {
  constructor(
    private readonly searchService: SearchService,
    private readonly categoryService: CategoryService,
    private readonly logger: Logger,
  ) {}

  @Get('search')
  async search(@Query('q') query: string): Promise {
    try {
      return await this.searchService.query(query, {
        timeout: 3000,
      });
    } catch (error) {
      this.logger.warn(`Search failed for query: ${query}`, error);
      
      // Fallback: extract category keywords and return category-level results
      const suggestedCategory = this.extractCategory(query);
      if (suggestedCategory) {
        const products = await this.categoryService.getProducts(
          suggestedCategory,
        );
        return products.map((p) => ({
          ...p,
          source: 'category-fallback',
          note: 'Search is unavailable; showing category results',
        }));
      }
      
      // Last resort: return popular searches
      return await this.getCachedPopularSearches();
    }
  }

  private extractCategory(query: string): string | null {
    const keywords = {
      electronics: ['phone', 'laptop', 'headphone'],
      home: ['chair', 'desk', 'lamp'],
    };
    
    for (const [category, terms] of Object.entries(keywords)) {
      if (terms.some((term) => query.toLowerCase().includes(term))) {
        return category;
      }
    }
    
    return null;
  }
}

Pair this with a category navigation system that doesn't depend on search at all. When search is down, users navigate by Home → Electronics → Phones. Not ideal, but functional.

Payments slow? Queue confirmations

This is the highest-stakes scenario. Payment processing is typically synchronous: user submits, payment gateway responds, you show success or failure. But payment gateways can be slow. I once watched an entire checkout flow block because a payment gateway was responding in 15+ seconds instead of the usual 2.

The graceful approach: when payment processing is slow, accept the payment optimistically, queue a confirmation job, and return success to the user immediately. The confirmation job retries until it gets a definitive answer. This turns a synchronous blocker into an asynchronous background task.

typescript

@Injectable()
export class PaymentService {
  constructor(
    private readonly paymentGateway: PaymentGateway,
    private readonly queue: Queue,
    private readonly orderService: OrderService,
    private readonly logger: Logger,
  ) {}

  async processPayment(orderId: string, amount: number): Promise {
    const timeoutMs = 5000; // 5-second timeout

    try {
      const result = await Promise.race([
        this.paymentGateway.charge({
          orderId,
          amount,
        }),
        this.delay(timeoutMs).then(() => {
          throw new Error('Payment timeout');
        }),
      ]);

      if (result.success) {
        await this.orderService.markPaymentConfirmed(orderId);
        return { success: true, transactionId: result.transactionId };
      }

      return { success: false, error: result.error };
    } catch (error) {
      // Timeout or transient error: queue for async confirmation
      this.logger.warn(`Payment processing slow for order ${orderId}`, error);

      // Optimistically create the order
      await this.orderService.createOrder(orderId, 'payment_pending');

      // Queue the payment confirmation job
      await this.queue.add('confirm-payment', {
        orderId,
        amount,
        attempts: 0,
        maxAttempts: 10,
      });

      // Return success to user
      return {
        success: true,
        pending: true,
        message: 'Payment is being confirmed. You will receive a confirmation email.',
      };
    }
  }

  private delay(ms: number): Promise {
    return new Promise((resolve) => setTimeout(resolve, ms));
  }
}

// Background job processor
@Processor('confirm-payment')
export class PaymentConfirmationJob {
  constructor(
    private readonly paymentGateway: PaymentGateway,
    private readonly orderService: OrderService,
  ) {}

  @Process()
  async handleConfirmation(job: Job): Promise {
    const { orderId, amount, attempts, maxAttempts } = job.data;

    try {
      const status = await this.paymentGateway.getTransactionStatus(orderId);
      
      if (status.confirmed) {
        await this.orderService.markPaymentConfirmed(orderId);
      } else if (status.failed) {
        await this.orderService.markPaymentFailed(orderId);
      } else if (attempts < maxAttempts) {
        // Retry after delay
        throw new Error('Payment still pending');
      }
    } catch (error) {
      if (job.attemptsMade < maxAttempts) {
        throw error; // Bull will retry
      }
      // Max attempts reached, mark as manual review needed
      await this.orderService.flagForManualReview(orderId);
    }
  }
}

This approach separates the user experience from the payment gateway's performance. If it's slow, users get confirmation instantly and move forward. The system confirms the payment asynchronously. If it ultimately fails, support has a clear audit trail.

Decide fallbacks before the incident

This is the meta-point, and it's the most important. Graceful degradation isn't something you figure out at 3 AM when production is on fire. You decide it during planning, design review, or team discussion.

Before building any critical feature, ask:

What if this dependency is down for 5 minutes? 10 minutes? An hour?
What's the acceptable user experience? Faster but stale data? Degraded functionality? A helpful message?
What's the cost of that fallback? Cache memory? Database load? Manual intervention?
How do we detect the failure? Timeouts, specific error codes, health checks?
How do we communicate it? Silently, with a banner, or in the UI?

Document these decisions in your architecture decision records (ADRs) or design docs. When the incident happens, the on-call engineer isn't guessing — they're executing a plan.

typescript

// Example: decision matrix in code
const FALLBACK_STRATEGY = {
  recommendations: {
    timeout: 2000,
    fallback: 'cached-popular-items',
    communication: 'silent',
    sla: 'best-effort',
  },
  search: {
    timeout: 3000,
    fallback: 'category-navigation',
    communication: 'banner',
    sla: 'best-effort',
  },
  payments: {
    timeout: 5000,
    fallback: 'async-confirmation',
    communication: 'ui-message',
    sla: 'critical',
  },
};

Common mistakes

No timeout. Hoping a slow dependency will eventually respond is not a strategy. Set explicit, reasonable timeouts everywhere.
Fallback without testing. You can't know if your fallback works until production hits it hard. Load-test your fallback paths.
Hiding failures completely. Users can often tell something's off (missing personalization, stale data). A honest message beats silent degradation: "We're showing popular items while personalization is unavailable."
Over-engineering fallbacks. You don't need a perfect fallback for every feature. Decide what's worth the complexity. Recommendations? Yes. A niche admin panel feature? Probably not.
Forgetting to monitor it. Set up alerts for when fallbacks activate. A production incident where every request uses the fallback should wake someone up, not go unnoticed.

Key takeaways

Graceful degradation means breaking a little when a dependency fails, not breaking everything.
Common patterns: fallback to cached data, degrade to a simpler feature (search → categories), or move synchronous work to async (payments).
Set hard timeouts on all external calls. 2–5 seconds is typical; 30 seconds is waiting for trouble.
Design fallbacks before the incident. Document them, test them, monitor them.
Communicate honestly with users about what's degraded. It builds trust more than silent failures.
Layer your fallbacks: primary → cache → default → manual review. Each layer buys time or delivers minimum value.
Not every feature needs a fallback. Prioritize by user impact and incident cost.

Graceful degradation in practice: lessons from production