Rate limiting outgoing requests | Expona Engineering Blog

Rate limiting outgoing requests: lessons from production

We've all hit it: a third-party API returning 429 Too Many Requests, followed by cascading errors through our system. Rate limiting outgoing requests is one of those invisible problems until it isn't—then it takes down a whole feature. Over years running payment processors, shipping integrations, and analytics pipelines in production, we've learned that respecting third-party API limits isn't optional; it's foundational to reliable systems. This guide walks you through the patterns we use to keep outgoing requests in check, from client-side throttling to intelligent retry logic and queue-based backpressure.

The case for rate.Limiter before each call

Go's golang.org/x/time/rate package gives us a token bucket implementation that's lightweight and effective. The idea is simple: before making a call to an external API, we ask the limiter if we can proceed. If we're under quota, we go. If we're over, we wait or drop the request depending on our strategy.

In a real e-commerce checkout flow, we integrate with a payment processor that allows 100 requests per second. Without rate limiting, a traffic spike could hammer their API and get us throttled or blocked entirely. With rate.Limiter, we enforce the contract ourselves:

package payment

import (
	"context"
	"fmt"
	"golang.org/x/time/rate"
)

type PaymentClient struct {
	limiter *rate.Limiter
	apiKey  string
}

// NewPaymentClient creates a client with 100 req/s limit
func NewPaymentClient(apiKey string) *PaymentClient {
	return &PaymentClient{
		limiter: rate.NewLimiter(rate.Limit(100), 1),
		apiKey:  apiKey,
	}
}

func (p *PaymentClient) Charge(ctx context.Context, amount int64) (string, error) {
	// Wait for a token or return early if context is done
	if err := p.limiter.Wait(ctx); err != nil {
		return "", fmt.Errorf("rate limit exceeded: %w", err)
	}

	// Now safe to call the API
	return p.chargeAPI(ctx, amount)
}

func (p *PaymentClient) chargeAPI(ctx context.Context, amount int64) (string, error) {
	// Actual HTTP call to payment processor
	// ...
	return "charge_id_123", nil
}

The rate.Limiter here is configured with 100 requests per second and a burst of 1. The burst allows one immediate request; beyond that, we queue. If your API allows bursts (like 20 requests instantly, then 5/sec sustained), set the burst higher. Always tune this to match the documented limits of the service you're calling.

One critical detail: the Wait(ctx) call respects context cancellation. If the request times out while waiting for a token, we fail fast instead of hanging indefinitely. This is especially important in request handlers where timeouts propagate from the HTTP server.

Retry 429s with backoff

Even with client-side rate limiting, you'll still hit 429 responses. Other clients might be pounding the same API, or the limit might be per-account and you're running multiple services. When you get a 429, the right move is to back off exponentially and retry—but do it intelligently.

The HTTP spec recommends checking the Retry-After header; if it's present, use that value. If not, implement exponential backoff with jitter:

package payment

import (
	"context"
	"fmt"
	"math"
	"math/rand"
	"net/http"
	"strconv"
	"time"
)

func (p *PaymentClient) chargeWithRetry(ctx context.Context, amount int64) (string, error) {
	const maxRetries = 3
	var lastErr error

	for attempt := 0; attempt <= maxRetries; attempt++ {
		// Wait for token before attempting
		if err := p.limiter.Wait(ctx); err != nil {
			return "", fmt.Errorf("rate limit wait failed: %w", err)
		}

		chargeID, resp, err := p.doCharge(ctx, amount)

		// If successful, return immediately
		if err == nil {
			return chargeID, nil
		}

		// Check if it's a 429
		if resp != nil && resp.StatusCode == http.StatusTooManyRequests {
			// Extract retry-after header if available
			retryAfter := extractRetryAfter(resp)
			if retryAfter == 0 {
				// Exponential backoff with jitter
				retryAfter = calculateBackoff(attempt)
			}

			lastErr = fmt.Errorf("attempt %d: 429 received, waiting %dms", attempt+1, retryAfter)

			if attempt < maxRetries {
				select {
				case <-time.After(time.Duration(retryAfter) * time.Millisecond):
					continue // Retry
				case <-ctx.Done():
					return "", ctx.Err()
				}
			}
		} else {
			// Non-429 error: don't retry
			return "", err
		}

		lastErr = err
	}

	return "", fmt.Errorf("failed after %d retries: %w", maxRetries, lastErr)
}

func extractRetryAfter(resp *http.Response) int64 {
	if retryAfter := resp.Header.Get("Retry-After"); retryAfter != "" {
		if ms, err := strconv.ParseInt(retryAfter, 10, 64); err == nil {
			return ms
		}
	}
	return 0
}

func calculateBackoff(attempt int) int64 {
	// Exponential backoff: 2^attempt * 100ms, capped at 30s
	base := math.Min(float64(math.Pow(2, float64(attempt))*100), 30000)
	// Add jitter: ±10%
	jitter := (rand.Float64() - 0.5) * 0.2 * base
	return int64(base + jitter)
}

func (p *PaymentClient) doCharge(ctx context.Context, amount int64) (string, *http.Response, error) {
	// Actual HTTP call; return chargeID, response, error
	// This is where you'd call http.NewRequestWithContext, execute, and parse
	return "", nil, nil
}

The key insights here: retry only on 429, use the Retry-After header when available, apply exponential backoff with jitter to avoid thundering herd, and always respect context cancellation so retries don't outlive their parent request.

Queue work when limits are hit

Client-side rate limiting works great for steady-state traffic, but what about bursts? If we have 10,000 pending charges during a flash sale and our limiter only allows 100/sec, we either queue internally or reject requests. Queuing is almost always the right call in production—it's better to be slow than to fail.

A simple work queue using buffered channels handles this well:

package payment

import (
	"context"
	"fmt"
	"sync"
)

type ChargeJob struct {
	ID     string
	Amount int64
	Done   chan error
}

type PaymentQueue struct {
	client    *PaymentClient
	jobs      chan *ChargeJob
	wg        sync.WaitGroup
	stopOnce  sync.Once
	stopChan  chan struct{}
}

func NewPaymentQueue(client *PaymentClient, bufferSize int, workers int) *PaymentQueue {
	pq := &PaymentQueue{
		client:   client,
		jobs:     make(chan *ChargeJob, bufferSize),
		stopChan: make(chan struct{}),
	}

	// Spawn worker goroutines
	for i := 0; i < workers; i++ {
		pq.wg.Add(1)
		go pq.worker()
	}

	return pq
}

func (pq *PaymentQueue) worker() {
	defer pq.wg.Done()
	for {
		select {
		case job := <-pq.jobs:
			if job == nil {
				return
			}
			// Process the charge; send result on Done channel
			chargeID, err := pq.client.Charge(context.Background(), job.Amount)
			if err == nil {
				fmt.Printf("Charged %d (ID: %s)\n", job.Amount, chargeID)
			}
			job.Done <- err
		case <-pq.stopChan:
			return
		}
	}
}

func (pq *PaymentQueue) Submit(job *ChargeJob) error {
	select {
	case pq.jobs <- job:
		return nil
	case <-pq.stopChan:
		return fmt.Errorf("queue is stopped")
	}
}

func (pq *PaymentQueue) Stop() {
	pq.stopOnce.Do(func() {
		close(pq.stopChan)
		pq.wg.Wait()
	})
}

In your HTTP handler, instead of calling Charge directly, you submit a job to the queue and wait for the result:

func (h *Handler) HandleCheckout(w http.ResponseWriter, r *http.Request) {
	amount := int64(9999) // cents

	job := &ChargeJob{
		ID:     r.Header.Get("X-Request-ID"),
		Amount: amount,
		Done:   make(chan error, 1),
	}

	if err := h.paymentQueue.Submit(job); err != nil {
		http.Error(w, "Queue full or stopped", http.StatusServiceUnavailable)
		return
	}

	select {
	case err := <-job.Done:
		if err != nil {
			http.Error(w, err.Error(), http.StatusBadRequest)
			return
		}
		w.WriteHeader(http.StatusOK)
		fmt.Fprintf(w, "Charge succeeded")
	case <-r.Context().Done():
		http.Error(w, "Request cancelled", http.StatusRequestTimeout)
	}
}

This pattern decouples the HTTP request from the actual rate-limited operation. Requests queue up safely, workers process them at the allowed rate, and the client gets a response once processing completes (or times out). Buffer size and worker count should be tuned based on expected concurrency and the API's limits.

Track usage per API key

Many production systems call multiple external APIs or use multiple API keys against the same service. Tracking usage per key helps you catch issues early and allocate capacity intelligently. A simple metrics store does the job:

package payment

import (
	"sync"
	"time"
)

type APIKeyMetrics struct {
	mu              sync.RWMutex
	successCount    int64
	errorCount      int64
	rateLimitCount  int64
	lastRateLimitAt time.Time
}

type MetricsStore struct {
	mu      sync.RWMutex
	metrics map[string]*APIKeyMetrics
}

func NewMetricsStore() *MetricsStore {
	return &MetricsStore{
		metrics: make(map[string]*APIKeyMetrics),
	}
}

func (ms *MetricsStore) getOrCreate(apiKey string) *APIKeyMetrics {
	ms.mu.Lock()
	defer ms.mu.Unlock()

	if m, exists := ms.metrics[apiKey]; exists {
		return m
	}

	m := &APIKeyMetrics{}
	ms.metrics[apiKey] = m
	return m
}

func (ms *MetricsStore) RecordSuccess(apiKey string) {
	m := ms.getOrCreate(apiKey)
	m.mu.Lock()
	defer m.mu.Unlock()
	m.successCount++
}

func (ms *MetricsStore) RecordRateLimit(apiKey string) {
	m := ms.getOrCreate(apiKey)
	m.mu.Lock()
	defer m.mu.Unlock()
	m.rateLimitCount++
	m.lastRateLimitAt = time.Now()
}

func (ms *MetricsStore) GetStatus(apiKey string) map[string]interface{} {
	m := ms.getOrCreate(apiKey)
	m.mu.RLock()
	defer m.mu.RUnlock()

	return map[string]interface{}{
		"success_count":      m.successCount,
		"error_count":        m.errorCount,
		"rate_limit_count":   m.rateLimitCount,
		"last_rate_limit_at": m.lastRateLimitAt,
	}
}

Call RecordSuccess and RecordRateLimit in your retry logic. Export these metrics to your observability stack (Datadog, Prometheus, CloudWatch) so you can alert if a single API key starts getting hammered with 429s. That's often a signal that the key's quota changed, the service scaled down, or a bug is firing requests in a tight loop.

Common mistakes

Ignoring Retry-After headers: Some APIs set strict limits and expect clients to read the header. Skipping it and retrying immediately wastes attempts.
Setting rate limits too high: A limiter of 1000 req/s does nothing if the API only allows 100. Read the docs and start conservative; you can always increase.
Retrying all errors: Only 429 should trigger backoff retry. A 400 Bad Request or 401 Unauthorized won't get better with time.
Not respecting context timeouts: limiter.Wait(ctx) can block indefinitely if ctx never times out. HTTP handlers have timeouts; use them.
Losing visibility into rate limit events: If you don't log or track 429s, you won't notice when you're pushing API limits. Metrics are essential.

Key takeaways

Use golang.org/x/time/rate.Limiter before each outgoing request to enforce API quotas client-side.
Retry 429s with exponential backoff and jitter; check the Retry-After header if present.
Implement a work queue with buffered channels to handle bursts gracefully without overwhelming the external API.
Track success, error, and rate-limit counts per API key and export to your monitoring system so you catch issues early.
Always respect context timeouts in rate limiters and retries to avoid cascading hangs across your system.

Rate limiting outgoing requests: lessons from production

Rate limiting outgoing requests: lessons from production

The case for rate.Limiter before each call

Retry 429s with backoff

Queue work when limits are hit

Track usage per API key

Common mistakes

Key takeaways

Comments

Leave a comment

Related articles

Why I still choose .NET for backends: a practical guide

Minimal APIs: a production-ready setup, explained simply

Program.cs, line by line: a hands-on guide for engineers

DI lifetimes: Singleton, Scoped, Transient — a deep dive