Limit async concurrency: real-world lessons and examples

We've all been there: you need to fetch data from an API for 10,000 users, so you write Promise.all(users.map(u => fetchUser(u))) and watch your application grind to a halt. Limit async concurrency: real-world lessons and examples shows you why this pattern fails in production and how to fix it. The difference between uncontrolled concurrency and bounded execution can be the difference between a responsive system and one that crashes under load.

Why Promise.all on thousands of requests is dangerous

When you fire off thousands of concurrent promises without limits, several things go wrong simultaneously. Your event loop becomes saturated, memory usage balloons as each request holds buffers and state, and you exhaust file descriptors or connection pools. Most critically, if you're hitting an external API, you're almost certainly triggering their rate limits or appearing to launch a distributed denial-of-service attack.

Consider a real scenario: you have 10,000 customer records and need to enrich them with data from a third-party service. If that service allows 100 requests per second, firing all 10,000 at once means 9,900 will fail immediately. Your retry logic kicks in, exponential backoff adds delays, and what should take 100 seconds takes 30 minutes.

// ❌ DO NOT DO THIS
const users = Array.from({ length: 10000 }, (_, i) => i);
const results = await Promise.all(
  users.map(id => fetchUserData(id))
);
// Crashes. Server-side rate limit hammered. Memory explodes.

Process in batches: sequential execution of parallel work

The simplest solution is batch processing: divide your work into chunks, process each batch in parallel, then move to the next batch. This gives you control over concurrency while keeping parallelism within safe bounds.

A common pattern processes 50 items in parallel at a time, then the next 50, and so on. This keeps memory usage predictable, respects downstream rate limits, and lets you monitor progress.

async function processBatch<T, R>(
  items: T[],
  batchSize: number,
  fn: (item: T) => Promise<R>
): Promise<R[]> {
  const results: R[] = [];
  
  for (let i = 0; i < items.length; i += batchSize) {
    const batch = items.slice(i, i + batchSize);
    const batchResults = await Promise.all(
      batch.map(item => fn(item))
    );
    results.push(...batchResults);
  }
  
  return results;
}

// Usage
const userIds = Array.from({ length: 10000 }, (_, i) => i);
const results = await processBatch(userIds, 50, fetchUserData);
// Processes 50 at a time, memory bounded, respects rate limits

This approach works well for one-off tasks, but batching requires you to decide batch sizes upfront. In production, you often want something more flexible.

Concurrency pools: the worker pattern

A concurrency pool maintains a fixed number of "workers" that pull tasks from a queue. As soon as a worker finishes a task, it grabs the next one. This decouples the number of concurrent operations from the total work, giving you fine-grained control.

We can implement a simple version using a counter and a loop. The key insight: spawn n async workers that all pull from the same queue until it's empty.

async function mapLimit<T, R>(
  xs: T[],
  n: number,
  fn: (x: T) => Promise<R>
): Promise<R[]> {
  const out: R[] = [];
  let i = 0;
  
  // Spawn n concurrent workers
  await Promise.all(
    Array.from({ length: n }, async () => {
      // Each worker processes items until none remain
      while (i < xs.length) {
        const k = i++;
        out[k] = await fn(xs[k]);
      }
    })
  );
  
  return out;
}

// Usage: 10,000 items processed by 20 concurrent workers
const results = await mapLimit(userIds, 20, fetchUserData);

This is elegant and concise. The pool pattern ensures exactly n concurrent operations at any time, with memory and CPU usage bounded predictably.

Using p-limit and community libraries

For production code, consider battle-tested libraries. The p-limit package (and similar utilities like pqueue) provide proven implementations with features like priority queues, error handling, and introspection.

import pLimit from 'p-limit';

const limit = pLimit(20); // Max 20 concurrent operations

const promises = userIds.map(id =>
  limit(() => fetchUserData(id))
);

const results = await Promise.all(promises);

The advantages of established libraries:

• Well-tested error handling and edge cases
• Support for priority queues if some tasks are more urgent
• Active maintenance and security updates
• Clear, familiar API that other team members recognize

For greenfield projects or when adding a dependency isn't viable, a simple custom pool like the one above works reliably. The trade-off: you maintain it yourself and miss out on advanced features.

Respect rate limits of downstream APIs

Even with concurrency limits in place, you must align your concurrency with what the API allows. Check documentation for:

• Requests per second: How many calls can the API handle?
• Burst limits: Can you do 100 requests per second for 10 seconds, then back off?
• Quota windows: Are limits per minute, hour, or day?
• Rate limit headers: Most APIs return headers like X-RateLimit-Remaining

A practical approach reads rate limit headers from responses and adjusts concurrency dynamically:

async function fetchWithRateLimit(
  url: string,
  state: { remaining: number; resetTime: number }
) {
  // If we're approaching the limit, wait
  if (state.remaining < 10) {
    const now = Date.now();
    const waitMs = Math.max(0, state.resetTime - now);
    if (waitMs > 0) {
      await new Promise(resolve => setTimeout(resolve, waitMs));
    }
  }
  
  const response = await fetch(url);
  
  // Update state from headers
  state.remaining = parseInt(
    response.headers.get('X-RateLimit-Remaining') || '0'
  );
  state.resetTime = parseInt(
    response.headers.get('X-RateLimit-Reset') || '0'
  ) * 1000;
  
  return response.json();
}

Combined with a concurrency limit, this keeps you well within API boundaries without guesswork.

Keep memory usage bounded

Unbounded concurrency kills memory. Each pending promise holds onto its closure, buffers, and intermediate state. With thousands of concurrent operations, you accumulate gigabytes of garbage that can't be collected until requests complete.

Concurrency pools naturally bound memory because you're limiting the number of in-flight operations. But watch for other leaks:

• Accumulating results: If you're building a large array of results, it lives in memory the whole time. Consider streaming or writing to disk for large datasets.
• Connection pools: HTTP clients like Node's http module or libraries like axios maintain connection pools. Ensure pool size matches your concurrency limit.
• Response buffers: Streaming large responses instead of loading them entirely into memory reduces pressure.

// ❌ Accumulates all results in memory
async function processLargeDataset(ids: string[]) {
  const limit = pLimit(20);
  const results = await Promise.all(
    ids.map(id => limit(() => fetchAndProcess(id)))
  );
  return results; // Potentially gigabytes
}

// ✅ Stream results to disk or external storage
async function processLargeDataset(ids: string[], outputFile: string) {
  const limit = pLimit(20);
  const stream = createWriteStream(outputFile);
  
  let completed = 0;
  for (const batch of chunkArray(ids, 100)) {
    await Promise.all(
      batch.map(id =>
        limit(async () => {
          const result = await fetchAndProcess(id);
          stream.write(JSON.stringify(result) + '\n');
          completed++;
          console.log(`Progress: ${completed}/${ids.length}`);
        })
      )
    );
  }
  
  stream.end();
}

Common mistakes

• Setting concurrency too high: Starting with 1,000 concurrent operations because "more is faster." In reality, resource contention and context-switching make it slower. Start with 10–20, measure, and tune.
• Ignoring rate limit headers: Hardcoding a concurrency limit based on outdated API docs. If the API changes its limits, you either throttle unnecessarily or exceed them. Read headers dynamically.
• Forgetting timeouts: Even with concurrency limits, a slow upstream API can leave workers idle waiting. Add request timeouts so workers move on to the next task.
• No monitoring or logging: When something goes wrong, you won't know if it's concurrency, rate limits, or a bug. Log concurrency depth, queue size, and error rates.
• Mixing sync and async work without limits: If your async work triggers CPU-intensive sync operations, high concurrency starves the event loop. Consider a separate worker thread pool for CPU work.

Key takeaways

• Never use Promise.all on thousands of items without bounds. You'll exhaust resources and trigger rate limits.
• Batch processing or concurrency pools keep memory usage predictable and allow controlled parallelism.
• Simple custom implementations like mapLimit work well; use libraries like p-limit for production robustness and features.
• Always respect downstream API rate limits by reading headers and backing off dynamically.
• Monitor concurrency depth, queue size, and error rates in production to catch issues early.
• Stream or write large result sets to disk rather than accumulating them in memory.
• Start conservative (10–20 concurrent workers) and tune based on metrics, not guesses.