Iterator pattern and generators: lessons from production

The Iterator pattern and generators form a powerful partnership for traversing collections without exposing their internals. After five years building real platforms, I've found that generators solve one of the most elegant problems in software engineering: how to produce sequences of values on-demand, without pre-computing them all into memory. Today we'll explore how they transform the way you work with collections, APIs, and massive datasets.

for...of works with any iterable

JavaScript's for...of loop is the syntactic sugar that makes the Iterator pattern accessible to everyone. Under the hood, it relies on the iterable protocol: any object that implements a Symbol.iterator method returning an iterator (an object with a next() method). The beauty is that you don't need to understand this contract to use it; you just write familiar loop syntax.

When I was building the product catalog system at a ride-hailing startup, we had collections scattered across Redis, PostgreSQL, and third-party APIs. Instead of forcing every consumer to know the storage layer, I created iterables that abstracted this away. The consuming code stayed clean:

async function filterActiveVehicles(fleet: VehicleFleet) {
  const active = [];
  for await (const vehicle of fleet.all()) {
    if (vehicle.status === 'active') {
      active.push(vehicle);
    }
  }
  return active;
}

Notice the await in the loop declaration. That's not magic—it's the Iterator pattern handling asynchronous values. The fleet doesn't expose whether vehicles come from a cache, database, or API. It simply implements the async iterable protocol, and callers consume it uniformly. This separation of concerns scales beautifully as requirements change.

Generators produce values lazily

Generator functions (marked with function*) are the most ergonomic way to implement iterables in JavaScript. They pause execution at each yield, resuming only when the next value is requested. This lazy evaluation is transformative for memory efficiency.

Consider a real scenario from my trading platform backend: we needed to process transaction logs that could span months. Materializing all of them into memory was impossible. A generator solved it elegantly:

function* rangeByDay(startDate: Date, endDate: Date, batchSize = 1000) {
  let current = new Date(startDate);
  while (current <= endDate) {
    const dayStart = new Date(current);
    const dayEnd = new Date(current);
    dayEnd.setDate(dayEnd.getDate() + 1);
    
    const transactions = fetchTransactionsSync(dayStart, dayEnd, batchSize);
    for (const tx of transactions) {
      yield tx;
    }
    current.setDate(current.getDate() + 1);
  }
}

// Usage: iterate through months of data without loading it all
for (const tx of rangeByDay(jan1, dec31)) {
  processTransaction(tx);
}

The key insight is that each day's batch is fetched on-demand. If you only iterate halfway through the year, you never fetch December's data. Compare this to a traditional approach that loads everything upfront—the memory savings are massive when dealing with millions of records.

Generators also compose beautifully. You can chain them, filter through them, and transform them without materializing intermediate collections:

function* filterByAmount(txs: IterableIterator, minAmount: number) {
  for (const tx of txs) {
    if (tx.amount >= minAmount) {
      yield tx;
    }
  }
}

// Chain generators without creating intermediate arrays
for (const tx of filterByAmount(rangeByDay(jan1, dec31), 1000)) {
  recordLargeTransaction(tx);
}

Async iterators stream paginated APIs

This is where generators become indispensable in modern backend development. Most APIs paginate results, returning a cursor or page number for the next batch. Async generators hide this pagination complexity behind a simple iteration interface.

Here's the code from your outline, and I use this pattern constantly when integrating third-party services:

async function* allLeads(api: Api) {
  let cursor: string | undefined;
  do {
    const page = await api.leads(cursor);
    yield* page.items;
    cursor = page.next;
  } while (cursor);
}

The yield* operator is subtle but powerful: it delegates to another iterable. So we yield each item in the page individually, not the page object itself. A consumer sees an infinite stream of leads, not pagination boundaries:

// Consumer code is blissfully unaware of pagination
for await (const lead of allLeads(crmApi)) {
  await enrichLeadProfile(lead);
  if (lead.score > 80) {
    await assignToSales(lead);
  }
}

I implemented this exact pattern at a travel startup where we synced hotel inventory from multiple suppliers. Each supplier had different pagination schemes. By wrapping each in an async generator, our sync engine never had to know those details. When a supplier changed their API format, we updated one function and everything downstream kept working.

The error handling is also elegant. If a single page fetch fails, you can catch the error at the call site:

try {
  for await (const lead of allLeads(crmApi)) {
    await processLead(lead);
  }
} catch (error) {
  logger.error('Lead sync failed', error);
  // Caller decides retry strategy
}

Process huge datasets with constant memory

The ultimate power of iterators and generators is processing datasets that dwarf available memory. In a recent SaaS project, we aggregated logs from millions of user interactions. A naive approach would load everything into an array, then sort and filter. That simply wasn't viable.

With generators and streaming, we maintained constant memory regardless of dataset size:

async function* streamUserEvents(userId: string, startDate: Date, endDate: Date) {
  const dayCount = Math.ceil((endDate.getTime() - startDate.getTime()) / (1000 * 60 * 60 * 24));
  
  for (let i = 0; i < dayCount; i++) {
    const date = new Date(startDate);
    date.setDate(date.getDate() + i);
    
    const events = await db.query(
      'SELECT * FROM user_events WHERE user_id = $1 AND DATE(created_at) = $2',
      [userId, date]
    );
    
    for (const event of events) {
      yield event;
    }
  }
}

// Process billions of events without memory growth
let heatmap = new Map();

for await (const event of streamUserEvents(userId, jan1, dec31)) {
  const key = `${event.action}:${event.page}`;
  heatmap.set(key, (heatmap.get(key) ?? 0) + 1);
  
  // Memory stays bounded; we're only holding the current event and aggregation state
}

This approach is critical for data pipelines and analytics. I've processed transaction histories spanning years without ever hitting memory limits. The trick is keeping your aggregation state small (just the summary you need) and discarding raw records immediately after use.

Common mistakes

• Forgetting async when calling async generators: for await (const x of gen()) requires both keywords. Miss either and you'll iterate over promises instead of resolved values, which catches many developers off guard.
• Yielding the entire collection instead of items: If you write yield page.items instead of yield* page.items, consumers get array objects, not individual elements. The whole abstraction breaks.
• Creating iterators that aren't reusable: If your generator consumes a cursor or state, calling it twice won't work the second time. Consider whether your iterator needs to be repeatable or if a one-shot generator is acceptable.
• Treating generators as lazy but forgetting error handling: Errors in generators only surface when you iterate. If you never consume the generator, the error silently doesn't happen. Always ensure iteration happens and is wrapped in try-catch.
• Building generators with side effects that matter: Since generators are lazy, side effects like logging or incrementing counters may not happen until iteration. This confuses debugging. Keep generators pure when possible.

Key takeaways

• The Iterator pattern with for...of and for await...of abstracts away collection internals, making code cleaner and more maintainable.
• Generators using function* implement iterables with minimal boilerplate, producing values on-demand for better memory efficiency.
• Async generators hide pagination and API complexity behind a familiar iteration interface, making it trivial to work with paginated endpoints.
• Constant memory processing of huge datasets becomes possible when you chain generators instead of materializing intermediate collections.
• The yield* operator elegantly delegates to nested iterables, enabling composition of generators into powerful data pipelines.