Python generators and iterators: lessons from production

Introduction

Memory exhaustion is a silent killer in production systems. We've all inherited—or written—code that loads a 500MB CSV file into a list, only to watch the service collapse under memory pressure at 3 AM. Python generators and iterators: lessons from production teaches us how to process huge datasets without loading them all into memory at once. The difference between a naive loop and a generator-powered pipeline can mean the difference between a service that scales smoothly and one that crashes when traffic spikes.

In this guide, we'll walk through how generators work, how to chain them into efficient pipelines, and the patterns we've found most reliable in e-commerce, SaaS, and data-heavy systems.

How yield produces values lazily

The core magic of a generator is the yield keyword. Unlike a function that builds a complete list and returns it all at once, a generator function suspends execution and returns one value at a time. The calling code pulls the next value when it's ready, not before.

Here's the intuitive difference:

# Naive approach: loads everything into memory
def get_users_list(user_ids):
    users = []
    for uid in user_ids:
        user = fetch_user_from_db(uid)  # Slow I/O call
        users.append(user)
    return users

# This blocks until all users are fetched and stored
all_users = get_users_list([1, 2, 3, 4, 5])
for user in all_users:
    process_user(user)

Now with a generator:

# Generator approach: yields one at a time
def get_users_generator(user_ids):
    for uid in user_ids:
        user = fetch_user_from_db(uid)
        yield user  # Pause here, return control

# Execution doesn't start until you iterate
for user in get_users_generator([1, 2, 3, 4, 5]):
    process_user(user)

In the second version, we only keep one user object in memory at a time. The function pauses at each yield, control returns to the caller, and the next iteration resumes the function right where it left off. This is lazy evaluation—work happens only when demanded.

In a real e-commerce system, this means exporting 10 million orders to a CSV file becomes practical: instead of building a list of 10 million objects, you generate and write them one at a time, keeping memory usage constant.

Generators keep state between calls

A generator function maintains its internal state across multiple calls. Local variables, loop counters, and the execution position all persist. This is why generators are so useful for stateful iteration—you don't need a separate class with instance variables.

Consider a real scenario: streaming database records in batches to avoid connection timeouts.

def batch_query_users(batch_size=1000):
    offset = 0
    while True:
        # State persists: offset remembers its value
        users = db.query("SELECT * FROM users LIMIT ? OFFSET ?", 
                        batch_size, offset)
        if not users:
            break
        
        offset += batch_size
        yield users  # Yield a batch, not a single record

# Pull batches one at a time
for batch in batch_query_users(batch_size=1000):
    process_batch(batch)  # Each batch is 1000 records
    # If this fails, we know exactly where we are

The generator remembers offset and knows where it left off. If processing a batch fails and we retry, we can resume from the last successful batch. This is much cleaner than manually managing pagination state in a loop.

Another classic pattern: generating infinite sequences with internal counters.

def fibonacci():
    a, b = 0, 1
    while True:
        yield a  # State: a and b persist
        a, b = b, a + b

# Pull as many Fibonacci numbers as needed
fib_gen = fibonacci()
for _ in range(10):
    print(next(fib_gen))

The variables a and b maintain their values across calls. No class, no instance attributes—just pure state encapsulation through the generator's execution context.

Chain them into pipelines

The real power emerges when you chain generators together. Each stage in the pipeline processes data lazily and passes it downstream. Memory usage stays constant even with long pipelines, and you can compose complex transformations declaratively.

Imagine a data-processing job: read orders from the database, filter by status, enrich with customer data, and write to a warehouse.

def read_orders_from_db():
    """Source: yield raw orders"""
    for order in db.query("SELECT * FROM orders"):
        yield order

def filter_status(orders, status):
    """Filter stage: only pass through matching orders"""
    for order in orders:
        if order['status'] == status:
            yield order

def enrich_with_customer(orders):
    """Enrich stage: add customer name"""
    for order in orders:
        customer = db.query("SELECT name FROM customers WHERE id = ?", 
                           order['customer_id'])
        order['customer_name'] = customer['name']
        yield order

def write_to_warehouse(orders):
    """Sink: consume all orders"""
    for order in orders:
        warehouse_db.insert('processed_orders', order)

# Build the pipeline
orders = read_orders_from_db()
orders = filter_status(orders, 'completed')
orders = enrich_with_customer(orders)
write_to_warehouse(orders)

Notice the chain: each generator takes an iterator as input, processes it, and yields the result to the next stage. The entire job pulls data on-demand from top to bottom. If you have 100 million orders, you never load more than one order in memory at any point. Backpressure works naturally: if enrichment is slow, the filter stage pauses, which pauses the database read.

This pattern scales beautifully. Adding a stage is a one-liner (insert another orders = some_transform(orders)), and each stage is independently testable.

itertools has the building blocks

Python's itertools module gives you high-performance building blocks for generator pipelines. These are implemented in C and far more efficient than hand-written loops. We use them constantly in production.

Here are the ones we rely on most:

• itertools.islice(iterable, start, stop) — grab a slice of an iterator without loading everything
• itertools.takewhile(predicate, iterable) — yield items while a condition is true
• itertools.dropwhile(predicate, iterable) — skip items while a condition is true, then yield the rest
• itertools.cycle(iterable) — repeat an iterable infinitely
• itertools.chain(*iterables) — concatenate multiple iterables into one
• itertools.combinations(iterable, r) — generate all r-length combinations
• itertools.groupby(iterable, key) — group consecutive equal items

Let's revisit our data pipeline using itertools:

from itertools import chain, islice

def read_orders_from_db():
    for order in db.query("SELECT * FROM orders"):
        yield order

def read_orders_from_archive():
    for order in archive_db.query("SELECT * FROM orders"):
        yield order

def enrich_with_customer(orders):
    for order in orders:
        customer = db.query("SELECT name FROM customers WHERE id = ?", 
                           order['customer_id'])
        order['customer_name'] = customer['name']
        yield order

# Combine two sources seamlessly
all_orders = chain(
    read_orders_from_db(),
    read_orders_from_archive()
)

# Process the first 10,000 for a test run
test_orders = islice(all_orders, 0, 10000)

# Enrich and write
enriched = enrich_with_customer(test_orders)
for order in enriched:
    warehouse_db.insert('processed_orders', order)

chain() stitches multiple iterables together without loading them all into memory. islice() lets you grab a window of items efficiently—perfect for pagination or testing on a subset. Combining these is how you build robust, composable ETL jobs.

Another practical example: batch processing with itertools.islice:

def batched(iterable, batch_size):
    """Yield successive batches from an iterable"""
    iterator = iter(iterable)
    while True:
        batch = list(islice(iterator, batch_size))
        if not batch:
            break
        yield batch

# Usage: process large dataset in batches
def process_all_users():
    for batch in batched(get_users_generator([1, 2, 3, ..., 1000000]), 
                        batch_size=1000):
        db.bulk_insert('processed_users', batch)

This keeps memory bounded: you only hold one batch (1,000 users) in memory at a time, no matter how many total users exist.

Common mistakes

Trying to iterate a generator twice. Generators are consumable. Once you iterate through them, they're exhausted. If you need to iterate multiple times, convert to a list or save to disk.

gen = get_users_generator([1, 2, 3])
for user in gen:
    print(user)  # Works: yields 1, 2, 3

for user in gen:
    print(user)  # Yields nothing: generator is exhausted

Forgetting that generators are lazy. If you need side effects to happen upfront (like logging or validation), a generator won't trigger them until iteration. Wrap expensive operations explicitly if you need eager evaluation for debugging.

Nesting generators without understanding the memory profile. A generator that yields lists is still a generator, but those lists live in memory. Be clear about what each stage yields—scalars, tuples, or objects—and measure memory usage under realistic load.

Ignoring exception handling in pipelines. If an error occurs mid-pipeline, the generator stops, and you might lose track of progress. Wrap generators in try-except blocks and log checkpoints, especially in background jobs.

def safe_enrich(orders):
    for order in orders:
        try:
            customer = db.query("SELECT name FROM customers WHERE id = ?", 
                               order['customer_id'])
            order['customer_name'] = customer['name']
            yield order
        except Exception as e:
            logger.error(f"Failed to enrich order {order['id']}: {e}")
            # Decide: skip, retry, or re-raise

Key takeaways

• yield makes functions lazy: Generators don't compute all values upfront; they produce them on demand, keeping memory constant.
• State persists between calls: Local variables in a generator function maintain their values across iterations, eliminating the need for external state management.
• Chain generators for composable pipelines: Connect multiple generators in sequence; each stage is independent, testable, and can handle massive datasets without loading them all at once.
• itertools is your friend: Use chain, islice, takewhile, and groupby to build efficient pipelines; they're faster and more readable than hand-written loops.
• Monitor and test under load: Generators hide complexity; always test your pipelines with realistic data volumes and measure memory usage to ensure they truly stay bounded.