Python multiprocessing for CPU work: a step-by-step guide

When you're running heavy computations in Python and notice that only one CPU core is busy while the others sit idle, you're seeing the Global Interpreter Lock (GIL) at work. Python's multiprocessing module lets you sidestep this entirely by spawning separate processes that each get their own GIL, effectively using every core on your machine. In this guide, we'll walk through Python multiprocessing for CPU work: a step-by-step guide with real production scenarios you'll encounter in backend services.

Why multiprocessing matters for CPU-bound tasks

Threading in Python works well for I/O-bound operations—network calls, database queries, file reads. But for CPU-bound work like data transformation, machine learning inference, or numeric calculations, threads don't help. The GIL ensures only one thread executes Python bytecode at a time, so your multi-threaded code actually runs slower than single-threaded due to context-switching overhead.

We faced this problem in a ride-hailing platform where we needed to compute optimal route candidates for thousands of rides per second. Our initial threaded solution was CPU-bound and hit a wall at around 40% CPU utilization across eight cores. Switching to multiprocessing unlocked all eight cores and reduced latency by 60%. That's the power we're talking about.

Multiprocessing creates separate Python processes, each with its own GIL, allowing true parallelism on multi-core systems. The trade-off is overhead—process creation and inter-process communication cost more than threads. So multiprocessing shines for coarse-grained parallel work, not fine-grained tasks.

ProcessPoolExecutor for simple parallel maps

The easiest entry point is concurrent.futures.ProcessPoolExecutor. It abstracts away process lifecycle management and gives you a clean API to map functions across data in parallel.

from concurrent.futures import ProcessPoolExecutor
import time

def expensive_computation(n):
    """Simulate CPU-bound work."""
    total = 0
    for i in range(n):
        total += i ** 2
    return total

# Sequential baseline
start = time.time()
results = [expensive_computation(10_000_000) for _ in range(4)]
print(f"Sequential: {time.time() - start:.2f}s")

# Parallel with ProcessPoolExecutor
start = time.time()
with ProcessPoolExecutor(max_workers=4) as executor:
    results = list(executor.map(expensive_computation, [10_000_000] * 4))
print(f"Parallel: {time.time() - start:.2f}s")

On a 4-core machine, sequential takes ~4 seconds (each task runs one after another), and parallel takes ~1.2 seconds (roughly one task per core, plus overhead). The speedup is real and scales with core count.

In a trading platform we worked on, we had to price 100,000 options contracts nightly. Each price calculation was independent and CPU-intensive. We chunked the contracts into 32 batches and used ProcessPoolExecutor(max_workers=8) to compute them in parallel. Processing time dropped from 45 minutes to 7 minutes.

The executor.map() method is your go-to for simple cases where you have a list of inputs and want results in the same order. For more complex scenarios—handling timeouts, retries, or variable result counts—use executor.submit() with a loop and collect futures.

Data is pickled between processes

Every argument you pass to a worker function and every return value gets serialized using Python's pickle module. This is the hidden cost most developers discover the hard way.

When you call executor.map(expensive_computation, [10_000_000] * 4), Python pickles each integer argument, sends it to a worker process, the worker unpickles it and runs the function, then pickles the result and sends it back to the main process, which unpickles it. For small data, this is negligible. For large data, it becomes the bottleneck.

from concurrent.futures import ProcessPoolExecutor
import pickle
import time

def process_data(data):
    """Worker function."""
    return sum(data)

# Large list: 10 million integers
large_data = list(range(10_000_000))

# Measure pickle overhead
start = time.time()
pickled = pickle.dumps(large_data)
unpickled = pickle.loads(pickled)
print(f"Pickle 10M integers: {time.time() - start:.2f}s")

# Now run it in a worker
start = time.time()
with ProcessPoolExecutor(max_workers=1) as executor:
    result = executor.submit(process_data, large_data).result()
print(f"With multiprocessing overhead: {time.time() - start:.2f}s")

On our machines, pickling 10 million integers takes about 0.3 seconds. If you're doing that repeatedly, it adds up fast. This is where chunking helps: instead of passing one huge list to one worker, split it into smaller chunks, process each chunk separately, and let the network of workers aggregate results.

Avoid sharing large objects

You might think: "I'll create a large lookup table in the main process and somehow share it with all workers to avoid re-pickling it every time." That instinct is correct, but the implementation is tricky.

Shared memory in multiprocessing requires deliberate setup with multiprocessing.Manager() or multiprocessing.shared_memory, and both have gotchas. In most production systems we've worked on, the overhead of inter-process synchronization and copying data outweighs the benefit of sharing.

Instead, the practical approach is to initialize shared data once per worker process and reuse it within that worker. Use a global variable inside your worker function or a class-based worker with initialization.

from concurrent.futures import ProcessPoolExecutor
import time

# Simulated lookup table: expensive to create
LOOKUP_TABLE = None

def create_lookup():
    """Expensive initialization."""
    global LOOKUP_TABLE
    LOOKUP_TABLE = {i: i ** 2 for i in range(1_000_000)}
    print(f"Worker initialized lookup table")

def lookup_operation(key):
    """Worker function using the pre-initialized lookup table."""
    if LOOKUP_TABLE is None:
        raise RuntimeError("Lookup table not initialized")
    return LOOKUP_TABLE.get(key, -1)

# Initialize once per worker using initializer
with ProcessPoolExecutor(max_workers=2, initializer=create_lookup) as executor:
    start = time.time()
    results = list(executor.map(lookup_operation, [100, 200, 300, 100, 200, 300]))
    print(f"Processed 6 lookups in {time.time() - start:.3f}s")
    print(f"Results: {results}")

The initializer parameter runs once per worker process when it starts, not on every task. This pattern is ideal for cached objects: ML models, database connections, compiled regex patterns, or lookup tables. Each worker builds its own copy during initialization, avoiding pickle overhead on repeated calls.

Consider NumPy or compiled code first

Before jumping to multiprocessing, ask: can NumPy do this faster? Or Numba? Or a compiled extension?

NumPy's vectorized operations run in C and are often faster than multiprocessing the same work in pure Python. A data platform we built needed to compute statistics across terabytes of numeric data. Our first solution used multiprocessing to parallelize Python loops. Our second solution used NumPy, and it was 10x faster on the same hardware, because NumPy avoided Python's per-loop overhead entirely.

Numba (a JIT compiler for NumPy code) and Cython (write performance-critical code in a C-like dialect) can squeeze more speed out of CPU-bound work without multiprocessing complexity. If you're computing financial metrics or scientific simulations, try those first.

That said, some tasks genuinely require multiprocessing:

• Running independent business logic (e.g., pricing multiple products) where the algorithm is already in Python and changing it isn't feasible.
• Combining Python with system calls or external libraries that release the GIL.
• Tasks that scale across machines (multiprocessing on one machine is a stepping stone to distributed work).

Common mistakes

Mistake 1: Creating a new ProcessPoolExecutor for every batch. Process creation has overhead. Reuse the same executor across multiple batches. Create it once at startup, keep it alive for the lifetime of your service.

Mistake 2: Not handling exceptions. When a worker raises an exception, executor.map() or future.result() will re-raise it. Wrap your calls in try-except or use concurrent.futures.as_completed() to handle partial failures gracefully.

Mistake 3: Passing mutable objects and expecting isolation. If you pass a list to a worker and modify it inside the worker, you're modifying a copy. The original list in the main process is untouched. This can be surprising if you expect side effects. Always return results explicitly.

Mistake 4: Undersizing max_workers. If your machine has 8 cores, try max_workers=8 for CPU-bound work. Don't default to a small number like 4. Measure and profile to find the sweet spot, but starting at core count is reasonable.

Mistake 5: Debugging parallelized code and finding it fails intermittently. Non-deterministic bugs often hide in multiprocessing. Always test with max_workers=1 first to confirm your logic works sequentially, then scale up.

Key takeaways

• Use ProcessPoolExecutor for CPU-bound parallelism. It's the most straightforward way to unlock multiple cores and works well for independent, coarse-grained tasks.
• Pickling overhead is real. Large data structures serialize on every transfer between processes. Minimize pickle size by passing small inputs and returning small outputs, or pre-initialize large data on workers.
• Initialize shared state once per worker, not per task. Use the initializer parameter to set up lookups, models, and connections that persist across all tasks in a worker process.
• Benchmark against single-threaded, NumPy, and compiled alternatives first. Multiprocessing isn't always the fastest solution. Profile your specific workload.
• Test with max_workers=1 to debug logic, then scale. Parallel bugs are hard to hunt. Start sequential and verify correctness before parallelizing.