Worker pools in Go: a practical guide for developers

Hook: Why worker pools matter

Imagine your API receives a spike of requests, each one needing to process a file, call an external service, or update a database. Without bounded concurrency, you could spawn thousands of goroutines at once—and crash. Worker pools in Go: a practical guide for developers shows you how to control this explosion by maintaining a fixed number of workers that safely process jobs from a queue. This pattern is foundational for building reliable systems that scale without melting your infrastructure.

The pattern — N workers read from a jobs channel and write to results

At its core, a worker pool is elegantly simple: you create a fixed number of goroutines (workers), each one listening on a shared jobs channel. As jobs arrive, workers pick them up, process them, and write results to a results channel. The main goroutine or caller feeds jobs in and collects results out. This decouples job submission from job execution—fast producers won't overwhelm slow processors.

Here's a minimal example: imagine processing a batch of URLs to fetch their status codes.

package main

import (
	"fmt"
	"net/http"
	"sync"
)

type Job struct {
	ID  int
	URL string
}

type Result struct {
	Job       Job
	StatusCode int
	Err        error
}

func worker(jobs <-chan Job, results chan<- Result, wg *sync.WaitGroup) {
	defer wg.Done()
	for job := range jobs {
		resp, err := http.Head(job.URL)
		result := Result{Job: job, Err: err}
		if err == nil {
			result.StatusCode = resp.StatusCode
		}
		results <- result
	}
}

func main() {
	numWorkers := 5
	jobs := make(chan Job, 10)
	results := make(chan Result, 10)
	var wg sync.WaitGroup

	// Start workers.
	for i := 0; i < numWorkers; i++ {
		wg.Add(1)
		go worker(jobs, results, &wg)
	}

	// Send jobs.
	go func() {
		for i := 1; i <= 20; i++ {
			jobs <- Job{ID: i, URL: fmt.Sprintf("https://example.com/%d", i)}
		}
		close(jobs)
	}()

	// Wait for workers to finish, then close results.
	go func() {
		wg.Wait()
		close(results)
	}()

	// Consume results.
	for result := range results {
		if result.Err != nil {
			fmt.Printf("Job %d: error %v\n", result.Job.ID, result.Err)
		} else {
			fmt.Printf("Job %d: %d\n", result.Job.ID, result.StatusCode)
		}
	}
}

Notice the flow: five goroutines start, each blocking on the jobs channel. As jobs are fed in (or closed), workers consume them. Results flow back out independently. If a producer adds jobs faster than workers can handle them, the buffered channel absorbs them, and workers catch up when ready.

Bound concurrency — Protect databases and APIs from overload

The real reason worker pools exist is backpressure. Without bounds, your code could spawn one goroutine per incoming request, exhausting memory and file descriptors. A fixed worker pool acts as a valve, ensuring you never overwhelm downstream resources.

Consider a service that needs to insert records into PostgreSQL. A naive approach—launching a goroutine for every row—will eventually hit connection limits on your pool (usually 100–300 per application). A worker pool lets you tune concurrency to match your database capacity.

package main

import (
	"context"
	"database/sql"
	"fmt"
	"sync"

	_ "github.com/lib/pq"
)

type InsertJob struct {
	ID   int
	Data string
}

type InsertResult struct {
	Job InsertJob
	Err error
}

func insertWorker(ctx context.Context, db *sql.DB, jobs <-chan InsertJob, results chan<- InsertResult, wg *sync.WaitGroup) {
	defer wg.Done()
	for job := range jobs {
		select {
		case <-ctx.Done():
			results <- InsertResult{Job: job, Err: ctx.Err()}
			return
		default:
		}

		_, err := db.ExecContext(ctx, "INSERT INTO my_table (id, data) VALUES ($1, $2)", job.ID, job.Data)
		results <- InsertResult{Job: job, Err: err}
	}
}

func main() {
	db, _ := sql.Open("postgres", "postgres://localhost/mydb")
	defer db.Close()

	// Set connection pool to match workers. Keep connections efficient.
	db.SetMaxOpenConns(10)
	db.SetMaxIdleConns(5)

	ctx := context.Background()
	numWorkers := 10
	jobs := make(chan InsertJob, 50)
	results := make(chan InsertResult, 50)
	var wg sync.WaitGroup

	// Start workers.
	for i := 0; i < numWorkers; i++ {
		wg.Add(1)
		go insertWorker(ctx, db, jobs, results, &wg)
	}

	// Feed jobs.
	go func() {
		for i := 1; i <= 1000; i++ {
			jobs <- InsertJob{ID: i, Data: fmt.Sprintf("record_%d", i)}
		}
		close(jobs)
	}()

	// Close results when done.
	go func() {
		wg.Wait()
		close(results)
	}()

	// Consume results.
	successCount := 0
	for result := range results {
		if result.Err != nil {
			fmt.Printf("Insert failed for job %d: %v\n", result.Job.ID, result.Err)
		} else {
			successCount++
		}
	}
	fmt.Printf("Inserted %d records\n", successCount)
}

Here, ten workers fight for the database connection pool, but never exceed it. The buffered channels provide breathing room: slow database writes don't block the job sender, and results accumulate safely until the consumer is ready. This pattern scales reliably—you can process millions of rows without crashing.

Wait for completion — sync.WaitGroup tracks workers

A critical detail: you must know when all workers have finished. sync.WaitGroup is the standard tool. Each worker calls Add(1) at startup and Done() when exiting. The main goroutine calls Wait(), which blocks until all workers are done. Only then is it safe to close the results channel and clean up.

The pattern is always the same: wg.Add(1) before launching a worker, defer wg.Done() inside the worker, and wg.Wait() in a cleanup goroutine that closes results. Miss any step and you'll either leak goroutines, deadlock on channel reads, or panic on a send to a closed channel.

Here's the cleanup loop refined:

// Start workers.
for i := 0; i < numWorkers; i++ {
	wg.Add(1)
	go worker(jobs, results, &wg)
}

// Close jobs to signal workers to exit.
go func() {
	for job := range incomingJobs {
		jobs <- job
	}
	close(jobs)
}()

// Wait for all workers and close results.
go func() {
	wg.Wait()
	close(results)
}()

// Safe to consume results now.
for result := range results {
	// handle result
}

The goroutine spawned to close results ensures that the main loop never tries to read from a closed channel prematurely. If you closed results directly in main, you might still have workers writing to it.

errgroup — golang.org/x/sync/errgroup cancels all on first error and supports limits

For production systems, the standard library's errgroup package (golang.org/x/sync/errgroup) abstracts away much of this boilerplate. It manages a WaitGroup internally, propagates errors, and optionally enforces a concurrency limit.

A huge advantage: if any worker returns an error, errgroup cancels the context, signaling all other workers to stop. This is invaluable for batch operations where one failure should halt the rest.

package main

import (
	"context"
	"fmt"
	"golang.org/x/sync/errgroup"
)

type Task struct {
	ID int
}

func processTask(ctx context.Context, task Task) error {
	// Simulate work.
	select {
	case <-ctx.Done():
		return ctx.Err()
	default:
		if task.ID == 15 {
			return fmt.Errorf("task %d failed", task.ID)
		}
		fmt.Printf("Processed task %d\n", task.ID)
		return nil
	}
}

func main() {
	ctx := context.Background()
	eg, ctx := errgroup.WithContext(ctx)

	// Limit to 5 concurrent tasks. (Go 1.20+)
	eg.SetLimit(5)

	// Queue 20 tasks.
	for i := 1; i <= 20; i++ {
		task := Task{ID: i}
		eg.Go(func() error {
			return processTask(ctx, task)
		})
	}

	// Wait and collect first error, if any.
	if err := eg.Wait(); err != nil {
		fmt.Printf("Batch failed: %v\n", err)
	}
}

With SetLimit(5), errgroup ensures at most five goroutines run concurrently. When task 15 fails, the context is cancelled and all pending tasks exit early. This is cleaner and safer than manual WaitGroup and channel plumbing, especially for complex scenarios.

Before Go 1.20, you'd implement limits manually or use a semaphore; now errgroup handles it natively. Use errgroup whenever you're spawning dynamic numbers of goroutines for a single batch operation.

Common mistakes

• Forgetting to close the jobs channel. Workers will hang forever waiting for jobs. Always close jobs when done sending, typically in a dedicated goroutine.
• Not closing results before the main loop exits. If you try to read results after the main work is done but before closing the channel, you'll deadlock. Use a goroutine to close results only after wg.Wait().
• Unbuffered channels with blocking workers. If results is unbuffered and workers write to it while the consumer sleeps, workers block. Add a buffer or consume results concurrently.
• Ignoring context cancellation. In production, always check ctx.Done() in worker loops so graceful shutdown works. Bare worker loops can orphan goroutines.
• Tuning worker count by guesswork. Profile your bottleneck (CPU, I/O, connection limits) and adjust worker count accordingly. Too few = slow; too many = resource contention.

Key takeaways

• A worker pool is a fixed set of goroutines reading jobs from a channel and writing results to another, decoupling producers from consumers.
• Bounded concurrency protects downstream resources (databases, APIs) from overload and keeps your system stable under load.
• sync.WaitGroup is essential: use Add(1) at startup, defer Done() in workers, and wait before cleanup.
• errgroup (golang.org/x/sync/errgroup) abstracts the pattern and adds context cancellation and concurrency limits with minimal code.
• Always close the jobs channel when done sending, and close results only after all workers exit. A separate cleanup goroutine prevents deadlocks.
• Profile and tune worker count to match your hardware and resource limits; sensible defaults are runtime.NumCPU() for CPU-bound work and 2–4× that for I/O-bound work.