Background jobs in Go: a practical guide for developers

Background jobs in Go: a practical guide for developers

Not every operation needs to happen before you send a response back to the client. In production systems handling thousands of requests per second, offloading work to run asynchronously can mean the difference between a responsive API and one that times out under load. Background jobs in Go allow you to decouple long-running tasks from request handlers, giving your team the flexibility to process email confirmations, generate reports, or sync external systems without slowing down your users.

Why background jobs matter in production

When you're building APIs that scale, synchronous operations become bottlenecks. A user signs up—you want to send them a confirmation email, but you don't want to wait 2–5 seconds for an SMTP call to complete before returning a 200 response. Similarly, processing a large dataset or calling a third-party webhook can introduce latency that compounds across your request path.

Background jobs solve this by letting you:

• Return responses faster to clients
• Isolate external service failures from user-facing requests
• Retry failed work automatically
• Scale job processing independently from your API tier

We've learned this across multiple production systems—once you start deferring non-critical work, both your response times and your team's sanity improve.

Goroutines for small in-process tasks

For lightweight, ephemeral work that doesn't need to survive a process restart, goroutines are your first tool. They're fast to spin up, require minimal overhead, and keep everything in-memory.

A simple example: after a user completes a payment, you want to log an analytics event, but you don't want to block the payment response.

func handlePayment(w http.ResponseWriter, r *http.Request) {
    // Process payment synchronously
    order, err := processPayment(r.Context())
    if err != nil {
        http.Error(w, "Payment failed", http.StatusBadRequest)
        return
    }

    // Fire off analytics asynchronously
    go logAnalyticsEvent(order.ID, order.Amount)

    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(map[string]string{"status": "success"})
}

The goroutine approach is low-friction: no external dependencies, no queue setup, instant execution. However, it has real limits. If your process crashes, that work is lost. If you restart during deployment, queued jobs evaporate. And if a background task panics, it can crash your entire service if you're not careful.

For this reason, we keep goroutine work truly small: metrics, lightweight logging, cache warming. Anything business-critical—especially anything that must complete—belongs in a proper queue.

Queues for durable jobs

Once you need durability, you need a queue. A queue gives you a durable record of work that needs to be done, decoupling producers (your API) from consumers (your job workers). If a job fails, it stays in the queue. You can scale workers up or down independently. You can inspect pending jobs in production.

Common queue patterns in Go production systems:

• Database-backed queues – rows in PostgreSQL, checked periodically. Simple, but polling introduces latency.
• Redis lists – fast, in-memory, but requires persistence tuning and cluster setup.
• Message brokers – Kafka, RabbitMQ, AWS SQS. Overkill for most teams starting out, but mature and reliable at scale.
• Purpose-built job libraries – asynq, go-jobs, faktory. These abstract away the transport layer.

The choice depends on your scale and operational comfort. A young startup might use a database queue. A trading or ride-hailing platform probably uses Kafka or a purpose-built system.

asynq for Redis-backed jobs

If you're looking for a sweet spot between simplicity and production readiness, asynq is worth serious consideration. It's a Redis-backed job queue with built-in retries, scheduling, and a web UI. Many teams we've worked with in e-commerce and SaaS have found it strikes the right balance.

Setting up asynq is straightforward:

import "github.com/hibiken/asynq"

// In your main server setup
redisOpt := asynq.RedisClientOpt{Addr: "localhost:6379"}
inspector := asynq.NewInspector(redisOpt)
client := asynq.NewClient(redisOpt)
defer client.Close()

// Create a job
task := asynq.NewTask(
    "email:send",
    map[string]interface{}{
        "user_id": 123,
        "email":   "user@example.com",
    },
)

// Enqueue it (returns immediately)
info, err := client.Enqueue(task)
if err != nil {
    log.Printf("Failed to enqueue: %v", err)
}
log.Printf("Enqueued task: %s", info.ID)

On the worker side, you register handlers and let asynq's server pull from Redis:

// Worker registration
mux := asynq.NewServeMux()
mux.HandleFunc("email:send", func(ctx context.Context, t *asynq.Task) error {
    var p map[string]interface{}
    if err := json.Unmarshal(t.Payload(), &p); err != nil {
        return fmt.Errorf("json.Unmarshal failed: %v: %w", err, asynq.SkipRetry)
    }

    userID := int(p["user_id"].(float64))
    email := p["email"].(string)

    // Actual work
    if err := sendEmail(ctx, email); err != nil {
        return err // Will be retried
    }

    log.Printf("Sent email to %s", email)
    return nil
})

// Start the server
srv := asynq.NewServer(redisOpt, asynq.Config{
    Concurrency: 10,
})

if err := srv.Run(mux); err != nil {
    log.Fatalf("Server failed: %v", err)
}

This approach gives you visibility (you can inspect pending jobs with the web UI), automatic retries with sensible defaults, and the ability to scale workers horizontally by running more instances of your worker binary.

Retries with backoff

Transient failures happen: a third-party API times out, a database is temporarily unavailable, a network blip occurs. Smart retry logic is crucial for reliable background job systems.

asynq supports exponential backoff out of the box. You configure it per queue:

srv := asynq.NewServer(redisOpt, asynq.Config{
    Concurrency: 10,
    // Retry failed tasks up to 5 times
    RetryDeltas: []time.Duration{
        10 * time.Second,  // First retry after 10 seconds
        30 * time.Second,  // Second retry after 30 seconds
        2 * time.Minute,   // Third retry after 2 minutes
        5 * time.Minute,   // Fourth retry after 5 minutes
        10 * time.Minute,  // Fifth retry after 10 minutes
    },
})

The key insight: not every error should be retried the same way. Distinguish between transient and permanent failures in your job handlers:

mux.HandleFunc("api:sync", func(ctx context.Context, t *asynq.Task) error {
    result, err := callExternalAPI(ctx)
    
    // Permanent error: don't retry
    if isValidationError(err) {
        log.Printf("Validation failed, skipping retry: %v", err)
        return asynq.SkipRetry
    }
    
    // Transient error: let it retry
    if isNetworkError(err) {
        return fmt.Errorf("Network error, will retry: %w", err)
    }
    
    // Success
    saveResult(result)
    return nil
})

This prevents your queue from filling up with jobs that will never succeed. We've seen teams waste resources repeatedly retrying bad data or malformed API calls.

Common mistakes

Spawning too many goroutines without a limit. Goroutines are cheap but not free. If you spawn one goroutine per request without bounding them, you'll eventually exhaust memory. Use worker pools or semaphores if you go the goroutine route.

Losing visibility into job failures. A job fails silently and nobody notices for days. Always log job execution, especially errors. asynq's web UI helps here, but pair it with structured logging to your observability stack.

Enqueuing jobs without error handling. If your queue is unreachable, you'll want to know immediately. Don't silently drop jobs—return an error to the caller or implement a fallback.

Making retry logic too aggressive. Retrying 50 times with 1-second delays doesn't help anyone. Think through what errors are actually transient in your domain. A 503 from an external API? Retry. A 400 Bad Request? Skip it.

Running a single job worker instance. If your only worker crashes, jobs accumulate in the queue until you restart it. Always run multiple worker instances, ideally across different machines or availability zones.

Key takeaways

• Use goroutines for fire-and-forget work that's truly ephemeral (analytics, metrics, lightweight logging).
• Move business-critical work to a durable queue as soon as you need guaranteed execution.
• asynq provides a production-ready Redis-backed job queue with retries and a web UI—a solid choice for most teams.
• Implement thoughtful retry logic: exponential backoff for transient failures, skip retries for permanent ones.
• Treat job workers as a separate tier: monitor them, scale them independently, log their failures.
• Always handle queue unavailability gracefully in your API; don't let a down Redis instance take out your service.