Sidekiq for high-volume jobs for Ruby on Rails developers

Why Sidekiq matters for Rails applications

By Day 30, you've built features that work well in request-response cycles, but you've likely hit a wall: sending emails to 50,000 users blocks your web server, generating reports ties up database connections, and webhooks to third-party APIs cause timeouts. Sidekiq for high-volume jobs for Ruby on Rails developers solves this by moving work off the request thread and into a dedicated background job system. When throughput matters and latency kills user experience, Sidekiq becomes essential.

We've run Sidekiq in production across e-commerce checkouts, CRM data syncs, and ride-hailing notifications. It's fast, reliable, and the mental model is straightforward: enqueue work, process it asynchronously, retry on failure. Let's dig into how to make it work at scale.

Redis-backed and very fast

Sidekiq uses Redis as its job queue and state store. Unlike delayed_job or Resque, which hit your primary database for every job, Sidekiq keeps the queue in memory. This means you can enqueue thousands of jobs per second with minimal overhead, and workers pull jobs from Redis without database contention.

A typical Rails app might run into trouble when ActiveJob with a database backend tries to process email notifications. Each job insert becomes a database write; each worker poll becomes a read. With Sidekiq, job throughput is bound by Redis I/O and network latency, not your Postgres connection pool.

Here's a concrete setup:

# Gemfile
gem 'sidekiq', '~> 7.0'
gem 'redis', '~> 5.0'

# config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
  config.redis = { url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/1') }
  config.max_retries = 5
  config.server_middleware do |chain|
    chain.add MyErrorTracker  # Custom middleware for error reporting
  end
end

Sidekiq.configure_client do |config|
  config.redis = { url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/1') }
end

In production, we typically dedicate a single Redis instance to Sidekiq, separate from cache. If you run Redis Cluster or Sentinel, configure connection pooling to avoid bottlenecks:

config.redis = {
  url: ENV['REDIS_URL'],
  network_timeout: 5,
  pool_size: 25,
  pool_timeout: 1
}

With a modest 4-core Sidekiq worker process and Redis on the same network, we've seen throughput of 2,000+ jobs per second for lightweight operations. For a high-volume e-commerce platform processing order confirmations and inventory updates, this becomes the difference between a smooth user experience and a cascading timeout.

Great dashboard

Sidekiq ships with a web UI that's intuitive and invaluable during operations. The dashboard shows real-time job stats, retry queues, dead-letter jobs, worker processes, and Redis memory usage. We mount it in a separate Rails engine to keep it away from your main app routes:

# config/routes.rb
require 'sidekiq/web'

Sidekiq::Web.use Rack::Auth::Basic do |username, password|
  username == ENV['SIDEKIQ_USER'] && password == ENV['SIDEKIQ_PASSWORD']
end

Rails.application.routes.draw do
  mount Sidekiq::Web => '/sidekiq'
end

The dashboard is crucial because it lets you answer questions without SSH-ing into production: How many jobs are currently retrying? What's the oldest job in the dead letter queue? Are workers consuming faster than enqueue rate? Is Redis memory climbing? These answers inform whether you need to scale horizontally, increase concurrency, or investigate a job that's failing repeatedly.

We've prevented several outages by noticing that a misconfigured webhook was filling the retry queue faster than workers could drain it. A quick glance at the dashboard showed the pattern; a quick code fix resolved it.

Retries with backoff

Network calls fail. Third-party APIs go down. Database locks occur. Sidekiq handles transient failures gracefully with built-in retry logic and exponential backoff. By default, Sidekiq retries failed jobs with exponential delays, so a flaky external API doesn't hammer your retry queue immediately.

Let's say you're syncing customer data with a CRM. The CRM API occasionally returns 503. You want to retry, but not immediately:

class SyncCustomerJob
  include Sidekiq::Job
  sidekiq_options retry: 10, dead: true

  def perform(customer_id)
    customer = Customer.find(customer_id)
    response = CrmClient.sync(customer)
    
    unless response.success?
      raise "CRM sync failed: #{response.message}"
    end
  end
end

Sidekiq retries this job up to 10 times with exponential backoff. The delay formula is roughly ((count + 15) * (rand + 1)) * 5 seconds, so the first retry happens after ~75 seconds, the second after minutes, and so on. After 10 failures, the job goes to the dead letter queue (because dead: true), where you can inspect it and replay it manually.

You can customize retry behavior with a block:

class SendWebhookJob
  include Sidekiq::Job
  
  sidekiq_retry_in do |count|
    count * 60  # Linear backoff: 60s, 120s, 180s, etc.
  end
  
  sidekiq_retries_exhausted do |msg, err|
    ErrorTracker.notify(err, context: msg)
  end

  def perform(webhook_url, payload)
    response = HTTP.post(webhook_url, json: payload)
    raise "Webhook failed: #{response.status}" unless response.status < 400
  end
end

The sidekiq_retries_exhausted block fires when retries are exhausted, giving you a chance to alert, clean up, or fall back gracefully. In real-world systems, this prevents silent failures that only surface weeks later when someone notices missing data.

Use for heavy job workloads

Not every background job needs Sidekiq. Sending a single email per request? Maybe Rails' built-in async job adapter with threads is enough. Processing 100,000 records? Definitely Sidekiq. The rule of thumb: if your job queue would grow faster than your workers can drain it during peak hours, you need Sidekiq.

Consider a ride-hailing platform on a Friday night. Drivers log in, each triggering a location sync, background matching algorithm, and push notification. Without Sidekiq, these pile up in a database queue, blocking each other. With Sidekiq, you enqueue thousands per second, spin up more workers, and drain the queue in seconds.

Here's a realistic example: daily digest emails to 2 million users:

class DailyDigestWorker
  include Sidekiq::Job
  sidekiq_options retry: 3, queue: 'mailers'

  def perform(user_id)
    user = User.find(user_id)
    digest_data = DigestBuilder.new(user).build
    UserMailer.daily_digest(user, digest_data).deliver_later
  end
end

# In a background task or scheduler:
User.find_in_batches(batch_size: 1000) do |batch|
  batch.each { |user| DailyDigestWorker.perform_async(user.id) }
end

With Sidekiq, you enqueue 2 million jobs quickly, then let 10–20 workers churn through them. Each job is small (fetch user, build digest, trigger email), so you get high throughput. If you did this synchronously in a Rake task, you'd wait hours; with Sidekiq, you distribute the load and complete in minutes.

For CPU-intensive work like image processing or report generation, Sidekiq scales similarly:

class GenerateReportJob
  include Sidekiq::Job
  sidekiq_options retry: 2, queue: 'heavy'

  def perform(report_id)
    report = Report.find(report_id)
    report.update(status: 'processing')
    
    begin
      csv_data = ReportGenerator.new(report).generate
      report.update(status: 'completed', file_path: upload_to_s3(csv_data))
    rescue => e
      report.update(status: 'failed', error_message: e.message)
      raise
    end
  end
end

By putting heavy work in a separate queue and scaling workers independently, you avoid starving fast jobs (emails, webhooks) with slow jobs (reports, image transcoding).

Common mistakes

• Not monitoring Redis memory: If Sidekiq queues grow faster than workers drain them (due to bugs or traffic spikes), Redis fills up. Set up alerts for Redis memory usage and queue depth. We've seen runaway jobs consume 60 GB of Redis in an hour.
• Forgetting to handle job idempotency: If a worker crashes after processing but before marking the job complete, Sidekiq retries it. Your jobs must be idempotent—running them twice should be safe. For example, use database upserts, not inserts. Check if a resource already exists before creating it.
• Using Sidekiq for real-time work: Sidekiq is asynchronous; there's inherent latency between enqueue and execution. If a user needs an answer in 50ms, don't enqueue a job. Run it inline or use a synchronous API.
• Overloading worker concurrency: Setting concurrency too high (e.g., 100 threads per process on a 4-core box) causes context switching overhead and memory bloat. A safe default is 5–10 threads per process; benchmark for your workload.
• Ignoring the dead letter queue: Dead jobs don't resurface unless you explicitly check the dashboard. Set up alerts when the DLQ grows; it's usually a sign of a code bug or third-party outage.

Key takeaways

• Sidekiq stores jobs in Redis, enabling throughput of thousands of jobs per second with minimal database overhead.
• The web dashboard gives you real-time visibility into queue depth, worker health, and retry patterns—essential for operating production systems.
• Retries with exponential backoff handle transient failures gracefully; the dead letter queue catches permanent failures for investigation.
• Use Sidekiq for high-volume, background-safe work: emails, webhooks, exports, data syncs. Not for latency-sensitive or one-off operations.
• Make jobs idempotent, monitor Redis memory, and keep separate queues for fast and slow work to avoid resource contention.