Background jobs with Active Job: lessons from production

Background jobs with Active Job: lessons from production

Requests that send emails, process payments, or generate reports will timeout, frustrate users, and crash under load. Background jobs with Active Job are how we move that slow work out of the request-response cycle. We've learned hard lessons running production systems where a synchronous email send became the bottleneck during high-traffic moments—and we're here to share what actually works.

Why background jobs matter in production

Every millisecond spent in a web request is a millisecond the user waits. When your checkout endpoint fires off a webhook to a payment processor, generates an invoice PDF, uploads it to S3, and sends a confirmation email—all synchronously—you're looking at 3-5 seconds of latency on a critical user path. That's money left on the table in e-commerce. The fix is straightforward: enqueue the work and return immediately.

Background jobs also give us retry logic, error handling, and observability that inline code doesn't. When a third-party API fails, a background job can retry with exponential backoff. When it eventually gives up, we can log it, alert on it, and investigate the root cause without having dropped a user request on the floor.

Create a job with bin/rails generate job

Rails makes job generation trivial. A single command scaffolds the boilerplate:

bin/rails generate job SendReceipt

This creates app/jobs/send_receipt_job.rb with the basic structure:

class SendReceiptJob < ApplicationJob
  queue_as :default

  def perform(*args)
    # TODO: implement the job logic here
  end
end

The perform method is where the actual work happens. It receives whatever arguments you pass when enqueuing. Let's fill in a real example: sending a receipt email after an order is placed.

class SendReceiptJob < ApplicationJob
  queue_as :default

  def perform(order_id)
    order = Order.find(order_id)
    OrderMailer.receipt(order).deliver_later
  rescue StandardError => e
    Rails.logger.error("SendReceiptJob failed for order #{order_id}: #{e.message}")
  end
end

Notice we pass order_id, not the order object itself. Active Job serializes arguments for storage in the queue backend, and IDs are safer and lighter than full objects. If the order is deleted between enqueue time and job execution, you'll get a clean RecordNotFound that you can handle explicitly.

Enqueue jobs with perform_later

Once a job is defined, enqueuing it is a one-liner from anywhere in your codebase—controllers, models, other jobs, rake tasks, webhooks, anywhere:

SendReceiptJob.perform_later(order.id)

In a controller, that's typically called after a successful order creation:

class OrdersController < ApplicationController
  def create
    @order = Order.create!(order_params)
    SendReceiptJob.perform_later(@order.id)
    redirect_to @order, notice: "Order placed"
  end
end

The request returns to the user in milliseconds. The job sits in the queue and executes asynchronously. You can also schedule a job for a specific time:

SendReceiptJob.set(wait: 5.minutes).perform_later(order.id)
SendReceiptJob.set(wait_until: 1.day.from_now).perform_later(order.id)

This is invaluable for reminders, cleanup tasks, and delayed notifications. A ride-hailing platform might schedule a job to cancel unpicked-up rides after 10 minutes. A SaaS might schedule a trial reminder after 3 days.

Solid Queue: database-backed queuing in Rails 8

Rails 8 ships with Solid Queue as the default job backend, replacing Sidekiq as the out-of-the-box choice. Solid Queue is database-backed—jobs live in your PostgreSQL or SQLite instance, not in a separate Redis cluster. For teams running lean infrastructure, this is a game-changer. No more Redis ops headaches for simple job queuing.

Install and configure it with:

bin/rails generate solid_queue:install

This creates the migrations and configuration. The setup creates job and semaphore tables and wires up Solid Queue as the queue adapter in config/queue.yml. Solid Queue also includes a web UI for monitoring queues, which we'll visit via /solid_queue in development.

For production, you'll start the queue processor as a separate process (or multiple processes for concurrency):

bin/rails solid_queue:start

On AWS, that's typically a separate ECS task. On a VPS, it's another systemd service. The key point: unlike Sidekiq, you don't need Redis. Your existing database becomes the queue. For small-to-medium production systems, this cuts infrastructure complexity significantly.

If you're running high volumes or need advanced features like job priorities or rate limiting, Sidekiq is still a solid choice. But for Day 28 of learning Rails, Solid Queue is what ships by default and works reliably for most use cases.

Retries and error handling with retry_on and discard_on

Jobs fail. APIs time out, databases lock, external services go down. Active Job gives us declarative retry and discard patterns that keep your code clean.

Use retry_on to retry on specific exceptions with exponential backoff:

class SendReceiptJob < ApplicationJob
  queue_as :default
  retry_on StandardError, wait: :exponentially_longer, attempts: 5

  def perform(order_id)
    order = Order.find(order_id)
    OrderMailer.receipt(order).deliver_now
  end
end

With wait: :exponentially_longer, the job retries after 1 second, then 4 seconds, then 9 seconds, then 16 seconds—giving transient failures time to recover without hammering the service. After 5 attempts, if it still fails, the job dies and logs are your only clue.

For permanent failures—like a malformed order ID or a customer email address that bounces permanently—use discard_on:

class SendReceiptJob < ApplicationJob
  queue_as :default
  retry_on StandardError, wait: :exponentially_longer, attempts: 5
  discard_on ActiveJob::DeserializationError

  def perform(order_id)
    order = Order.find(order_id)
    OrderMailer.receipt(order).deliver_now
  end
end

If the order is deleted before the job runs, Order.find raises RecordNotFound, which triggers a deserialization error. discard_on catches it, logs it, and removes the job from the queue without retrying. You avoid infinite retry loops on data that no longer exists.

For more granular control, handle exceptions in the job itself and raise or return based on context:

class SendReceiptJob < ApplicationJob
  queue_as :default
  retry_on Timeout::Error, wait: 5.seconds, attempts: 3
  discard_on ActiveJob::DeserializationError

  def perform(order_id)
    order = Order.find(order_id)
    begin
      OrderMailer.receipt(order).deliver_now
    rescue Net::OpenTimeout => e
      Rails.logger.warn("Timeout sending receipt for order #{order_id}, will retry")
      raise e
    rescue Errno::ECONNREFUSED => e
      Rails.logger.error("Email service down, discarding job #{job_id}")
      # Don't re-raise; let the job complete silently
    end
  end
end

In production, pair this with alerting. If a job is discarded, you want to know. Log to a service like Datadog or New Relic, and set up alerts on discard events.

Common mistakes

Passing full objects instead of IDs: Jobs are serialized for storage. Passing SendReceiptJob.perform_later(order) serializes the entire order, bloating queue tables. Pass order.id instead and fetch the record in the job.

Forgetting to start the queue processor: In development, bin/rails server auto-runs jobs in-process by default (via the :inline adapter). In production, if you forget to start bin/rails solid_queue:start on a separate process, jobs enqueue silently and never execute. Painful to debug.

Over-retrying on permanent errors: If a job always fails because of bad input, retrying 100 times wastes resources. Use discard_on to catch known failure modes early. Monitor discard events to catch logic bugs.

Jobs that depend on external state: If your job assumes a user record exists but the user was deleted, the job fails. Always check that dependent records still exist and handle RecordNotFound gracefully.

Ignoring queue backlog: In production, if your queue is falling behind, jobs pile up and latency climbs. Monitor queue depth and job duration. Scale your queue processors if the backlog grows consistently.

Key takeaways

• Background jobs move slow work out of web requests. Use them for emails, webhooks, file uploads, and any I/O that can be deferred.
• Create jobs with bin/rails generate job and enqueue them with ClassName.perform_later(args).
• Pass IDs to jobs, not full objects, to keep queue payloads small and avoid serialization headaches.
• Solid Queue (Rails 8 default) is database-backed and eliminates the need for a separate Redis instance in many cases.
• Use retry_on with exponential backoff for transient failures and discard_on for permanent ones.
• Monitor queue depth, job latency, and discard rates in production. Silent job failures are silent debt.
• Test jobs locally and in staging before running them at scale. A bug in a job that fires millions of times is expensive.