Kafka, RabbitMQ, SQS or Redis Streams: a practical guide

Kafka, RabbitMQ, SQS or Redis Streams?

Over five years building production systems, I've watched teams make messaging tool decisions based on blog posts, hype, or what their CTO used at their last job. The truth is simpler: each tool solves a different problem, and most mature platforms use more than one. Let me walk you through how to pick the right tool for your specific need, because the wrong choice will cost you either in operational complexity or in lost data replay capabilities when something breaks at 2 AM.

Quick guide: Tool vs Best For

Let's establish a mental model first. Here's how I think about the four main options:

This table might feel abstract, so let me ground it. At a ride-hailing platform I worked on, we used Kafka for the immutable stream of ride events (pickup, dropoff, payment). We used Redis Streams for real-time driver location updates. And we used RabbitMQ for background email notifications. Each solved a different problem without overlap.

Replay is the key difference

Here's the single biggest conceptual difference: Kafka stores messages. Queues typically discard them after consumption.

In Kafka, once a consumer processes a message, it's still there. You can rewind, replay from any point, or spin up a new consumer and process the entire history. This is powerful for:

• Event sourcing and audit logs
• Rebuilding read models
• Recovering from bugs where you need to reprocess data
• Analytics and historical analysis

RabbitMQ, SQS, and Redis Streams are not built for replay. Once a message is acknowledged, it's gone. This is actually a feature for background work—you don't want to retry a payment refund twice.

Let me show this with a practical example. Say you're processing user signup events:

// Kafka: consumer can seek to any offset
const kafka = new Kafka({
  clientId: 'analytics-app',
  brokers: ['kafka:9092'],
});

const consumer = kafka.consumer({ groupId: 'analytics-group' });
await consumer.subscribe({ topic: 'user-events' });

// Rewind to one hour ago and reprocess
await consumer.seek({ topic: 'user-events', partition: 0, offset: '0' });

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const event = JSON.parse(message.value.toString());
    await analytics.track(event);
  },
});

With RabbitMQ, you'd need to have saved the messages elsewhere or have them TTL'd into a dead-letter queue. With SQS, once a message is deleted, it's gone forever. This is why event-driven architectures often start with Kafka—the replay capability is invaluable when you inevitably need to debug production issues.

Operations matter more than you think

Here's something I wish someone had told me earlier: a managed queue on AWS might save you more money and headache than a self-hosted Kafka cluster, even if Kafka is technically better suited to your use case.

Running Kafka in production means:

• Maintaining broker clusters (usually 3-5 nodes minimum)
• Handling rebalancing and partition reassignment
• Monitoring disk usage (Kafka holds data for retention periods)
• JVM tuning
• Zookeeper or KRaft controller management

At a trading platform I worked on, we initially self-hosted Kafka. We spent so much time tuning retention policies, dealing with disk space issues, and managing unplanned broker restarts that we switched to AWS MSK (Managed Streaming for Kafka). The cost increase was worth the operational simplicity.

SQS is even simpler. It's pay-per-request, scales automatically, and you never think about it:

// SQS: literally just send and forget
const sqs = new SQSClient({ region: 'us-east-1' });

await sqs.send(new SendMessageCommand({
  QueueUrl: 'https://sqs.us-east-1.amazonaws.com/123456/orders',
  MessageBody: JSON.stringify({
    orderId: '12345',
    userId: 'user-456',
    amount: 99.99,
  }),
  MessageAttributes: {
    Priority: { StringValue: 'high', DataType: 'String' },
  },
}));

RabbitMQ sits in the middle. It's relatively lightweight to operate (single binary, no JVM), but you still need to provision capacity and manage upgrades.

My rule of thumb: if you're a team of fewer than 10 engineers without a dedicated platform team, start with a managed solution. As you grow and your use cases become more sophisticated, graduate to self-hosted complexity.

Use both, but for different reasons

This is where the real insight lives. The best platforms don't pick one tool. They use multiple messaging systems deliberately.

Here's the pattern I've seen work consistently:

• Kafka for facts: Immutable events that happened in your system. Orders placed, payments processed, users signed up. Retain for 30 days at minimum.
• A queue (RabbitMQ or SQS) for work: Background jobs that need reliable delivery but don't need replay. Email notifications, invoice generation, analytics aggregation.
• Redis Streams for ephemeral streams: Real-time data that's only valuable in the moment. User location updates, stock ticks, live notification feeds. Retention measured in hours, not days.

At a CRM platform I built, we shipped our architecture like this:

// Kafka: contact lifecycle events (immutable, replayable)
await kafkaProducer.send({
  topic: 'contact-events',
  messages: [{
    key: contactId,
    value: JSON.stringify({
      type: 'contact.created',
      contactId,
      email,
      timestamp: Date.now(),
    }),
  }],
});

// RabbitMQ: send email notification (work queue)
await rabbitmq.publish('email-queue', {
  to: email,
  templateId: 'welcome',
  contactId,
});

// Redis Streams: real-time sync to dashboard (ephemeral)
await redis.xadd(
  'dashboard-updates',
  '*',
  'action', 'contact_created',
  'contactId', contactId,
);

Each message went to the tool that made sense for its lifecycle. Kafka kept our audit trail. RabbitMQ ensured emails were delivered even if our notification service crashed. Redis Streams kept the dashboard snappy without overloading our main pipeline.

Common mistakes

Mistake 1: Using Kafka for fire-and-forget work. If you're just sending background emails or processing invoices, Kafka's replay complexity adds waste. Use a proper task queue.

Mistake 2: Using SQS for events you need to replay. Once you need to rebuild a read model or audit what happened, you'll wish you had Kafka. Migrations at that point are painful.

Mistake 3: Assuming managed means serverless. AWS MSK is managed, but you still provision throughput and pay for it running. SQS is the truly serverless option. Know the difference.

Mistake 4: Over-engineering for scale you don't have. I've seen startups run multi-broker Kafka clusters for 1,000 messages per day. Redis Streams would have been fine. Start simple, measure, then optimize.

Mistake 5: Forgetting about monitoring. RabbitMQ and Kafka both need active monitoring. Set up alerts on queue depth, consumer lag, and message age before you go to production. SQS has built-in CloudWatch metrics, which is another reason it's easier.

Key takeaways

• Kafka solves event streaming and replay. Everything else is a compromise on those capabilities.
• RabbitMQ is excellent for reliable task queues with complex routing. It's the Swiss Army knife of messaging.
• SQS is the right default if you're on AWS and need simple, managed queueing. Don't overthink it.
• Redis Streams is underrated for real-time, low-latency work that doesn't need long-term persistence.
• The operational cost of self-hosted Kafka often exceeds its technical benefits for teams under 50 engineers. Start managed.
• Production platforms almost always use multiple messaging systems. That's not a failure of design—it's a recognition that different problems need different tools.
• Replay capability is the primary decision point. If you need it, Kafka. If you don't, a queue is usually simpler.