Managed Kafka options for Kafka & WebSockets developers

After running self-hosted Kafka in production long enough, you start wondering whether you're solving the right problem. Operating brokers, managing replication, patching versions, and monitoring disk space—it works, but it's not your competitive advantage. Managed Kafka options let you run Kafka without running Kafka, shifting operations to providers who do it at scale. In this article, I'll walk through the major managed Kafka offerings, how their pricing works, and what hidden costs you need to watch.

Amazon MSK and MSK Serverless

AWS Managed Streaming for Apache Kafka (MSK) gives you a fully managed Kafka cluster provisioned across multiple availability zones. You choose broker instance types, storage, and partition count—AWS handles patching, replication, and hardware maintenance. I've run production workloads on MSK, and the appeal is straightforward: you get a Kafka cluster that behaves like your self-hosted version, but without the operational burden.

With standard MSK, you provision specific broker nodes—kafka.m5.large, kafka.m5.xlarge, and so on. Each broker stores data on EBS volumes. You pay an hourly rate per broker, plus data storage and data transfer charges. For a three-broker cluster on m5.large instances with 100 GB storage per broker, you're looking at roughly $200–250 USD per month in compute alone, depending on region.

MSK Serverless flips the model: you don't choose broker types. Instead, you provision throughput in MB/sec for producers and consumers separately. You pay based on what you actually use—provisioned throughput and data stored. Serverless is ideal if your traffic is bursty or unpredictable. A steady-state producer pushing 10 MB/sec might cost $50–80 USD per month on Serverless, versus fixed monthly broker costs on standard MSK.

Here's a practical example: suppose you have a real-time order processing pipeline. During business hours, you produce 50 MB/sec; at night, nearly zero. Standard MSK forces you to pay for three always-on brokers. Serverless lets you provision just enough throughput and scales automatically.

// Connecting to MSK from a Node.js producer
import { Kafka } from 'kafkajs';

const kafka = new Kafka({
  clientId: 'order-producer',
  brokers: [
    'b-1.cluster.xxxxx.kafka.us-east-1.amazonaws.com:9092',
    'b-2.cluster.xxxxx.kafka.us-east-1.amazonaws.com:9092',
    'b-3.cluster.xxxxx.kafka.us-east-1.amazonaws.com:9092',
  ],
  ssl: true,
  sasl: {
    mechanism: 'scram-sha-512',
    username: 'kafka-user',
    password: process.env.KAFKA_PASSWORD,
  },
});

const producer = kafka.producer();
await producer.connect();
await producer.send({
  topic: 'orders',
  messages: [{ key: orderId, value: JSON.stringify(order) }],
});

One thing to note: MSK requires you to place brokers in a VPC, and cross-AZ traffic incurs charges. More on that later.

Confluent Cloud with schema registry and connectors

Confluent Cloud is a fully managed Kafka service operated by Confluent, the company founded by Kafka's creators. It's platform-agnostic—you can run it on AWS, GCP, or Azure—and it comes with extras like Schema Registry, ksqlDB, and connectors bundled in or readily available.

Confluent Cloud pricing is based on partitions and throughput. You provision cluster throughput in CKUs (Confluent Kafka Units). One CKU gets you 1 MB/sec ingress and 1 MB/sec egress per partition, plus 50 GB storage. A small cluster with 10 partitions at 1 CKU costs roughly $1 per hour, or around $730 USD per month. That's higher than standard MSK on small workloads, but the Schema Registry and connector ecosystem add real value if you're building data pipelines.

The Schema Registry is the differentiator here. It's a centralized store for Avro, Protobuf, and JSON schemas. Every producer and consumer can enforce schema compatibility, preventing breaking changes from reaching production. If you're managing hundreds of schemas across teams, this alone saves weeks of debugging.

// Using Confluent Cloud Schema Registry with a producer
import { Kafka, SchemaRegistry, SchemaType } from '@kafkajs/confluent-schema-registry';

const registry = new SchemaRegistry({
  host: 'https://psrc-xxxxx.us-east-1.provider.confluent.cloud',
  auth: {
    username: process.env.REGISTRY_KEY,
    password: process.env.REGISTRY_SECRET,
  },
});

const schema = {
  type: 'record',
  name: 'Order',
  fields: [
    { name: 'id', type: 'string' },
    { name: 'amount', type: 'float' },
    { name: 'timestamp', type: 'long' },
  ],
};

const schemaId = await registry.register(schema);

const kafka = new Kafka({
  clientId: 'order-producer',
  brokers: ['pkc-xxxxx.us-east-1.provider.confluent.cloud:9092'],
  sasl: {
    mechanism: 'plain',
    username: process.env.CONFLUENT_KEY,
    password: process.env.CONFLUENT_SECRET,
  },
  ssl: true,
});

const producer = kafka.producer();
await producer.connect();

const orderEvent = { id: '12345', amount: 99.99, timestamp: Date.now() };
const encoded = await registry.encode(schemaId, orderEvent);

await producer.send({
  topic: 'orders',
  messages: [{ key: orderEvent.id, value: encoded }],
});

Confluent Cloud also offers managed connectors—pre-built integrations for Salesforce, Databricks, PostgreSQL, and dozens more. Running a connector on Confluent costs extra (roughly $10–20 per connector per hour), but you avoid building and maintaining ETL pipelines yourself.

Comparing pricing per partition and throughput

Let's ground this in numbers. Here's a realistic comparison for a mid-size workload: 50 partitions, 100 MB/sec sustained throughput, 1 TB storage.

MSK standard is the cheapest if you have predictable, steady throughput. If your throughput is bursty or you want built-in schema governance, Confluent becomes more competitive once you factor in the operational savings. MSK Serverless sits in the middle—higher per-unit cost than standard MSK, but zero provisioning overhead.

The pricing model also matters psychologically. With Confluent, you scale partitions freely (more partitions = more CKUs), so cost is directly tied to your data model complexity. With MSK, adding partitions doesn't add cost—only the brokers matter. If you have 100 partitions across three brokers, that's the same cost as 10 partitions on three brokers.

Watch networking costs across AZs

Here's where most engineers get surprised: data transfer charges. Both AWS MSK and Confluent Cloud charge for inter-AZ traffic.

On AWS MSK, cross-AZ data transfer costs $0.01 per GB. If you have a three-broker cluster in three different AZs (which is the default for high availability), replication traffic between brokers incurs this charge. For a busy cluster replicating 500 GB per day across AZs, that's $5 per day, or $150 per month—a hidden cost on top of your broker fees.

Confluent Cloud is similar. Cross-AZ traffic (they call it "inter-region" data transfer within a single region's deployment) is charged separately from ingress/egress charges.

To minimize cross-AZ costs, co-locate your producers and consumers with your brokers. If your producer is in us-east-1a and all your brokers are in us-east-1b and us-east-1c, every message crosses an AZ boundary. Architects often overlook this because single-AZ deployments feel risky—fair point—but at least be aware of the cost.

Here's a practical scenario: I was running a trade execution pipeline on MSK. Orders came in from a client-facing API in us-east-1a. The Kafka cluster was in 1a, 1b, 1c for redundancy. But our stream processors (reading from Kafka) were in separate ASGs spread across 1b and 1c. Every message was being read and processed across AZ boundaries. We consolidated the stream processors to 1a and saved $40/month on a $400 Kafka bill—10% reduction just by thinking about locality.

// Example: explicit AZ awareness when configuring clients
// Place this in an EC2 instance in us-east-1a to minimize cross-AZ traffic

import { Kafka } from 'kafkajs';

const kafka = new Kafka({
  clientId: 'trade-consumer',
  brokers: [
    'b-1.cluster.xxxxx.kafka.us-east-1.amazonaws.com:9092', // 1a
    'b-2.cluster.xxxxx.kafka.us-east-1.amazonaws.com:9092', // 1b (fallback)
    'b-3.cluster.xxxxx.kafka.us-east-1.amazonaws.com:9092', // 1c (fallback)
  ],
  ssl: true,
});

const consumer = kafka.consumer({ groupId: 'trade-processor' });
await consumer.connect();
await consumer.subscribe({ topic: 'trades' });

// KafkaJS will prefer brokers with lower latency; co-locating helps
await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    console.log(`Trade: ${message.value}`);
  },
});

Common mistakes

• Underprovisioning on day one. You can scale Confluent Cloud partitions and MSK brokers, but not instantly. Plan for 2–3x your current throughput forecast to avoid midnight scaling emergencies.
• Ignoring schema drift. Managed Kafka without schema governance is just self-hosted Kafka in the cloud. If you pick MSK, set up Confluent Schema Registry separately or use AWS Glue Schema Registry—both are small additional costs that prevent production incidents.
• Forgetting to budget for egress. Confluent and AWS charge egress (data leaving the cluster). A consumer draining 100 GB to S3 for backups can add $1 per GB. Ten backups per month on a 100 GB partition is $1,000 in egress.
• Confusing CKU math on Confluent. One CKU gives throughput per partition, not per cluster. If you have 50 partitions and want 2 MB/sec per partition, you need 100 CKUs, not 2. This is easy to miscalculate during planning.
• Over-engineering single-region deployments. Three-AZ replication is standard, but for non-critical workloads, a single AZ saves cross-AZ transfer costs. Evaluate your SLA before defaulting to multi-AZ.

Key takeaways

• MSK standard is cost-optimal for predictable, steady-state workloads; MSK Serverless wins for bursty traffic.
• Confluent Cloud costs more per MB/sec but includes Schema Registry, connectors, and ksqlDB—valuable if you're building data platforms.
• Compare pricing not just on compute, but also storage, egress, and cross-AZ transfer; the latter can add 10–20% to your monthly bill.
• Co-locate producers, consumers, and brokers in the same AZ whenever possible to minimize inter-AZ data transfer charges.
• Schema governance (Schema Registry or equivalent) is non-optional in production, regardless of which managed Kafka you choose.
• Start with a managed offering if you have <2 engineers; self-host only if you have dedicated Kafka expertise and a reason to own the operational risk.