Run Kafka locally with Docker: best practices explained

Opening: Why local Kafka matters in your development workflow

Getting Kafka running on your machine shouldn't require wrestling with infrastructure or waiting for a staging environment to test message flows. In this guide, we'll show you how to run Kafka locally with Docker: best practices explained so you can develop and debug streaming features at full speed. Whether you're building a real-time notification system, event sourcing layer, or asynchronous task queue, having a reliable local Kafka setup saves weeks of frustrated debugging downstream.

One container, no ZooKeeper

The old way of running Kafka locally meant spinning up at least two containers: one for ZooKeeper and one for the broker. That added complexity, networking overhead, and created yet another dependency to manage. KRaft mode—Kafka Raft—eliminated ZooKeeper entirely by using Raft consensus for cluster metadata. For development, this means you get a single-container setup that mirrors modern production architectures.

Here's a minimal docker-compose.yml that runs a complete Kafka broker without external coordination:

version: '3.8'
services:
  kafka:
    image: apache/kafka:4.0.0
    container_name: local-kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://kafka:29092,CONTROLLER://kafka:29093,PLAINTEXT_HOST://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LOG_DIRS: /tmp/kafka-logs
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
    volumes:
      - kafka_data:/tmp/kafka-logs
    networks:
      - local-network

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    ports:
      - "8080:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:29092
      KAFKA_CLUSTERS_0_METRICS_PORT: 9999
    depends_on:
      - kafka
    networks:
      - local-network

volumes:
  kafka_data:

networks:
  local-network:
    driver: bridge

A few key points in this configuration:

• KAFKA_NODE_ID: Unique identifier for this broker. In single-node KRaft, it's typically 1.
• KAFKA_PROCESS_ROLES: Set to broker,controller to run both roles in one process, which is safe for local development.
• KAFKA_LISTENERS: Three protocols—one internal for broker-to-broker communication, one for the controller, one for external clients.
• KAFKA_ADVERTISED_LISTENERS: Maps external connections to localhost:9092 so your host machine can reach the broker.
• KAFKA_CONTROLLER_QUORUM_VOTERS: Points to itself (1@kafka:29093) for quorum voting.

Spin it up with docker-compose up -d and your Kafka broker is live in seconds. No ZooKeeper daemon, no separate container orchestration—just one lightweight process running broker and controller logic.

Add Kafka UI for browsing topics

Command-line tools like kafka-console-consumer.sh have their place, but when you're iterating fast, a visual UI dramatically speeds up debugging. Kafka UI gives you a web interface to explore topics, consumer groups, messages, and partitions without SSH-ing into containers or memorizing CLI flags.

The docker-compose.yml above includes Kafka UI already. Point your browser to http://localhost:8080 and you'll see:

• A list of all topics and their partition counts
• Message content for each topic (inspect actual events in real time)
• Consumer group lag and offset tracking
• Broker metrics and cluster health

This is invaluable when you're testing a producer that's pushing 50 events per second and need to verify the exact payload structure or check if messages are being distributed across partitions evenly. Rather than writing temporary shell scripts to dump messages, you click through the UI in seconds.

Seed topics with scripts

Before your application connects to Kafka, you need topics to exist. You could create them manually each time you start the compose stack, but that's error-prone and slows down onboarding for new team members. Instead, automate topic creation with an init script that runs on startup.

Create a init-topics.sh file in your project:

#!/bin/bash
set -e

# Wait for Kafka to be ready
sleep 10

# Create topics
docker exec local-kafka kafka-topics.sh --bootstrap-server kafka:29092 \
  --create --if-not-exists \
  --topic orders \
  --partitions 3 \
  --replication-factor 1

docker exec local-kafka kafka-topics.sh --bootstrap-server kafka:29092 \
  --create --if-not-exists \
  --topic payments \
  --partitions 2 \
  --replication-factor 1

docker exec local-kafka kafka-topics.sh --bootstrap-server kafka:29092 \
  --create --if-not-exists \
  --topic notifications \
  --partitions 1 \
  --replication-factor 1

echo "Topics created successfully"

Then modify your compose file to include a service that runs this script:

  init-kafka:
    image: apache/kafka:4.0.0
    container_name: init-kafka
    entrypoint: /scripts/init-topics.sh
    volumes:
      - ./init-topics.sh:/scripts/init-topics.sh
    depends_on:
      - kafka
    networks:
      - local-network

Now when you run docker-compose up, Kafka automatically creates your three topics with appropriate partition counts. Teams working on order processing, payment events, and user notifications can all start developing immediately without manual setup steps.

For even more control, you can seed topics with sample data. Add a producer service that publishes test events at startup—useful when you're developing consumer logic and need realistic data:

  seed-data:
    image: apache/kafka:4.0.0
    container_name: seed-kafka-data
    entrypoint: bash -c "
      sleep 15 &&
      echo 'order-123' | kafka-console-producer.sh --bootstrap-server kafka:29092 --topic orders &&
      echo 'payment-456' | kafka-console-producer.sh --bootstrap-server kafka:29092 --topic payments
    "
    depends_on:
      - kafka
    networks:
      - local-network

Match production versions

A common pitfall in local development is using an arbitrary Kafka version that differs from production. This can mask serialization issues, protocol incompatibilities, and broker feature differences that only surface when you deploy.

Check your production environment's Kafka version first. If you're running Kafka 4.0.0 in production (or 3.8, 3.7, etc.), use that same version locally. Update the image tag:

  kafka:
    image: apache/kafka:4.0.0  # Match your production broker version

Beyond the broker version, ensure your client libraries match the expected protocol version. If your NestJS or Node.js backend uses kafkajs, lock its version in package.json to a release tested against your broker:

{
  "dependencies": {
    "kafkajs": "^2.2.4"
  }
}

A mismatch between client and broker versions can cause subtle bugs—offsets not persisting correctly, consumer group rebalances behaving unpredictably, or authentication/SASL configuration being ignored. Spending five minutes to align versions saves hours of head-scratching later.

Common mistakes

Not waiting for broker readiness: Services that connect immediately after the Kafka container starts often fail because the broker hasn't yet initialized its logs or finished quorum setup. Add explicit health checks or use depends_on with conditions. The sleep 10 in init scripts is a crude but effective guard; better yet, use the broker's readiness endpoint in a wait loop.

Using replication-factor > 1 for single-node setups: If you set --replication-factor 2 but only have one broker, topics won't initialize. For local development, always use --replication-factor 1. Production defaults should differ, but local is one-broker-only.

Forgetting to persist data: Without volumes, your Kafka logs vanish when containers restart. The compose file above includes kafka_data:/tmp/kafka-logs, so messages survive restarts. This is crucial for testing consumer group offset recovery or replay scenarios.

Misconfiguring listener addresses: The trio of KAFKA_LISTENERS, KAFKA_ADVERTISED_LISTENERS, and client bootstrap addresses must align. If your producer connects to localhost:9092 but the broker advertises kafka:9092, you'll get connection timeouts or NAT-related issues. Double-check that external clients see PLAINTEXT_HOST://0.0.0.0:9092 and internal services see PLAINTEXT://kafka:29092.

Running old ZooKeeper-based images: Apache's older images still require ZooKeeper by default. Ensure you're pulling from apache/kafka version 3.3 or later, where KRaft is stable and enabled via environment variables.

Key takeaways

• Use KRaft mode with a single container—no ZooKeeper, simpler networking, closer to modern production Kafka.
• Add Kafka UI to visualize topics and messages during development; it's faster than CLI tools for iterative debugging.
• Automate topic creation with init scripts so new developers get a working setup on first docker-compose up.
• Always match your local Kafka version to production to catch version-related bugs early.
• Persist Kafka logs with Docker volumes so your development environment survives container restarts.
• Be explicit about listener configuration—separate internal and external addresses to avoid connectivity surprises.