Kafka with Go: concepts, trade-offs and production lessons

Kafka has become the backbone of event-driven architectures in production systems, yet many teams ship Go services that lose messages, process duplicates, or scramble the order of critical events. Understanding how to produce and consume events reliably is not optional—it's the difference between a system you can sleep on and one that wakes you at 3 AM.

This guide covers Kafka with Go: concepts, trade-offs and production lessons, distilled from real deployments in e-commerce, ride-hailing, and SaaS environments. We'll walk through client choice, partitioning strategy, commit semantics, and idempotency patterns that turn a working prototype into a system you trust.

Clients: franz-go vs confluent-kafka-go

Your first decision is the client library. We typically choose between two strong options: franz-go (pure Go) and confluent-kafka-go (librdkafka binding).

franz-go is a modern, pure-Go implementation with excellent ergonomics. It's easier to deploy (no C dependencies), has better error messages, and integrates cleanly with Go's standard patterns. For greenfield services or when you control the full stack, franz-go is our go-to choice. It's maintained actively and performs well in production.

confluent-kafka-go wraps librdkafka, a battle-tested C library used across many organizations. If your team already standardizes on it across services or languages, staying consistent reduces cognitive load. The trade-off: you ship a C dependency, must manage CGO, and debugging feels less idiomatic. In containerized environments, this is manageable but adds friction.

For new Go projects, we recommend franz-go unless you have a strong organizational reason to use confluent-kafka-go.

Keys for Ordering: Same Key, Same Partition, Ordered Events

One of Kafka's most misunderstood features is partitioning. Messages sent to a topic with the same key always land in the same partition, preserving order within that partition. This is critical for workflows where sequence matters.

Consider an e-commerce system. A customer's order history must be processed in order: create order → capture payment → update inventory → notify warehouse. If these events hit different partitions, a slow consumer might process "notify warehouse" before "capture payment" completes, leaving the warehouse confused.

The solution: use customer ID as the key.

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/segmentio/kafka-go"
)

func produceOrderEvents(ctx context.Context, writer *kafka.Writer, customerID string) error {
	events := []struct {
		EventType string
		Payload   string
	}{
		{"order_created", "order_123"},
		{"payment_captured", "txn_456"},
		{"inventory_updated", "sku_789"},
	}

	for _, evt := range events {
		msg := kafka.Message{
			Key:   []byte(customerID), // Same customer ID → same partition
			Value: []byte(fmt.Sprintf(`{"event":"%s","data":"%s"}`, evt.EventType, evt.Payload)),
		}

		if err := writer.WriteMessages(ctx, msg); err != nil {
			return fmt.Errorf("write failed: %w", err)
		}
		log.Printf("Produced %s for customer %s", evt.EventType, customerID)
	}

	return nil
}

Without a key, Kafka distributes messages round-robin across partitions. With a key, all messages for that customer go to the same partition, and consumers within a consumer group process them in order. This is per-partition ordering, not global—but it's usually what you need.

One pitfall: don't assume all partitions process at the same speed. If partition 0 is slow and partition 1 is fast, partition 1's messages will be processed first. This is fine as long as different customers don't depend on each other's order. For workflows within a single entity (customer, order, user), partition-level ordering is sufficient.

Commit After Processing: Avoid Losing Messages on Crashes

A consumer reads messages from Kafka and must track which ones it has processed. This is done via commits—telling Kafka "I've successfully handled messages up to offset X in partition Y." If your consumer crashes before committing, Kafka will redeliver those unacked messages when it restarts.

The trade-off is between safety and speed. Manual commits after successful processing are slower but safer; automatic commits are fast but risky if your handler crashes mid-processing.

In production, we default to manual commits with processing-first semantics: process the message, commit only after it's safely persisted or sent downstream.

package main

import (
	"context"
	"log"

	"github.com/segmentio/kafka-go"
)

func consumeOrdersManually(ctx context.Context, reader *kafka.Reader) error {
	for {
		msg, err := reader.ReadMessage(ctx)
		if err != nil {
			log.Printf("Read error: %v", err)
			continue
		}

		// Step 1: Process the message
		if err := processOrderEvent(ctx, string(msg.Value)); err != nil {
			log.Printf("Processing failed (will retry): %v", err)
			// Don't commit; consumer will reprocess on restart
			continue
		}

		// Step 2: Commit only after successful processing
		if err := reader.CommitMessages(ctx, msg); err != nil {
			log.Printf("Commit failed: %v", err)
			// Handler succeeded but commit failed; reprocess next time
			// (idempotency handles this—see below)
		}

		log.Printf("Message committed: offset %d", msg.Offset)
	}
}

func processOrderEvent(ctx context.Context, payload string) error {
	// Simulate processing: write to DB, call external API, etc.
	log.Printf("Processing: %s", payload)
	return nil
}

Notice the asymmetry: a processing failure means we skip the commit and reprocess. A commit failure means the processing succeeded (it's already written to the database or sent to the API) but Kafka doesn't know. On restart, we'll reprocess—which is why idempotency is non-negotiable.

One caveat: committing every message is expensive in high-throughput scenarios. If you're processing 100k messages per second, committing each one triggers network round-trips. Batching commits (e.g., every 100 messages or every 5 seconds) is a practical optimization, but only if your handler is truly idempotent.

Idempotent Handlers: Messages Can Be Delivered Twice

Kafka guarantees at-least-once delivery, not exactly-once. A message can be delivered twice if a consumer crashes between processing and commit, or if a broker rebalance happens mid-commit. Your handler must tolerate this.

Idempotency means: running the same operation twice produces the same result as running it once. For most state changes, this requires a unique identifier and an upsert pattern.

Example: an order-creation event includes an idempotency key (UUID). On the first delivery, we check if an order with that key exists; if not, we create it. On the second delivery (reprocess after a crash), the key already exists, so we return success without creating a duplicate.

package main

import (
	"context"
	"database/sql"
	"fmt"
	"log"
)

type Order struct {
	IdempotencyKey string
	CustomerID     string
	Amount         float64
}

func createOrderIdempotent(ctx context.Context, db *sql.DB, order Order) error {
	// Use INSERT ... ON CONFLICT to handle duplicates
	query := `
		INSERT INTO orders (idempotency_key, customer_id, amount, created_at)
		VALUES ($1, $2, $3, NOW())
		ON CONFLICT (idempotency_key) DO NOTHING
		RETURNING id
	`

	var orderID int
	err := db.QueryRowContext(ctx, query, order.IdempotencyKey, order.CustomerID, order.Amount).Scan(&orderID)

	if err == sql.ErrNoRows {
		// Key already exists; idempotent success
		log.Printf("Order already exists for key %s", order.IdempotencyKey)
		return nil
	}
	if err != nil {
		return fmt.Errorf("insert failed: %w", err)
	}

	log.Printf("Order created: ID=%d, key=%s", orderID, order.IdempotencyKey)
	return nil
}

The idempotency key must come from the producer. If your event originates from a user action (place order, send payment), generate a UUID upstream and include it in the Kafka message. If the event is generated internally (scheduled job, webhook), ensure each event has a unique, deterministic key (e.g., hash of timestamp + entity ID).

Database constraints are your friend: add a unique index on the idempotency key column. The database enforces idempotency even if your application logic has bugs.

Common mistakes

• Committing before processing. If you commit the offset and then the handler crashes, you lose that message forever. Always process first, commit after.
• Using auto-commit with long processing times. Auto-commit fires on a timer, not after the handler finishes. If processing takes 30 seconds but auto-commit runs every 5 seconds, a crash mid-process causes duplicate delivery. Manual commits are slower but predictable.
• Ignoring consumer group rebalances. When a new consumer joins the group, Kafka triggers a rebalance. During rebalance, all consumers pause. If you're mid-commit, the rebalance might interrupt you. Use heartbeat and session timeout tuning to minimize rebalance frequency, but don't ignore it.
• Forgetting idempotency keys in event design. If your Kafka message payload doesn't include a way to deduplicate it, you'll struggle to add idempotency later. Include UUIDs from the start.
• Over-partitioning or under-partitioning. Too many partitions means each consumer might own dozens; too few means you bottleneck on one slow partition. Start with partitions equal to your peak concurrent consumer instances, then scale.

Key takeaways

• Choose franz-go for new Go projects unless your organization standardizes on confluent-kafka-go.
• Use message keys to route related events to the same partition, preserving order within a customer, order, or entity.
• Commit offsets only after successfully processing messages; use manual commits to control exactly when Kafka marks an offset as consumed.
• Design all handlers as idempotent: include idempotency keys in event payloads and use database upserts to tolerate duplicate deliveries.
• Test failure scenarios: simulate crashes mid-commit, slow consumers, and broker rebalances to verify your system handles at-least-once delivery correctly.