Stateless vs stateful processing: lessons from production

After 37 days of building Kafka pipelines, you've probably processed millions of events. But once you need to aggregate, join, or correlate data across time, you hit a wall: stateless vs stateful processing becomes the decision that either scales elegantly or breaks under load. State is where stream processing gets hard, and understanding when to use each approach will save you from building systems that crumble when a late event arrives or a second instance spins up.

Stateless scales by adding instances

Stateless processing is the simplest form of stream handling. Each event enters, gets transformed or filtered, and exits independently. No memory of previous events, no shared computation context. Think of it as a pure function: same input always produces the same output, regardless of what came before.

The beauty here is horizontal scalability. If you have a stateless processor consuming 10,000 events/sec and you need to handle 20,000, you spin up another instance and add it to the consumer group. Kafka automatically rebalances partitions, and you're done. No coordination, no synchronization, no distributed locking.

I learned this the hard way with an e-commerce platform. We had a payment event processor that validated transactions, applied currency conversion, and logged outcomes. All stateless. When Black Friday traffic spiked, we went from 3 to 8 instances in minutes without any code changes. The system scaled linearly because each instance processed its assigned partitions independently.

Here's what a stateless processor looks like:

async function processPaymentEvent(event: PaymentEvent): Promise {
  const converted = convertCurrency(event.amount, event.currency, 'USD');
  const isValid = validateAmount(converted);
  
  if (isValid) {
    await publishEvent('payment.validated', {
      ...event,
      amountUsd: converted,
      processedAt: Date.now()
    });
  } else {
    await publishEvent('payment.rejected', event);
  }
}

This function takes an event, does computation, produces output. No side effects beyond publishing. No dependency on state from previous events. Run it across 1 instance or 100, the behavior remains identical. The partition offset advances, and Kafka tracks where you left off.

The trade-off is obvious: you can't calculate per-user sums, detect fraud patterns, or join with historical data. Those require memory of the past, which brings us to stateful processing.

Stateful needs state stores and changelogs

Stateful processing maintains intermediate results across events. You're building a picture of "what happened so far" for each key. A rolling 5-minute window sum, a user session, a cart total, a fraud score—these all require state.

The challenge is scale and fault tolerance. If your processor crashes after 2 hours of computation, you can't lose that state. If you add a second instance, both can't update the same state simultaneously. Kafka Streams solves this with local state stores backed by changelog topics.

Here's the architecture: your processor maintains a local RocksDB (or similar) with state keyed by event key. Every state change gets published to an internal changelog topic in Kafka. When a processor fails, a standby replica or a restarted instance rebuilds the entire state by replaying the changelog topic from the beginning. When you scale horizontally, each instance owns a partition of keys, so state remains local and single-threaded.

A practical example: I built a ride-hailing matching system where we tracked driver availability in real time. For each driver, we stored location, current load, and earnings this hour. Every status update (pickup, drop-off, app logout) changed this state. Here's how it looked:

interface DriverState {
  driverId: string;
  lat: number;
  lng: number;
  currentLoadPercent: number;
  earningsThisHour: number;
  lastUpdatedAt: number;
}

const driverStateStore = new KeyValueStore<string, DriverState>('driver-state');

async function processDriverEvent(event: DriverStatusEvent): Promise<void> {
  const currentState = driverStateStore.get(event.driverId) || createEmptyState(event.driverId);
  
  const updatedState = {
    ...currentState,
    lat: event.lat,
    lng: event.lng,
    currentLoadPercent: event.loadPercent,
    earningsThisHour: currentState.earningsThisHour + (event.tripEarnings || 0),
    lastUpdatedAt: Date.now()
  };
  
  driverStateStore.put(event.driverId, updatedState);
  
  // State change is automatically published to changelog topic
}

Now the problem: if I try to scale this to 2 instances without careful partitioning, both might try to update the same driver's state. Kafka Streams handles this by ensuring all events for a given key (driverId) route to the same partition and instance. The changelog topic preserves ordering, so if the instance fails, a new one replays events in sequence and reconstructs exact state.

But there's overhead: changelog topics consume disk, network, and CPU. State size grows over time. Changelog log compaction helps, but large states still hurt restart times. For a trading platform I worked on, we had per-symbol order books with thousands of orders each. State size was manageable, but changelog replay on restart took minutes.

The key insight: stateful processing doesn't scale by simply adding instances. You scale by partitioning your state space by key and ensuring each partition's events go to one instance. Adding instances without increasing keys doesn't help—it just spreads the load unevenly.

Late events complicate windows

Stateful processing often involves time windows: "sum sales in 5-minute buckets" or "count users per 10-second interval." Kafka tracks event time (when the event happened) separately from processing time (when you processed it). Late events—data that arrives after its window closed—create chaos.

Imagine a travel booking system aggregating reservations by hour. Your 2pm window closes at 2:59:59. But at 3:15pm, a delayed event arrives for a reservation actually made at 2:30pm. You've already emitted the 2pm aggregate and maybe triggered downstream billing or reporting. Do you recalculate? Hold the window open? Emit a correction?

Kafka Streams lets you configure grace periods: windows stay open for N milliseconds after the watermark passes, accepting late events. But you have to handle the consequences. Downstreams need idempotency or must expect updates to past results.

Here's a window processor with grace handling:

const bookingAggregation = stream
  .windowedBy(TimeWindows.of(Duration.ofMinutes(5)).grace(Duration.ofMinutes(2)))
  .aggregate(
    () => ({ count: 0, totalRevenue: 0 }),
    (key, event, state) => ({
      count: state.count + 1,
      totalRevenue: state.totalRevenue + event.price
    }),
    Materialized.as('booking-window-store')
  );

bookingAggregation.toStream().foreach((key, result) => {
  console.log(`Window ${key.window().startTime()}: ${result.count} bookings, $${result.totalRevenue}`);
});

The grace period of 2 minutes means events arriving up to 2 minutes after the window closes can still update the aggregate. Emissions happen twice: once when the window closes (partial result) and again if a late event arrives (corrected result). Subscribers must be prepared to handle re-emissions or else you'll have data corruption downstream.

Late events also require more state. A longer grace period or sliding windows means the processor holds state for more windows simultaneously, increasing memory and changelog churn. I've seen systems grind to a halt because they configured 15-minute grace periods without realizing that meant keeping 3x more state in memory.

Start stateless whenever you can

My experience across five different production systems points to one pattern: stateless is faster to build, easier to debug, and simpler to scale. Start there. Only move to stateful when you genuinely can't solve the problem without it.

Ask yourself: can I buffer events and batch-process them in a database? Can I denormalize data upstream so I don't need joins? Can I split this into a stateless stream processor plus a separate batch job? Often the answer is yes, and you've avoided the complexity.

For a CRM platform, we initially tried a stateful processor to update customer scores in real time. Then we realized: why not emit raw events to a Kafka topic, let a batch job process them nightly, and subscribers query the database? Instant simplification. No state stores, no recovery logic, no grace period tuning.

When you do need state, keep it small and ephemeral. Prefer bounded state (drop old keys) over unbounded. Use changelog log compaction. Monitor state size and restart times religiously.

Common mistakes

• Mixing state and logic: Putting complex business logic inside state mutations causes bugs during recovery. Keep mutations simple; do calculations downstream.
• Ignoring changelog size: Changelog topics grow without bounds if you don't enable log compaction. I've seen a single processor consume 500GB of storage because nobody set cleanup.policy=compact.
• Assuming late events are rare: In mobile or IoT systems, 10-20% of events can arrive late. Configuring zero grace period silently drops data. Test with realistic delays.
• Scaling by adding instances without repartitioning keys: If all your state is on one key, one instance handles all the work. Adding instances doesn't help. You must partition by domain.
• Not idempotenting state updates: If a processor restarts mid-window, it replays events and re-applies state mutations. If mutations aren't idempotent, counts double. Always use upserts, never increments.

Key takeaways

• Stateless processors scale horizontally by adding instances; each instance processes its assigned partitions independently with zero coordination.
• Stateful processors require local state stores backed by Kafka changelog topics; scaling requires partitioning your state space by key so each instance owns disjoint keys.
• Late events and time windows introduce complexity: grace periods keep windows open but force downstream systems to handle re-emissions and updates to past results.
• Start stateless whenever possible; move to stateful only when windowing, joins, or per-key aggregation are unavoidable.
• Stateful recovery depends on changelog replay; keep state small, enable log compaction, and test restart times under realistic scale.
• Late events are common in real systems; configure grace periods based on measured delays, not assumptions, and ensure all state mutations are idempotent.