Stream processing basics: a hands-on guide for engineers

Stream processing basics: a hands-on guide for engineers

Most backend engineers deal with request-response patterns all day—a customer clicks, you query, you respond. But what happens when events arrive faster than you can process them individually, or when you need to compute insights across hundreds of thousands of events per second? That's where stream processing comes in. This guide covers the fundamentals of computing over events as they arrive, moving beyond simple event consumption toward real production patterns we've built for e-commerce checkouts, ride-hailing platforms, and trading systems.

Why stream processing matters in modern systems

Stream processing sits between real-time dashboards and batch jobs. When you need aggregations, joins, or stateful transformations that happen continuously as data flows through Kafka, you're doing stream processing. Unlike batch jobs that run on fixed schedules, stream processors react to events immediately. This matters when your business cares about latency—fraud detection needs answers in milliseconds, not hours. Real-time inventory updates can't wait for nightly ETL. A ride-hailing platform needs to match drivers to riders as requests arrive, not batch them up.

The trade-off is complexity. Stateless filtering is simple. Adding state, maintaining consistency under failure, and handling out-of-order events—that's where engineering discipline becomes essential.

Stateless operations—filter, map, and route events

Start here if you're new to stream processing. Stateless operations work on each event independently, with no memory of previous events. They're the easiest to build, test, and scale horizontally.

Filter removes unwanted events. Map transforms each event into a new shape. Route sends events to different topics based on their content. In an e-commerce system, you might filter out test orders, map order events into a standardized format for downstream consumers, and route high-value orders to a premium fulfillment topic.

Here's a practical example using Kafka Streams:

const kafka = require('kafkajs').Kafka;

const client = new kafka.Kafka({
  clientId: 'order-processor',
  brokers: ['localhost:9092']
});

const consumer = client.consumer({ groupId: 'order-group' });
const producer = client.producer();

async function processOrders() {
  await consumer.connect();
  await producer.connect();
  
  await consumer.subscribe({ topic: 'orders' });
  
  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      const order = JSON.parse(message.value);
      
      // Filter: skip test orders
      if (order.env === 'test') return;
      
      // Map: normalize currency
      const normalized = {
        ...order,
        amountUsd: order.currency === 'USD' 
          ? order.amount 
          : order.amount / order.exchangeRate
      };
      
      // Route: send to different topics
      const targetTopic = normalized.amountUsd > 1000 
        ? 'premium-orders' 
        : 'standard-orders';
      
      await producer.send({
        topic: targetTopic,
        messages: [{ value: JSON.stringify(normalized) }]
      });
    }
  });
}

processOrders().catch(console.error);

This pattern scales linearly. Add more partitions, add more consumer instances, and throughput grows. No shared state means no locks, no coordination, no bottlenecks.

Stateful operations—counts, joins, and aggregations

Stateless operations only get you so far. The moment you need to know "how many orders today" or "match this purchase to the customer's previous browsing history," you need state. Stateful operations remember context across events, which adds complexity but unlocks powerful capabilities.

State stores are typically local to each stream processor instance. Kafka Streams maintains them as embedded RocksDB instances. They're backed by changelog topics in Kafka, so if a processor crashes, it rebuilds state from the beginning. This gives you fault tolerance without external databases.

Consider a ride-hailing platform tracking active driver counts by zone:

// Simplified Kafka Streams topology in JavaScript via node-rdkafka
const Kafka = require('node-rdkafka');

const builder = new Kafka.streams.StreamsBuilder();

const driverEvents = builder.stream('driver-status-changes');

driverEvents
  .map((key, value) => {
    const event = JSON.parse(value);
    return [event.zone, JSON.stringify({ driverId: event.driverId })];
  })
  .groupByKey()
  .count()  // Counts occurrences per zone
  .toStream()
  .to('active-drivers-by-zone');

const streams = builder.build();
streams.start();

Here, we're maintaining a count per zone. When a driver comes online or goes offline, the count updates automatically. Behind the scenes, Kafka Streams keeps this count in a state store. If we need higher availability, we can run multiple instances of this topology; Kafka Streams handles state migration automatically.

Joins combine two streams or a stream with a table. Matching a payment event to its corresponding order requires a join. So does enriching a click stream with customer attributes. Stream-to-stream joins are harder than stream-to-table joins because both sides are moving; you must decide on a time window. Stream-to-table joins are simpler—the table (typically a compacted topic or an external database) is treated as reference data.

A common pattern in trading systems: join trade executions (stream) with open positions (table) to calculate portfolio impact:

const trades = builder.stream('trade-executions');
const positions = builder.table('customer-positions');  // Compacted topic

trades
  .join(
    positions,
    (trade, position) => {
      return {
        customerId: trade.customerId,
        security: trade.security,
        newPosition: (position?.quantity || 0) + trade.quantity,
        impact: trade.quantity * trade.price
      };
    },
    Kafka.streams.Joined.with({
      name: 'trade-position-join',
      keySerde: 'string',
      valueSerde: 'json'
    })
  )
  .to('position-updates');

State grows with your data volume. A count per key scales linearly. A join needs to hold state for both sides. In production, monitor state store size. If you're joining millions of distinct keys, a separate database might be more efficient than local state.

Windows—tumbling, hopping, and session windows

Windows slice time into digestible chunks. They answer questions like "orders per minute" or "users active in the last 5 minutes." Different window types serve different use cases.

Tumbling windows are non-overlapping. A 1-minute tumbling window covers [00:00, 01:00), [01:00, 02:00), etc. Use them for dashboards and alerts. "Flag zones with more than 10 accident reports per 5 minutes."

Hopping windows overlap. A 1-minute window hopping every 30 seconds covers [00:00, 01:00), [00:30, 01:30), [01:00, 02:00), etc. They smooth out variations. SLA dashboards often use hopping windows to avoid sudden jumps.

Session windows group events by inactivity. If a user is inactive for 10 minutes, the session ends. Ideal for user engagement metrics in CRM systems or ride-hailing apps. A driver's "shift" is naturally a session—online continuously, then offline.

In an e-commerce platform, you might track checkout abandonment using session windows:

const checkoutEvents = builder.stream('checkout-events');

checkoutEvents
  .groupByKey()  // Group by session ID
  .windowedBy(
    Kafka.streams.SessionWindows.with(
      60 * 1000  // 60-second inactivity timeout
    )
  )
  .aggregate(
    () => ({ events: [], lastEventTime: null }),
    (key, event, aggregate) => ({
      events: [...aggregate.events, event.type],
      lastEventTime: event.timestamp
    })
  )
  .toStream()
  .filter((key, aggregate) => {
    // Emit only if the last event was checkout_abandoned
    return aggregate.events[aggregate.events.length - 1] === 'checkout_abandoned';
  })
  .to('abandoned-checkouts');

Windows require careful timestamp handling. Use event time (when the order was placed), not processing time (when your processor saw it). Events may arrive out of order—a network delay can cause an old event to arrive late. Configure grace periods to allow late arrivals. Too short, and you lose data; too long, and memory grows.

Tools—Kafka Streams, Apache Flink, or ksqlDB

You have three main options, each with strengths.

Kafka Streams is a library, not a separate service. Deploy it inside your application. Minimal operational overhead. Works well for teams already running Java or Node.js. State management is built in. No external cluster to manage. Best for: teams wanting simplicity and tight integration with their apps.

Apache Flink is a distributed stream processor. More powerful than Kafka Streams—better windowing semantics, more sophisticated state backends, savepoints for versioning. Steeper learning curve and operational complexity. You manage a Flink cluster. Best for: large-scale deployments where you need the power, or when your team is already running it.

ksqlDB is SQL on Kafka. Write SELECT * FROM orders WHERE amount > 1000; and it becomes a stream processor. Rapid prototyping. No coding. Good for: analysts, dashboards, and quick alerts. Less flexible for complex business logic.

Our experience: start with Kafka Streams if you're a small team with engineering depth. Move to Flink if you hit scaling limits or need advanced features. Use ksqlDB alongside for operational dashboards and ad-hoc queries.

Common mistakes

• Ignoring late events. Networks are unreliable. Set appropriate grace periods on windows. Test with synthetic delays.
• Unbounded state growth. If you're grouping by a high-cardinality key (user ID, transaction ID), state grows forever. Use windowing and explicit retention policies.
• Processing time instead of event time. Causes aggregations to be skewed. Always use the event's timestamp, not when you processed it.
• Synchronous external calls in stream handlers. One slow downstream service blocks your entire pipeline. Use local aggregation and async pushes, or accept the latency trade-off.
• No backpressure handling. If your processor can't keep up, it buffers. Kafka eventually fills. Monitor lag. Scale early.
• Treating state stores as databases. They're optimized for streaming, not random access. For lookups outside your topology, use a real database.

Key takeaways

• Stateless operations (filter, map, route) are your foundation. They're simple, testable, and infinitely scalable. Use them liberally.
• Stateful operations (aggregations, joins, counts) unlock insights but require careful state management. Monitor state store size and configure retention policies.
• Windows group events in time. Use tumbling for dashboards, hopping for smoothing, and session windows for user-behavior analysis.
• Choose your tool based on team expertise and operational appetite. Kafka Streams for simplicity, Flink for power, ksqlDB for speed.
• Event time, grace periods, and late arrival handling are non-negotiable in production. Test failure modes aggressively.
• Start stateless, add state only when needed, and always have a strategy for state cleanup and recovery.