What migrating Kafka from JSON to Avro taught me explained

Opening: Why I Finally Made the Switch

Last year, I migrated a ride-hailing platform's entire event streaming backbone from JSON to Avro, and I walked in thinking the win would be smaller payloads. I walked out understanding that the real value was something far more important: a contract enforcement layer that prevented bugs before they shipped. What migrating Kafka from JSON to Avro taught me fundamentally changed how my team thinks about event-driven architecture.

When you're running Kafka topics that power critical services—booking confirmations, driver assignments, payment events—schema fragmentation becomes a silent killer. JSON let us be flexible. Too flexible. And that flexibility had a cost.

Payload Size Dropped Noticeably

Let's start with the obvious win. I ran a quick audit across three high-volume topics in our platform. Our driver location event, published roughly 50 million times per day, was averaging 340 bytes in JSON. The same event serialized with Avro sat at 86 bytes. That's a 75% reduction.

Here's what a typical location event looked like in JSON:

{
  "event_id": "uuid-here-32-chars",
  "driver_id": 12345678,
  "latitude": 6.927079,
  "longitude": 80.771145,
  "timestamp": "2024-01-19T14:23:45.123Z",
  "accuracy_meters": 8.5,
  "speed_kmh": 42.3,
  "heading_degrees": 245,
  "event_type": "location_update",
  "source": "gps",
  "is_active": true,
  "region_code": "CMB"
}

The Avro schema for the same event defined all types upfront, eliminated redundant field names in every serialized message, and used efficient binary encoding:

{
  "type": "record",
  "name": "DriverLocationEvent",
  "namespace": "com.ridehail.events",
  "fields": [
    {"name": "event_id", "type": "string"},
    {"name": "driver_id", "type": "long"},
    {"name": "latitude", "type": "double"},
    {"name": "longitude", "type": "double"},
    {"name": "timestamp", "type": "long"},
    {"name": "accuracy_meters", "type": ["null", "float"], "default": null},
    {"name": "speed_kmh", "type": ["null", "float"], "default": null},
    {"name": "heading_degrees", "type": ["null", "float"], "default": null},
    {"name": "event_type", "type": "string"},
    {"name": "source", "type": "string"},
    {"name": "is_active", "type": "boolean"},
    {"name": "region_code", "type": "string"}
  ]
}

Multiply 254 bytes saved per event by 50 million events daily, and you're looking at roughly 12.7 GB of storage saved per day. Over a month, that's 381 GB. For a high-traffic system, that cascades into real infrastructure savings—fewer broker disks, lower S3 costs for retention, faster replication across availability zones.

But here's what surprised me: the storage savings weren't the team's main concern after a few weeks. It was what Avro prevented.

Producers Could No Longer Send Malformed Events

In our old JSON world, nothing stopped a producer from sending this:

{
  "driver_id": "not-a-number",
  "latitude": "invalid",
  "timestamp": "whenever"
}

A consumer somewhere downstream would eventually hit that garbage, fail silently, trigger a dead-letter queue, and create a support ticket three days later. We had defensive parsing everywhere: null checks, try-catch blocks, type coercion logic, metric emissions for malformed events. That code was expensive to write and maintain.

With Avro, the producer itself couldn't serialize this. Before the message ever touches Kafka, the Avro serializer validates:

• The field driver_id must be a long (64-bit integer), not a string.
• The field latitude must be a double, not a string.
• The field timestamp is required and must be a long.
• If accuracy_meters is provided, it must be a float or null—nothing else.

An invalid event throws an exception at the point of production, not silently in a consumer hours later. We pushed validation left, which meant better observability and faster feedback loops.

In our NestJS producer code, we integrated the @confluent/kafka client with Avro serialization. If a service tried to publish an event with the wrong schema, it failed immediately during local testing or CI:

import { Kafka } = from '@confluentinc/kafka-javascript';
import { SchemaRegistry } from '@confluent/schema-registry';

const schemaRegistry = new SchemaRegistry({
  host: 'http://schema-registry:8081'
});

const kafka = new Kafka({
  clientId: 'ride-hail-producer',
  brokers: ['localhost:9092']
});

const producer = kafka.producer();

async function publishLocationEvent(event: any) {
  const schemaId = await schemaRegistry.getLatestSchemaId(
    'com.ridehail.events.DriverLocationEvent-value'
  );
  
  const serialized = await schemaRegistry.encode(schemaId, event);
  
  await producer.send({
    topic: 'driver-locations',
    messages: [{ value: serialized }]
  });
}

The encode step fails if event doesn't match the schema. No guessing, no surprises in production.

Schema Evolution Rules Forced Better Design

This is where the philosophy shifted. Avro's compatibility rules—forward, backward, and full compatibility—forced us to think about contracts differently.

Backward compatibility means new consumers can read old data. Forward compatibility means old consumers can read new data. Full compatibility means both. We opted for full compatibility as a team standard.

When we needed to add a new field battery_percentage to the location event, we couldn't just drop it in. We had to make it optional with a default:

{
  "name": "battery_percentage",
  "type": ["null", "int"],
  "default": null
}

When we wanted to rename heading_degrees to bearing_degrees, we couldn't just change the name. We had to alias it and deprecate the old field:

{
  "name": "bearing_degrees",
  "type": "float",
  "aliases": ["heading_degrees"]
}

These rules sound restrictive at first. They are. But they're restrictive in a healthy way. They forced us to think about the production rollout strategy before writing code. We couldn't deploy a consumer that expected a field that didn't exist yet. We couldn't deploy a producer that removed a field old consumers were still reading.

This eliminated entire categories of deployment ordering bugs. Previously, we'd had incidents where a new service got deployed, started publishing a different JSON shape, and upstream services couldn't parse it. With Avro's rules, that contract violation happened at serialization time, caught during testing.

Plan Dual-Write or a Translation Period

The migration itself isn't trivial. You can't flip a switch and convert all your producers and consumers in one moment. We planned a three-week dual-write window.

Week one: new producers published to a parallel Avro topic alongside the existing JSON topic. Consumers didn't change. This let us validate that Avro serialization worked and captured all the data fidelity we needed.

Week two: a translation service consumed from the JSON topic and republished to the Avro topic. We ran both the old and new consumer code in parallel, comparing results to ensure they were equivalent. Real-time reconciliation caught subtle differences in how we handled null values or timestamp precision.

Week three: we migrated consumers one service at a time. The booking service switched first (smallest risk), then the driver app (moderate risk), then the backend orchestration service (highest risk). If any consumer had trouble, we could roll back to the JSON topic immediately.

Here's the consumer migration pattern we used:

async function consumeLocationEvents() {
  const consumer = kafka.consumer({ groupId: 'booking-service' });
  await consumer.subscribe({ 
    topic: 'driver-locations-avro',
    fromBeginning: false 
  });

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      try {
        // Deserialize Avro
        const schemaId = message.headers?.schemaId 
          ? parseInt(message.headers.schemaId.toString())
          : await schemaRegistry.getLatestSchemaId(
              'com.ridehail.events.DriverLocationEvent-value'
            );
        
        const event = await schemaRegistry.decode(schemaId, message.value);
        
        // Process
        await handleLocationUpdate(event);
      } catch (error) {
        // Log the failure with the raw message for debugging
        logger.error('Failed to process location event', {
          error: error.message,
          rawValue: message.value.toString('base64')
        });
        throw error; // Trigger a retry or DLQ
      }
    }
  });
}

The key was that we could run old and new consumers simultaneously, which meant zero downtime and a genuine safety net.

Common Mistakes

Not versioning schemas in the registry

We initially didn't track which version of the schema produced a message. This made debugging old messages in dead-letter queues nearly impossible. Always include the schema ID or version in your message headers.

Treating evolution rules as suggestions

Someone on the team tried to remove a field without checking backward compatibility. It compiled fine locally but broke consumers in production. Schema registry can enforce this, but only if you enable it. Make it mandatory, not optional.

Forgetting about tooling dependencies

Every consumer and producer needs Avro libraries and schema registry access. We discovered one legacy microservice we'd forgotten about that still expected JSON. It crashed silently for six hours until monitoring caught it. Audit all services before the migration.

Setting strict compatibility without understanding the cost

Full compatibility is the safest but requires discipline. Some teams find backward-only or forward-only sufficient for their use case. Discuss it explicitly with your team instead of defaulting.

Key Takeaways

• Payload size reduction (in our case, 75%) is real and compounds over time, but it's the bonus, not the main event.
• Schema enforcement at serialization time catches bugs before they reach production, saving debugging time and support tickets.
• Evolution rules feel restrictive but encode best practices: they force you to think about deployments, backward compatibility, and consumer expectations upfront.
• Plan a dual-write and validation window. The migration is not instantaneous, and that's okay—safety margins are valuable.
• Invest in schema registry access, versioning, and enforcement policies. The infrastructure pays for itself in operational stability.
• What migrating Kafka from JSON to Avro taught me was that serialization format isn't really about size or speed—it's about building shared contracts between producers and consumers, written into the data layer itself.