Schema Registry and compatibility in Kafka & WebSockets

On Day 20 of our Kafka and WebSockets journey, we're tackling one of the most painful problems in event-driven systems: how do you change your event schemas without crashing every consumer downstream? Schema Registry and compatibility modes give you the framework to evolve your events safely, and getting this right early saves you from incident calls at 2 AM.

Why Schema Registry Matters in Production

I've seen teams burn hours debugging mismatched message formats between producers and consumers. A payment service shipped a new field, the order processor didn't know about it, and suddenly reconciliation reports went silent. Without a central source of truth for event structure, you're flying blind.

Confluent's Schema Registry (or similar tools like AWS Glue Schema Registry) solve this by:

• Enforcing a single definition of what an event looks like
• Versioning schemas automatically
• Embedding a schema ID in each message so consumers know exactly which version they're reading
• Rejecting producers that violate compatibility rules

In practice, your producer registers a schema once, Kafka messages get a tiny schema ID header, and your consumer reads that ID to deserialize correctly. No surprises, no runtime parsing errors.

Central Schemas — Producers Register, Messages Carry the ID

Here's the mental model: your Schema Registry is a library. When you want to emit an event, you first ask the registry "Does this schema already exist?" If not, you register it and get back an ID. Every message then carries that ID.

Let's say you have an order service emitting OrderCreated events:

// Your schema (Avro format)
const orderSchema = {
  type: 'record',
  namespace: 'com.myshop.events',
  name: 'OrderCreated',
  fields: [
    { name: 'orderId', type: 'string' },
    { name: 'customerId', type: 'string' },
    { name: 'amount', type: 'double' },
    { name: 'timestamp', type: 'long' }
  ]
};

// Register with Schema Registry
const registryClient = new SchemaRegistryClient({ host: 'localhost:8081' });
const { id } = await registryClient.register(orderSchema);
// Returns: { id: 1, version: 1, subject: 'OrderCreated-value' }

When you produce a message, the Kafka client automatically includes that schema ID:

const producer = kafka.producer({
  schemaRegistry: registryClient
});

await producer.send({
  topic: 'orders',
  messages: [
    {
      key: 'order-123',
      value: {
        orderId: 'order-123',
        customerId: 'cust-456',
        amount: 99.99,
        timestamp: Date.now()
      }
    }
  ]
});
// Message on Kafka contains: [schema_id: 1][serialized_value]

On the consumer side, you get that schema ID back automatically:

const consumer = kafka.consumer({
  schemaRegistry: registryClient
});

consumer.on('message', (message) => {
  // Client automatically looks up schema ID 1, deserializes
  const event = message.value;
  console.log(event.orderId, event.amount);
});

The beauty is decoupling: producers and consumers don't need to be deployed together. The schema ID is the contract.

Compatibility Modes — BACKWARD, FORWARD, FULL

Schema Registry enforces three compatibility modes. Understanding which one you use prevents breaking changes before they hit production.

BACKWARD Compatibility

Rule: New schemas must be readable by old consumers.

This means you can add optional fields or remove fields, but you cannot change a field type or remove a required field.

// v1 (registered)
const schemaV1 = {
  type: 'record',
  name: 'OrderCreated',
  fields: [
    { name: 'orderId', type: 'string' },
    { name: 'customerId', type: 'string' },
    { name: 'amount', type: 'double' }
  ]
};

// v2 (compatible)
const schemaV2 = {
  type: 'record',
  name: 'OrderCreated',
  fields: [
    { name: 'orderId', type: 'string' },
    { name: 'customerId', type: 'string' },
    { name: 'amount', type: 'double' },
    { name: 'currency', type: 'string', default: 'USD' } // New optional field
  ]
};

// A consumer running the v1 schema will ignore currency, but won't crash

Scenario: Your payment processor is still running the old schema. You ship a new currency field. The old processor sees the message, ignores the currency field it doesn't know about, and processes the order fine. No outage.

FORWARD Compatibility

Rule: Old schemas must be readable by new consumers.

New consumers can understand messages written with old schemas. You can remove optional fields but cannot add required fields or remove required ones.

// v1
const schemaV1 = {
  type: 'record',
  name: 'OrderCreated',
  fields: [
    { name: 'orderId', type: 'string' },
    { name: 'customerId', type: 'string' },
    { name: 'amount', type: 'double' },
    { name: 'shippingAddress', type: 'string' } // Will be removed
  ]
};

// v2 (forward compatible)
const schemaV2 = {
  type: 'record',
  name: 'OrderCreated',
  fields: [
    { name: 'orderId', type: 'string' },
    { name: 'customerId', type: 'string' },
    { name: 'amount', type: 'double' }
    // shippingAddress removed, but messages with it still parse
  ]
};

// A new consumer can read v1 messages (it just won't have shippingAddress)

Scenario: Your shipping service has been storing the address in a separate table for a year. You want to stop including it in the event. Old producers still send it; the new consumer ignores it gracefully.

FULL Compatibility

Rule: Both old and new consumers can read both old and new messages.

This is the safest mode but most restrictive. Only add optional fields with defaults, and avoid removing anything that a new consumer might expect. It's BACKWARD + FORWARD combined.

In my experience, most teams start with BACKWARD (new consumers read old data) and rarely need FULL. FORWARD is rarer unless you're decommissioning fields.

Backward is a Good Default — New Consumers Read Old Data

Why default to BACKWARD? Because it matches real deployments:

• Your producer ships version 2 of the schema (adds a new field).
• An old consumer is still running version 1 (deployed yesterday).
• The old consumer receives a v2 message, ignores the new field, and processes the order correctly.
• You redeploy the consumer at your own pace.

This gives you breathing room. You're not forced into synchronized deployments across 15 services.

From my production experience with payment and ride-hailing platforms, I've never regretted defaulting to BACKWARD. It aligns with the reality that producers and consumers move independently.

Here's how to set it in the registry:

// Set BACKWARD as the global compatibility mode
await registryClient.setCompatibility('BACKWARD');

// Or per subject (topic)
await registryClient.setCompatibilityForSubject('OrderCreated-value', 'BACKWARD');

Version Deliberately — Breaking Changes Go to New Topics or Major Versions

Not all changes are additive. Sometimes you need to break the contract: renaming a field, changing a type, or restructuring the entire event.

Rule: Don't force breaking changes through compatibility modes. Use a new topic or new major version instead.

Let's say you realize customerId should be customer_id and the type changes from string to UUID:

// This violates BACKWARD compatibility
const schemaV2_breaking = {
  type: 'record',
  name: 'OrderCreated',
  fields: [
    { name: 'orderId', type: 'string' },
    { name: 'customer_id', type: 'string', logicalType: 'uuid' }, // Renamed & type changed
    { name: 'amount', type: 'double' }
  ]
};

// Registry rejects it. Registry logs:
// "org.apache.kafka.common.errors.TopicAuthorizationException:
// Could not update schema"

// Instead, create a new topic or subject
const schemaV2_explicit = {
  type: 'record',
  namespace: 'com.myshop.events',
  name: 'OrderCreatedV2', // Explicit major version bump
  fields: [
    { name: 'orderId', type: 'string' },
    { name: 'customer_id', type: 'string', logicalType: 'uuid' },
    { name: 'amount', type: 'double' }
  ]
};

// Register under a different subject
await registryClient.register(schemaV2_explicit); // New registration
// Produce to 'orders-v2' topic or dual-write to both during transition

In a real scenario:

• You plan the breaking change 2 sprints out.
• You create orders-v2 topic with the new schema.
• You dual-write to both topics for 2 weeks (producers emit to both).
• You migrate consumers from orders to orders-v2 one by one.
• After all consumers have migrated, you stop dual-writing and decommission orders.

I've done this with the trading platform I worked on when we redesigned the portfolio event format. It took discipline but zero outages.

Common Mistakes

1. Ignoring the compatibility mode. Teams register schemas without setting a mode, then change types or remove fields and wonder why production breaks. Set your mode explicitly.

2. Over-relying on FULL compatibility. FULL is safe but restrictive. Most systems thrive on BACKWARD. Don't lock yourself into never removing fields.

3. Not testing schema evolution locally. Before registering a schema change, spin up a test registry and consumer, produce a message with the new schema, and verify the old consumer can handle it. Automate this in CI.

4. Neglecting schema versioning in your source code. Keep a schemas/ folder in your repo with versioned Avro files. Your future self will thank you when you need to trace a production incident back to a schema change from 6 months ago.

5. Mixing breaking and non-breaking changes. If you add a field and rename another in one go, you've muddied the git history. Make one type of change per pull request.

Key Takeaways

• Schema Registry centralizes event contracts and embeds schema IDs in messages, eliminating deserializer guesswork.
• BACKWARD, FORWARD, and FULL compatibility modes automatically reject breaking changes at registration time, preventing production outages.
• Default to BACKWARD compatibility: new producers can ship while old consumers catch up on their own schedule.
• For unavoidable breaking changes, create new topics or explicit major versions rather than fighting compatibility rules.
• Treat schemas like code: version them, review them, test them in CI, and document the why behind each change.