Leader election: Design Patterns lessons from production

Imagine 2 AM on a Tuesday: your scheduled job that reconciles accounts just ran on two instances simultaneously, creating duplicate charges for hundreds of customers. Leader election: Design Patterns lessons from production is the pattern that prevents exactly this scenario. When you have multiple instances of the same service running, you need a way to ensure only one performs critical work at any given time.

Scheduled jobs and single consumers

The most common place we encounter leader election problems is with scheduled jobs. You deploy your service across three availability zones for redundancy, but your nightly billing job runs on all three simultaneously. You've just tripled your costs and sent your customer support team into crisis mode.

Similarly, when consuming from a Kafka topic or SQS queue, you might have multiple instances of your consumer service. Without coordination, each instance will process the same messages. Most queue systems handle this at the protocol level—SQS gives you only one consumer per message, Kafka uses consumer groups—but you still need to think about this at the application layer when you have custom solutions or when you're building cross-service coordination.

The naive solution is to hardcode a single instance: give it a specific hostname, IP, or environment variable. This works until you need to scale or that instance crashes. Now your job never runs. We need something more robust.

Consider a notification digest service: every hour, we send users a summary of their activity. We run three instances for availability. Each instance needs to know: should I be the one sending digests right now? If I crash, can another instance take over within a reasonable time?

// Naive approach - don't do this
const isLeader = process.env.HOSTNAME === 'notifications-prod-1';

if (isLeader) {
  await sendDigests();
}

This fails catastrophically when that one instance goes down.

Distributed locks with leases

The production-ready solution uses distributed locks with time-based leases. One instance acquires a lock, holds it for a defined period, and renews it periodically. If an instance crashes or becomes unresponsive, its lease expires and another instance can acquire the lock.

We typically implement this using Redis, PostgreSQL, or a dedicated coordination service. The key insight is the lease duration: it needs to be long enough that network hiccups don't cause leader flapping, but short enough that failover is acceptable.

Here's a practical Redis-based implementation:

interface LeaderElectionOptions {
  lockKey: string;
  leaseDurationMs: number;
  renewIntervalMs: number;
  instanceId: string;
}

class LeaderElection {
  private redis: RedisClient;
  private leaseToken: string;
  private isLeader = false;
  private options: LeaderElectionOptions;

  constructor(redis: RedisClient, options: LeaderElectionOptions) {
    this.redis = redis;
    this.options = options;
    this.leaseToken = `${options.instanceId}-${Date.now()}`;
  }

  async start(): Promise {
    this.attemptLeadership();
    setInterval(() => this.attemptLeadership(), this.options.renewIntervalMs);
  }

  private async attemptLeadership(): Promise {
    try {
      const acquired = await this.redis.set(
        this.options.lockKey,
        this.leaseToken,
        'EX',
        Math.ceil(this.options.leaseDurationMs / 1000),
        'NX' // Only set if not exists
      );

      if (acquired) {
        if (!this.isLeader) {
          this.isLeader = true;
          console.log('Became leader');
          this.onLeadershipGained();
        }
      } else {
        if (this.isLeader) {
          this.isLeader = false;
          console.log('Lost leadership');
          this.onLeadershipLost();
        }
      }
    } catch (error) {
      console.error('Leadership election error:', error);
      this.isLeader = false;
    }
  }

  private onLeadershipGained(): void {
    // Start leader-only work
  }

  private onLeadershipLost(): void {
    // Stop leader-only work
  }

  async executeIfLeader(fn: () => Promise): Promise {
    if (!this.isLeader) return null;
    return fn();
  }
}

// Usage
const election = new LeaderElection(redis, {
  lockKey: 'digest-sender-leader',
  leaseDurationMs: 30000, // 30 second lease
  renewIntervalMs: 10000, // Try to renew every 10 seconds
  instanceId: process.env.HOSTNAME || 'instance-' + Math.random()
});

await election.start();

// In your scheduled job
await election.executeIfLeader(async () => {
  await sendDigests();
});

Notice the lease expiration strategy. A 30-second lease means if a leader crashes, another instance can take over within 30 seconds. We try to renew every 10 seconds, giving us three renewal attempts before expiration. This balance prevents flapping while keeping failover reasonable.

With PostgreSQL, you'd use an advisory lock or a dedicated table with a timestamp and instance identifier. The principle remains the same: atomic compare-and-set with expiration.

Kubernetes lease objects

If you're running on Kubernetes, there's a native abstraction: Lease objects. Kubernetes provides coordination.k8s.io/v1 leases, designed exactly for this use case. Instead of managing your own distributed lock, you declare a Lease resource and let Kubernetes handle the coordination.

Here's what a Lease looks like:

interface KubernetesLease {
  apiVersion: 'coordination.k8s.io/v1';
  kind: 'Lease';
  metadata: {
    name: string;
    namespace: string;
  };
  spec: {
    holderIdentity: string; // Pod name or instance ID
    leaseDurationSeconds: number;
    acquireTime?: MicroTime;
    renewTime?: MicroTime;
  };
}

Libraries like client-go (Go) and kubernetes-client (Node.js) provide leader election helpers that manage Lease objects for you:

// Pseudo-code using a hypothetical Kubernetes client library
const leaderElection = new KubernetesLeaderElector({
  leaseName: 'digest-sender',
  namespace: 'production',
  identity: process.env.POD_NAME,
  leaseDurationSeconds: 30
});

leaderElection.on('leader', async () => {
  console.log('Became leader, starting digest sender');
  await digestSenderLoop();
});

leaderElection.on('follower', () => {
  console.log('Lost leadership');
  // Cleanup happens automatically
});

await leaderElection.run();

The advantage: Kubernetes etcd provides strong consistency guarantees. You're not relying on Redis TTLs or database polling. The control plane manages leader transitions consistently across your cluster.

Use Kubernetes Leases when you're fully on Kubernetes and can accept the overhead of API calls. For external systems or when you need lower latency, Redis or PostgreSQL are faster.

Handle leader changes gracefully

The worst bugs happen during leadership transitions. A job is halfway through when the leader crashes. Another instance takes over and starts the job again. You've now double-processed critical data.

Design your leader-only work to be idempotent or have clear boundaries. Here's a pattern that works well:

interface WorkItem {
  id: string;
  status: 'pending' | 'processing' | 'completed';
  processedAt?: Date;
  processedBy?: string;
}

class SafeLeaderWork {
  private instanceId: string;
  private workTimeout = 60000; // 1 minute per item

  async processWorkItem(item: WorkItem): Promise {
    // Check if another instance already processed this
    if (item.status === 'completed') {
      return;
    }

    // Atomic update: mark as processing by us
    const updated = await db.items.update(
      { id: item.id, status: 'pending' },
      {
        status: 'processing',
        processedBy: this.instanceId,
        processingStartedAt: new Date()
      }
    );

    if (!updated) {
      // Another instance claimed this work
      return;
    }

    try {
      await this.doWork(item);
      await db.items.update(
        { id: item.id },
        { status: 'completed', processedAt: new Date() }
      );
    } catch (error) {
      // Mark as failed but not completed; another instance can retry
      await db.items.update(
        { id: item.id },
        { status: 'pending', processedBy: null }
      );
      throw error;
    }
  }

  private async doWork(item: WorkItem): Promise {
    // Actual work here
  }
}

The strategy: use database state to track work progress. When leadership changes, the new leader sees pending items and can resume. Items in "processing" state longer than the timeout are assumed to have failed and are retried.

Another approach is to use a distributed transaction coordinator, but that adds complexity. Keep it simple: write state to a consistent store, have leaders respect that state.

Common mistakes

• Lease duration too short: Your network has a blip, every instance thinks it lost leadership, all three try to acquire the lock, none succeeds reliably. Set lease duration to at least 2-3x your expected network latency.
• Not handling leadership loss: Your leader crashes while processing. You never clean up partial state because you assumed you were always the leader. Always implement a onLeadershipLost handler that stops work and cleans up.
• No monitoring on leader transitions: You have a bug that causes leadership to flip every 10 seconds. Without alerting on leadership changes, you won't catch this until your data is corrupted. Log every transition and alert on excessive churn.
• Trusting timestamps across instances: You log "became leader at 14:23:45" from instance A and "became leader at 14:23:47" from instance B. Network clocks are never perfectly synchronized. Use server-assigned timestamps from your lock storage, not client clocks.
• Forgetting about network partitions: If your Redis connection times out, you're not the leader anymore. Don't assume you can keep doing work just because you haven't been explicitly removed. Always check before critical operations.

Key takeaways

• Leader election prevents duplicate work in distributed systems; without it, multiple instances will race to process the same jobs, messages, or data.
• Distributed locks with leases work by granting one instance temporary ownership; leases must be long enough to avoid flapping but short enough for acceptable failover.
• Redis-based election is fast and simple; Kubernetes Leases provide consistency if you're running on Kubernetes; PostgreSQL works when you want to keep dependencies minimal.
• Always make leader-only work idempotent or track state in a consistent store so new leaders can resume safely after crashes.
• Monitor leadership changes, handle transitions gracefully, and never assume you're still the leader if network conditions degrade.