Cron jobs with multiple instances for NestJS developers

The multiple-instance problem

You deploy your NestJS service to three pods in Kubernetes. Your scheduled cron job runs every day at 2 AM to send daily report emails. Sounds simple—until you check your inbox and realize you've received the same report three times. Cron jobs with multiple instances is one of those distributed systems problems that catches teams off guard. When you scale horizontally, every running instance executes the same schedule independently, and unless you add coordination, you get duplicate work.

This article walks through practical patterns I've used in production: Redis-backed distributed locks, BullMQ repeatable jobs, dedicated worker deployments, and idempotent design. By the end, you'll know which approach fits your constraints.

Why duplication happens in scaled environments

When you use NestJS's @Cron() decorator or ScheduleModule, each instance independently tracks time and fires the job. There's no built-in coordination. In development with a single pod, this is invisible. Move to production with three replicas, and your email provider logs three identical sends for the same report. Database imports run three times. Webhook notifications fire tripled. The root cause is the same across all scenarios: no distributed coordination.

The naive fix—"just skip the job on instances 2 and 3"—doesn't scale. How do you know which instance is primary? What happens if the primary crashes mid-execution? You need a mechanism that works across network boundaries, handles failures gracefully, and doesn't require hardcoding instance IDs.

Pattern 1: Distributed locks in Redis

A Redis-backed lock is the simplest approach for existing cron jobs. Before executing, acquire a lock. If you can't acquire it, another instance already claimed the job. You release the lock after completion.

import { Injectable } from '@nestjs/common';
import { Cron } from '@nestjs/schedule';
import { RedisService } from './redis.service';
import { Logger } from '@nestjs/common';

@Injectable()
export class ReportJobService {
  private readonly logger = new Logger(ReportJobService.name);

  constructor(private readonly redis: RedisService) {}

  @Cron('0 2 * * *')
  async sendDailyReport() {
    const lockKey = 'cron:daily-report-lock';
    const lockValue = Date.now().toString();
    const lockTtl = 300; // 5 minutes

    // Try to acquire lock
    const acquired = await this.redis.set(
      lockKey,
      lockValue,
      'PX',
      lockTtl * 1000,
      'NX',
    );

    if (!acquired) {
      this.logger.log('Lock already held, skipping daily report');
      return;
    }

    try {
      this.logger.log('Acquired lock, executing daily report');
      // Generate and send reports
      await this.generateReports();
    } finally {
      // Safely delete lock only if we still own it
      await this.redis.eval(
        `if redis.call("get", KEYS[1]) == ARGV[1] then
          return redis.call("del", KEYS[1])
        else
          return 0
        end`,
        1,
        lockKey,
        lockValue,
      );
    }
  }

  private async generateReports() {
    // Your report generation logic
  }
}

Key details in this pattern:

• NX means "only set if the key doesn't exist," so only one instance succeeds.
• PX sets a millisecond timeout. If an instance crashes, the lock expires and another can retry.
• The lockValue is a timestamp. We use Lua scripting on release to ensure only the owner deletes it (preventing accidental cleanup by a slower instance).
• The TTL should be longer than your job's expected runtime, plus a safety buffer.

Pros: minimal code changes, works with existing @Cron decorators, low overhead. Cons: requires operational knowledge of lock expiry times; if a job runs longer than expected, another instance may claim the lock and execute in parallel.

Pattern 2: BullMQ repeatable jobs

For more sophisticated scheduling, BullMQ is a battle-tested queue library that handles distribution natively. Instead of @Cron, you define repeatable jobs stored in Redis. BullMQ ensures only one worker processes each job occurrence, even across multiple instances.

import { Injectable, OnModuleInit } from '@nestjs/common';
import { Queue } from 'bullmq';
import { InjectQueue } from '@nestjs/bullmq';
import { Logger } from '@nestjs/common';

@Injectable()
export class ReportJobService implements OnModuleInit {
  private readonly logger = new Logger(ReportJobService.name);

  constructor(@InjectQueue('reports') private reportQueue: Queue) {}

  async onModuleInit() {
    // Register repeatable job: every day at 2 AM
    await this.reportQueue.add(
      'daily-report',
      {},
      {
        repeat: {
          pattern: '0 2 * * *', // cron expression
        },
      },
    );
  }
}

@Injectable()
export class ReportJobConsumer {
  private readonly logger = new Logger(ReportJobConsumer.name);

  @Process('daily-report')
  async handleDailyReport() {
    this.logger.log('Processing daily report');
    // Generate and send reports
    await this.generateReports();
  }

  private async generateReports() {
    // Your logic
  }
}

In your module setup:

import { BullModule } from '@nestjs/bullmq';

@Module({
  imports: [
    BullModule.registerQueue({
      name: 'reports',
      connection: { host: 'redis', port: 6379 },
    }),
  ],
  providers: [ReportJobService, ReportJobConsumer],
})
export class ReportModule {}

BullMQ automatically stores repeating job schedules in Redis and distributes them across workers. Each instance becomes a worker; only one claims a job per occurrence. If a worker dies, the job remains in Redis until another worker picks it up.

Pros: clean separation of concern, built-in job retry, monitoring dashboard available (Bull Arena). Cons: adds a queue library dependency and operational complexity (you need to monitor queue health).

Pattern 3: Dedicated worker deployment

The heavyweight but safest approach: run cron jobs on a single dedicated pod or a separate deployment entirely. Your main API fleet handles requests; a separate "worker" pod handles scheduled tasks.

// Pseudo-code for deployment split
// main-app.module.ts
@Module({
  imports: [
    ScheduleModule.forRoot({
      disable: process.env.DISABLE_CRONS === 'true',
    }),
  ],
})
export class MainAppModule {}

// In Kubernetes manifests
// deployment-api.yaml: replicas: 3, env DISABLE_CRONS=true
// deployment-worker.yaml: replicas: 1, env DISABLE_CRONS=false (omitted)

By setting an environment variable, the API pods skip cron registration entirely. Only the worker pod enables scheduling. If that pod crashes, Kubernetes restarts it. If you need HA, run two worker pods but add Redis locking to ensure they don't duplicate.

Pros: simple mental model, zero coordination overhead, easy to scale workers independently. Cons: requires infrastructure changes, adds operational burden (monitor the worker separately).

Pattern 4: Design jobs to be idempotent

Regardless of which locking pattern you choose, design every cron job to be idempotent. This means running it twice produces the same result as running it once. If your lock fails and two instances execute, idempotency prevents corruption.

@Cron('0 2 * * *')
async reconcileDailyMetrics() {
  const today = new Date().toISOString().split('T')[0];
  const metricsKey = `metrics:${today}`;

  // DON'T DO THIS (not idempotent)
  const currentValue = await this.redis.get(metricsKey);
  const newValue = (parseInt(currentValue) || 0) + 100;
  await this.redis.set(metricsKey, newValue);

  // DO THIS (idempotent)
  // Compute the final state from raw data, not incremental updates
  const rawEvents = await this.db.query(`
    SELECT COUNT(*) as count FROM events 
    WHERE DATE(created_at) = ?
  `, [today]);
  
  const finalMetrics = rawEvents.count * 100;
  await this.redis.set(metricsKey, finalMetrics);
}

In the first example, if two instances run, you increment twice. In the second, both instances compute the same result from source data, so duplication doesn't matter. This principle applies to database updates, file writes, and API calls. If you're sending emails, deduplicate by record ID and send timestamp; if an email was sent in the last hour for that ID, skip it.

Common mistakes

• Lock TTL too short: A 30-second TTL on a 10-minute report job causes another instance to steal the lock mid-execution, running it twice. Set TTL to job_duration + buffer (at least 2–3x expected runtime).
• Forgetting to release the lock: If an exception kills the process before lock cleanup, the lock hangs until TTL expires. Use try-finally to guarantee release.
• Mixing patterns: Don't use Redis locks on a job that's also defined as a BullMQ repeatable job. One mechanism handles it; adding another creates confusion and potential races.
• Assuming instance count: Hardcoding "instance 0 runs crons, instances 1–2 don't" breaks when you scale to 5 pods. Use dynamic coordination (locks, queues, env vars).
• No monitoring: A silent lock failure (e.g., Redis down) means your job never runs, and you won't know until a report deadline passes. Log every lock acquisition and failure.

Key takeaways

• Multiple instances running identical cron jobs cause duplicate executions unless you add coordination.
• Redis distributed locks are the lightest solution for wrapping existing @Cron decorators; use NX and appropriate TTL.
• BullMQ repeatable jobs handle distribution natively and scale cleanly; consider it for new projects or when you want job history and retries.
• A dedicated worker deployment (single pod) is simplest operationally but requires infrastructure separation.
• Idempotent job design is your safety net: compute final state from source data, not incremental updates, so duplication is harmless.
• Always monitor lock acquisition and failures; silent degradation is worse than an alert.
• Choose one pattern and stick with it; mixing locks and queues creates race conditions.