Dark launches: ship code, release later, explained simply

Dark launches: ship code, release later

Most teams release features the moment the code is ready—and pay the price when bugs hit production and real users are affected immediately. After 5+ years shipping platforms used by thousands, I've learned that dark launches: ship code, release later is the only way to separate deployment from release. In this approach, you merge code behind a flag that is off by default, enable it for internal users first, roll out by percentage, and maintain a kill switch if metrics drop. This strategy removes the panic of binary releases and gives you the safety net to catch problems before they affect your revenue-generating customers.

Merge behind a flag that is off

The foundation of dark launches is the feature flag—a code path that sits dormant in production until you flip a switch. This means your main branch is always deployable, and features can land days or weeks before they're visible to anyone.

Here's how I structure this in practice. Every significant feature lives behind a flag:

export async function createOrder(input: CreateOrderInput) {
  const order = await db.orders.create(input);
  
  if (featureFlags.isEnabled('new-payment-processor', { userId: input.userId })) {
    // New code path
    await processWithNewPaymentGateway(order);
  } else {
    // Existing code path
    await processWithLegacyPaymentGateway(order);
  }
  
  return order;
}

The key is that both code paths live side by side. Deployment is decoupled from release. You deploy to production confident that the new code won't run unless you explicitly enable it. This removes the Friday deployment dread because even if the feature is buggy, it stays off.

In a NestJS or Express app, I usually inject a feature flag service at the dependency level:

@Injectable()
export class OrderService {
  constructor(
    private db: Database,
    private featureFlags: FeatureFlagService,
    private paymentNew: NewPaymentProcessor,
    private paymentLegacy: LegacyPaymentProcessor,
  ) {}

  async createOrder(input: CreateOrderInput): Promise {
    const order = await this.db.orders.create(input);
    
    const processor = this.featureFlags.isEnabled('new-payment-processor', {
      userId: input.userId,
    })
      ? this.paymentNew
      : this.paymentLegacy;
    
    await processor.process(order);
    return order;
  }
}

This approach keeps the business logic clean and makes it trivial to toggle behavior at runtime without redeploying.

Enable for internal users first

Once code is live but flagged off, the next step is to validate it works in the real production environment—with real data, real load, real network conditions. You do this by enabling the flag for internal users first: your team, beta testers, or employees who use your product.

Internal users are low-risk. They're patient, they'll report issues clearly, and they're familiar enough with the product to spot UX problems. I always spend 24–48 hours running a feature for our team before any customer sees it.

To enable a flag for specific users, I store flag config in a database or cache and check user identity during the flag evaluation:

export interface FeatureFlagContext {
  userId?: string;
  email?: string;
  organizationId?: string;
  isInternal?: boolean;
}

@Injectable()
export class FeatureFlagService {
  async isEnabled(flagName: string, context: FeatureFlagContext): Promise {
    const flag = await this.cache.get(`flag:${flagName}`);
    
    if (!flag) return false;
    
    // Internal users always get new features
    if (context.isInternal) return true;
    
    // Check if user is in allowlist
    if (flag.allowlist?.includes(context.userId)) return true;
    
    // Check percentage rollout
    if (flag.percentage > 0) {
      const hash = this.hashUserId(context.userId);
      return (hash % 100) < flag.percentage;
    }
    
    return false;
  }
  
  private hashUserId(userId: string): number {
    return userId.split('').reduce((acc, char) => acc + char.charCodeAt(0), 0);
  }
}

During this internal phase, you're looking for crashes, slowness, data corruption, or incorrect calculations. I always log what path users are taking (both old and new) so I can spot divergence in metrics.

Real example: at a ride-hailing platform, we built a new matching algorithm to reduce wait times. We enabled it for our internal team for two days. On day one, we noticed that some edge cases (riders in remote areas) weren't getting matched at all. The code had a bug in the fallback logic that didn't show up in testing. We fixed it before any customer was affected.

Roll out by percentage

Once internal validation passes, you gradually release to real customers using percentage-based rollout. This is the canary phase: you enable the flag for a small slice of traffic—say 5%—measure what happens, then increase it to 10%, 25%, 50%, and eventually 100%.

Percentage rollout gives you a feedback loop. With 5% of your traffic, if there's a problem, it affects only a small set of users and your team catches it quickly. By the time you hit 50%, you have high confidence the feature is stable.

I track a few key signals during rollout:

• Error rate: Any spike in 5xx errors or exceptions in the new code path
• Latency: Does the new feature add tail latency that hurts user experience
• Business metrics: If it's a revenue feature, conversion rate and order size. If it's a UX feature, engagement or retention.
• Database impact: Are you causing N+1 queries or long-running transactions that lock tables

Here's how I set up a rollout config in Redis:

interface FlagConfig {
  name: string;
  enabled: boolean;
  percentage: number; // 0-100
  allowlist: string[]; // user IDs
  excludeList: string[]; // user IDs to never show
  updatedAt: number;
  owner: string; // Slack handle or email for on-call
}

// Example in Redis
const flagConfig: FlagConfig = {
  name: 'new-payment-processor',
  enabled: true,
  percentage: 15, // 15% of users
  allowlist: ['user-123', 'user-456'], // internal testers
  excludeList: ['vip-user-789'], // exclude high-value customer
  updatedAt: Date.now(),
  owner: 'payments-team@company.com',
};

await redis.set(
  `flag:${flagConfig.name}`,
  JSON.stringify(flagConfig),
  'EX',
  86400 // 24h TTL
);

During a rollout at an e-commerce platform, we tested a new checkout flow. At 5%, error rate was normal. At 10%, we noticed abandoned cart rate jumped 2%. At 25%, we saw the same pattern. We paused, investigated, and found that the new flow had an extra validation step that wasn't clear to users. We fixed the UX, retested at 5%, and then rolled out to 100% over the next week.

Percentage rollout is not instant. It can take days or a week to reach 100%. But that patience saves you from outages and angry support tickets.

Kill switch if metrics drop

No matter how thorough your testing, production always finds edge cases. That's why a kill switch is non-negotiable. At any point—during internal rollout or at 50% traffic—if metrics degrade, you must be able to disable the flag instantly without a redeploy.

I've had features that looked perfect in staging, passed internal validation, but broke at 20% traffic due to database contention or a race condition that only happens under real load. The kill switch saved us: we disabled the flag in seconds, the error rate returned to baseline, and we had time to investigate without customer impact.

To make this fast, I always:

1. Cache flag config in memory with a short TTL (30-60 seconds)
2. Set up a monitoring alert that triggers if the new code path's error rate exceeds baseline + 5%
3. Document a runbook: which flag to disable, how to disable it, who to notify
4. Test the kill switch manually at least once before shipping

Here's a kill switch check in a monitoring layer:

@Injectable()
export class FeatureFlagMonitor {
  async checkAndDisableOnMetricsDrop(
    flagName: string,
    threshold: { errorRate: number; p99Latency: number }
  ) {
    const metrics = await this.prometheus.query({
      flagName,
      timeRange: '5m',
    });

    if (metrics.errorRate > threshold.errorRate) {
      console.error(`[ALERT] ${flagName} error rate spiked to ${metrics.errorRate}`);
      
      // Auto-disable or notify
      await this.slack.notify({
        channel: '#incidents',
        message: `Feature flag ${flagName} is degrading. Error rate: ${metrics.errorRate}. Review: runbook-link`,
      });
      
      // Optional: auto-kill switch
      // await this.featureFlagService.disable(flagName);
    }

    if (metrics.p99Latency > threshold.p99Latency) {
      console.warn(`[WARN] ${flagName} p99 latency increased to ${metrics.p99Latency}ms`);
    }
  }
}

I use Prometheus and Grafana to surface the key metrics for each feature flag in a central dashboard. My team knows to check it every hour during a rollout. If something looks odd, we disable the flag, investigate offline, and re-enable once we've fixed it.

Common mistakes

Mistake 1: Forgetting to remove the flag after 100% rollout. Dead code and old flags clutter your codebase. Set a deadline (e.g., 2 weeks after 100%) to remove the flag and clean up. I add a comment with the flag name and removal date so it's searchable.

Mistake 2: Feature flags for every tiny change. Flags add cognitive overhead. Use them for significant logic changes, especially in critical paths like payment or search ranking. Don't flag minor UI tweaks or small refactors.

Mistake 3: Ignoring the allowlist performance hit. Checking if a user is in an allowlist array on every request can slow things down if your list is large. Use a set or cache it. At scale, lookups matter.

Mistake 4: No observability during rollout. You deploy a flag at 5% and assume it's fine because you didn't hear complaints. But complaints are lagging indicators. Use structured logging and distributed tracing to see what the new code is actually doing. Log every flag evaluation with context so you can debug later.

// Example: structured logging for flag evaluation
logger.debug('flag_evaluated', {
  flagName: 'new-payment-processor',
  userId: context.userId,
  enabled: true,
  percentage: 15,
  matched: true,
  timestamp: Date.now(),
  duration_ms: 2, // Time to evaluate the flag
});

Mistake 5: No rollback plan. If you disable a flag at 50% traffic, what happens to requests mid-flight? Make sure both code paths handle failures gracefully. If the new processor is half-deployed and you kill it, orders should still complete using the legacy path.

Key takeaways

• Dark launches decouple deployment from release: merge behind a flag that is off, then enable it when you're ready
• Always validate on internal users first to catch bugs in production without affecting customers
• Use percentage rollout to gain confidence gradually—5% to 10% to 25% to 100% over days or a week
• Monitor error rate, latency, and business metrics during every rollout phase
• Maintain a kill switch and test it: if metrics drop, disable the flag instantly without a redeploy
• Remove the flag and clean up dead code once you hit 100% and are confident the feature is stable
• Log flag evaluations and outcomes so you can debug issues even after they've been disabled