Building a workflow automation engine

I built my first workflow automation system for a CRM platform four years ago, and the experience taught me that automation isn't just about stringing actions together—it's about orchestrating them reliably across unpredictable timing and failure scenarios. Building a workflow automation engine requires understanding how to decompose business logic into reusable, testable components that can adapt to changing requirements without rewriting the entire system. In this article, I'll walk you through the core patterns that make workflow engines reliable and maintainable.

Workflow anatomy: The building blocks

Every workflow automation engine rests on the same fundamental structure. A workflow contains a trigger (the event that starts execution), conditions (the gates that decide whether to proceed), actions (what actually happens), and delays (temporal spacing between events). Understanding this anatomy is crucial because it shapes every architectural decision downstream.

Let's use a concrete CRM example: when a lead is created from Facebook, we want to check if the lead's company size meets our criteria, assign them to a sales rep, tag them, send a welcome email, wait 2 hours, and then send a follow-up SMS. That entire sequence is a workflow with one trigger, multiple conditions, several actions, and a delay.

The key insight is that triggers, conditions, actions, and delays are all first-class entities in your system. You store them, serialize them, and execute them independently. This separation allows your engine to handle retries, scheduling, and monitoring without rebuilding logic every time a workflow changes.

typescript

interface WorkflowDefinition {
  id: string;
  name: string;
  trigger: TriggerConfig;
  nodes: WorkflowNode[];
  enabled: boolean;
}

interface WorkflowNode {
  id: string;
  type: 'condition' | 'action' | 'delay';
  config: Record<string, unknown>;
  nextNodeId?: string;
  fallbackNodeId?: string;
}

interface TriggerConfig {
  event: string;
  filters?: Record<string, unknown>;
}

In production, I store workflow definitions as JSON in PostgreSQL. When an event fires (like "lead.created"), the system looks up all workflows subscribed to that event, evaluates them, and kicks off execution for each matching workflow.

Command pattern for actions

The command pattern is your most valuable ally when building actions. Each action—assign, tag, notify, create task—is a self-contained command that knows how to execute and potentially rollback.

Why the command pattern? Because workflows live for hours or days. An action that executes at 2 AM might fail temporarily, need retry logic, or require manual intervention. Commands are serializable, versioned, and auditable. You can store them in a queue, replay them, and build comprehensive logs.

typescript

interface Command<T = any> {
  type: string;
  id: string;
  payload: T;
  timestamp: number;
  version: number;
}

interface CommandHandler<T> {
  canHandle(command: Command): boolean;
  execute(command: Command<T>, context: ExecutionContext): Promise<CommandResult>;
  rollback?(command: Command<T>, context: ExecutionContext): Promise<void>;
}

class AssignCommandHandler implements CommandHandler<AssignPayload> {
  canHandle(command: Command): boolean {
    return command.type === 'assign_lead';
  }

  async execute(
    command: Command<AssignPayload>,
    context: ExecutionContext
  ): Promise<CommandResult> {
    const { leadId, userId } = command.payload;
    
    const lead = await this.leadRepo.findById(leadId);
    if (!lead) {
      return { success: false, error: 'Lead not found' };
    }

    const assignedLead = await this.leadRepo.update(leadId, { assignedTo: userId });
    
    await this.eventBus.publish('lead.assigned', {
      leadId,
      assignedTo: userId,
      timestamp: Date.now()
    });

    return { success: true, data: assignedLead };
  }

  async rollback(command: Command<AssignPayload>, context: ExecutionContext): Promise<void> {
    const { leadId } = command.payload;
    await this.leadRepo.update(leadId, { assignedTo: null });
  }
}

The handler registry maintains a map of command types to handlers. When a workflow node says "execute assign_lead", the registry finds the right handler, executes it, and records the result. This decoupling means you can add new action types without touching the workflow engine itself.

State machine for workflow runs

A workflow run is the execution instance of a workflow definition against a specific entity (like a lead). Each run progresses through states: pending, running, paused, completed, or failed. A state machine ensures runs can only transition in valid ways and lets you enforce invariants.

I've learned the hard way that without a state machine, you end up with runs stuck in ambiguous states—partially executed, unclear whether they completed or not, impossible to retry safely. A state machine makes the entire system auditable and debuggable.

typescript

enum WorkflowRunState {
  PENDING = 'pending',
  RUNNING = 'running',
  PAUSED = 'paused',
  COMPLETED = 'completed',
  FAILED = 'failed',
  CANCELLED = 'cancelled'
}

interface WorkflowRun {
  id: string;
  workflowId: string;
  entityId: string;
  entityType: string;
  state: WorkflowRunState;
  currentNodeId?: string;
  executedCommands: CommandExecution[];
  failureReason?: string;
  createdAt: number;
  updatedAt: number;
  completedAt?: number;
}

interface CommandExecution {
  commandId: string;
  command: Command;
  nodeId: string;
  state: 'pending' | 'executing' | 'success' | 'failed' | 'rolled_back';
  result?: CommandResult;
  error?: string;
  executedAt?: number;
  retriesRemaining: number;
}

class WorkflowRunStateMachine {
  canTransition(from: WorkflowRunState, to: WorkflowRunState): boolean {
    const validTransitions: Record<WorkflowRunState, WorkflowRunState[]> = {
      [WorkflowRunState.PENDING]: [WorkflowRunState.RUNNING, WorkflowRunState.CANCELLED],
      [WorkflowRunState.RUNNING]: [WorkflowRunState.PAUSED, WorkflowRunState.COMPLETED, WorkflowRunState.FAILED],
      [WorkflowRunState.PAUSED]: [WorkflowRunState.RUNNING, WorkflowRunState.CANCELLED],
      [WorkflowRunState.COMPLETED]: [],
      [WorkflowRunState.FAILED]: [WorkflowRunState.RUNNING],
      [WorkflowRunState.CANCELLED]: []
    };

    return validTransitions[from]?.includes(to) ?? false;
  }

  async transition(run: WorkflowRun, to: WorkflowRunState): Promise<WorkflowRun> {
    if (!this.canTransition(run.state, to)) {
      throw new Error(`Cannot transition from ${run.state} to ${to}`);
    }

    const updated = { ...run, state: to, updatedAt: Date.now() };
    if (to === WorkflowRunState.COMPLETED || to === WorkflowRunState.FAILED) {
      updated.completedAt = Date.now();
    }
    return updated;
  }
}

Store the entire execution history in the database. When debugging a workflow, you want to see exactly which commands executed, in what order, with what results. This becomes invaluable when investigating why a lead was assigned twice or a notification wasn't sent.

Queue-driven execution with delays and retries

The final piece is the execution engine, which I always build on top of a job queue. Whether you use Bull (Redis-backed), AWS SQS, or Kafka depends on your scale, but the pattern is the same: workflow steps become jobs that are enqueued, processed asynchronously, and retried on failure.

Delays are handled naturally by the queue system. Instead of blocking a thread for 2 hours, you enqueue a delayed job that processes at the scheduled time. This lets you run workflows across thousands of entities without consuming server resources.

typescript

interface WorkflowExecutionJob {
  workflowRunId: string;
  nodeId: string;
  command: Command;
  attempt: number;
  maxRetries: number;
  delayMs?: number;
}

class WorkflowExecutor {
  async executeNode(
    run: WorkflowRun,
    nodeId: string,
    queue: JobQueue
  ): Promise<void> {
    const workflow = await this.workflowRepo.findById(run.workflowId);
    const node = workflow.nodes.find(n => n.id === nodeId);

    if (!node) {
      throw new Error(`Node ${nodeId} not found`);
    }

    if (node.type === 'delay') {
      const delayConfig = node.config as { delayMs: number };
      const job: WorkflowExecutionJob = {
        workflowRunId: run.id,
        nodeId,
        command: { type: 'noop', id: `delay-${Date.now()}`, payload: {}, timestamp: Date.now(), version: 1 },
        attempt: 1,
        maxRetries: 0,
        delayMs: delayConfig.delayMs
      };
      await queue.scheduleJob(job, delayConfig.delayMs);
      return;
    }

    if (node.type === 'condition') {
      const conditionConfig = node.config as { expression: string };
      const conditionMet = await this.evaluateCondition(conditionConfig.expression, run);
      
      const nextNodeId = conditionMet ? node.nextNodeId : node.fallbackNodeId;
      if (nextNodeId) {
        await this.executeNode(run, nextNodeId, queue);
      }
      return;
    }

    if (node.type === 'action') {
      const command = this.buildCommand(node, run);
      const job: WorkflowExecutionJob = {
        workflowRunId: run.id,
        nodeId,
        command,
        attempt: 1,
        maxRetries: 3
      };
      await queue.enqueueJob(job);
    }
  }

  async processJob(job: WorkflowExecutionJob): Promise<void> {
    const run = await this.runRepo.findById(job.workflowRunId);
    const handler = this.handlerRegistry.get(job.command.type);

    if (!handler) {
      throw new Error(`No handler for command type: ${job.command.type}`);
    }

    try {
      const result = await handler.execute(job.command, { run });
      
      await this.runRepo.recordCommandExecution(run.id, {
        commandId: job.command.id,
        command: job.command,
        nodeId: job.nodeId,
        state: 'success',
        result,
        executedAt: Date.now()
      });

      const workflow = await this.workflowRepo.findById(run.workflowId);
      const node = workflow.nodes.find(n => n.id === job.nodeId);
      
      if (node?.nextNodeId) {
        await this.executeNode(run, node.nextNodeId, this.queue);
      } else {
        await this.runRepo.updateState(run.id, WorkflowRunState.COMPLETED);
      }
    } catch (error) {
      if (job.attempt < job.maxRetries) {
        await this.queue.enqueueJob({
          ...job,
          attempt: job.attempt + 1,
          delayMs: Math.pow(2, job.attempt) * 1000 // exponential backoff
        });
      } else {
        await this.runRepo.recordCommandExecution(run.id, {
          commandId: job.command.id,
          command: job.command,
          nodeId: job.nodeId,
          state: 'failed',
          error: error.message
        });

        await this.runRepo.updateState(run.id, WorkflowRunState.FAILED, error.message);
        await this.alerting.notifyAdmins(`Workflow run ${run.id} failed`, error);
      }
    }
  }
}

In production, I use exponential backoff for retries. A transient database connection error gets retried with increasing delays, giving dependent services time to recover. Permanent errors are caught quickly and escalated to monitoring.

Common mistakes

Storing workflow runs in memory: Workflows can outlive your application. Always persist runs to a database so you can resume execution after deployments or crashes.
Skipping the state machine: Without state guards, runs can be executed twice, partially executed, or left in inconsistent states. Always enforce valid transitions.
Making actions non-idempotent: A retry must produce the same outcome as the first attempt. If assigning a lead twice creates two assignments, your rollback and retry logic breaks. Use upsert patterns or uniqueness constraints.
Ignoring command serialization: If you can't serialize a command to JSON and deserialize it weeks later, you can't replay workflows or handle long-running delays. Keep payloads simple and versioned.
Blocking the main thread on delays: Using sleep() in your application code will exhaust your worker pool. Always offload delays to the job queue.

Key takeaways

Decompose workflows into triggers, conditions, actions, and delays—each is independently evaluable and retryable.
Use the command pattern to make actions serializable, auditable, and rollbackable.
Implement a state machine to ensure workflow runs progress through valid states without ambiguity.
Rely on a job queue for all execution, delays, and retries—never block threads or assume synchronous timing.
Store complete execution history so you can debug, replay, and monitor every workflow run in production.

Building a workflow automation engine in Design Patterns