Execution Lifecycle Manager

Execution Lifecycle Manager

Centralized state management for running DAG executions with graceful shutdown patterns.

When to Use

✅ Use for:

• Implementing execution start/stop/pause/resume controls
• Graceful process termination (SIGTERM → SIGKILL)
• Tracking active executions across the system
• Cleaning up orphaned processes
• Implementing abort handlers with cost tracking

❌ NOT for:

• Cost estimation or pricing calculations (use cost-accrual-tracker)
• Building or modifying DAG structures
• Skill matching or selection
• Process spawning (use the executor directly)

Core Patterns

1. Graceful Shutdown Pattern

Always use SIGTERM first, then escalate to SIGKILL:

// CORRECT: Two-phase shutdown
const GRACEFUL_TIMEOUT_MS = 2000;

async function terminateProcess(proc: ChildProcess): Promise<void> {
  proc.kill('SIGTERM');

  const forceKillTimer = setTimeout(() => {
    if (!proc.killed) {
      proc.kill('SIGKILL');
    }
  }, GRACEFUL_TIMEOUT_MS);

  await waitForExit(proc);
  clearTimeout(forceKillTimer);
}

2. AbortController Pattern

Use AbortController for cancellation propagation:

// Parent (DAGExecutor)
const abortController = new AbortController();

// Pass signal to child executors
await executor.execute({
  ...request,
  abortSignal: abortController.signal,
});

// To abort all children:
abortController.abort();

3. Execution Registry Pattern

Track active executions for monitoring and cleanup:

interface ActiveExecution {
  executionId: string;
  abortController: AbortController;
  status: 'running' | 'stopping' | 'stopped' | 'completed' | 'failed';
  startedAt: number;
  stoppedAt?: number;
}

class ExecutionManager {
  private executions: Map<string, ActiveExecution> = new Map();

  create(id: string): ActiveExecution { /* ... */ }
  stop(id: string, reason: string): Promise<StopResult> { /* ... */ }
  listActive(): ActiveExecution[] { /* ... */ }
}

Anti-Patterns

SIGKILL Without SIGTERM

Novice thinking: "Just kill it immediately"

Reality: SIGKILL doesn't allow cleanup. Processes can't:

• Flush buffers to disk
• Close network connections gracefully
• Release locks
• Save partial progress

Timeline:

• Always: SIGTERM allows graceful shutdown
• If stuck after 2-5s: Then use SIGKILL

Correct approach: Always SIGTERM first, SIGKILL as fallback.

Missing Abort Signal Propagation

Novice thinking: "Just track the top-level execution"

Reality: Without signal propagation, child processes become orphans:

• Parent dies, children keep running
• Resources leak
• Costs continue accruing

Correct approach: Pass AbortSignal through entire execution tree.

Synchronous Stop Handler

Novice thinking: "Stop should return immediately"

Reality: Stopping is async - processes need time to terminate:

• Network requests need to timeout
• File handles need to close
• Costs need final calculation

Correct approach: Return Promise with final state after cleanup completes.

State Machine

         ┌──────────┐
         │  idle    │
         └────┬─────┘
              │ start()
              ▼
         ┌──────────┐
    ┌───►│ running  │◄───┐
    │    └────┬─────┘    │
    │         │          │ resume()
    │         │ pause()  │
    │         ▼          │
    │    ┌──────────┐    │
    │    │ paused   │────┘
    │    └────┬─────┘
    │         │ stop()
    │         ▼
    │    ┌──────────┐
    └────│ stopping │ (transitional - 2-10s)
         └────┬─────┘
              │
     ┌────────┴────────┐
     ▼                 ▼
┌──────────┐     ┌──────────┐
│ stopped  │     │  failed  │
└──────────┘     └──────────┘

API Design

Stop Endpoint Response

interface StopResponse {
  status: 'stopped';
  executionId: string;
  reason: string;  // 'user_abort' | 'timeout' | 'error'
  finalCostUsd: number;
  stoppedAt: number;
  summary: {
    nodesCompleted: number;
    nodesFailed: number;
    nodesTotal: number;
    durationMs: number;
  };
}

Cleanup on Server Shutdown

// In server.ts
process.on('SIGINT', async () => {
  console.log('Shutting down...');

  // Stop all active executions gracefully
  const active = executionManager.listActive();
  await Promise.all(
    active.map(e => executionManager.stop(e.executionId, 'server_shutdown'))
  );

  server.close();
});

Integration Points

References

See /references/process-signals.md for Unix signal handling details.

Unix Process Signals Reference

Signal Types for Execution Control

Best Practices

1. Always Try SIGTERM First

// Give process time to clean up
proc.kill('SIGTERM');

// Wait for graceful exit
await Promise.race([
  waitForExit(proc),
  sleep(TIMEOUT_MS),
]);

// Only force kill if still running
if (!proc.killed) {
  proc.kill('SIGKILL');
}

2. Handle Signals in Your Process

// In spawned process
process.on('SIGTERM', async () => {
  console.log('Received SIGTERM, cleaning up...');

  // Flush pending writes
  await flushBuffers();

  // Close connections
  await closeConnections();

  // Save progress
  await saveCheckpoint();

  process.exit(0);
});

3. Child Process Groups

When spawning with detached: false (default), children die with parent.

For independent children, use detached: true and track PIDs manually:

const proc = spawn(cmd, args, { detached: true });
proc.unref(); // Parent can exit without waiting

// To kill later:
process.kill(-proc.pid, 'SIGTERM'); // Negative PID = process group

Node.js Specifics

spawn() is Preferred

Always use spawn() for execution control:

• Signals go directly to the process
• No shell injection vulnerabilities
• Streams available for real-time output

// CORRECT: Use spawn with shell: false
const proc = spawn('claude', ['-p', prompt], {
  shell: false,  // Direct execution, no shell
});

AbortSignal Integration

const controller = new AbortController();

const proc = spawn('claude', ['-p', prompt], {
  signal: controller.signal, // Node 15+
});

// Later:
controller.abort(); // Sends SIGTERM

Timeout Recommendations

Debugging Orphaned Processes

# Find processes by name
ps aux | grep claude

# Find processes by parent PID
pstree -p <parent-pid>

# Kill process group
kill -TERM -<pgid>

# List all node processes
pgrep -a node

Common Issues

1. Process Doesn't Die on SIGTERM

Cause: Process ignores or doesn't handle SIGTERM Solution: Use SIGKILL after timeout

2. Zombie Processes

Cause: Parent doesn't wait() for child exit Solution: Always handle close event:

proc.on('close', (code) => {
  cleanup(proc.pid);
});

3. Shell Absorbs Signal

Cause: Using shell execution which spawns an intermediate shell Solution: Use spawn() with shell: false