Go Process Cli

Go Process-Management CLIs

Make Go command-line tools that spawn, supervise, and shut down processes correctly — the part the stdlib makes easy to get subtly wrong.

When to Apply

Reach for this skill when building or reviewing a Go CLI whose job is to control other processes or concurrent workloads: a daemon, a process supervisor, a job/worker runner, a deploy or migration tool, a test harness that shells out, or any program that must stop cleanly when an orchestrator says so. It targets Go 1.22+ on Unix-like systems (Linux, macOS); a few rules note where Windows differs. It assumes you already know Go — it only corrects the specific defaults that go wrong in this domain.

Categories

Quick Reference

The defaults this skill exists to correct:

• `signal.NotifyContext(ctx, SIGINT, SIGTERM)` — not a channel that only watches Ctrl-C. Orchestrators send SIGTERM.
• `exec.CommandContext` + `cmd.Cancel`/`cmd.WaitDelay` — plain CommandContext SIGKILLs children with no cleanup.
• `Setpgid` + `kill(-pgid)` — signalling the direct child orphans grandchildren (sh -c, make, npm).
• `cmd.Output()` for capture — reading a pipe after Wait() deadlocks.
• `errgroup.WithContext` + `SetLimit(n)` — a goroutine per task means a process per task: PID/FD exhaustion.
• `ctx` as an argument — never a struct field; `defer cancel()` on every derived context.
• `select { case <-ticker.C: case <-ctx.Done(): }` — time.Sleep makes shutdown wait out the interval.
• `main → run() error → os.Exit(1)` — log.Fatal/os.Exit deep in code skip every defer.
• `proc.Signal(syscall.Signal(0))` — os.FindProcess always succeeds on Unix; it never proves liveness.
• `os.OpenFile(O_CREATE|O_EXCL)` for PID files — check-then-create is a race; `cmd.Wait()` every child or it zombifies.

How to Use

1. Identify which category your task falls under (see table above). 2. Read the relevant rule files in references/ — each is a focused, self-contained pattern naming the wrong default it corrects. 3. Apply the pattern; follow the Reference link in each rule for the authoritative source.

Rules cross-link (e.g. graceful shutdown ties signals → context → child-process kill → cleanup), so follow the links when a task spans categories.

When extending this skill, copy assets/templates/_template.md: WHY first, one canonical example with realistic names, a foil only if the wrong way is a genuine trap.

Source Authority

Every rule cites primary sources — the Go standard library reference on pkg.go.dev, the official Go blog (go.dev/blog), and the cobra documentation — chosen because they are maintainer-authored and version-current. No content farms, listicles, or undated tutorials.

Related Skills

• radical-simplification — when a supervisor design has accreted accidental complexity.
• unix-cli / cli-for-agents — broader CLI ergonomics beyond process management.

Go

Version 0.1.0 dot-skills May 2026

---

Abstract

Distilled patterns for Go CLIs that manage processes and concurrent workloads — daemons, supervisors, job runners, deploy tools. Targets Go 1.22+ on Unix-like systems and corrects the specific standard-library defaults that go wrong in this domain: catching SIGTERM (not just Ctrl-C) via signal.NotifyContext, killing children gracefully with exec.Cmd Cancel/WaitDelay before SIGKILL, signalling whole process groups so grandchildren don't orphan, avoiding os/exec pipe deadlocks, bounding concurrency with errgroup so a goroutine-per-task doesn't become a process-per-task, threading context for cancellation, funnelling exit through run() error so defers run, choosing stdlib flag vs cobra and wiring ExecuteContext, and supervising state (signal-0 liveness, atomic PID files, zombie reaping, structured slog). Each rule names the wrong default it corrects, shows one canonical example, and cites the primary source.

---

Table of Contents

1. Signals & Graceful Shutdown

• 1.1 Bind a context to SIGINT and SIGTERM with signal.NotifyContext
• 1.2 Let a second signal force-quit a wedged shutdown

2. Child Processes

• 2.1 Cancel a child with SIGTERM before SIGKILL using Cancel and WaitDelay
• 2.2 Capture child output with Output, not a pipe you read after Wait
• 2.3 Kill the process group, not just the direct child
• 2.4 Read the child's real exit code from exec.ExitError
• 2.5 Use exec.CommandContext so a child dies with its context

3. Concurrency & Workloads

• 3.1 Bound concurrency instead of one goroutine per task
• 3.2 Give every goroutine a cancellation path so it cannot leak
• 3.3 Only the sender closes a channel, exactly once
• 3.4 Use errgroup for fan-out with cancel-on-first-error

4. Context Propagation

• 4.1 Always defer cancel() from WithCancel, WithTimeout, and WithDeadline
• 4.2 Make blocking waits cancellable instead of time.Sleep
• 4.3 Pass context as an explicit argument, never store it in a struct

5. Errors & Exit Codes

• 5.1 Aggregate failures across workloads with errors.Join
• 5.2 Funnel main through a run() error so deferred cleanup runs

6. CLI Framework & Flags

• 6.1 Choose stdlib flag or cobra by the command surface, not by habit
• 6.2 Return errors from RunE; never os.Exit inside a command
• 6.3 Use a FlagSet per subcommand when staying on the stdlib
• 6.4 Wire the signal-bound context into cobra with ExecuteContext

7. Process State & Supervision

• 7.1 Create the PID or lock file atomically with O_CREATE and O_EXCL
• 7.2 Log process-state transitions as structured slog records
• 7.3 Probe liveness with signal 0 — os.FindProcess lies on Unix
• 7.4 Wait on every child you start so it doesn't become a zombie

---

References

1. https://pkg.go.dev/os/signal#NotifyContext 2. https://pkg.go.dev/os/exec 3. https://pkg.go.dev/golang.org/x/sync/errgroup 4. https://pkg.go.dev/context 5. https://pkg.go.dev/log/slog 6. https://go.dev/blog/pipelines 7. https://cobra.dev/

---

Source Files

This document was compiled from individual reference files. For detailed editing or extension:

{Same text as title}

{1–3 sentences naming the wrong default a competent Go developer would otherwise reach for, and the concrete consequence in a process-management CLI — orphaned children, leaked goroutines, skipped cleanup, unbounded shutdown, etc.}

{Canonical, copy-pasteable example with realistic names — never foo/bar.
 Show the correct pattern; keep it focused on the one decision the rule settles.}

{Optional: an Incorrect/Correct foil ONLY if the wrong way is a genuine, common trap. Keep the diff minimal so the contrast is the lesson. Link related rules with title when a task spans categories.}

Reference: {source title}

Sections

This document defines the category structure and file-name prefixes used by every rule in references/. Categories are ordered by importance × frequency — the decisions a process-management CLI gets wrong most often, and most expensively, come first. These rules target Go 1.22+ on Unix-like systems (Linux, macOS); Windows process semantics differ where noted.

1. Signals & Graceful Shutdown (sig)

Description: How the CLI process reacts to termination requests. Binding a context.Context to OS signals, catching SIGTERM (not just Ctrl-C) so orchestrators can stop the process cleanly, and letting a second signal force-quit a wedged shutdown. This is the signature correctness concern for any long-running or supervising CLI.

2. Child Processes (exec)

Description: Spawning and supervising external processes with os/exec. Tying a child's lifetime to a context, terminating it gracefully before SIGKILL, killing the whole process group so grandchildren don't orphan, reading its output without deadlock, and extracting its exit code. The most footgun-dense area of the standard library for this domain.

3. Concurrency & Workloads (work)

Description: Running concurrent work safely. Using errgroup for fan-out with cancel-on-first-error, bounding concurrency instead of spawning a goroutine per task, giving every goroutine a cancellation path so it cannot leak, and respecting channel ownership so a close never panics.

4. Context Propagation (ctx)

Description: Threading cancellation through the program. Passing context.Context as an explicit argument rather than storing it, releasing the resources a derived context holds, and making blocking waits cancellable so shutdown is responsive instead of hanging on a time.Sleep.

5. Errors & Exit Codes (err)

Description: Turning failures into clean process exits. Funnelling all paths through a run() error so deferred cleanup actually runs, and aggregating failures across many workloads instead of surfacing only the first.

6. CLI Framework & Flags (cli)

Description: Structuring the command-line surface. Choosing stdlib flag versus cobra by the shape of the tool, wiring the signal-bound context into a cobra command tree via ExecuteContext, returning errors from RunE instead of exiting inside a command, and the canonical stdlib subcommand pattern for when a framework is overkill.

7. Process State & Supervision (state)

Description: Observing and tracking processes. Probing real liveness (os.FindProcess lies on Unix), creating PID/lock files atomically to prevent two instances racing, reaping children so they don't become zombies, and logging state transitions as structured slog records a supervisor can parse.

Wire the signal-bound context into cobra with ExecuteContext

A cobra app that calls rootCmd.Execute() gives its commands a context of context.Background() — one that is never cancelled. Commands then either ignore cancellation entirely or, worse, mint their own context.Background() internally, so the SIGTERM you carefully captured in main never reaches the work. rootCmd.ExecuteContext(ctx) threads your signal-bound context into the command tree; inside any RunE, cmd.Context() returns it, so shutdown propagates to every child process and goroutine the command spawns.

func main() {
	ctx, stop := signal.NotifyContext(context.Background(),
		syscall.SIGINT, syscall.SIGTERM)
	defer stop()

	// Pass the cancellable context into cobra, not Execute().
	if err := rootCmd.ExecuteContext(ctx); err != nil {
		os.Exit(1)
	}
}

var startCmd = &cobra.Command{
	Use: "start",
	RunE: func(cmd *cobra.Command, args []string) error {
		ctx := cmd.Context() // the signal-bound context, not Background()
		return supervise(ctx) // SIGTERM now reaches everything below
	},
}

Always read the context from cmd.Context() inside a command rather than capturing a package-level variable — it keeps the command testable (a test passes its own context) and guarantees you use the one cobra actually propagated.

Reference: pkg.go.dev — cobra.Command.ExecuteContext

Choose stdlib flag or cobra by the command surface, not by habit

Two opposite default mistakes: pulling in cobra for a tool that has one job, or hand-rolling subcommand dispatch with the global flag set for a tool that has grown ten commands. Pick by the shape of the CLI. A single action with a few flags (myd --config x --port 8080) needs only stdlib flag — no dependency, no boilerplate. A tree of verbs and nouns (proc start, proc stop, proc ls --json) with shared persistent flags, generated help, and completion is exactly what cobra exists for; rebuilding that on flag means reimplementing dispatch, help text, and flag inheritance by hand.

// Single-purpose tool: stdlib flag is the right size.
func main() {
	cfg := flag.String("config", "/etc/myd.yaml", "config file path")
	port := flag.Int("port", 8080, "listen port")
	flag.Parse()
	os.Exit(run(*cfg, *port))
}

// Multi-command tool: cobra carries the dispatch, help, and shared flags.
var rootCmd = &cobra.Command{Use: "proc", Short: "manage worker processes"}

func main() {
	rootCmd.AddCommand(startCmd, stopCmd, statusCmd)
	rootCmd.PersistentFlags().String("socket", "/run/proc.sock", "control socket")
	// ... see cli-execute-context-wires-signals for ExecuteContext
}

The decision is reversible but not free — migrating flag → cobra later is mechanical, so start with flag and adopt cobra when a second subcommand appears, not in anticipation of one. (spf13/pflag, which cobra uses, adds GNU-style --long/-s flags if you want those without the full framework.)

Reference: pkg.go.dev — flag · cobra.dev

Return errors from RunE; never os.Exit inside a command

Cobra offers both Run func(...) and RunE func(...) error. Using Run and calling os.Exit/log.Fatal inside it has the same defect as anywhere else — it skips deferred cleanup — plus two cobra-specific ones: the command becomes untestable (a test can't assert on a process that exited), and there is no single place to decide the exit code. Use RunE, return errors, and let one handler in main map them to an exit code. Then suppress cobra's reflex to print full usage on a runtime error (usage text belongs to argument errors, not "the database was down").

var stopCmd = &cobra.Command{
	Use:           "stop [name]",
	Args:          cobra.ExactArgs(1),
	SilenceUsage:  true, // don't dump usage on a runtime failure
	SilenceErrors: true, // we print the error ourselves, once, in main
	RunE: func(cmd *cobra.Command, args []string) error {
		if err := stopProcess(cmd.Context(), args[0]); err != nil {
			return fmt.Errorf("stop %s: %w", args[0], err)
		}
		return nil
	},
}

func main() {
	if err := rootCmd.ExecuteContext(ctx); err != nil {
		slog.Error("command failed", "err", err)
		os.Exit(1)
	}
}

With SilenceErrors, ExecuteContext still returns the error, so main owns both the message and the code — and a richer mapping (e.g. errors.As to a *NotFoundError → exit 2) lives in exactly one spot. Argument-validation errors from cobra's Args validators still print usage, which is what you want for those.

Reference: pkg.go.dev — cobra.Command (RunE, SilenceUsage)

Use a FlagSet per subcommand when staying on the stdlib

When a tool has a couple of subcommands but you don't want a framework, the trap is leaning on the global flag.CommandLine set — every subcommand's flags collide in one namespace, and flag.Parse() chokes on the verb. The stdlib already supports subcommands cleanly: os.Args[1] selects the command, and each command gets its own flag.NewFlagSet parsing os.Args[2:]. Each set has independent flags, its own usage, and its own error handling.

func main() {
	if len(os.Args) < 2 {
		fmt.Fprintln(os.Stderr, "usage: proc <start|stop> [flags]")
		os.Exit(2)
	}

	switch os.Args[1] {
	case "start":
		fs := flag.NewFlagSet("start", flag.ExitOnError)
		port := fs.Int("port", 8080, "listen port")
		_ = fs.Parse(os.Args[2:]) // parse only this subcommand's args
		os.Exit(runStart(*port))
	case "stop":
		fs := flag.NewFlagSet("stop", flag.ExitOnError)
		force := fs.Bool("force", false, "SIGKILL instead of SIGTERM")
		_ = fs.Parse(os.Args[2:])
		os.Exit(runStop(*force))
	default:
		fmt.Fprintf(os.Stderr, "unknown command %q\n", os.Args[1])
		os.Exit(2)
	}
}

This stays dependency-free and scales to a handful of commands. Once subcommands need shared persistent flags, nested verbs, or generated completion, that is the signal to switch to cobra rather than grow this switch into a parser — see Choose stdlib flag or cobra by the command surface.

Reference: pkg.go.dev — flag.NewFlagSet

Make blocking waits cancellable instead of time.Sleep

time.Sleep is uninterruptible. A poll loop or retry backoff built on time.Sleep(interval) cannot react to shutdown: when SIGTERM arrives mid-sleep, the process sits idle for the rest of the interval before noticing, so a 30-second poll loop can take up to 30 seconds to die — long enough for a supervisor to escalate to SIGKILL. Replace the sleep with a select over ctx.Done() and a timer, so the wait ends the moment either the interval elapses or cancellation arrives.

// Unresponsive: a pending SIGTERM waits out the full interval.
func pollBad(ctx context.Context) {
	for {
		check()
		time.Sleep(30 * time.Second) // ignores ctx
	}
}

// Responsive: cancellation wins the race against the tick.
func poll(ctx context.Context) error {
	ticker := time.NewTicker(30 * time.Second)
	defer ticker.Stop()
	for {
		check()
		select {
		case <-ticker.C: // next interval
		case <-ctx.Done(): // shutdown — leave immediately
			return ctx.Err()
		}
	}
}

The same shape covers a cancellable single wait: select { case <-time.After(d): case <-ctx.Done(): return ctx.Err() }. The principle is that any wait inside a long-running process must be racing the context, so shutdown latency is bounded by how often you reach a select, not by your longest sleep.

Reference: pkg.go.dev — time.Ticker · pkg.go.dev — context.Context

Always defer cancel() from WithCancel, WithTimeout, and WithDeadline

Every derived context holds resources — at minimum a goroutine and, for timeouts, a time.Timer — that are released only when its cancel func is called. Drop the cancel on the floor and that context lives until its parent is cancelled, which for a root context is program exit. In a supervisor that creates a timeout per child, that is a steady leak of timers and goroutines. The discipline is mechanical: bind both return values and defer cancel() on the next line, even for WithTimeout where the deadline will fire anyway — cancel also frees resources immediately when the work finishes early, and go vet flags the missing call.

func probe(ctx context.Context, addr string) error {
	ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
	defer cancel() // releases the timer the instant probe returns

	cmd := exec.CommandContext(ctx, "healthcheck", addr)
	return cmd.Run()
}

defer cancel() is correct even on the success path: cancelling an already-finished or already-timed-out context is a harmless no-op, so there is never a reason to omit it. The only mistake is not calling it.

Reference: pkg.go.dev — context.WithTimeout

Pass context as an explicit argument, never store it in a struct

It is tempting to stash a context.Context in a struct field so methods don't each need a ctx parameter. The Go team explicitly warns against this, and for a process manager the bug is concrete: a context's scope is a single operation, but a struct's lifetime is the whole program. A Supervisor{ctx} captures the context that was current when it was built; every later Start/Stop call then uses a cancellation scope that no longer matches the request, so a per-command timeout or a fresh shutdown signal never reaches the work. Pass ctx as the first parameter of each method that does cancellable work.

// Wrong: the context is frozen at construction time.
type Supervisor struct{ ctx context.Context }
func (s *Supervisor) Start(spec Spec) error { return run(s.ctx, spec) }

// Right: each call carries the context that matches its scope.
type Supervisor struct{ /* config, no context */ }

func (s *Supervisor) Start(ctx context.Context, spec Spec) error {
	return run(ctx, spec)
}

The exception the docs allow is narrow: a struct that is a single request (and is discarded with it) may hold a context. A long-lived manager, registry, or client is not that. When in doubt, thread it through — the explicit parameter makes the cancellation scope visible at every call site.

Reference: pkg.go.dev — context (package overview) · go.dev/blog — Contexts and structs

Aggregate failures across workloads with errors.Join

When a CLI stops, kills, or checks a whole fleet of processes, "return the first error" is the wrong report: if three of ten shutdowns fail, the operator needs all three, not whichever lost the race. errgroup is built to cancel on the first error — the right tool when one failure should abort the rest, but the wrong one when every item must be attempted and every failure recorded. For the run-them-all case, iterate, collect, and combine with errors.Join (Go 1.20+), which wraps multiple errors into one that still works with errors.Is/errors.As.

func stopAll(ctx context.Context, procs []*Process) error {
	var errs []error
	for _, p := range procs {
		if err := p.Stop(ctx); err != nil {
			// Don't bail — record and keep going so every process is attempted.
			errs = append(errs, fmt.Errorf("stop %s: %w", p.Name, err))
		}
	}
	return errors.Join(errs...) // nil if errs is empty; combined otherwise
}

errors.Join returns nil when the slice is empty, so the happy path needs no special case, and its message lists each failure on its own line. Reach for it whenever completeness matters more than failing fast; reach for errgroup when the first failure should cancel the others.

Reference: pkg.go.dev — errors.Join

Funnel main through a run() error so deferred cleanup runs

os.Exit — and log.Fatal, which calls it — terminates the process immediately, without running any deferred functions. Call either deep inside the program and every defer above it is skipped: the PID file is not removed, the child process is not killed, buffered logs are not flushed, the lock is not released. The fix is structural: main does nothing but call a run() error that owns all the defers, and os.Exit appears in exactly one place — after run has returned and its defers have unwound.

func main() {
	ctx, stop := signal.NotifyContext(context.Background(),
		syscall.SIGINT, syscall.SIGTERM)
	defer stop()

	if err := run(ctx); err != nil {
		slog.Error("fatal", "err", err)
		os.Exit(1) // the ONLY os.Exit; reached after run's defers ran
	}
}

func run(ctx context.Context) error {
	pidFile, err := acquirePIDFile("/run/myd.pid")
	if err != nil {
		return err
	}
	defer pidFile.Remove() // actually runs, because run returns normally

	srv := startServer(ctx)
	defer srv.Shutdown() // actually runs

	return srv.Wait(ctx)
}

This keeps every cleanup co-located with the resource it cleans up, and guarantees it executes on every exit path. The moment you reach for log.Fatal in a helper, you have silently disabled all of it — return an error instead and let it propagate to the single exit point.

Reference: pkg.go.dev — os.Exit

Cancel a child with SIGTERM before SIGKILL using Cancel and WaitDelay

exec.CommandContext's built-in cancellation calls Process.Kill() — an unconditional SIGKILL. A database, a server, or a job runner killed that way loses in-flight work and leaves corrupt state, because SIGKILL cannot be trapped. Go 1.20 added two fields that fix this without hand-rolling a goroutine: Cmd.Cancel lets you choose how the process is stopped when the context ends, and Cmd.WaitDelay bounds how long to wait afterward before forcing a kill and closing I/O pipes.

func startServer(ctx context.Context) error {
	cmd := exec.CommandContext(ctx, "myserver", "--config", "prod.yaml")

	// On ctx cancellation, ask politely first.
	cmd.Cancel = func() error {
		return cmd.Process.Signal(syscall.SIGTERM)
	}
	// If it hasn't exited 10s after SIGTERM, the runtime SIGKILLs it
	// and unblocks any pipe reads.
	cmd.WaitDelay = 10 * time.Second

	err := cmd.Run()
	switch {
	case ctx.Err() != nil:
		// We initiated the stop; a signal-exit here is the expected
		// outcome, not a crash. Check the context, not the error type.
		slog.Info("server stopped on shutdown signal")
		return nil
	case errors.Is(err, exec.ErrWaitDelay):
		// Exited 0 but left I/O pipes open past the grace window.
		slog.Warn("server exited but I/O pipes lingered past WaitDelay")
		return nil
	default:
		return err // a genuine, unsolicited failure
	}
}

This is the process-management equivalent of cancel-cooperatively-then-abort: well-behaved children flush and exit within the grace window; stuck ones are still bounded by WaitDelay.

ErrWaitDelay is narrower than it looks — the runtime returns it only when the child exits successfully but leaves its I/O pipes open past the grace window, not when a child is killed for ignoring SIGTERM (that surfaces as a signal-exit *exec.ExitError). So to tell "we stopped it" from "it crashed," check ctx.Err(), not the error type. WaitDelay bounds two distinct hazards: a child that won't die after Cancel, and a child that exited but left inherited pipes open — without it, such a pipe can keep Wait blocked indefinitely even after you signal the child.

Reference: pkg.go.dev — os/exec.Cmd (Cancel, WaitDelay)

Use exec.CommandContext so a child dies with its context

exec.Command creates a child whose lifetime is independent of your program's cancellation. When the CLI is told to shut down, that child keeps running — now an orphan holding a port, a lock, or GPU memory. exec.CommandContext ties the process to a context.Context: when the context is cancelled (shutdown signal, timeout, parent error), the runtime kills the child automatically. For a tool whose whole job is managing processes, this binding is the default you want, not the exception.

func startWorker(ctx context.Context, addr string) error {
	// When ctx is cancelled, the child is killed automatically.
	cmd := exec.CommandContext(ctx, "worker", "--listen", addr)
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr

	if err := cmd.Run(); err != nil {
		return fmt.Errorf("worker %s: %w", addr, err)
	}
	return nil
}

Be aware of the default kill behavior: plain CommandContext sends SIGKILL the instant the context is done, giving the child no chance to clean up. For anything stateful, override that with a graceful-then-forceful policy — see Cancel a child with SIGTERM before SIGKILL.

Reference: pkg.go.dev — os/exec.CommandContext

Read the child's real exit code from exec.ExitError

A process supervisor must distinguish why a child failed: exit 0 (clean), exit 1 (generic error), exit 137 (SIGKILL/OOM), exit 2 (config error), and so on. The naive handler collapses everything to "err != nil → exit 1", discarding the one number a supervisor needs to decide whether to restart, alert, or give up. cmd.Run() returns an *exec.ExitError when the child ran but exited non-zero; unwrap it with errors.As and call ExitCode(). A nil error means exit 0; a non-ExitError error (e.g. binary not found) means the child never ran at all.

func runChild(ctx context.Context, name string, args ...string) (code int, err error) {
	cmd := exec.CommandContext(ctx, name, args...)
	cmd.Stdout, cmd.Stderr = os.Stdout, os.Stderr

	err = cmd.Run()
	if err == nil {
		return 0, nil
	}
	var exitErr *exec.ExitError
	if errors.As(err, &exitErr) {
		// Child ran and exited non-zero; ExitCode() is its real status
		// (-1 if it was terminated by a signal).
		return exitErr.ExitCode(), nil
	}
	// Failed to start: bad path, permission denied, context cancelled.
	return -1, fmt.Errorf("could not run %s: %w", name, err)
}

Propagating the child's code to your own process lets shells and CI scripts react correctly: os.Exit(code) at the top level makes your wrapper transparent. Folding it to 1 makes every failure look identical.

Reference: pkg.go.dev — os/exec.ExitError · pkg.go.dev — os.ProcessState.ExitCode

Kill the process group, not just the direct child

Signalling cmd.Process reaches only the process you spawned. But CLIs routinely run sh -c "...", make, npm run, or any wrapper that forks its own children — and those grandchildren survive when you kill the parent, becoming orphans that still hold ports and files. The Unix answer is to put the child in its own process group with Setpgid, then signal the whole group by sending to the negated PID. A negative PID means "every process in this group."

func runScript(ctx context.Context, script string) error {
	cmd := exec.Command("sh", "-c", script)
	// Put the child in a new process group so we can signal its whole tree.
	cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}

	if err := cmd.Start(); err != nil {
		return err
	}
	pgid := cmd.Process.Pid // equals the group id because of Setpgid

	go func() {
		<-ctx.Done()
		// Negative pid = signal the entire process group, grandchildren included.
		_ = syscall.Kill(-pgid, syscall.SIGTERM)
	}()

	return cmd.Wait()
}

Use this instead of relying on CommandContext's killer when the child spawns its own children — the context killer signals only the direct child, leaving the group behind. (The negative PID survives Go's int→uintptr conversion inside syscall.Kill; the kernel reads it back as a process group. Setpgid and group signalling are Unix-specific — on Windows you'd use a Job Object.)

Reference: pkg.go.dev — syscall.SysProcAttr · pkg.go.dev — syscall.Kill

Capture child output with Output, not a pipe you read after Wait

Two classic os/exec deadlocks: (1) writing to a child's stdin and reading its stdout in the same goroutine — the OS pipe buffer fills, the child blocks writing, you block writing, neither drains; (2) calling cmd.Wait() and then reading from StdoutPipe — Wait closes the pipe as soon as the child exits, so the read races the close. The docs state plainly that it is incorrect to call Wait before all pipe reads complete. For the common case — run a command and collect its output — cmd.Output() and cmd.CombinedOutput() drain in the background and return a fully-read buffer, sidestepping both traps.

// Simple capture — no pipe management, no deadlock.
func gitHead(ctx context.Context) (string, error) {
	out, err := exec.CommandContext(ctx, "git", "rev-parse", "HEAD").Output()
	if err != nil {
		return "", fmt.Errorf("git rev-parse: %w", err)
	}
	return strings.TrimSpace(string(out)), nil
}

When you genuinely need streaming (tailing a long-running child's logs), read the pipe to completion before calling Wait:

stdout, _ := cmd.StdoutPipe()
if err := cmd.Start(); err != nil {
	return err
}
sc := bufio.NewScanner(stdout)
for sc.Scan() { // drain fully first
	slog.Info("child", "line", sc.Text())
}
return cmd.Wait() // only after the pipe is exhausted

Reference: pkg.go.dev — os/exec.Cmd.StdoutPipe

Bind a context to SIGINT and SIGTERM with signal.NotifyContext

The reflex is to catch os.Interrupt (Ctrl-C) on a channel. But a CLI that manages workloads is almost always stopped by a supervisor — systemd, Docker, Kubernetes, a parent process — and those send SIGTERM, not SIGINT. A process that only watches SIGINT ignores the polite stop and gets SIGKILL'd a few seconds later with no cleanup. signal.NotifyContext (Go 1.16+) collapses the channel boilerplate into a context whose cancellation is the shutdown signal, so the same ctx you already thread everywhere becomes the shutdown trigger.

func main() {
	ctx, stop := signal.NotifyContext(context.Background(),
		syscall.SIGINT, syscall.SIGTERM)
	defer stop() // restores default handler; lets a later signal kill normally

	if err := run(ctx); err != nil {
		slog.Error("exited with error", "err", err)
		os.Exit(1)
	}
}

func run(ctx context.Context) error {
	// ctx.Done() fires on the first SIGINT/SIGTERM.
	// Pass ctx down so every worker, child process, and blocking wait
	// observes the same cancellation.
	return supervise(ctx)
}

stop() is important: after the first signal you usually want a second one to terminate the process the default way (see Let a second signal force-quit). Calling stop() unregisters the handler so the program no longer swallows signals once shutdown is underway.

Reference: pkg.go.dev — os/signal.NotifyContext

Let a second signal force-quit a wedged shutdown

Graceful shutdown can hang — a child ignores SIGTERM, a flush blocks on a dead socket. If the only signal handler is the one that started the graceful path, the user hits Ctrl-C again and nothing happens, because the program is still swallowing the signal. The fix is to stop intercepting after the first signal so the next one reaches the default handler and kills the process. signal.NotifyContext's stop() does exactly this: once called, the next SIGINT/SIGTERM is no longer caught and terminates the program normally.

func main() {
	ctx, stop := signal.NotifyContext(context.Background(),
		syscall.SIGINT, syscall.SIGTERM)

	go func() {
		<-ctx.Done() // first signal arrived
		slog.Info("shutdown requested; press Ctrl-C again to force quit")
		stop() // stop catching signals — a second one now kills us
	}()

	if err := run(ctx); err != nil {
		os.Exit(1)
	}
}

This gives the operator an escape hatch without any extra signal plumbing: first signal = drain, second signal = die. Pair it with a deadline on the graceful path (see Cancel a child with SIGTERM before SIGKILL) so an unattended supervisor also gets bounded shutdown time, not just an interactive user.

Reference: pkg.go.dev — os/signal.NotifyContext

Create the PID or lock file atomically with O_CREATE and O_EXCL

The "single instance" guard is usually written as check then create: if _, err := os.Stat(pidFile); os.IsNotExist(err) { write(pidFile) }. That is a time-of-check/time-of-use race — two copies of the daemon launched together both pass the Stat, both write, and both run. The atomic primitive is os.OpenFile with O_CREATE|O_EXCL: the kernel guarantees the create-only-if-absent is a single uninterruptible operation, so exactly one process wins and the loser gets EEXIST.

func acquirePIDFile(path string) (*os.File, error) {
	// O_EXCL makes this fail if the file already exists — atomically.
	f, err := os.OpenFile(path, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o644)
	if errors.Is(err, os.ErrExist) {
		// Someone holds it — but is that process still alive?
		if pid, _ := readPID(path); !isAlive(pid) {
			os.Remove(path) // stale file from a crash; retry once
			return acquirePIDFile(path)
		}
		return nil, fmt.Errorf("already running (pidfile %s)", path)
	}
	if err != nil {
		return nil, err
	}
	fmt.Fprintf(f, "%d\n", os.Getpid())
	return f, nil // caller defers f.Close() + os.Remove(path)
}

A crash leaves a stale PID file, so the EEXIST branch must distinguish "another instance is genuinely running" from "leftover from a process that died" using a signal-0 liveness check. For multi-writer coordination beyond start-up exclusion, prefer an advisory lock (flock) which the kernel releases automatically on process exit — no stale-file cleanup needed.

Reference: pkg.go.dev — os.OpenFile

Wait on every child you start so it doesn't become a zombie

On Unix, a child that exits is not gone — the kernel keeps its exit status in the process table as a zombie until the parent calls wait. Start children with cmd.Start() and never call cmd.Wait() (or Run/Output, which call it for you) and zombies accumulate one per finished child, eventually exhausting the PID table so the supervisor can no longer fork anything. Wait reaps the entry and, equally important, releases the goroutines and file descriptors the os/exec machinery allocated for that child. Every Start needs a matching Wait.

func supervise(ctx context.Context, spec Spec) error {
	cmd := exec.CommandContext(ctx, spec.Bin, spec.Args...)
	if err := cmd.Start(); err != nil {
		return err
	}
	slog.Info("started", "name", spec.Name, "pid", cmd.Process.Pid)

	// Always reap: Wait collects the exit status and frees resources.
	err := cmd.Wait()
	slog.Info("exited", "name", spec.Name, "code", cmd.ProcessState.ExitCode())
	return err
}

If you start a child and return without waiting (fire-and-forget), spawn a goroutine whose sole job is cmd.Wait() — the reap must happen somewhere. The one case you don't manage directly is a child that outlives the parent: when the supervisor itself exits, its orphaned children are re-parented to PID 1 (init), which reaps them.

Reference: pkg.go.dev — os/exec.Cmd.Wait

Probe liveness with signal 0 — os.FindProcess lies on Unix

os.FindProcess on Unix always succeeds — it never checks whether the PID is alive, it just wraps the number. So proc, err := os.FindProcess(pid); err == nil tells you nothing; the returned *os.Process is equally non-nil for a running process, a dead one, and a PID that was recycled to a different program. To actually test liveness, send signal 0: the kernel performs all the permission and existence checks for a real signal but delivers nothing. nil means the process exists and you may signal it; ESRCH means it's gone; EPERM means it exists but belongs to another user.

func isAlive(pid int) bool {
	proc, err := os.FindProcess(pid)
	if err != nil {
		return false // effectively never happens on Unix
	}
	err = proc.Signal(syscall.Signal(0)) // probe, delivers nothing
	switch {
	case err == nil:
		return true // exists and signalable
	case errors.Is(err, syscall.EPERM):
		return true // exists but owned by another user
	default: // ESRCH and friends
		return false
	}
}

This matters most when reading a PID from a stale PID file: the number may now belong to an unrelated process, so "the PID exists" is not "my process is running." Treat signal-0 liveness as necessary but not sufficient — confirm identity (PID file age, a control socket, a cmdline check) before acting on a recycled PID.

Reference: pkg.go.dev — os.FindProcess

Log process-state transitions as structured slog records

A process supervisor's logs are read by machines — another supervisor, a log pipeline, an alerting rule — as often as by humans. fmt.Printf("worker %s exited with %d\n", name, code) forces every one of those consumers to parse free text with brittle regexes, and it loses the fields the moment the sentence is reworded. log/slog (Go 1.21+) emits each transition as key/value pairs that a JSON handler turns into queryable records: filter by pid, alert on event=exited code!=0, join on name — without parsing prose.

func main() {
	// JSON handler → one structured record per line, machine-parseable.
	slog.SetDefault(slog.New(slog.NewJSONHandler(os.Stderr, nil)))
	// ...
}

func onExit(name string, pid, code int, d time.Duration) {
	slog.Info("process transition",
		"event", "exited",
		"name", name,
		"pid", pid,
		"code", code,
		"uptime", d,
	)
}

Keep the key set stable across transitions (event, name, pid) so records for started, exited, and restarted line up in queries. Attach per-process context once with logger := slog.With("name", name, "pid", pid) and reuse it, so you don't repeat the fields — and never restate them. Use a TextHandler for interactive runs and the JSONHandler under a supervisor; the call sites don't change.

Reference: pkg.go.dev — log/slog

Bound concurrency instead of one goroutine per task

for _, t := range tasks { go process(t) } looks harmless because goroutines are cheap — but each one here spawns an OS process, opens files, and grabs sockets. With ten thousand tasks you get ten thousand concurrent children: PID exhaustion, too many open files, a thrashed scheduler, and an OOM kill. The number of in-flight goroutines is unrelated to how many processes the machine can sustain. Cap it. errgroup.SetLimit(n) makes g.Go block until a slot frees, so at most n workloads run at once with no manual semaphore.

func processAll(ctx context.Context, tasks []Task) error {
	g, ctx := errgroup.WithContext(ctx)
	g.SetLimit(runtime.NumCPU()) // cap concurrent children

	for _, t := range tasks {
		t := t
		g.Go(func() error {
			return runTask(ctx, t) // blocks here until a slot is free
		})
	}
	return g.Wait()
}

Pick the limit from the bottleneck the workloads actually contend for: CPU-bound children → runtime.NumCPU(); processes hammering one database → its connection limit; I/O-bound → higher. The point is that the limit is a deliberate number tied to a real resource, not "however many tasks happened to arrive." For finer control across multiple call sites, golang.org/x/sync/semaphore offers a weighted variant.

Reference: pkg.go.dev — errgroup.Group.SetLimit

Use errgroup for fan-out with cancel-on-first-error

The hand-rolled fan-out — sync.WaitGroup plus a buffered error channel — has two recurring defects: it keeps running every other goroutine after one fails (wasting work and time), and it surfaces only whichever error happened to land in the channel first. errgroup.WithContext solves both: the first goroutine to return a non-nil error cancels a shared context, so siblings that respect that context stop early, and Wait() returns that first error. It is the idiomatic fan-out primitive for managing N concurrent workloads.

func launchAll(ctx context.Context, specs []WorkerSpec) error {
	g, ctx := errgroup.WithContext(ctx)
	for _, spec := range specs {
		spec := spec // capture before Go (pre-1.22 loops)
		g.Go(func() error {
			// ctx is cancelled the moment any sibling returns an error,
			// so a CommandContext child here is killed automatically.
			return runWorker(ctx, spec)
		})
	}
	// Waits for all; returns the first non-nil error.
	return g.Wait()
}

The cancellation only helps goroutines that actually watch ctx — a worker that ignores it runs to completion regardless. That is exactly why every child here should be launched with exec.CommandContext(ctx, ...): the group's cancellation then propagates all the way down to the OS process.

Reference: pkg.go.dev — golang.org/x/sync/errgroup

Give every goroutine a cancellation path so it cannot leak

A goroutine blocked forever on a channel send or receive is never collected — it leaks its stack, and anything it captured, for the life of the process. In a long-running supervisor these accumulate until memory or FDs run out. The default trap is a goroutine that does a bare ch <- v or <-ch with no second exit: if the reader (or writer) goes away, that goroutine is stuck. Every goroutine that blocks on a channel must also select on ctx.Done() so cancellation drains it.

// Leaks: if nobody ever reads `results`, this goroutine blocks on send forever.
func watchBad(events <-chan Event, results chan<- Result) {
	for e := range events {
		results <- handle(e) // no escape if the reader is gone
	}
}

// Drains on cancellation: the ctx.Done() case lets the goroutine exit.
func watch(ctx context.Context, events <-chan Event, results chan<- Result) {
	for {
		select {
		case e, ok := <-events:
			if !ok {
				return
			}
			select {
			case results <- handle(e):
			case <-ctx.Done(): // reader gone / shutting down → exit
				return
			}
		case <-ctx.Done():
			return
		}
	}
}

The rule generalizes: a goroutine's lifetime must be bounded by something — a closed input channel, a cancelled context, or a WaitGroup the owner joins. If you cannot point to what stops a goroutine, it leaks. defer wg.Done() and a parent that calls wg.Wait() give the owner a way to confirm the goroutine actually finished before shutdown completes.

Reference: go.dev/blog — Go Concurrency Patterns: Pipelines and cancellation

Only the sender closes a channel, exactly once

close(ch) on an already-closed channel, or a send on a closed channel, is an unrecoverable panic — and in a worker pool it is easy to trigger by accident: two goroutines both think they own the channel, or a receiver closes it to "signal done." Go's convention exists precisely to make this impossible to get wrong: a channel is closed by its sole sender, and only to broadcast "no more values are coming." Receivers never close; they detect closure via the two-value receive. With multiple senders, none of them closes — a separate sync.WaitGroup coordinates the close.

func produce(ctx context.Context, jobs []Job) <-chan Job {
	out := make(chan Job)
	go func() {
		defer close(out) // the single sender owns the close
		for _, j := range jobs {
			select {
			case out <- j:
			case <-ctx.Done():
				return // defer still closes out exactly once
			}
		}
	}()
	return out
}

// Fan-in from many senders: close once, after all of them finish.
func merge(cs ...<-chan Result) <-chan Result {
	out := make(chan Result)
	var wg sync.WaitGroup
	wg.Add(len(cs))
	for _, c := range cs {
		go func(c <-chan Result) {
			defer wg.Done()
			for r := range c {
				out <- r
			}
		}(c)
	}
	go func() { wg.Wait(); close(out) }() // one closer, after all senders done
	return out
}

Returning a receive-only channel (<-chan T) from a constructor encodes the ownership in the type: callers literally cannot close what they only receive from.

Reference: go.dev/blog — Go Concurrency Patterns: Pipelines and cancellation