Golang Performance

Go Performance

This skill is about measurement-first optimization in Go.

When to activate

Use this skill when you need to:

• Confirm a performance regression (latency/throughput/CPU/memory)
• Identify hot paths with pprof (CPU / heap / mutex / block)
• Reduce allocations and GC pressure in a measured hotspot
• Fix contention (mutex, scheduler, channel backpressure)
• Validate improvements with benchmarks and repeatable runs

If you need general idioms and patterns (not measurement), use golang-patterns.

---

Rules of engagement

• Profile before optimizing. A fast guess beats a slow change.
• Change one thing at a time. Measure after each change.
• Keep a baseline. Every claim should have “before vs after”.
• Don’t optimize the cold path. Make the hot path boring.

---

Outcome expectations

• Performance claims are backed by reproducible before/after measurements.
• Profile type selection matches the observed symptom.
• Optimizations are incremental, attributable, and regression-resistant.

---

Workflow

1. Make it measurable

• Add a benchmark (or a reproducible load test) for the suspected hotspot.
• Run multiple iterations; record mean + variance.

2. Capture evidence

• CPU profile for time
• Heap/allocs profile for memory
• Mutex/block profiles for contention
• Trace when the scheduler / GC behavior matters

3. Analyze before changing code

• Identify top offenders (top, top -cum)
• Inspect annotated source (list)
• Confirm whether you are bound by CPU, allocations, syscalls, or contention

4. Apply targeted fixes

• Allocation and GC: reduce allocations, reuse buffers, avoid retaining large backing arrays
• Data layout: improve locality, avoid interface boxing in hot loops
• Concurrency: reduce contention, bound goroutines, add backpressure

5. Verify and document

• Re-run the benchmark/profile
• Ensure correctness isn’t traded away
• Record the change and its measured impact

---

Symptom to first profile mapping

• High CPU -> CPU profile
• Memory growth -> heap profile (compare snapshots)
• High allocation churn / GC pressure -> allocs profile
• Latency spikes without CPU spike -> block profile
• Lock contention suspicion -> mutex profile
• Scheduler/pathological latency behavior -> runtime trace

---

Safety note: exposing pprof

net/http/pprof endpoints can leak sensitive runtime data. Prefer:

• bind to localhost
• protect with auth / firewall
• enable only in dev / controlled environments

---

Resources

Load these references on demand:

• references/profiling.md — pprof + trace collection and analysis commands
• references/benchmarks.md — stable benchmarks, -benchmem, benchstat, hygiene
• references/allocations-gc.md — allocation patterns, slice retention, sync.Pool guidance
• references/contention.md — mutex/block profiles, contention patterns, backpressure

Allocations and GC pressure

The goal

In hot paths, fewer allocations usually means:

• less GC work
• better cache locality
• less latency variability

Measure first with /debug/pprof/allocs and b.ReportAllocs().

---

Common allocation sources

Growing slices without capacity

// Prefer: make with capacity when size is known.
out := make([]T, 0, n)
for i := 0; i < n; i++ {
    out = append(out, f(i))
}

Building strings in loops

var b strings.Builder
b.Grow(n) // if you can estimate
for _, s := range parts {
    b.WriteString(s)
}
return b.String()

Interface boxing in hot loops

If a hot loop converts to interface{} (or any), values may escape. Prefer concrete types and typed helpers.

---

Slice retention (backing array kept alive)

// BAD: keeps the whole backing array alive
small := big[:10]

// GOOD: copy only what you need
small := make([]byte, 10)
copy(small, big[:10])

This often shows up as “mystery memory” in heap profiles.

---

sync.Pool (use with care)

sync.Pool can reduce allocations for short-lived, frequently allocated objects.

Rules:

• Pool buffers, not business objects.
• Always Reset() buffers before returning to the pool.
• Treat pooled objects as temporary: the runtime may drop pool contents at any GC.
• Measure. Pooling can increase CPU due to contention or cache misses.

---

Escape analysis

go build -gcflags="-m" ./...

Use this output to understand why values move to heap. Don’t cargo-cult “avoid pointers”: correctness first, measure impact.

Retention diagnostics

If heap stays high after load drops, investigate retained references:

• long-lived caches/maps holding large values
• slices or strings referencing oversized backing arrays
• goroutines capturing large objects in closures

References

• https://go.dev/doc/diagnostics
• https://go.dev/blog/escape-analysis

Go Benchmarks (stable, repeatable)

Basic benchmark skeleton

func BenchmarkThing(b *testing.B) {
    b.ReportAllocs()

    setup := makeInput()
    b.ResetTimer()

    for i := 0; i < b.N; i++ {
        _ = Thing(setup)
    }
}

Run benchmarks

go test -run=^$ -bench=. -benchmem ./...

# Longer benches reduce noise
go test -run=^$ -bench=. -benchmem -benchtime=3s ./...

# Fix CPU variability where possible
GOMAXPROCS=1 go test -run=^$ -bench=. -benchmem ./...

Compare before/after (benchstat)

benchstat summarizes change with statistics.

# Install once
go install golang.org/x/perf/cmd/benchstat@latest

# Capture output
go test -run=^$ -bench=BenchmarkThing -benchmem ./... > before.txt
# apply change
go test -run=^$ -bench=BenchmarkThing -benchmem ./... > after.txt

benchstat before.txt after.txt

Benchmark hygiene

• Keep setup outside the timer (b.ResetTimer())
• Avoid allocations in the benchmark loop unless you’re measuring them
• Use b.StopTimer() / b.StartTimer() for expensive setup between iterations
• Use sub-benchmarks (b.Run) for comparisons

Additional stability tips:

• Avoid running heavyweight background processes while benchmarking.
• Prefer fixed input datasets so before/after runs are comparable.
• Capture multiple runs and compare with benchstat instead of single-run conclusions.

References

• https://pkg.go.dev/testing
• https://go.dev/blog/benchmarks

Contention and backpressure

What to measure

• Mutex contention: pprof/mutex
• Blocking on channels: pprof/block
• Scheduler behavior / goroutine churn: go tool trace

Typical causes

• One global lock protecting a map/cache
• Using unbuffered channels on high-throughput paths
• Spawning unbounded goroutines (burst load)
• Holding locks while doing I/O

Patterns that help

Shard the lock

Instead of one lock, split by hash prefix.

Reduce lock scope

Do the minimum work while holding the lock. Compute outside, then commit.

Bound concurrency

Use a semaphore or worker pool to cap goroutines.

sem := make(chan struct{}, max)
for _, item := range items {
    item := item
    sem <- struct{}{}
    go func() {
        defer func() { <-sem }()
        process(item)
    }()
}

Add backpressure

Prefer bounded queues. If producers can outpace consumers forever, memory becomes the buffer.

Mitigation order (practical)

1. confirm contention with mutex/block profiles 2. shrink critical sections 3. reduce shared state / shard locks 4. bound producer rate (queue + workers) 5. re-profile before considering lock-free redesign

References

• https://go.dev/blog/pipelines
• https://go.dev/doc/effective_go#concurrency

Profiling Go with pprof and trace

Use this reference when you need concrete commands and a minimal workflow.

Enable pprof (HTTP)

import (
    "net/http"
    _ "net/http/pprof"
)

// Bind to localhost to reduce exposure.
go func() {
    _ = http.ListenAndServe("127.0.0.1:6060", nil)
}()

CPU profile

go tool pprof http://127.0.0.1:6060/debug/pprof/profile?seconds=30

# Inside pprof
(pprof) top
(pprof) top -cum
(pprof) list YourFunc
(pprof) web

Heap and allocs

go tool pprof http://127.0.0.1:6060/debug/pprof/heap
go tool pprof http://127.0.0.1:6060/debug/pprof/allocs

(pprof) top
(pprof) top -cum
(pprof) list YourFunc

Tips:

• Use /allocs to find allocation sites.
• Use /heap to find live objects (retention).

Contention profiles (mutex/block)

• Mutex profile: time spent waiting on sync.Mutex/RWMutex
• Block profile: goroutines blocked on channel ops, select, etc.

# mutex and block endpoints exist when the runtime has profiling enabled
go tool pprof http://127.0.0.1:6060/debug/pprof/mutex
go tool pprof http://127.0.0.1:6060/debug/pprof/block

For accurate mutex/block data, set sampling rates in code where appropriate:

runtime.SetMutexProfileFraction(1)
runtime.SetBlockProfileRate(1)

Use lower sampling rates in production if overhead is a concern.

go tool trace

Trace is useful when the scheduler and GC behavior are part of the problem.

curl -o trace.out http://127.0.0.1:6060/debug/pprof/trace?seconds=5
go tool trace trace.out

Minimal analysis checklist

1. Confirm the hotspot appears consistently across runs 2. Confirm whether it is CPU, allocs/GC, syscalls, or contention 3. Make one change and re-measure 4. Keep profile type, load shape, and duration consistent when comparing runs

References

• pprof: https://pkg.go.dev/net/http/pprof
• Go blog (pprof): https://go.dev/blog/pprof
• Go blog (trace): https://go.dev/blog/trace