Performance Engineer

Performance Engineer

When to Use

• Profile services under load — CPU, heap, goroutines, GC, lock contention
• Trace requests end-to-end — identify critical path and fan-out
• Load test — baseline, soak, stress, spike; interpret saturation and errors
• Define budgets — p50/p95/p99 targets, error budget, per-dependency ceilings
• Tune databases, caches, queues, and hot code paths with measured before/after
• Detect regressions — benchmark suites, CI gates, release comparisons
• Model capacity — headroom, scaling triggers, cost vs latency trade-offs
• Report findings — reproducible steps, flame graphs, ranked recommendations

When NOT to Use

• Build or fix Jenkins/GitHub Actions pipelines → devops
• Canary/blue-green rollout plans → deployment-strategist
• Star schema, dbt layers, warehouse ELT design → data-warehouse-engineer, analytics-data-engineer
• Implement React components or a11y → senior-frontend-software-engineer
• General feature delivery without perf focus → senior-software-engineer
• DC power/cooling/rack utilization programs → data-center-compute-supply-efficiency
• LLM token benchmarks and compression research → research-engineer-scientist-tokens
• Enterprise architecture NFR sign-off only → senior-system-architecture

Related skills

Core Workflows

1. Profiling and diagnostics

CPU, memory, I/O, contention.

See `references/profiling_diagnostics.md`.

2. Load and capacity testing

Scenarios, harness, interpretation.

See `references/load_capacity_testing.md`.

3. Latency, throughput, and SLOs

Percentiles, budgets, error budget.

See `references/latency_slo_budgets.md`.

4. Database and data-path tuning

Queries, indexes, caches.

See `references/database_query_performance.md`.

5. Frontend and client runtime

CWV, network, rendering.

See `references/frontend_runtime_performance.md`.

6. Regression detection and reporting

Baselines, CI, executive summary.

See `references/regression_ci_reporting.md`.

Outputs

• Perf investigation brief — symptom, scope, hypothesis, reproduction
• Trace + profile pack — flame graph, span waterfall, top offenders
• Load test plan and results — scenario matrix, graphs, bottlenecks
• Optimization backlog — ranked by impact × effort with measurements
• SLO proposal — targets, measurement points, alert thresholds
• Regression report — build-over-build comparison with root cause

Principles

• Measure first — no optimization without baseline and success metric
• One variable — isolate changes; document environment (hardware, data size, version)
• User-centric metrics — tail latency and error rate over averages
• Saturation-aware — distinguish queueing from inefficient work
• Reproducible — scripts, seeds, and configs checked in or attached to report

Database and data-path performance

Table of contents

1. Investigation flow 2. Query analysis 3. Indexing and schema 4. Caching and denormalization 5. Warehouse vs OLTP

Investigation flow

1. Identify slow queries from APM, DB slow log, or EXPLAIN 2. Capture plan, row counts, buffer hits, execution time 3. Correlate with traffic — one query vs N+1 storm 4. Reproduce on staging with similar cardinality 5. Measure after fix under load test

Query analysis

Always compare plans before/after; watch plan regression on data growth.

Indexing and schema

• Index equality + range columns in selective order
• Avoid over-indexing write-heavy tables
• Partial indexes for filtered subsets
• Consider read replicas for read-heavy, eventually-consistent OK paths

For warehouse-specific modeling and partition keys → data-warehouse-engineer.

Caching and denormalization

Document cache stampede mitigation (single-flight, jittered TTL).

Warehouse vs OLTP

Route deep star-schema and ELT design to warehouse skill; stay on app-issued SQL and connection behavior here.

Frontend and client runtime performance

Table of contents

1. Core Web Vitals 2. Network and assets 3. Rendering 4. Measurement 5. Handoff to FE engineering

Core Web Vitals

Set budgets per template (e.g. LCP < 2.5s p75 field data).

Network and assets

• Critical path: fewer serial requests, HTTP/2/3, preconnect
• Bundle size — code split, tree-shake, defer non-critical JS
• Images: modern formats, responsive srcset, lazy below fold
• Third-party scripts: audit cost; load after consent/interaction

Lab tools (Lighthouse) ≠ field RUM — track both.

Rendering

• Avoid unnecessary re-renders (memoization with measurement)
• Virtualize long lists
• Prefer CSS over JS layout thrashing
• SSR/streaming: balance TTFB vs hydration cost (Next.js patterns)

Deep component implementation → senior-frontend-software-engineer.

Measurement

• RUM: real users by device, region, connection
• Lab: CI Lighthouse or equivalent on representative pages
• Filmstrip and long-task attribution for INP fixes
• Compare before/after on same throttling profile

Handoff to FE engineering

Performance engineer delivers:

• Ranked issues with repro URL and trace
• Suggested fix class (defer script, resize image, split route)
• Validated improvement in lab or staging

FE engineer owns merge, a11y regression check, and design alignment.

Latency, throughput, and SLOs

Table of contents

1. Metric selection 2. SLO structure 3. Latency budgets 4. Error budget 5. Alerting

Metric selection

Prefer histograms over averages. Align window with traffic (1m, 5m rollups).

SLO structure

Example API SLO:

• Availability: 99.9% successful (non-5xx) over 30d
• Latency: 99% of requests < 300ms at gateway
• Freshness (if applicable): 95% of reads < 60s stale

Document:

• SLI — what is measured (synthetic vs real user)
• Scope — which routes, regions, tenants
• Exclusions — maintenance, client errors (4xx policy)

Coordinate with devops for implementation in observability backend.

Latency budgets

Decompose end-to-end budget across hops:

client → CDN → gateway → service A → service B → DB
  50ms     20ms    30ms      80ms       60ms    40ms  (example)

Rules:

• Sum of sub-budgets ≤ user-facing target with margin
• Parallel calls use max(sub-budgets), not sum
• Reserve slack for retries and jitter

Publish budget table in RFC or runbook; flag violations in traces.

Error budget

error_budget = (1 - SLO_target) × eligible_requests

When budget burns:

1. Freeze risky releases 2. Prioritize reliability over features 3. Post-incident review if exhaustion from single cause

Performance work that improves tail often restores budget faster than mean fixes.

Alerting

• Page on SLO burn rate (multi-window), not raw CPU
• Separate symptom (latency) from cause (DB connections)
• Load test failures should block release if SLO regression > threshold

Avoid alert fatigue — pair with devops and incident-management-engineer process.

Load and capacity testing

Table of contents

1. Test types 2. Scenario design 3. Harness requirements 4. Interpreting results 5. Capacity model

Test types

Always define success criteria before running (max error rate, p99 ceiling).

Scenario design

Model realistic mix:

• Read/write ratio, auth vs anonymous, heavy vs light endpoints
• Think time if mimicking users; zero think time for API soak
• Seed data size matching prod order of magnitude (not empty DB)
• Idempotent writes or isolated test tenants

Avoid testing only health checks — include business-critical paths.

Harness requirements

• Distributed workers if single client cannot generate load
• Warm-up phase excluded from metrics
• Capture: RPS, latency histogram, errors by class, saturation (CPU, pool wait)
• Store artifacts: config, git SHA, env vars (redacted), result files
• Integrate with CI for smoke/load on release candidates

Tools vary by stack — skill is tool-agnostic; document chosen tool and version.

Interpreting results

Do not extrapolate linearly past knee of curve — document max sustainable RPS.

Capacity model

headroom = (max_sustainable_rps - current_peak_rps) / max_sustainable_rps

Include:

• Per-service and per-dependency limits
• Autoscale rules and minimum instances
• Cost per 1k requests at target utilization
• Time to scale out vs spike duration

Hand off infra scaling changes to devops / infrastructure-engineer with evidence.

Profiling and diagnostics

Table of contents

1. Triage checklist 2. CPU profiling 3. Memory and leaks 4. I/O and blocking 5. Concurrency

Triage checklist

Capture request ID, release version, traffic mix, and data volume before profiling.

CPU profiling

• Sample at steady state under representative load, not idle
• Prefer wall-clock + CPU profiles when I/O-bound
• Compare profiles before/after change on same machine class
• Attribute cost to frames you own vs libraries vs runtime

Document: profiler tool, duration, sample rate, build flags (debug symbols).

Memory and leaks

• Heap dumps or allocation profiles at peak and after GC
• Track RSS vs heap, off-heap (Netty, mmap), container limits
• Watch for retained collections, session caches, global singletons
• Validate object churn driving GC pressure

For leaks: reproduce with soak test; bisect release; fix and verify with 2× soak duration.

I/O and blocking

• Trace syscall wait vs application time
• Check connection pool wait time metrics
• N+1 queries and synchronous fan-out show as wide span trees
• Large payloads: compression, pagination, streaming

Concurrency

• Lock profiling: mutex, RW lock, channel blocking
• Thread/goroutine pool exhaustion
• Unbounded parallelism amplifying downstream load
• Coordinated omission in benchmarks — use open-loop with care

Pair with latency_slo_budgets.md for queueing theory context (Little's law: L = λW).

Regression detection and reporting

Table of contents

1. Baselines 2. CI performance gates 3. Release comparison 4. Report template

Baselines

Store per scenario:

• Git SHA, branch, date
• Environment (CPU, memory, region)
• Load profile (RPS, duration, mix)
• Metrics: p50/p95/p99, error rate, CPU, memory

Version baselines when intentional perf change ships; annotate in changelog.

CI performance gates

Policies:

• Fail PR on regression > X% vs main (flake budget: rerun N times)
• Store trends in time-series for dashboards
• Separate noisy tests; quarantine with owner

Coordinate pipeline wiring with devops; own scenario and thresholds here.

Release comparison

For each release candidate:

1. Run same harness as last green baseline 2. Diff percentiles and errors 3. Attribute to commits via bisect if needed 4. Block or waive with documented risk

Include statistical caution — small samples lie; require minimum request count.

Report template

## Summary
- Symptom / ticket:
- Environment:
- Verdict: improved | regressed | inconclusive

## Baseline vs experiment
| Metric | Baseline | Experiment | Δ |
| p99 latency | | | |
| Throughput | | | |
| Error rate | | | |

## Root cause
(Top 3 offenders with evidence — trace id, query, frame)

## Recommendations
1. [impact] …
2. …

## Reproduction
(steps, scripts, links)

## Risks / follow-ups

Share with engineering lead; link PRs for fixes. Escalate SLO breach to incident-management-engineer if production-impacting.