High Concurrency Scalability

High Concurrency & Scalability

When to Use

• Choose or refactor concurrency models—threads, async/await, actors, coroutines—for target throughput and latency
• Reduce lock contention and design low-contention, lock-free, or partitioned data paths
• Size connection pools, file descriptors, thread pools, and memory limits per dependency
• Design caching layers, TTL strategy, and stampede / thundering-herd mitigation
• Plan horizontal scaling, load balancing, session affinity, and stateless vs sticky tradeoffs
• Apply backpressure, bounded queues, rate limiting, and bulkheads under overload
• Scale the data layer—read replicas, routing, sharding concepts, pool tuning, hot keys
• Profile bottlenecks, model capacity, and tie scale triggers to SLOs and error budgets
• Define autoscaling signals, warm pools, and cold-start vs cost tradeoffs
• Architect multi-region read paths and CDN/edge caching at a design level

When NOT to Use

• Decompose monoliths into bounded contexts and inter-service contracts only → microservices-developer
• Event schemas, broker selection, and messaging topology only → event-driven-architecture
• General feature delivery, RFCs, or CRUD without scale focus → senior-software-engineer
• Org-wide SLO program, on-call, incident response, and error-budget policy → site-reliability-engineer
• Deep flame graphs, load-test harnesses, and p99 regression hunts as the main task → performance-engineer
• Kubernetes platform golden paths and IDP product work → platform-engineer
• VPC, managed service provisioning, and landing-zone IaC → cloud-engineer
• Cloud spend optimization and unit economics only → cloud-economist, finops-analyst

Related skills

Core Workflows

1. Scope and constraints

Clarify traffic shape, SLOs, statefulness, and failure modes.

See `references/high_concurrency_scalability_scope.md`.

2. Concurrency and synchronization

Pick execution model; partition work; minimize shared mutable state.

See `references/concurrency_models_and_synchronization.md`.

3. Caching and data-layer scale

Cache hierarchy, replica routing, sharding and hot-key mitigation.

See `references/caching_and_data_layer_scale.md`.

4. Throughput, backpressure, and queues

Bounded queues, shedding, rate limits, and async pipelines.

See `references/throughput_backpressure_and_queues.md`.

5. Horizontal scale and load distribution

Replicas, LB algorithms, affinity, autoscaling triggers.

See `references/horizontal_scaling_and_load_distribution.md`.

6. Capacity, observability, and SLO-driven scale

Metrics, headroom models, scale policies tied to objectives.

See `references/capacity_planning_observability_slo.md`.

Outputs

• Scale brief — workload profile, bottlenecks, target RPS/latency, state assumptions
• Concurrency note — model choice, pool sizes, contention risks, partitioning plan
• Cache and data plan — layers, TTL, invalidation, replica/shard routing, hot-key mitigations
• Overload matrix — backpressure, rate limits, bulkheads, degradation modes
• Capacity model — headroom, scale triggers, cold-start impact, cost sensitivity
• Observability checklist — saturation, queue depth, pool wait, cache hit rate, tail latency

Principles

• Measure saturation—CPU, memory, I/O, pool wait, queue depth—not averages alone
• Bound everything—connections, threads, queue length, in-flight requests
• Prefer partition over lock—shard by key, actor mailbox, or isolated replica
• Design for overload—shed load deliberately; never unbounded retry or queue growth
• Scale on SLO signals—error rate and tail latency, not CPU alone
• Keep hot paths stateless where possible; isolate stateful tiers explicitly

Caching and data-layer scale

Table of contents

1. Cache hierarchy 2. Stampede and thundering herd 3. Invalidation strategies 4. Read replicas and routing 5. Sharding concepts 6. Connection pool tuning 7. Hot keys and skew

Cache hierarchy

Typical layers (closest to client first):

1. CDN / edge — static and cacheable API responses; respect Cache-Control 2. Application local — Caffeine, in-process LRU; watch memory and consistency 3. Distributed cache — Redis/Memcached; shared across instances 4. Database buffer pool — not a substitute for app cache design

Cache-aside (lazy load): App reads cache → on miss, load DB → populate cache. Simple; risk of stampede on miss.

Read-through / write-through: Cache library coordinates DB. Stronger consistency options; more coupling.

Write-behind: Fast writes to cache; async flush. High throughput; complex failure handling.

Stampede and thundering herd

Mitigations when many clients miss the same key simultaneously:

For caching stampede incidents: log miss rate spikes, key cardinality, and loader latency; fix coalescing before adding cache memory.

Invalidation strategies

• TTL-only — acceptable eventual staleness; simplest ops
• Explicit delete — on write path; risk missed invalidation bugs
• Versioned keys — entity:{id}:v{ver}; invalidate by bumping version
• Pub/sub fan-out — local cache purge on change events; handle delivery gaps
• Tag-based purge — invalidate groups (e.g., all keys for tenant:42)

Document maximum staleness product can tolerate per entity type.

Read replicas and routing

• Route read-only queries to replicas; writes to primary
• Handle replication lag — “read your writes” may require primary stickiness or session tokens
• Use connection pool per target (primary vs replica); don’t share one pool blindly
• Monitor replica lag seconds; shed replica traffic when lag exceeds SLO
• Read replica scale helps read-heavy workloads; does not fix write hot spots

Sharding concepts

• Choose shard key with high cardinality and even distribution (avoid monotonic insert hotspots where possible)
• Consistent hashing or directory service for routing; plan resharding early
• Cross-shard queries are expensive—design APIs to be single-shard when possible
• Sharding is not free: ops complexity, rebalancing, cross-shard transactions avoided or sagas

Connection pool tuning

• Pool size ≈ (expected_concurrent_requests × avg_query_time) / target_latency—validate with load tests
• Too large pools hurt DB (context switch, lock contention on DB side)
• Set acquire timeout; fail fast rather than hang threads
• Recycle connections on firewall idle timeout mismatches
• Separate pools for OLTP vs analytics or long reports

Hot keys and skew

Symptoms: one shard or replica at 100% while peers idle.

Mitigations:

• Split hot key into sub-keys (e.g., counter:42:0..N aggregate)
• Local combine — batch increments in app memory, flush periodically
• Dedicated partition for celebrity/tenant outliers
• Rate limit per key at edge to protect shared infrastructure

Capacity planning, observability, and SLO-driven scale

Table of contents

1. Capacity planning process 2. Headroom and growth models 3. Profiling and bottleneck analysis 4. Observability for scale 5. SLO-driven scaling 6. Game days and validation

Capacity planning process

1. Baseline current peak (RPS, concurrent users, messages/sec) with seasonality 2. Model per-tier capacity (RPS per instance at target p99) 3. Apply growth — marketing events, tenant onboarding, compound monthly growth 4. Add headroom — typically 30–50% below saturation for unexpected bursts 5. Price tradeoffs — replica count vs larger instances vs cache spend 6. Document triggers — when to scale manually vs autoscale vs procure capacity

Capacity planning outputs: instance count ranges, DB size/IOPS, cache memory, network egress, runbook thresholds.

Headroom and growth models

Simple model:

required_instances = ceil(peak_rps / rps_per_instance_at_slo) × burst_factor

Refine with:

• Burst factor — flash sale multiplier over steady peak
• Efficiency loss — deploys, noisy neighbors, retry traffic
• Dependency ceilings — DB connections, partner rate limits

Track utilization at the constraining resource (not average CPU across idle cores).

Profiling and bottleneck analysis

Order of investigation for throughput collapse:

1. Saturation — CPU, memory, disk I/O, network, FDs, thread pools 2. Queueing — pool acquire wait, broker lag, servlet queue 3. Lock contention — mutex wait, DB row locks, hot keys 4. Algorithmic — O(n²) paths, N+1 queries, serialization overhead 5. External — slowest dependency on critical path

Use traces to find critical path; use profiles for hot functions.

Deep tool ownership → performance-engineer; this skill frames what to measure and architectural fixes.

Observability for scale

USE method for resources: Utilization, Saturation, Errors.

Correlate deploys with metric shifts; keep synthetic probes for edge-to-origin path.

SLO-driven scaling

Define SLIs (latency, availability, freshness) and SLO targets.

• Tie autoscaling to SLI burn (error budget consumption), not vanity metrics
• Scale policies should cite which SLO is protected (e.g., checkout p99 < 300ms)
• Error budget policy: when budget is low, freeze risky changes; favor stability over new capacity experiments

Program-level SLO governance → site-reliability-engineer.

Game days and validation

Before production peaks:

• Load test to declared peak × burst factor (harness detail → performance-engineer)
• Validate autoscaling lag and cold-start impact on p99
• Exercise degradation and shedding paths
• Fail a zone or replica—confirm LB health checks and traffic shift
• Run cache expiry drill with stampede protections enabled

Record results in the capacity plan; update rps_per_instance constants quarterly.

Concurrency models and synchronization

Table of contents

1. Model selection 2. Threads and thread pools 3. Async and coroutines 4. Actors and message passing 5. Locks, contention, and lock-free patterns 6. Resource limits

Model selection

Rule: Match the model to the dominant wait type (I/O vs CPU) and team operability—not fashion.

Threads and thread pools

• Size pools from measured queue wait and dependency latency, not cores × constant alone
• Separate pools for CPU-bound vs blocking I/O work to avoid starvation
• Use bounded submit queues; reject or shed when saturated (CallerRuns is often a latency trap)
• Avoid unbounded new Thread per request; cap in-flight work at the edge
• For servlet-style stacks: align accept queue, worker pool, and downstream pool limits

Async and coroutines

• Never call blocking APIs on the event loop without a dedicated executor bridge
• Limit in-flight async operations per request and globally (semaphores)
• Propagate cancellation and timeouts through the async tree
• Watch head-of-line blocking on single-loop designs; consider sharded loops or processes
• For async scalability: scale out replicas; each instance still has one loop’s constraints

Actors and message passing

• One actor owns mutable state; communicate via immutable messages
• Mailbox depth is implicit queue—monitor and apply backpressure (drop, shed, or slow producers)
• Partition by stable key (user_id, session_id) for locality and even load
• Supervision: define restart vs escalate policy for poison messages

Locks, contention, and lock-free patterns

Reduce contention:

• Shrink critical sections; prefer read-copy-update or per-shard locks
• Partition data so hot paths rarely share locks (striped counters, per-bucket maps)
• Use lock ordering discipline to prevent deadlocks when multiple locks are unavoidable

Lock-free / low-lock (use carefully):

• Atomic counters and compare-and-swap for stats and id generation
• Ring buffers for single-producer/single-consumer paths
• Verify ABA and memory-order semantics; test under ARM weak memory

When locks are fine: Low contention, short sections, clear invariants—simplicity beats exotic structures.

Resource limits

Document limits in runbooks; alert on sustained pool wait before exhaustion.

High concurrency and scalability scope

Table of contents

1. Role focus 2. Typical deliverables 3. Workload dimensions 4. Decision boundaries 5. Anti-patterns

Role focus

Typical deliverables

• Traffic and concurrency profile (RPS, burst factor, fan-out, payload size)
• Bottleneck assessment with evidence (profile, queue depth, pool wait)
• Concurrency architecture note (threads vs async vs actors; partition keys)
• Cache hierarchy diagram with TTL, invalidation, and stampede controls
• Data-scale plan (replicas, routing, shard keys, hot-key mitigations)
• Overload playbook (rate limits, shedding order, bulkhead map)
• Autoscaling policy draft (metrics, thresholds, min/max, warm pool)
• Capacity spreadsheet or headroom model with growth scenarios

Workload dimensions

Capture these before recommending scale patterns:

Decision boundaries

Invest in horizontal scale when:

• Work is embarrassingly parallel or partitionable by key
• Single-node CPU/memory/network saturates before SLO is met
• Stateless tiers can scale behind a load balancer
• Data tier supports read scale-out or shard routing

Prefer vertical scale or optimization first when:

• Strong single-node affinity (large in-memory working set)
• Cross-shard transactions dominate and cannot be redesigned
• Coordination overhead of many small instances exceeds benefit
• Profiling shows algorithmic or query inefficiency—not saturation

Add async queues when:

• Producers can outpace consumers temporarily
• Work can be processed with acceptable lag (clear lag SLO)
• Backpressure at the edge protects synchronous path

Anti-patterns

• Unbounded resources—threads, connections, or queues with no ceiling
• Scale without measure—more replicas while pool wait or lock time dominates
• Cache as database—no invalidation story; stale reads violate product rules
• Retry amplification—retries on overloaded dependencies without jitter or caps
• CPU-only autoscaling—ignores latency, errors, or queue depth until users suffer
• Shared mutable hot row—all writers on one key regardless of “microservices”
• Thundering herd—mass expiry or deploy cold cache with no coalescing

Horizontal scaling and load distribution

Table of contents

1. Horizontal vs vertical scale 2. Stateless replicas 3. Load balancing 4. Sticky sessions and affinity 5. Autoscaling 6. Multi-region and CDN edge

Horizontal vs vertical scale

Horizontal scaling adds instances behind a load balancer. Requires shared-nothing app tier or externalized state.

Stateless replicas

• Push session state to Redis, DB, or signed cookies (understand security/size limits)
• Externalize uploads to object storage; don’t rely on local disk
• Use health checks that reflect readiness (dependencies up), not only process alive
• Graceful shutdown: drain connections, stop accept, finish in-flight with deadline

Load balancing

Placement options: hardware LB, cloud LB (ALB/NLB/GCLB), service mesh, client-side (gRPC xDS).

Load balancing at L7 can route by path, header, or gRPC service name.

Enable connection reuse (HTTP/2, keep-alive) to reduce handshake overhead at high concurrency.

Sticky sessions and affinity

Sticky sessions route the same client to the same backend (cookie, IP hash).

Pros: local cache warmth, legacy session in memory.

Cons: uneven load; painful deploys; lost stickiness on scale-in.

Prefer state externalization over stickiness when SLO and elasticity matter.

If affinity is required, use consistent hashing with bounded impact on node add/remove.

Autoscaling

Autoscaling adjusts replica count (or serverless concurrency) from signals.

Policies:

• Scale-out fast, scale-in slow — avoid flapping; protect cold instances
• Min replicas > 0 for latency-sensitive paths; accept cost or use warm pools
• Cold-start tradeoffs — JVM/.NET warmup, serverless init; use provisioned concurrency or always-warm min
• Predictive scale — schedule ahead of known events (marketing, payroll)

Document autoscaling limits (max replicas, budget caps) in capacity plans.

Multi-region and CDN edge

Architecture-level patterns (implementation → cloud-engineer):

• Active-passive — simpler consistency; failover RTO/RPO defined
• Active-active — lower latency globally; conflict resolution required
• Read local, write global — replicas per region; write routing to primary or CRDT/merge strategy
• CDN — cache static and cacheable API at edge; short TTL for semi-dynamic; purge APIs for incidents

Health-check cross-region dependencies; avoid circular failover. Measure cross-region latency on critical paths.

Throughput, backpressure, and queues

Table of contents

1. Throughput vs latency 2. Backpressure fundamentals 3. Queueing patterns 4. Rate limiting 5. Bulkheads and isolation 6. Load shedding and degradation

Throughput vs latency

• Throughput — work completed per unit time (RPS, messages/sec)
• Latency — time per unit of work (p50/p95/p99)
• Under load, queueing theory applies: as utilization → 100%, latency grows nonlinearly (Kingman-style behavior)
• Optimizing throughput alone can inflate tails if queues grow unbounded
• Define success as both SLO latency and sustainable throughput at peak factor

Backpressure fundamentals

Backpressure signals upstream that downstream cannot accept more work yet.

Mechanisms:

Without backpressure: memory grows, GC thrashes, timeouts cascade, retries amplify load.

Implement backpressure closest to the bottleneck and propagate a consistent error contract to callers.

Queueing patterns

Queue depth metrics: depth, age (oldest message), consumer rate, DLQ rate.

Set max depth alerts before consumers are hours behind.

Rate limiting

Rate limiting protects shared resources and enforces fairness.

Algorithms:

• Token bucket — smooth bursts with sustained cap
• Leaky bucket — constant outflow; stricter smoothing
• Fixed/sliding window — simple per-interval caps
• Distributed counters — Redis/sidecar; watch clock and failure modes

Dimensions: per IP, API key, user, tenant, endpoint, downstream.

Return 429 with Retry-After; prefer reject fast over slow timeout.

Combine with admission control at the edge (API gateway, service mesh).

Bulkheads and isolation

Bulkhead — partition resources so one tenant or feature cannot exhaust shared pools.

Examples:

• Separate thread pools per dependency or tenant tier
• Dedicated connection pool for admin vs customer traffic
• Cell-based architecture (failure and load isolation by slice)

Pair bulkheads with timeouts on cross-bulkhead calls.

Load shedding and degradation

When limits are hit, shed in priority order:

1. Non-critical features (recommendations, analytics beacons) 2. Expensive optional paths (full-text enrichments) 3. New requests while finishing in-flight critical work 4. Hard fail only when safety or consistency requires it

Document degradation modes in the scale brief; test them in game days.

Avoid retry storms: cap retries, use jitter, retry only on idempotent operations and specific error classes.