Iterate Lessons Log

<!-- PM-Skills | https://github.com/product-on-purpose/pm-skills | Apache 2.0 -->

Lessons Log

A lessons log entry captures significant learning from projects, incidents, or experiences in a format that's useful to future teams who weren't there. Unlike retrospectives (which focus on team improvement), lessons logs focus on organizational knowledge that transcends individual teams.patterns, anti-patterns, and hard-won wisdom.

When to Use

• After completing a significant project or initiative
• Following a major incident, outage, or failure
• When you realize something important that others should know
• After discovering a pattern that keeps recurring
• When experienced team members leave (capture their knowledge)
• During post-mortems to preserve learnings

When NOT to Use

• You are facilitating the team ceremony itself -> use iterate-retrospective; this skill banks the durable entry that outlives it
• You are deciding whether to change direction based on results -> use iterate-pivot-decision
• The learning is an experiment readout -> use measure-experiment-results first, then bank the transferable lesson here
• You are updating stakeholders on what was learned -> use foundation-stakeholder-update

Instructions

When asked to create a lessons log entry, follow these steps:

1. Choose a Descriptive Title Write a title that someone searching for this topic would find. Include keywords that describe the situation and the learning. Avoid generic titles like "Project X lessons."

2. Provide Context Explain the situation fully enough that someone who wasn't there can understand it. Include the project, timeline, team, and any relevant constraints. Future readers need this context to assess applicability.

3. Describe What Happened Write a factual account of what occurred. Be specific about actions taken, decisions made, and outcomes observed. Avoid blame.focus on events and systems.

4. Extract the Lesson Articulate what you learned clearly. The lesson should be actionable.something others can apply. Distinguish between what you observed and your interpretation of why it matters.

5. Formulate Recommendations Provide specific guidance for future teams facing similar situations. What should they do? What should they avoid? What questions should they ask?

6. Define Applicability Help readers know when this lesson applies. What situations trigger relevance? What context makes it more or less applicable?

7. Add Tags for Searchability Include keywords and categories that will help future searchers find this entry. Think about what someone would search for when facing a similar situation.

Output Format

Use the template in references/TEMPLATE.md to structure the output. A complete entry fills every template section: Metadata; Summary; Context; What Happened; The Lesson; Recommendations; Applicability; Supporting Evidence; Tags and Categories; and Review and Updates.

Quality Checklist

Before finalizing, verify:

• [ ] Title is descriptive and searchable
• [ ] Context is complete enough for someone who wasn't there
• [ ] Lesson is clearly articulated and actionable
• [ ] Recommendations are specific, not vague
• [ ] Entry stands alone (doesn't require external context)
• [ ] Tags enable future discovery

Examples

See references/EXAMPLE.md for a completed example.

iterate-lessons-log - Version History

2.1.0 (2026-06-10)

Quality-convergence minor (F-12 Batch 3): added a "When NOT to Use" section with boundary pointers to neighboring skills, and the Output Format now enumerates the template sections a complete artifact fills. No template or example changes.

2.0.1 (2026-06-10)

Description-only patch (F-12 Batch 0, from the 2026-06-09 repo audit): the trigger-surface description was rewritten to disambiguate collision pairs with an explicit boundary pointer to the sibling skill. No body, template, or behavior changes.

2.0.0 (2026-01-26)

Baseline row for the prior published version; see git history for its changes.

Load Testing with Production Traffic Replay Prevents Scale Failures

Metadata

---

Summary

Our payment service experienced a 47-minute outage during Black Friday due to database connection pool exhaustion under 10x normal load. Despite running load tests, we didn't catch this because our synthetic load tests didn't replicate real traffic patterns.specifically the bursty, concentrated nature of flash sale traffic. Production traffic replay would have caught this.

---

Context

Background

Black Friday is our highest traffic day, typically 8-10x normal volume. The platform team conducted load testing in October, simulating 10x traffic with good results. We also completed a capacity review and provisioned additional infrastructure. Despite this preparation, our payment service failed at 9:03 AM EST when traffic spiked.

Timeline

Team and Stakeholders

• Platform Engineering team (incident response)
• Payment team (service owners)
• Customer Support (user communication)
• Executive team (business impact)
• Affected: ~12,000 customers couldn't complete purchases during outage

Constraints

• Black Friday is immovable.no option to delay or reschedule
• Payment service is critical path.no graceful degradation possible
• Real traffic replay wasn't available (privacy concerns, infrastructure gap)

---

What Happened

Sequence of Events

1. Sale launch (09:00): Traffic increased as expected. Systems initially handling load well.

2. Flash sale spike (09:02): A limited-quantity flash sale created concentrated traffic.5,000 users hitting checkout simultaneously rather than distributed load.

3. Connection pool saturation (09:03): Payment service database connections exhausted. Each flash sale request held connections longer due to inventory checks under contention.

4. Cascade begins (09:05): New requests queue, timeouts trigger retries, retries amplify load. Database CPU hits 100%.

5. Service degradation (09:07): Payment API latency exceeds 30 seconds. Frontend shows errors. Customers start contacting support.

6. Incident response (09:15): War room assembled. Initially suspected DDoS attack due to traffic pattern.

7. Root cause identified (09:35): Database team identified connection pool exhaustion. Pool sized for 10x sustained load, not 10x burst load with long-held connections.

8. Remediation (09:50): Connection pool increased from 100 to 500, connection timeout reduced. Service begins recovering.

9. Resolution (10:15): Service fully restored. Estimated $340,000 in lost revenue during outage.

Key Decisions Made

Outcome

47-minute service outage during our highest-revenue hour. Estimated $340,000 direct revenue loss, unknown brand damage. Customer trust impacted.we received 400+ complaint tickets in the following week referencing the Black Friday experience.

---

The Lesson

What We Learned

Primary Lesson: Synthetic load tests with evenly distributed traffic don't catch burst-pattern failures. Production traffic replay.using anonymized real traffic patterns.reveals timing, distribution, and correlation patterns that synthetic tests miss.

Supporting Observations:

• Our load test sent 100 requests/second evenly distributed. Real traffic came in bursts of 5,000 in 10 seconds.
• Synthetic tests didn't simulate correlated requests (many users hitting the same flash sale item).
• Connection pool math was correct for throughput but wrong for concurrency under burst.
• The failure mode was predictable in retrospect.database contention under burst is a known pattern.

Why This Matters

Load testing is expensive and time-consuming. If we're going to invest in it, we need it to actually predict production behavior. Synthetic tests gave us false confidence. Production traffic replay is the only way to truly simulate real-world patterns, especially for events with unusual traffic characteristics.

Root Cause Analysis

• Immediate cause: Connection pool exhaustion under burst load
• Contributing cause: Load test didn't simulate burst patterns
• Root cause: No mechanism to replay real traffic patterns in test environments
• Systemic cause: Load testing treated as checkbox, not continuous practice

---

Recommendations

Do This

1. Implement production traffic replay capability. Capture and anonymize real traffic patterns. Replay them in staging environments before major events. Investment pays off on the first prevented outage.

2. Test for burst patterns specifically. Design load tests that include concentrated spikes, not just sustained throughput. Model flash sale dynamics, viral traffic patterns, and marketing campaign spikes.

3. Size for concurrency, not just throughput. When capacity planning, consider maximum concurrent connections, not just requests per second. These require different calculations.

4. Do chaos engineering for resource exhaustion. Regularly test what happens when connection pools, thread pools, and memory approach limits. Don't wait for production to discover failure modes.

Avoid This

1. Don't assume synthetic load tests prove production readiness. Synthetic tests prove a specific pattern works.they don't prove all patterns work. Know what you haven't tested.

2. Don't treat load testing as a one-time checkbox. Traffic patterns change. Run load tests regularly, especially before major events or after significant changes.

3. Don't dismiss production traffic replay due to privacy concerns. Traffic patterns can be anonymized and replayed without exposing user data. The technique is well-established.

Questions to Ask

• What are the busiest patterns in our real traffic, not just the average?
• Have we tested what happens when N requests hit the same resource simultaneously?
• What's our failure mode when [resource] is exhausted? Do we fail gracefully?
• When was the last time we load tested with realistic patterns?

---

Applicability

When This Applies

• Preparing for known high-traffic events (sales, launches, campaigns)
• Operating services with shared resources (connection pools, caches)
• Running systems where traffic is bursty, not steady
• After scaling capacity based on throughput calculations

When This May Not Apply

• Services with naturally even traffic distribution
• Systems designed for graceful degradation (non-critical paths)
• Very early-stage products where traffic is low and predictable

Related Situations

• Marketing campaign launches with concentrated traffic spikes
• Product launches with viral potential
• Any "thundering herd" scenario (cache expiration, failover recovery)
• Services downstream of rate-limiters that can "dam break"

---

Supporting Evidence

Metrics/Data

Quotes

"The load test results looked great. We hit 10x traffic with no issues. What we didn't realize is that we were testing a fantasy version of 10x traffic." - Sarah Chen, Platform Engineering

"In retrospect, of course flash sales create burst patterns. We just didn't think to simulate it." - Mike Johnson, Payment Service Owner

Artifacts

• Black Friday Incident Postmortem (internal doc)
• Load Testing Report - October 2025 (internal doc)
• Production Traffic Replay RFC (internal doc)

---

Tags and Categories

Primary Category: Infrastructure

Tags: #load-testing #scale-failure #connection-pools #traffic-patterns #black-friday #incident #database #capacity-planning

Related Lessons: LL-2024-008 (Cache thundering herd), LL-2025-003 (Database connection limits)

---

Review and Updates

---

This lesson was captured to help future teams avoid repeating our mistakes and build on our successes.

[Descriptive Title of the Lesson]

Metadata

---

Summary

<!-- One paragraph summary of the lesson for quick scanning -->

[Brief summary of what happened and what was learned.someone should be able to decide if this is relevant from this paragraph alone]

---

Context

Background

<!-- What was the situation? What were we trying to do? -->

[Description of the project, initiative, or situation]

Timeline

Team and Stakeholders

• [Who was involved]
• [Who was affected]

Constraints

<!-- What limitations or pressures existed? -->

• [Constraint 1]
• [Constraint 2]

---

What Happened

Sequence of Events

<!-- Factual account of what occurred -->

1. [Event 1] 2. [Event 2] 3. [Event 3] 4. [Event 4]

Key Decisions Made

Outcome

<!-- What was the result? -->

[Description of the outcome.good, bad, or mixed]

---

The Lesson

What We Learned

<!-- Clear articulation of the learning -->

Primary Lesson: [Clear statement of the core learning]

Supporting Observations:

• [Observation 1]
• [Observation 2]
• [Observation 3]

Why This Matters

<!-- Why is this learning significant? -->

[Explanation of the importance and implications]

Root Cause Analysis

<!-- If applicable, what underlying factors led to this? -->

• [Root cause 1]
• [Root cause 2]

---

Recommendations

Do This

<!-- Positive guidance for future teams -->

1. [Specific recommendation] 2. [Specific recommendation] 3. [Specific recommendation]

Avoid This

<!-- Anti-patterns to prevent -->

1. [What to avoid and why] 2. [What to avoid and why]

Questions to Ask

<!-- Diagnostic questions for future teams facing similar situations -->

• [Question 1]?
• [Question 2]?
• [Question 3]?

---

Applicability

When This Applies

<!-- Situations where this lesson is relevant -->

• [Situation 1]
• [Situation 2]
• [Situation 3]

When This May Not Apply

<!-- Situations where this lesson may not be relevant -->

• [Exception 1]
• [Exception 2]

Related Situations

<!-- Similar contexts where the lesson might help -->

• [Related situation]
• [Related situation]

---

Supporting Evidence

Metrics/Data

<!-- If applicable, quantitative evidence -->

Quotes

<!-- Relevant quotes from team members or stakeholders -->

"[Quote]" - [Attribution]

Artifacts

<!-- Links to relevant documents, postmortems, etc. -->

• [Link to related document]
• [Link to related document]

---

Tags and Categories

Primary Category: [Infrastructure / Process / Communication / Technical / Strategy / People]

Tags: [#tag1] [#tag2] [#tag3] [#tag4]

Related Lessons: [LL-XXXX-XXX], [LL-XXXX-XXX]

---

Review and Updates

---

This lesson was captured to help future teams avoid repeating our mistakes and build on our successes.