Science

Customization

Before executing, check for user customizations at: ~/.claude/PAI/USER/SKILLCUSTOMIZATIONS/Science/

If this directory exists, load and apply any PREFERENCES.md, configurations, or resources found there. These override default behavior. If the directory does not exist, proceed with skill defaults.

🚨 MANDATORY: Voice Notification (REQUIRED BEFORE ANY ACTION)

You MUST send this notification BEFORE doing anything else when this skill is invoked.

1. Send voice notification:

   curl -s -X POST http://localhost:31337/notify \
     -H "Content-Type: application/json" \
     -d '{"message": "Running the WORKFLOWNAME workflow in the Science skill to ACTION"}' \
     > /dev/null 2>&1 &

2. Output text notification:

   Running the **WorkflowName** workflow in the **Science** skill to ACTION...

This is not optional. Execute this curl command immediately upon skill invocation.

Science - The Universal Algorithm

The scientific method applied to everything. The meta-skill that governs all other skills.

The Universal Cycle

GOAL -----> What does success look like?
   |
OBSERVE --> What is the current state?
   |
HYPOTHESIZE -> What might work? (Generate MULTIPLE)
   |
EXPERIMENT -> Design and run the test
   |
MEASURE --> What happened? (Data collection)
   |
ANALYZE --> How does it compare to the goal?
   |
ITERATE --> Adjust hypothesis and repeat
   |
   +------> Back to HYPOTHESIZE

The goal is CRITICAL. Without clear success criteria, you cannot judge results.

---

Workflow Routing

Output when executing: Running the **WorkflowName** workflow in the **Science** skill to ACTION...

Core Workflows

Diagnostic Workflows

---

Resource Index

---

Domain Applications

---

Scale of Application

---

Integration Points

---

Key Principles (Quick Reference)

1. Goal-First - Define success before starting 2. Hypothesis Plurality - NEVER just one idea (minimum 3) 3. Minimum Viable Experiments - Smallest test that teaches 4. Falsifiability - Experiments must be able to fail 5. Measure What Matters - Only goal-relevant data 6. Honest Analysis - Compare to goal, not expectations 7. Rapid Iteration - Cycle speed > perfect experiments

---

Anti-Patterns

---

Quick Start

1. Goal - What does success look like? 2. Observe - What do we know? 3. Hypothesize - At least 3 ideas 4. Experiment - Minimum viable tests 5. Measure - Collect goal-relevant data 6. Analyze - Compare to success criteria 7. Iterate - Adjust and repeat

The answer emerges from the cycle, not from guessing.

Gotchas

• Hypothesis-test-analyze is the core loop. Don't skip the hypothesis step — going straight to testing is just trial-and-error, not science.
• Minimum 3 hypotheses before testing. Single-hypothesis testing is confirmation bias.
• Measurements must be specific and reproducible. "It seems better" is not a measurement.
• Full cycle is for systematic investigation. For quick debugging, use quick diagnosis mode.

Examples

Example 1: Quick diagnosis

User: "figure out why Surface time filters show stale items"
→ Quick diagnosis mode
→ Hypothesis: timestamp format mismatch in D1
→ Test: query D1 for actual stored format
→ Analyze: compare stored vs expected format
→ Result: ISO string vs Unix timestamp mismatch

Example 2: Full systematic investigation

User: "experiment with different prompt structures for better output"
→ Full cycle mode
→ 3+ hypotheses generated
→ Controlled experiments with measurements
→ Analysis identifies winning approach
→ Iterates until convergence

Execution Log

After completing any workflow, append a single JSONL entry:

echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","skill":"Science","workflow":"WORKFLOW_USED","input":"8_WORD_SUMMARY","status":"ok|error","duration_s":SECONDS}' >> ~/.claude/PAI/MEMORY/SKILLS/execution.jsonl

Replace WORKFLOW_USED with the workflow executed, 8_WORD_SUMMARY with a brief input description, and SECONDS with approximate wall-clock time. Log status: "error" if the workflow failed.

Science Examples

Worked examples showing the scientific method at different scales.

---

Example 1: Quick Debug (Micro-Scale, 5 minutes)

Problem: App throws 500 error on login. Users can't log in.

Goal: Users can log in without 500 error

Hypotheses (60 seconds):

H1: Database connection issue - 500s often mean backend failure
H2: Recent deploy broke something - worked yesterday
H3: Auth service down - login specifically fails
H4: Rate limiting triggered - traffic spike possible

Test Order: H1 (fastest), H3 (fast), H4 (medium), H2 (slow)

Test H1:

$ psql -c "SELECT 1"  # Connection timeout
$ docker ps | grep postgres  # Empty - container stopped!

Root Cause: PostgreSQL container stopped.

Fix: docker-compose up -d postgres

Verify: curl -I /api/login returns 200

Total Time: 5 minutes vs 30+ minutes of random flailing

Key Insight: 2 minutes of structured thinking saved 25+ minutes.

---

Example 2: TDD Cycle (Micro-Scale, 10 minutes)

Goal: Function isValidEmail(email: string): boolean that validates emails

Hypotheses (implementation approaches):

• H1: Simple regex
• H2: Library (email-validator)
• H3: Step-by-step parsing

Test H1 (TDD Red-Green-Refactor):

// RED: Write failing test
test('returns true for valid email', () => {
  expect(isValidEmail('user@example.com')).toBe(true);
});
// FAILS - function doesn't exist

// GREEN: Minimal code to pass
function isValidEmail(email: string): boolean {
  return /^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(email);
}
// PASSES

// Continue adding edge case tests...
// All pass - H1 CONFIRMED

Results: 6 tests pass, 100% coverage

Analysis: H1 (simple regex) confirmed - works for our needs

Outcome: Ship simple solution, refactor with JSDoc

---

Example 3: Feature Experiment (Meso-Scale, 2 weeks)

Problem: 40% cart abandonment rate. Need to reduce to under 25%.

Goal:

• Abandonment < 25%
• No conversion regression
• No complaint increase

Hypotheses:

Experiment (A/B Test H1):

• Control: Current checkout
• Treatment: + Security badges
• N: 5,000 per group
• Duration: 7 days

Results:

H1 CONFIRMED - 4 point reduction, p=0.0003

Iteration 2: Test H4 on treatment group Iteration 3: Test H3 on combined

Final Outcome (3 iterations):

Goal Achieved: 24% < 25%

---

Example 4: Prompt Iteration (Meso-Scale, 2 hours)

Problem: Summarization prompt produces inconsistent results.

Goal:

• Length: 100-150 words (currently 50-300)
• Key points: >90% captured (currently ~70%)
• Format: 100% bullets (currently ~60%)
• Eval score: >85% (baseline: 62%)

Baseline Prompt:

Summarize the following document:
{document}

Hypotheses:

• H1: Add explicit length constraint
• H2: Specify exact format (3-5 bullets)
• H3: Add key point extraction guidance
• H4: Few-shot examples

Experiment (Evals):

Winner: D (combined constraints) at 88%

Iteration 2: Add stronger enforcement

Final Prompt:

Summarize the following document.

Requirements:
- Length: 100-150 words
- Format: 3-5 bullet points, each 20-30 words
- You MUST use bullet points. No paragraphs allowed.
- Content: Identify main argument, key evidence, conclusion
- Verify: main claim, supporting evidence, implications

Document:
{document}

Final Score: 93% (from 62% baseline)

Key Learning: Combined constraints outperform few-shot examples.

---

Scale Summary

Science Methodology - The Deep Dive

This document elaborates on each phase of the scientific method as applied within PAI.

The power of this methodology is its universality and its iteration speed. Apply it at micro-scale (TDD cycles) or macro-scale (product strategy). The pattern remains the same.

---

Phase 0: GOAL - The North Star

Why Goal First?

You cannot judge experiments without success criteria.

Every failed project shares one trait: vague or shifting goals. "Make it better" is not a goal. "Reduce customer churn by 20% within 3 months" is a goal.

The Goal Specification Process

1. Outcome Statement What will exist or change when we succeed?

• Be specific about the end state
• Describe it as if it already happened
• Make it concrete and observable

2. Measurable Indicators How will we KNOW we succeeded?

• Quantitative metrics (numbers we can measure)
• Qualitative indicators (observable behaviors)
• Leading indicators (early signals)
• Lagging indicators (final confirmation)

3. Success Threshold What level constitutes "success"?

• Minimum viable success (acceptable)
• Target success (goal)
• Stretch success (exceeds expectations)

4. Constraints What are the boundaries?

• Time constraints (deadlines)
• Resource constraints (budget, people)
• Quality constraints (must maintain X)
• Technical constraints (must work with Y)

5. Anti-Goals What are we explicitly NOT trying to do?

• Prevents scope creep
• Clarifies trade-offs
• Keeps focus sharp

Goal Quality Checklist

Example Goal Specifications

Weak Goal:

"Improve the login experience"

Strong Goal:

Outcome: Users can log in without frustration, as measured by reduced support tickets and improved satisfaction

>

Indicators:

- Login-related support tickets drop from 50/week to <10/week

- Time-to-login decreases from 15s avg to <5s avg

- User satisfaction survey shows >4.5/5 for login experience

>

Threshold:

- Minimum: 50% reduction in tickets

- Target: 80% reduction + <5s login

- Stretch: Zero tickets + <3s login + biometric support

>

Constraints:

- Must maintain existing auth integrations (OAuth, SAML)

- No new third-party dependencies without security review

- Ship within 2 sprints

>

Anti-Goals:

- Not redesigning the entire auth system

- Not adding new auth methods (that's a separate project)

---

Phase 1: OBSERVE - Understanding Current State

Why Observation First?

You cannot solve what you do not understand.

Jumping to solutions without understanding the problem space leads to:

• Solving symptoms instead of root causes
• Reinventing existing solutions
• Missing critical constraints
• Building on false assumptions

The Observation Process

1. Gather Existing Knowledge

• What do we already know about this problem?
• What has been tried before?
• What do experts say?
• What does research show?

2. Map the Current State

• How does the system work now?
• Where are the pain points?
• What are the workarounds people use?
• What data do we have?

3. Identify Gaps

• What don't we know?
• What assumptions are we making?
• What could we be wrong about?
• What would change our approach if true?

4. Establish Baseline

• What are the current metrics?
• What is "normal" performance?
• What does the trend look like?
• What are the benchmarks?

Observation Quality Checklist

Tools for Observation

---

Phase 2: HYPOTHESIZE - Generating Candidates

The Plurality Principle

NEVER generate just one hypothesis.

Single-hypothesis thinking leads to:

• Confirmation bias (you'll see evidence that supports it)
• Missed alternatives (better solutions exist)
• Overinvestment (sunk cost in one approach)
• Narrow framing (the real solution might be orthogonal)

Minimum: 3 hypotheses. Better: 5-10 for important problems.

The Hypothesis Generation Process

1. Divergent Phase (Quantity over Quality)

• Generate as many ideas as possible
• No judgment yet - all ideas welcome
• Build on others' ideas
• Explore orthogonal approaches
• Include contrarian ideas

2. Categorization Phase

• Group similar hypotheses
• Identify different "axes" of solution space
• Note which assumptions each hypothesis challenges

3. Refinement Phase

• Make each hypothesis specific and testable
• Clarify what success looks like for each
• Identify what would DISPROVE each (falsifiability)

4. Prioritization Phase

• Rank by expected value (likelihood × impact)
• Rank by cost to test (cheaper experiments first)
• Identify dependencies between hypotheses

Hypothesis Quality Checklist

Hypothesis Template

## Hypothesis: [Short Name]

**Claim:** [What we believe will work]

**Rationale:** [Why we think this might work]

**Testable Prediction:** If this hypothesis is correct, then [observable outcome]

**Falsification Criteria:** This hypothesis is WRONG if [observable outcome]

**Cost to Test:** [Time/resources needed]

**Confidence Level:** [Low/Medium/High] - [Why]

Tools for Hypothesis Generation

---

Phase 3: EXPERIMENT - Designing Tests

The Minimum Viable Experiment

Design the smallest experiment that tests the hypothesis.

Not every experiment needs to be comprehensive. The goal is LEARNING, not perfection.

Ask:

• What's the fastest way to learn if this works?
• What's the cheapest way to get meaningful data?
• What would DISPROVE the hypothesis?
• What's the minimum sample size for confidence?

The Experiment Design Process

1. Define Success Criteria

• What outcome confirms the hypothesis?
• What outcome refutes the hypothesis?
• What outcome is inconclusive (need more data)?

2. Design the Test

• What will we do?
• What will we measure?
• How will we measure it?
• What's the control (if applicable)?

3. Identify Variables

• Independent variable (what we're changing)
• Dependent variable (what we're measuring)
• Control variables (what we're keeping constant)
• Confounding variables (what might skew results)

4. Plan Data Collection

• What data do we need?
• How will we collect it?
• When will we collect it?
• How much is enough?

5. Define Stopping Criteria

• When do we stop the experiment?
• What constitutes "enough" data?
• When do we declare success/failure/inconclusive?

Experiment Quality Checklist

Experiment Template

## Experiment: [Name]

**Testing Hypothesis:** [Which hypothesis this tests]

**Method:**
1. [Step 1]
2. [Step 2]
3. [Step 3]

**Measuring:**
- Primary metric: [What we're measuring]
- Secondary metrics: [Other useful data]

**Success Criteria:**
- Hypothesis CONFIRMED if: [condition]
- Hypothesis REFUTED if: [condition]
- INCONCLUSIVE if: [condition]

**Duration:** [How long to run]

**Resources Needed:** [What's required]

Tools for Experimentation

---

Phase 4: EXECUTE - Running Experiments

Parallel Execution

When experiments are independent, run them in parallel.

This is where PAI shines:

• Multiple worktrees for code experiments
• Multiple agents for research experiments
• Multiple eval runs for prompt experiments

Sequential experiments waste time when parallel is possible.

The Execution Process

1. Preparation

• Verify all prerequisites are met
• Ensure measurement infrastructure is ready
• Confirm success/failure criteria are clear
• Set up logging/data collection

2. Execution

• Run the experiment as designed
• Don't change parameters mid-experiment (unless aborting)
• Document any unexpected events
• Capture all relevant data

3. Observation

• Monitor progress (don't just set and forget)
• Note anomalies
• Be ready to abort if experiment is clearly failing
• Don't peek at results prematurely (avoid bias)

Execution Quality Checklist

---

Phase 5: MEASURE - Collecting Data

Data Collection Principles

1. Measure What Matters Only collect data that relates to the goal. More data ≠ better data.

2. Quantify Where Possible Numbers enable comparison. "It felt faster" < "Load time dropped 40%"

3. Capture Context Numbers without context are meaningless. Record conditions, anomalies, environment.

4. Preserve Raw Data Always keep raw data. Derived metrics can be recalculated; raw data cannot be recovered.

The Measurement Process

1. Collect Primary Metrics

• The main thing you're testing
• Direct measurement of success criteria

2. Collect Secondary Metrics

• Supporting context
• Early warning indicators
• Sanity checks

3. Capture Qualitative Data

• Observations
• Feedback
• Anomalies
• Surprises

4. Validate Data Quality

• Are there outliers?
• Is the data complete?
• Are there obvious errors?
• Do the numbers make sense?

Measurement Template

## Results: [Experiment Name]

**Date:** [When run]
**Duration:** [How long]

**Primary Metrics:**
| Metric | Before | After | Change |
|--------|--------|-------|--------|
| [Name] | [Value] | [Value] | [Δ%] |

**Secondary Metrics:**
| Metric | Value | Notes |
|--------|-------|-------|
| [Name] | [Value] | [Context] |

**Observations:**
- [Notable event 1]
- [Notable event 2]

**Raw Data Location:** [Link/path]

---

Phase 6: ANALYZE - Comparing to Goal

The Analysis Process

1. Compare to Success Criteria

• Did we meet the goal?
• By how much did we exceed or fall short?
• Is the difference statistically significant?

2. Determine Hypothesis Status

• CONFIRMED: Evidence supports the hypothesis
• REFUTED: Evidence contradicts the hypothesis
• INCONCLUSIVE: Need more data

3. Identify Learnings

• What did we learn?
• What surprised us?
• What does this change about our understanding?

4. Generate New Questions

• What new questions emerged?
• What would we do differently?
• What should we explore next?

Analysis Quality Checklist

Analysis Template

## Analysis: [Experiment Name]

**Goal Comparison:**
| Success Criteria | Required | Actual | Status |
|-----------------|----------|--------|--------|
| [Metric 1] | [Value] | [Value] | ✅/❌ |
| [Metric 2] | [Value] | [Value] | ✅/❌ |

**Hypothesis Status:** [CONFIRMED / REFUTED / INCONCLUSIVE]

**Key Learnings:**
1. [Learning 1]
2. [Learning 2]
3. [Learning 3]

**Surprises:**
- [What we didn't expect]

**New Questions:**
- [Question that emerged]

**Implications for Next Iteration:**
- [How this changes our approach]

---

Phase 7: ITERATE - The Feedback Loop

The Iteration Process

1. Update World Model

• What do we now know that we didn't before?
• How does this change our understanding?
• What assumptions were validated or invalidated?

2. Refine Hypotheses

• Which hypotheses should we continue exploring?
• Which should we abandon?
• What new hypotheses emerged?

3. Plan Next Cycle

• What's the next experiment?
• What's the new goal (if it changed)?
• What's the new observation needed?

4. Decide: Continue or Ship

• Have we learned enough?
• Is it time to ship and learn from production?
• Is further experimentation valuable?

When to Stop Iterating

Stop when:

• Goal is achieved
• Further iteration has diminishing returns
• Time/resource constraints require shipping
• The problem has changed (pivot)

Don't stop because:

• We're frustrated
• We want to avoid negative results
• The experiment is taking longer than expected
• We're attached to our hypothesis

The Iteration Anti-Pattern

Infinite iteration without shipping is as bad as shipping without iteration.

The goal is PROGRESS, not perfection. At some point, ship and learn from production.

---

Meta: Science Applied to Science

This methodology applies to itself.

Goal: Improve how we solve problems

Observe: Track which experiments produce insights

Hypothesize: Try different experiment designs

Experiment: Apply methodology variations

Measure: Track learning rate, success rate

Analyze: Compare methodology versions

Iterate: Evolve the methodology itself

---

Integration with PAI Skills

Development Skill

Science manifests as TDD:

• Goal = Test (defines expected behavior)
• Hypothesize = Implementation approach
• Experiment = Write code
• Measure = Run tests
• Analyze = All green?
• Iterate = Refactor

Evals Skill

Science manifests as prompt evaluation:

• Goal = Eval criteria
• Hypothesize = Prompt variations
• Experiment = Run evals
• Measure = Scores
• Analyze = Compare to baseline
• Iterate = Refine prompt

Research Skill

Science manifests as investigation:

• Goal = Research question
• Observe = Literature review
• Hypothesize = Possible answers
• Experiment = Investigation
• Measure = Evidence gathering
• Analyze = Synthesize findings
• Iterate = Deeper questions

Council Skill

Science manifests as collaborative exploration:

• Observe = Initial positions
• Hypothesize = Multiple perspectives
• Experiment = Debate rounds
• Analyze = Synthesis
• Iterate = Refined positions

---

Summary

The scientific method is not complicated. It's:

1. Know what you want (Goal) 2. Understand where you are (Observe) 3. Guess how to get there (Hypothesize) 4. Try your guesses (Experiment) 5. See what happened (Measure) 6. Compare to what you wanted (Analyze) 7. Adjust and repeat (Iterate)

The power is in the discipline of the cycle, not any individual step.

Apply this everywhere. To code. To products. To ideas. To life.

This is how progress happens.

The Science Protocol

Skills don't call Science. They implement it.

This document defines the Science Protocol - the interface that skills embody when they apply systematic, evidence-based iteration to their domain.

---

The Core Insight

Science is not a service that other skills invoke. It's a protocol they implement.

Like how TCP/IP defines communication patterns without caring what's being communicated, Science defines the iteration pattern without caring what domain it's applied to.

The Distinction:

---

The Protocol Interface

Any Science-compliant workflow exhibits these behaviors:

interface ScienceProtocol {
  // Phase 0: What are we trying to achieve?
  goal: {
    successCriteria: string[];      // How will we know we succeeded?
    measurableIndicators: Metric[]; // What numbers define success?
    constraints: string[];          // What are our limits?
    antiGoals?: string[];           // What are we NOT trying to do?
  };

  // Phase 1: What is the current state?
  observe(): Observation;

  // Phase 2: What might work? (MUST be plural)
  hypothesize(): Hypothesis[];  // Minimum 3

  // Phase 3-4: Design and run tests
  experiment(hypothesis: Hypothesis): ExperimentResult;

  // Phase 5-6: What happened? How does it compare?
  measure(result: ExperimentResult): Measurement;
  analyze(measurement: Measurement, goal: Goal): Analysis;

  // Phase 7: What's next?
  iterate(analysis: Analysis): NextAction;
}

---

How Skills Implement Science

Skills don't import Science. They declare compliance through structure and behavior.

Development (TDD)

skill: Development
implements: Science
mapping:
  goal: Test assertion defines expected behavior
  observe: Read existing code, understand context
  hypothesize: Generate implementation approaches
  experiment: Write code following TDD Red-Green-Refactor
  measure: Run tests, check coverage
  analyze: All green? Meets spec?
  iterate: Refactor, or move to next feature

The Cycle:

Write Test (GOAL) → Run Test (OBSERVE - it fails) →
Think of Fix (HYPOTHESIZE) → Write Code (EXPERIMENT) →
Run Test (MEASURE) → Pass/Fail? (ANALYZE) →
Refactor or Continue (ITERATE)

Evals (Prompt Optimization)

Evals is the canonical Science implementation for prompts.

When doing prompt experiments, DO NOT build ad-hoc evaluation. Invoke Evals directly.

skill: Evals
implements: Science
mapping:
  goal: Eval criteria (threshold, metrics) - pre-committed before running
  observe: Baseline prompt performance measurement
  hypothesize: Prompt variations with FALSIFIABLE claims
  experiment: Run eval suite with position swapping, multi-judge panels
  measure: Scores, SEM, confidence intervals
  analyze: Compare to pre-committed threshold, statistical significance
  iterate: Refine prompt, try next hypothesis, or declare success

The Cycle:

Define Use Case (GOAL) → Run Baseline (OBSERVE) →
Generate Variants (HYPOTHESIZE) → Compare Prompts (EXPERIMENT) →
Collect Scores (MEASURE) → Statistical Analysis (ANALYZE) →
Pick Winner or Refine (ITERATE)

Evals adds domain-specific rigor:

• Position swapping - mitigates LLM positional bias
• Multi-judge panels - reduces model quirks (7x cheaper than single large judge)
• Reasoning-first scoring - 13%+ accuracy improvement
• Paradigm check - when to question the entire eval framework

Research (Investigation)

skill: Research
implements: Science
mapping:
  goal: Research question to answer
  observe: Literature review, existing knowledge
  hypothesize: Possible answers to investigate
  experiment: Deep investigation of each hypothesis
  measure: Evidence gathering, source quality
  analyze: Synthesize findings
  iterate: Deeper questions emerge

Council (Debate)

skill: Council
implements: Science
mapping:
  goal: Decision to make
  observe: Initial positions from each perspective
  hypothesize: Multiple viewpoints as competing hypotheses
  experiment: Structured debate rounds
  measure: Arguments, counterarguments
  analyze: Synthesis, convergence points
  iterate: Refined positions until consensus

Worktree (Parallel Experiments)

skill: Development/Worktree
implements: Science
mapping:
  goal: Find best implementation approach
  observe: Base branch state
  hypothesize: Named variant approaches
  experiment: Parallel development in worktrees
  measure: Code quality, test results, performance
  analyze: Compare implementations
  iterate: Pick winner, merge, cleanup

RedTeam (Adversarial Analysis)

skill: RedTeam
implements: Science
mapping:
  goal: Strongest possible argument/product
  observe: Original argument or design
  hypothesize: Attack vectors (32 agents generate them)
  experiment: Multi-agent assault on every angle
  measure: Vulnerability scores
  analyze: Steelman response to each attack
  iterate: Refined, attack-hardened output

---

Protocol Compliance Markers

Skills can declare compliance in their SKILL.md frontmatter:

---
name: Development
description: ...
implements: Science
science_cycle_time: micro  # micro, meso, macro
---

This is documentation of the mapping, not runtime coupling.

---

Integration Points (When Science Orchestrates)

Science explicitly orchestrates only when:

1. Cross-domain problems - The problem spans multiple skills 2. Explicit experimentation requests - "Try 3 approaches and compare" 3. Novel problems - No existing skill pattern fits

Orchestration Flow:

User: "Figure out best auth approach"
           │
           ▼
┌─────────────────────────────────────────┐
│           SCIENCE ORCHESTRATES           │
└─────────────────────────────────────────┘
           │
           ├──► Goal: Research Skill helps define success criteria
           │
           ├──► Observe: Research Skill gathers context
           │
           ├──► Hypothesize: Council Skill generates approaches
           │
           ├──► Experiment: Worktree runs parallel implementations
           │
           ├──► Measure: Evals Skill compares approaches
           │
           ├──► Analyze: Science synthesizes results
           │
           └──► Iterate: Pick winner or refine

---

Handoff Boundaries

What Science Provides to Skills

What Skills Provide to Science

---

Scale-Appropriate Protocol Adherence

Key Principle: Don't over-formalize micro-scale work. Let the protocol be implicit until you need its structure.

---

The Self-Reference Property

Science applies to itself.

The Science skill MUST be able to improve its own methodology using the scientific method:

1. Goal: Improve problem-solving effectiveness 2. Observe: Track experiment outcomes across domains 3. Hypothesize: Try methodology variations 4. Experiment: Apply variations to real problems 5. Measure: Learning rate, success rate, iteration speed 6. Analyze: Compare methodology versions 7. Iterate: Update Science documentation

This is tracked in Meta/ directory.

---

Bidirectional Skill Integration

Science → Evals

When Science encounters prompt optimization:

1. Recognize the domain - "This is a prompt experiment" 2. Delegate to specialist - Invoke Evals skill directly 3. Use Evals infrastructure - Don't reinvent position swapping, multi-judge, etc. 4. Accept Evals output - Scores, statistics, recommendations

The directive: If experimenting with prompts, use Evals. Period.

Evals → Science

When Evals gets stuck or needs broader framing:

1. Paradigm check trigger - 3+ iterations without improvement 2. Invoke Science explicitly - Science/Workflows/StructuredInvestigation.md 3. Question the frame - Is the use case defined correctly? Are we measuring the right thing? 4. Return with clarity - New hypothesis, new direction, or confirmation to continue

The escalation path: Evals runs implicitly as Science. When implicit fails, go explicit.

When Science Orchestrates Across Skills

For problems spanning prompt AND code AND research:

User: "Improve the summarization pipeline"
           │
           ▼
┌─────────────────────────────────────────┐
│           SCIENCE ORCHESTRATES           │
└─────────────────────────────────────────┘
           │
           ├──► Goal: What does "better" mean? (Science/DefineGoal)
           │
           ├──► Observe: Current performance across dimensions
           │
           ├──► Hypothesize: Multiple improvement paths
           │    ├─ Better prompt (→ Evals)
           │    ├─ Better code (→ Development/Worktree)
           │    └─ Better architecture (→ Research)
           │
           ├──► Experiment: Parallel experiments via appropriate skills
           │
           ├──► Measure: Aggregate results from each skill
           │
           └──► Iterate: Pick winner, next cycle

Science orchestrates; specialist skills execute.

---

Summary

Science is not a skill you call. It's a pattern you embody.

• Skills implement the protocol independently
• No runtime coupling required
• Science orchestrates only for cross-domain coordination
• Scale determines formality level
• The protocol improves itself through its own application
• Evals is the canonical implementation for prompts
• When Evals stalls, escalate to explicit Science workflows

Science Templates

Reusable templates for structured scientific thinking. Copy and adapt as needed.

---

Goal Template

## Goal: [Name]

**Created:** [Date]
**Status:** [ACTIVE / ACHIEVED / ABANDONED]

### Outcome
[What will be true when we succeed - specific and observable]

### Success Criteria
**Quantitative:**
- [ ] [Metric 1]: [target]
- [ ] [Metric 2]: [target]

**Qualitative:**
- [ ] [Observable behavior 1]

### Thresholds
- **Minimum:** [Bare minimum win]
- **Target:** [What we're aiming for]
- **Stretch:** [Outstanding success]

### Constraints
- **Time:** [Deadline]
- **Resources:** [Budget, people]
- **Quality:** [Standards to maintain]

### Anti-Goals
- [Out of scope item 1]
- [Out of scope item 2]

Quick Version:

### Goal: [Name]
- **Success:** [One-sentence outcome]
- **Metric:** [Primary measurable target]
- **Deadline:** [When]
- **NOT doing:** [Key anti-goal]

---

Hypothesis Template

## Hypothesis: [Name]

**ID:** H[N]
**Status:** [UNTESTED / TESTING / CONFIRMED / REFUTED / INCONCLUSIVE]

### Claim
**Statement:** [Precise, testable statement]
**If-Then:** If [condition], then [outcome], because [mechanism]

### Rationale
1. [Reason 1]
2. [Reason 2]

### Falsification
**WRONG if:** [Specific observable outcome that disproves it]

### Testing
- **Cost:** [Time/resources]
- **Minimum Experiment:** [Briefest test description]
- **Confidence:** [Low/Med/High] - [Why]

Quick Version:

### H[N]: [Name]
- **Claim:** [What we think]
- **Because:** [Why]
- **Wrong if:** [Falsification]
- **Test cost:** [Low/Med/High]

---

Experiment Template

## Experiment: [Name]

**ID:** EXP-[N]
**Testing:** H[N]
**Status:** [DESIGNED / RUNNING / COMPLETE / ABORTED]

### Success Criteria (PRE-COMMITTED)
- **CONFIRMED if:** [Condition]
- **REFUTED if:** [Condition]
- **INCONCLUSIVE if:** [Condition]

### Variables
- **Independent (we change):** [Variable]
- **Dependent (we measure):** [Metric]
- **Controls (constant):** [Variables]

### Method
1. [Step 1]
2. [Step 2]
3. [Step 3]

### Data Collection
| Metric | How Measured | Target |
|--------|--------------|--------|
| [Name] | [Method]     | [Value]|

### Timeline
- **Duration:** [How long]
- **Stopping:** [When to stop early]

Quick Version:

### EXP: [Name]
- **Testing:** H[N]
- **Method:** [Brief description]
- **Measuring:** [Primary metric]
- **Success if:** [Condition]
- **Failure if:** [Condition]

---

Results Template

## Results: [Experiment Name]

**Experiment:** EXP-[N]
**Hypothesis:** H[N]
**Date:** [When run]

### Summary
**Verdict:** [CONFIRMED / REFUTED / INCONCLUSIVE]
**One-Sentence:** [What happened]

### Data
| Metric | Baseline | Result | Change | Target | Met? |
|--------|----------|--------|--------|--------|------|
| [Name] | [Value]  | [Value]| [Delta]| [Value]| Y/N  |

### Observations
- [Key observation 1]
- [Key observation 2]

### Analysis
| Criterion | Required | Achieved | Status |
|-----------|----------|----------|--------|
| [Crit 1]  | [Value]  | [Value]  | Y/N    |

### Learnings
1. [Key insight]
2. [Surprise finding]

### Next Steps
- [Recommended action]
- [Follow-up question]

Quick Version:

### Results: [Experiment]
- **Tested:** H[N]
- **Verdict:** [CONFIRMED/REFUTED/INCONCLUSIVE]
- **Key Metric:** [Value] vs [Target]
- **Learning:** [Main insight]
- **Next:** [Action]

Analyze Results Workflow

Phase 6 of the Scientific Cycle

Compare results to the goal. This workflow ensures honest, objective analysis without confirmation bias.

---

The Core Question

"Did we achieve what we set out to achieve?"

Compare results to PRE-DEFINED success criteria. Not to your expectations. Not to what you hoped for. To what you committed to before starting.

---

The Analysis Process

Step 1: Compare to Success Criteria

Pull up the goals defined in DefineGoal. Compare directly.

## Goal Comparison

| Success Criterion | Required | Actual | Status |
|-------------------|----------|--------|--------|
| [Criterion 1]     | [Value]  | [Value] | ✅/❌ |
| [Criterion 2]     | [Value]  | [Value] | ✅/❌ |
| [Criterion 3]     | [Value]  | [Value] | ✅/❌ |

**Overall Goal Status:** [ACHIEVED / PARTIALLY ACHIEVED / NOT ACHIEVED]

Step 2: Determine Hypothesis Status

For each hypothesis tested:

## Hypothesis Verdicts

### H1: [Name]
**Status:** [CONFIRMED / REFUTED / INCONCLUSIVE]

**Evidence:**
- [What the data showed]
- [How it relates to falsification criteria]

**Confidence:** [High/Medium/Low] because [reasoning]

### H2: [Name]
[Same structure]

Decision Framework:

Step 3: Identify Learnings

What did we learn, beyond the hypothesis verdict?

## Key Learnings

**What We Learned:**
1. [Insight that changes our understanding]
2. [Something we now know that we didn't before]

**What Surprised Us:**
1. [Unexpected finding 1]
2. [Unexpected finding 2]

**What Changed About Our Mental Model:**
- [Before: We thought X]
- [After: We now understand Y]

Step 4: Acknowledge Limitations

Be honest about what the data can and cannot tell us.

## Limitations

**Methodological Limitations:**
- [Limitation 1 - how it affects conclusions]
- [Limitation 2 - how it affects conclusions]

**Generalizability:**
- [Can we generalize these results? Why/why not?]

**Alternative Interpretations:**
- [Other explanations for the data]
- [What would change our interpretation?]

Step 5: Generate New Questions

Good analysis raises new questions.

## New Questions

**Questions That Emerged:**
1. [New question from the findings]
2. [Follow-up investigation needed]

**Hypotheses for Next Cycle:**
- [New hypothesis suggested by data]
- [Refinement of existing hypothesis]

---

Confirmation Bias Countermeasures

1. Check Against Pre-Registered Criteria

Did you define success criteria before gathering data? Compare ONLY to those.

If you find yourself saying "Well, we didn't hit the metric, but..." — that's a red flag.

2. Consider the Opposite

Ask: "If I wanted to argue the opposite conclusion, what would I say?"

3. Seek Disconfirming Interpretation

Ask: "What interpretation of this data would make me LESS confident in my conclusion?"

4. Steel-Man the Failure

If hypothesis was refuted, don't dismiss. Ask: "What can we learn from this failure?"

5. Red Team Your Analysis

Use RedTeam skill to attack your conclusions.

---

Statistical Considerations

When sample sizes and quantitative data warrant it:

## Statistical Analysis

**Test Used:** [Name and justification]

**Results:**
- Test statistic: [Value]
- p-value: [Value]
- Effect size: [Value and interpretation]
- Confidence interval: [Range]

**Interpretation:**
- [What this means in practical terms]

**Caveats:**
- [Statistical limitations]
- [Assumption violations if any]

Rules of Thumb:

• p < 0.05 is conventional significance, but effect size matters more
• Large samples can make tiny effects "significant"
• Small samples can hide real effects
• Confidence intervals tell you more than p-values

---

Analysis Quality Checklist

---

Template

# Analysis: [Experiment Name]

**Analysis Date:** [When]
**Analyst:** [Who]

## Goal Comparison
| Criterion | Required | Achieved | Status |
|-----------|----------|----------|--------|
| [Crit 1]  | [Value]  | [Value]  | ✅/❌  |

**Overall:** [ACHIEVED / NOT ACHIEVED]

## Hypothesis Verdicts
- **H1:** [CONFIRMED/REFUTED/INCONCLUSIVE] - [brief reasoning]
- **H2:** [CONFIRMED/REFUTED/INCONCLUSIVE] - [brief reasoning]

## Key Learnings
1. [Main insight]
2. [Secondary insight]

## Surprises
- [What we didn't expect]

## Limitations
- [Methodological concern]

## New Questions
1. [Follow-up question]

## Recommendations
- [What to do next]

---

Common Pitfalls

Pitfall 1: Moving Goalposts

❌ "We didn't hit the number, but qualitatively it's better..." ✅ "We didn't hit the number. That's a failure. What did we learn?"

Pitfall 2: Cherry-Picking

❌ Highlighting only data that supports the hypothesis ✅ Presenting all data, including contradictory evidence

Pitfall 3: Over-Interpreting

❌ Drawing sweeping conclusions from limited data ✅ Matching confidence to evidence strength

Pitfall 4: Dismissing Failures

❌ "The experiment didn't work, let's move on" ✅ "The experiment failed - why? What does that teach us?"

Pitfall 5: Confirmation Interpretation

❌ Interpreting ambiguous data as supporting your belief ✅ Explicitly considering alternative interpretations

---

Integration with Next Phase

Analysis feeds into Iterate.

Based on analysis:

• Goal achieved → Document, ship, move on
• Hypothesis confirmed → Implement and monitor
• Hypothesis refuted → Try next hypothesis or generate new ones
• Inconclusive → Design better experiment
• Paradigm shift needed → Return to fundamentals

The cycle continues until the goal is achieved or explicitly abandoned.

Define Goal Workflow

Phase 0 of the Scientific Cycle

You cannot judge experiments without success criteria. This workflow ensures crystal-clear goals before any investigation begins.

---

The Core Question

"What does success look like?"

If you can't answer this precisely, stop. Define it before proceeding.

---

The Goal Specification Process

Step 1: Outcome Statement

What will exist or change when we succeed?

## Outcome
[Describe the end state as if it already happened]

Examples:

• ❌ "Make the app faster" (vague)
• ✅ "Page load time is under 1 second for 95% of users"

• ❌ "Fix the bug" (vague)
• ✅ "Users can complete checkout without encountering the 500 error"

• ❌ "Improve the content" (vague)
• ✅ "Blog post generates 50+ comments and 1000+ shares"

Step 2: Measurable Indicators

How will we KNOW we succeeded?

## Indicators

**Quantitative:**
- [Metric 1]: [target value]
- [Metric 2]: [target value]

**Qualitative:**
- [Observable behavior 1]
- [Observable behavior 2]

**Leading (early signals):**
- [What we'll see first if on track]

**Lagging (final confirmation):**
- [What confirms we truly succeeded]

Step 3: Success Thresholds

What level constitutes success?

## Thresholds

**Minimum (acceptable):**
- [Bare minimum to call it a win]

**Target (goal):**
- [What we're actually aiming for]

**Stretch (exceeds expectations):**
- [Outstanding success looks like]

This prevents binary thinking. Sometimes "partial success" is valuable information.

Step 4: Constraints

What are our boundaries?

## Constraints

**Time:** [Deadline or time budget]
**Resources:** [Budget, people, tools available]
**Quality:** [Standards that must be maintained]
**Technical:** [System limitations]
**Ethical:** [What we won't do]

Step 5: Anti-Goals

What are we explicitly NOT trying to do?

## Anti-Goals
- [Out of scope item 1]
- [Out of scope item 2]
- [Out of scope item 3]

This prevents scope creep and clarifies trade-offs.

---

Goal Quality Checklist

Before proceeding, verify:

If any box is unchecked, refine the goal.

---

Pre-Registration Principle

Lock your success criteria BEFORE gathering evidence.

This is crucial for avoiding confirmation bias. Once you've defined success:

1. Write it down 2. Timestamp it 3. Don't change it post-hoc

If you find yourself wanting to move goalposts after seeing data, that's a red flag. Either your original goal was wrong (and you should acknowledge that), or you're falling into confirmation bias.

---

Template

# Goal Specification: [Project/Problem Name]

**Date:** [When this was defined]
**Owner:** [Who is accountable]

## Outcome
[What will be true when we succeed]

## Success Criteria
- [ ] [Measurable criterion 1]
- [ ] [Measurable criterion 2]
- [ ] [Measurable criterion 3]

## Thresholds
- **Minimum:** [Acceptable level]
- **Target:** [Actual goal]
- **Stretch:** [Outstanding]

## Constraints
- **Time:** [Deadline]
- **Resources:** [Limits]
- **Quality:** [Standards]

## Anti-Goals
- [Not trying to do X]
- [Not trying to do Y]

## Pre-Registration
This goal specification was locked at [timestamp] and will not be modified after data collection begins.

---

Common Pitfalls

Pitfall 1: Vague Outcomes

❌ "Make it better" ✅ "Reduce load time from 3s to under 1s"

Pitfall 2: No Falsifiability

❌ "Improve user experience" (how would you fail?) ✅ "Increase task completion rate from 60% to 80%"

Pitfall 3: Moving Goalposts

❌ "Well, we didn't hit the metric, but qualitatively it feels better..." ✅ "We didn't hit the metric. That's a failure. What did we learn?"

Pitfall 4: Missing Constraints

❌ "Optimize the system" (with unlimited time and resources?) ✅ "Optimize the system within 2 weeks using existing infrastructure"

Pitfall 5: Scope Creep via Unstated Anti-Goals

❌ Project keeps expanding because boundaries weren't defined ✅ Explicitly stated: "We are NOT redesigning the entire auth system"

---

Integration with Next Phase

Once the goal is defined, proceed to GatherContext or GenerateHypotheses.

A well-defined goal enables:

• Focused observation (what to look for)
• Relevant hypotheses (what might achieve the goal)
• Meaningful experiments (tests that relate to success criteria)
• Honest analysis (clear comparison to targets)

Design Experiment Workflow

Phase 3 of the Scientific Cycle

Design the smallest experiment that meaningfully tests the hypothesis. This workflow ensures experiments are efficient, valid, and actionable.

---

The Core Principle

Minimum Viable Experiment

Not every experiment needs to be comprehensive. The goal is LEARNING, not perfection.

Ask:

• What's the fastest way to learn if this works?
• What's the cheapest way to get meaningful data?
• What would DISPROVE the hypothesis?
• What's the minimum sample size for confidence?

---

The Experiment Design Process

Step 1: Define Success Criteria

Before running, know what outcomes mean.

## Success Criteria

**Hypothesis CONFIRMED if:**
[Specific observable outcome that supports the hypothesis]

**Hypothesis REFUTED if:**
[Specific observable outcome that contradicts the hypothesis]

**INCONCLUSIVE if:**
[Conditions that mean we need more data]

The key insight: Define these BEFORE running the experiment. This prevents post-hoc rationalization.

Step 2: Identify Variables

## Variables

**Independent Variable (what we're changing):**
[The thing we're manipulating]

**Dependent Variable (what we're measuring):**
[The outcome we're observing]

**Control Variables (what we're holding constant):**
- [Variable 1]: held at [value]
- [Variable 2]: held at [value]

**Confounding Variables (what might skew results):**
- [Potential confounder 1]: mitigated by [strategy]
- [Potential confounder 2]: mitigated by [strategy]

Step 3: Design the Test

## Experiment Design

**Method:**
1. [Step 1]
2. [Step 2]
3. [Step 3]

**Control (baseline):**
[What we're comparing against]

**Treatment:**
[What we're testing]

**Measurement Approach:**
- Primary metric: [What we're measuring]
- How measured: [Collection method]
- When measured: [Timing]
- By whom: [Who collects]

Step 4: Plan Data Collection

## Data Collection Plan

**Data Points:**
- [Metric 1]: collected via [method]
- [Metric 2]: collected via [method]

**Sample Size:**
[How many data points needed]

**Duration:**
[How long to run]

**Collection Schedule:**
[When/how often to collect]

**Data Storage:**
[Where raw data will be preserved]

Step 5: Define Stopping Rules

## Stopping Rules

**Stop early if:**
- [Condition for early success]
- [Condition for early failure]
- [Safety condition]

**Continue until:**
- [Minimum duration/sample reached]

**Declare done when:**
- [Clear endpoint condition]

---

Experiment Quality Checklist

---

Domain-Specific Experiment Patterns

Code Experiments

Tool: Worktrees for parallel experiments

## Code Experiment Design

**Hypothesis:** [Implementation approach X will work]

**Branches:**
- control: Current implementation
- treatment-1: Approach A
- treatment-2: Approach B

**Test Method:**
- Run test suite on each branch
- Measure performance benchmarks
- Compare code complexity metrics

**Success Criteria:**
- All tests pass
- Performance meets threshold
- Complexity doesn't increase

Prompt Experiments

Tool: Evals Skill (MANDATORY)

For ALL prompt experiments, invoke the Evals skill directly.

Evals implements the Science Protocol for prompt engineering. Don't reinvent evaluation - use the battle-tested methodology.

Invocation:

→ Invoke Evals skill, ComparePrompts workflow
→ Follow Evals' Science Protocol Alignment checklist
→ Use Evals' statistical rigor (SEM, confidence intervals)

What Evals Provides:

• Position swapping (mitigates LLM positional bias)
• Multi-judge panels (reduces individual model quirks)
• Statistical significance testing
• Pre-commitment enforcement
• Paradigm check when stuck

## Prompt Experiment Design

**Hypothesis:** [Prompt variant X will perform better]
**Falsified if:** [Variant X performance ≤ baseline, or improvement < threshold]

**Variants (aim for 3+):**
- baseline: Current prompt
- variant-1: [Modified prompt A]
- variant-2: [Modified prompt B]
- variant-3: [Modified prompt C - different direction]

**Eval Method (via Evals skill):**
- Run eval suite with each variant
- N=100 samples per variant
- Position swapping enabled
- Measure accuracy, format compliance

**Success Criteria (pre-committed):**
- Variant exceeds baseline by > 5%
- Statistical significance p < 0.05
- No regression on secondary metrics

CLI Quick Reference:

# Run prompt comparison via Evals skill
bun run ~/.claude/skills/Evals/EvalServer/cli-run.ts \
  --use-case <name> \
  --compare prompts/baseline.md prompts/variant-1.md \
  --position-swap

Feature Experiments

Tool: Feature flags, A/B tests

## Feature Experiment Design

**Hypothesis:** [Feature X will improve metric Y]

**Groups:**
- control: 50% of users (no feature)
- treatment: 50% of users (with feature)

**Measurement:**
- Primary: [Conversion rate]
- Secondary: [Engagement, retention]

**Duration:** 2 weeks minimum

**Success Criteria:**
- Lift > 5% with 95% confidence
- No degradation in secondary metrics

Research Experiments

Tool: Parallel research agents

## Research Experiment Design

**Hypothesis:** [Theory X explains phenomenon Y]

**Investigation Approach:**
- Agent 1: Search for supporting evidence
- Agent 2: Search for contradicting evidence
- Agent 3: Search for alternative explanations

**Evidence Evaluation:**
- Source quality rating
- Recency
- Consensus level

**Success Criteria:**
- Preponderance of high-quality evidence
- No strong contradicting evidence

---

Parallel Experiment Design

When hypotheses are independent, design experiments to run simultaneously.

Hypothesis 1 ─────► Experiment 1 ─────► Results 1 ─┐
                                                   │
Hypothesis 2 ─────► Experiment 2 ─────► Results 2 ─┼─► Analysis
                                                   │
Hypothesis 3 ─────► Experiment 3 ─────► Results 3 ─┘

Benefits:

• Faster time to insight
• Reduces sequential bias
• Enables direct comparison

Requirements:

• Experiments must be independent
• Resources available for parallel execution
• Results can be meaningfully compared

---

Template

# Experiment Design: [Name]

**Testing Hypothesis:** [Which hypothesis]
**Date:** [When designed]

## Success Criteria
- CONFIRMED if: [condition]
- REFUTED if: [condition]
- INCONCLUSIVE if: [condition]

## Variables
- **Independent:** [What we change]
- **Dependent:** [What we measure]
- **Controls:** [What we hold constant]

## Method
1. [Step 1]
2. [Step 2]
3. [Step 3]

## Data Collection
- **Metrics:** [What we collect]
- **Sample Size:** [How many]
- **Duration:** [How long]

## Stopping Rules
- Stop early if: [condition]
- Continue until: [condition]

## Resources Needed
- [Resource 1]
- [Resource 2]

---

Common Pitfalls

Pitfall 1: Over-Engineering

❌ Building elaborate test infrastructure for a simple question ✅ Finding the minimum viable test that answers the question

Pitfall 2: Confirmation Design

❌ Designing tests that can only succeed ✅ Designing tests that could definitively fail

Pitfall 3: Moving Targets

❌ Changing success criteria after seeing results ✅ Pre-committing to criteria before running

Pitfall 4: Ignoring Confounders

❌ Assuming all differences are due to the treatment ✅ Identifying and controlling for other variables

Pitfall 5: Insufficient Sample

❌ Declaring results from N=3 ✅ Understanding minimum sample for statistical validity

---

Integration with Next Phases

A well-designed experiment enables:

• Clear execution - steps are defined
• Objective measurement - metrics are specified
• Honest analysis - success criteria are pre-committed
• Efficient iteration - learn quickly, move on

Full Cycle Workflow

Level 3 Science - Research-grade methodology for macro-scale problems

For problems spanning weeks to months, requiring statistical rigor, stakeholder review, or external publication. This is the complete scientific method with all safeguards.

---

When to Use

• Product strategy or architectural decisions
• Research for publication or sharing
• Multi-stakeholder decisions requiring documented methodology
• Long-running experiments (A/B tests, pilot programs)
• When you need to convince skeptics with rigorous evidence
• Anything where getting it wrong is very expensive

Anti-Triggers (Don't Use When)

• Problem can be solved in a day
• Quick iteration is more valuable than rigor
• Premature formalism would delay learning
• The stakes don't justify the overhead

---

The Complete Scientific Cycle

Phase 0: Research Protocol Design

Before starting, document the methodology itself.

# Research Protocol: [Project Name]

**Principal Investigator:** [Who owns this]
**Date:** [Start date]
**Expected Duration:** [Timeline]

## Research Question
[The fundamental question we're trying to answer]

## Significance
[Why this matters - what decisions hinge on the answer?]

## Methodology Overview
[High-level approach]

## Ethical Considerations
[Any concerns about how we're testing]

## Peer Review
[Who will review methodology before execution?]

## Version History
| Version | Date | Changes |
|---------|------|---------|
| 1.0     | Date | Initial protocol |

Phase 1: Goal Definition with Pre-Registration

Pre-commit to success criteria before gathering evidence.

This prevents moving goalposts and confirmation bias.

## Pre-Registered Goal Specification

**Primary Outcome:**
[The main thing we're measuring]

**Success Criteria (PRE-REGISTERED - cannot change post-hoc):**
- [ ] [Quantitative criterion 1 with exact threshold]
- [ ] [Quantitative criterion 2 with exact threshold]
- [ ] [Qualitative criterion with clear rubric]

**Secondary Outcomes:**
- [Exploratory measurement 1]
- [Exploratory measurement 2]

**Statistical Significance Threshold:**
[e.g., p < 0.05, or Bayesian equivalent]

**Minimum Effect Size:**
[What delta matters practically, not just statistically?]

**Stopping Rules:**
- Stop early if: [condition]
- Continue until: [condition]

**Pre-Registration Timestamp:** [When this was locked]
**Pre-Registration Witness:** [Who reviewed before data collection]

Phase 2: Comprehensive Context Gathering

Literature review and baseline establishment.

## Context Report

### Literature Review
| Source | Key Finding | Relevance | Quality |
|--------|-------------|-----------|---------|
| [Ref]  | [Finding]   | [How it applies] | [High/Med/Low] |

### Prior Art
- [What exists in this space]
- [What has been tried before]
- [Why previous attempts succeeded/failed]

### Baseline Measurements
| Metric | Current Value | Collection Method | Confidence |
|--------|---------------|-------------------|------------|
| [Name] | [Value ± error] | [How measured] | [High/Med/Low] |

### Stakeholder Map
| Stakeholder | Interest | Influence | Concerns |
|-------------|----------|-----------|----------|
| [Name]      | [What they care about] | [High/Med/Low] | [Worries] |

### Constraints Analysis
- **Time:** [Hard deadline or flexible?]
- **Budget:** [Resource limits]
- **Technical:** [System constraints]
- **Political:** [Organizational dynamics]
- **Ethical:** [Boundaries we won't cross]

Phase 3: Hypothesis Generation with Devil's Advocate

Multiple hypotheses with active attempts to disprove each.

## Hypothesis Set

### Hypothesis 1: [Name]

**Claim:** [Precise, testable statement]

**Theoretical Basis:** [Why might this be true?]

**Supporting Evidence:**
1. [Evidence point 1]
2. [Evidence point 2]

**Challenging Evidence (Devil's Advocate):**
1. [Counter-evidence 1]
2. [Counter-evidence 2]

**Falsification Criteria:**
- This hypothesis is FALSE if: [specific observable outcome]

**Bayesian Prior:** [Initial probability estimate before testing]

**Competing Hypotheses:** [Which other hypotheses conflict with this one?]

---
[Repeat for minimum 5 hypotheses]

Hypothesis Quality Checklist:

• [ ] Is it specific enough to be testable?
• [ ] Is it falsifiable (could evidence prove it wrong)?
• [ ] Have we seriously considered counter-evidence?
• [ ] Is it distinct from other hypotheses?
• [ ] Would confirming it actually tell us what to do?

Phase 4: Experiment Design with Controls

Rigorous experimental design.

## Experiment Design: [Name]

### Variables
**Independent Variable (what we're changing):**
[Precise definition]

**Dependent Variable (what we're measuring):**
[Precise definition]

**Control Variables (what we're holding constant):**
- [Variable 1]: held at [value]
- [Variable 2]: held at [value]

**Confounding Variables (what might skew results):**
- [Confounder 1]: mitigated by [strategy]
- [Confounder 2]: mitigated by [strategy]

### Control Group
**Design:** [How we're establishing baseline]
**Size:** [N required for statistical power]

### Treatment Group(s)
**Design:** [What intervention/change]
**Size:** [N required]

### Randomization
[How subjects/cases are assigned to groups]

### Blinding
- Single-blind: [who is blinded]
- Double-blind: [if applicable]

### Power Analysis
**Effect size expected:** [Minimum delta to detect]
**Sample size required:** [N]
**Power:** [e.g., 80%]

### Data Collection Protocol
1. [Step 1 - exactly how to collect]
2. [Step 2]
3. [Step 3]

### Analysis Plan (PRE-REGISTERED)
[Exactly what statistical tests will be run]
[How will we handle missing data?]
[What corrections for multiple comparisons?]

Phase 5: Execution with Audit Trail

Run experiments with complete documentation.

## Execution Log

### Experiment: [Name]
**Started:** [Timestamp]
**Ended:** [Timestamp]
**Executed By:** [Who]

### Protocol Deviations
| Date | Deviation | Reason | Impact Assessment |
|------|-----------|--------|-------------------|
| [Date] | [What changed] | [Why] | [How it affects validity] |

### Data Collection Log
| Timestamp | Data Point | Value | Collector | Notes |
|-----------|-----------|-------|-----------|-------|
| [Time]    | [Metric]  | [Value] | [Who] | [Context] |

### Anomalies Observed
- [Unexpected event 1]: [Description and impact]
- [Unexpected event 2]: [Description and impact]

### Raw Data Location
[Path to preserved raw data - never modify raw data]

Phase 6: Statistical Analysis

Rigorous analysis with appropriate methods.

## Statistical Analysis

### Descriptive Statistics
| Group | N | Mean | SD | Median | Range |
|-------|---|------|----|---------| ------|
| Control | [N] | [M] | [SD] | [Med] | [Min-Max] |
| Treatment | [N] | [M] | [SD] | [Med] | [Min-Max] |

### Inferential Statistics
**Test Used:** [Name of test and justification]
**Results:**
- Test statistic: [Value]
- p-value: [Value]
- Confidence interval: [Range]
- Effect size: [Value and interpretation]

### Assumption Checks
- Normality: [Met/Violated] - [How addressed]
- Homogeneity of variance: [Met/Violated] - [How addressed]
- Independence: [Met/Violated] - [How addressed]

### Bayesian Analysis (if applicable)
**Prior:** [Distribution]
**Posterior:** [Updated distribution]
**Bayes Factor:** [Value and interpretation]

### Sensitivity Analysis
[How robust are results to different assumptions?]

Phase 7: Analysis Against Pre-Registered Goals

Honest comparison - no moving goalposts.

## Results vs. Pre-Registered Criteria

### Primary Outcome
| Pre-Registered Criterion | Required | Achieved | Status |
|-------------------------|----------|----------|--------|
| [Criterion 1]           | [Value]  | [Value]  | ✅/❌  |
| [Criterion 2]           | [Value]  | [Value]  | ✅/❌  |

### Hypothesis Verdicts
| Hypothesis | Prior | Posterior | Verdict | Confidence |
|------------|-------|-----------|---------|------------|
| H1         | [%]   | [%]       | CONFIRMED/REFUTED | [High/Med/Low] |
| H2         | [%]   | [%]       | CONFIRMED/REFUTED | [High/Med/Low] |

### Unexpected Findings
[Results that emerged but weren't pre-registered]
*Note: These are exploratory and require replication*

### Limitations
1. [Methodological limitation 1]
2. [Methodological limitation 2]
3. [Generalizability concerns]

### Alternative Interpretations
[What other explanations for results are possible?]

Phase 8: Iteration Planning

Based on findings, plan next steps.

## Iteration Plan

### If Primary Goal Met
- [ ] Document methodology for future reference
- [ ] Communicate findings to stakeholders
- [ ] Implement winning approach
- [ ] Set up ongoing monitoring

### If Primary Goal Not Met
- [ ] Analyze why hypotheses failed
- [ ] Generate new hypotheses based on learnings
- [ ] Design next cycle experiments
- [ ] Consider paradigm shift (is the whole frame wrong?)

### Knowledge Transfer
**Key Learnings:**
1. [Generalizable insight 1]
2. [Generalizable insight 2]

**Methodology Improvements for Next Time:**
1. [Process improvement 1]
2. [Process improvement 2]

**Follow-Up Research Questions:**
1. [New question emerged 1]
2. [New question emerged 2]

---

Artifacts Produced

A Full Cycle produces these documentation artifacts:

1. Research Protocol - Pre-registered methodology 2. Context Report - Literature review and baseline 3. Hypothesis Document - All hypotheses with priors 4. Experiment Design - Detailed methodology 5. Execution Log - Complete audit trail 6. Statistical Report - Analysis results 7. Findings Report - Conclusions and recommendations 8. Iteration Plan - Next steps

---

Peer Review Checkpoints

---

When to Pivot or Stop

Pivot the entire approach when:

• Fundamental assumption proves false
• Environment changed significantly
• Better opportunity emerged
• Stakeholder priorities shifted

Stop early when:

• Clear failure with no recovery path
• Resources better spent elsewhere
• Ethical concerns emerged
• Goal achieved early

Document the decision: Even pivots and stops generate learnings.

Generate Hypotheses Workflow

Phase 2 of the Scientific Cycle

Never test just one idea. This workflow ensures you generate multiple competing hypotheses before experimentation.

---

The Plurality Principle

NEVER generate just one hypothesis.

Single-hypothesis thinking leads to:

• Confirmation bias - you'll see evidence that supports it
• Missed alternatives - better solutions exist
• Overinvestment - sunk cost in one approach
• Narrow framing - the real solution might be orthogonal

Minimum: 3 hypotheses. Better: 5-10 for important problems.

---

The Hypothesis Generation Process

Step 1: Divergent Phase (Quantity > Quality)

Generate as many ideas as possible without judgment.

Techniques:

• Brainstorming - rapid idea generation
• Reverse brainstorming - "How could we make this worse?"
• Analogy - "How do other domains solve this?"
• Constraint removal - "If we had unlimited [time/money/expertise]..."
• Orthogonal thinking - "What if the entire frame is wrong?"

## Raw Hypothesis List
1. [Idea 1]
2. [Idea 2]
3. [Idea 3]
...
[Generate 10-20 before filtering]

Use Council Skill for multi-perspective brainstorming. Use parallel agents to generate ideas from different angles.

Step 2: Categorization Phase

Group hypotheses by type and approach.

## Hypothesis Categories

**Root Cause Hypotheses:**
- [The problem is X because of Y]

**Solution Hypotheses:**
- [Approach A would solve this]
- [Approach B would solve this]

**Paradigm Hypotheses:**
- [The entire framing is wrong because...]

**Null Hypothesis:**
- [Nothing is actually wrong / current state is optimal]

Step 3: Refinement Phase

Make each hypothesis specific and testable.

For each hypothesis:

## Hypothesis: [Short Name]

**Claim:** [Precise statement of what we believe]

**Rationale:** [Why we think this might be true]

**Testable Prediction:** If this hypothesis is correct, then [observable outcome]

**Falsification Criteria:** This hypothesis is WRONG if [observable outcome]

**Cost to Test:** [Time/resources needed]

**Confidence Level:** [Low/Medium/High] - [Why]

Step 4: Prioritization Phase

Rank hypotheses for testing order.

## Prioritized Hypothesis List

| Rank | Hypothesis | Likelihood | Impact | Test Cost | Priority Score |
|------|-----------|------------|--------|-----------|----------------|
| 1    | [Name]    | [H/M/L]    | [H/M/L] | [H/M/L]  | [L×I/C]       |
| 2    | [Name]    | [H/M/L]    | [H/M/L] | [H/M/L]  | [Score]       |
| 3    | [Name]    | [H/M/L]    | [H/M/L] | [H/M/L]  | [Score]       |

Priority Formula: (Likelihood × Impact) / Cost to Test

Test cheap, high-likelihood hypotheses first.

---

Hypothesis Quality Checklist

For each hypothesis, verify:

---

The Falsification Focus

Don't ask "How can I prove I'm right?" Ask "How can I prove I'm wrong?"

For each hypothesis, complete this sentence:

"This hypothesis would be DISPROVEN if [specific observable outcome]"

If you can't complete this sentence, the hypothesis isn't testable.

---

Confirmation Bias Countermeasures

1. Pre-Mortem

Before testing: "Imagine this hypothesis failed completely. What went wrong?"

2. Devil's Advocate

Actively argue AGAINST your favorite hypothesis.

3. Seek Disconfirming Evidence

For each hypothesis, ask: "What evidence would make me LESS confident?"

4. Red Team

Use RedTeam skill to attack your hypotheses from 32 angles.

5. Consider the Null

Always include: "Maybe nothing is wrong" or "Maybe the current approach is optimal"

---

Multi-Agent Hypothesis Generation

For important problems, use parallel agents:

Agent 1: "Generate hypotheses from a technical perspective"
Agent 2: "Generate hypotheses from a user perspective"
Agent 3: "Generate hypotheses that challenge our assumptions"
Agent 4: "Generate orthogonal hypotheses from analogous domains"

Merge and deduplicate the results.

---

Template

# Hypothesis Document: [Problem Name]

**Goal:** [Reference to defined goal]
**Date:** [When generated]

## Hypothesis Set

### H1: [Name]
**Claim:** [What we believe]
**Rationale:** [Why we think this]
**Evidence For:** [Supporting data]
**Evidence Against:** [Challenging data]
**Falsification:** WRONG if [outcome]
**Test Cost:** [Time/resources]
**Confidence:** [L/M/H]

### H2: [Name]
[Same structure]

### H3: [Name]
[Same structure]

[Minimum 3, preferably 5+]

## Prioritization
1. [H#] - highest priority because [reason]
2. [H#] - second because [reason]
3. [H#] - third because [reason]

## Null Hypothesis
[What if nothing needs to change?]

---

Common Pitfalls

Pitfall 1: Single Hypothesis

❌ "I think it's the database" ✅ "It could be: database, cache, API, or client-side rendering"

Pitfall 2: Unfalsifiable Hypotheses

❌ "The code quality is poor" ✅ "Specific module X has cyclomatic complexity > 20"

Pitfall 3: Confirmation Bias

❌ Looking only for evidence that supports your favorite idea ✅ Actively seeking evidence that would disprove each hypothesis

Pitfall 4: Attachment to Initial Hypothesis

❌ "I'm 90% sure it's H1, so let's focus on that" ✅ "Let's test the cheapest hypotheses first regardless of confidence"

Pitfall 5: Missing the Null

❌ Assuming something is definitely wrong ✅ Considering: "What if this is working as intended?"

---

Integration with Next Phase

Hypotheses feed into DesignExperiment.

Each hypothesis should have:

• A clear falsification test
• Known cost to test
• Priority ranking

This enables efficient experiment design - test cheap, high-value hypotheses first.

Iterate Workflow

Phase 7 of the Scientific Cycle

The feedback loop. Based on analysis, decide what's next and close the cycle.

---

The Core Principle

The power is in the cycle speed, not individual experiments.

Fast cycles > perfect experiments.

The goal is continuous learning and convergence toward the goal, not getting it right the first time.

---

The Iteration Decision Tree

Based on your analysis:

Analysis Complete
       │
       ▼
┌──────────────┐
│ Goal Achieved? │
└──────┬───────┘
       │
   ┌───┴───┐
   │       │
  YES      NO
   │       │
   ▼       ▼
 SHIP    Continue
   │       │
   │   ┌───┴────────────────────┐
   │   │                        │
   │   ▼                        ▼
   │ Hypothesis          All Hypotheses
   │ Confirmed?              Tested?
   │   │                        │
   │ ┌─┴─┐                  ┌───┴───┐
   │ YES NO                 YES     NO
   │ │   │                   │       │
   │ ▼   ▼                   ▼       ▼
   │ Implement  Test Next   Generate   Test Next
   │ Solution   Hypothesis  New Hyp.   Hypothesis
   │                           │
   │                           │
   └────── Monitor ◄───────────┘

---

Iteration Paths

Path 1: Goal Achieved - SHIP

## Iteration: Goal Achieved

**Status:** SUCCESS

**Actions:**
- [ ] Document the solution
- [ ] Implement in production
- [ ] Set up monitoring
- [ ] Capture learnings for future reference
- [ ] Archive experiment materials

**Learnings to Preserve:**
- [What worked and why]
- [What we'd do differently]
- [Generalizable insights]

Key Warning: Don't over-iterate. When the goal is achieved, ship. Perfect is the enemy of good.

Path 2: Hypothesis Confirmed - Implement & Monitor

## Iteration: Hypothesis Confirmed

**Confirmed Hypothesis:** [Name]

**Implementation Plan:**
1. [Implementation step 1]
2. [Implementation step 2]
3. [Implementation step 3]

**Monitoring Plan:**
- Metric to watch: [What]
- Check frequency: [How often]
- Rollback trigger: [If this happens]

**Next Cycle:**
- Goal still applies
- Watch for new issues that emerge

Path 3: Hypothesis Refuted - Try Next

## Iteration: Hypothesis Refuted

**Refuted Hypothesis:** [Name]
**Why It Failed:** [Brief explanation]

**Learning:** [What this taught us]

**Next Hypothesis to Test:** [Name]
**Why This One Next:** [Rationale]

**Adjustments to Approach:**
- [Any changes based on what we learned]

Path 4: All Hypotheses Exhausted - Generate New

## Iteration: Need New Hypotheses

**Hypotheses Tested:**
- H1: REFUTED - [why]
- H2: REFUTED - [why]
- H3: REFUTED - [why]

**What We Learned From Failures:**
- [Pattern in failures]
- [Assumptions that were wrong]

**New Direction:**
- [Insight from failures that suggests new approach]

**New Hypotheses to Generate:**
- Consider orthogonal approaches
- Challenge fundamental assumptions
- Look at analogous problems

**Action:** Return to GenerateHypotheses with updated context

Path 5: Inconclusive - Better Experiment Needed

## Iteration: Inconclusive Results

**Why Inconclusive:**
- [Data quality issues]
- [Sample size too small]
- [Confounding variables]

**Improved Experiment Design:**
- [What to change]
- [How to get cleaner data]

**Action:** Return to DesignExperiment with refined approach

Path 6: Paradigm Shift - Reframe the Problem

## Iteration: Paradigm Shift Needed

**The Original Frame:**
[How we were thinking about the problem]

**Why It's Wrong:**
[Evidence that the whole approach is off]

**New Frame:**
[How we should think about it instead]

**New Goal:**
[Revised goal based on new understanding]

**Action:** Return to DefineGoal with fundamentally new approach

---

The Anti-Pattern: Infinite Iteration

Infinite iteration without shipping is as bad as shipping without iteration.

Signs you're stuck:

• Same hypotheses cycling without progress
• Scope keeps expanding
• "Just one more experiment" repeatedly
• Fear of declaring success or failure

When to Stop:

• Goal is achieved (even partially)
• Diminishing returns on further experiments
• Time/resource constraints require decision
• Problem has changed (environment shifted)
• Better opportunity elsewhere

The Rule: Set iteration limits upfront. "We will run 5 cycles max, then decide."

---

Updating the World Model

Each iteration should update your understanding:

## World Model Update

**Before This Cycle:**
- We believed: [previous understanding]
- We assumed: [previous assumptions]

**After This Cycle:**
- We now know: [updated understanding]
- We've learned: [new knowledge]

**Confidence Changes:**
- [Belief X]: [increased/decreased] because [evidence]
- [Belief Y]: [increased/decreased] because [evidence]

**Bayesian Update:**
- Prior: [what we thought before]
- Evidence: [what we observed]
- Posterior: [what we think now]

---

Documentation for Future Cycles

Before closing an iteration, capture:

## Iteration Record: [Cycle Number]

**Date:** [When]
**Duration:** [How long this cycle took]

**Starting State:**
- Goal: [What we were trying to achieve]
- Hypotheses: [What we were testing]

**Experiments Run:**
- [Experiment 1]: [Result]
- [Experiment 2]: [Result]

**Outcome:**
- [GOAL ACHIEVED / CONTINUE / PIVOT]

**Key Learning:**
- [Most important insight from this cycle]

**Next Action:**
- [What happens next]

---

Template

# Iteration Decision: [Cycle N]

**Analysis Summary:** [One sentence on what we learned]

**Decision:** [SHIP / CONTINUE / PIVOT / STOP]

**Rationale:** [Why this decision]

**If Continuing:**
- Next hypothesis: [Name]
- Next experiment: [Brief description]

**If Shipping:**
- Implementation plan: [Brief]
- Monitoring plan: [Brief]

**If Pivoting:**
- New direction: [Description]
- Why: [What changed]

**If Stopping:**
- Why stopping: [Reason]
- What we learned anyway: [Takeaway]

**World Model Update:**
- [Key belief change]

**Time to Next Decision:** [When we'll evaluate again]

---

Common Pitfalls

Pitfall 1: Premature Stopping

❌ Giving up after one failed hypothesis ✅ Systematic exploration of hypothesis space

Pitfall 2: Refusing to Stop

❌ Endless iteration seeking perfection ✅ Shipping when goal is achieved

Pitfall 3: Not Learning from Failure

❌ "That didn't work, let's try something else" ✅ "That didn't work because X, which teaches us Y"

Pitfall 4: Ignoring Paradigm Shifts

❌ Staying in the same frame despite evidence it's wrong ✅ Recognizing when fundamental reframing is needed

Pitfall 5: No Documentation

❌ Moving on without recording learnings ✅ Capturing insights for future reference

---

Integration: Completing the Cycle

Iteration closes the loop:

GOAL → OBSERVE → HYPOTHESIZE → EXPERIMENT → MEASURE → ANALYZE → ITERATE
  ▲                                                              │
  │                                                              │
  └──────────────────────────────────────────────────────────────┘

The cycle continues until:

• Goal is achieved
• Problem is resolved
• Decision is made to stop

Each cycle should be FASTER than the last as you learn and refine.

Measure Results Workflow

Phase 5 of the Scientific Cycle

Collect data from experiments. This workflow ensures measurement is accurate, complete, and unbiased.

---

Data Collection Principles

1. Measure What Matters

Only collect data that relates to the goal. More data ≠ better data.

Ask: "Does this measurement help us evaluate the hypothesis against our success criteria?"

2. Quantify Where Possible

Numbers enable comparison. "It felt faster" < "Load time dropped 40%"

But don't force quantification when qualitative insights are what matter.

3. Capture Context

Numbers without context are meaningless. Record conditions, anomalies, environment.

4. Preserve Raw Data

Always keep raw data. Derived metrics can be recalculated; raw data cannot be recovered.

Never modify original data files.

---

The Measurement Process

Step 1: Execute Collection Protocol

Follow the plan defined in DesignExperiment.

## Data Collection Log

**Experiment:** [Name]
**Collection Started:** [Timestamp]
**Collector:** [Who]

| Timestamp | Data Point | Value | Collection Method | Notes |
|-----------|-----------|-------|-------------------|-------|
| [Time]    | [Metric]  | [Value] | [How collected] | [Context] |
| [Time]    | [Metric]  | [Value] | [How collected] | [Context] |

Step 2: Collect Primary Metrics

The main thing you're testing - the direct measure of hypothesis success/failure.

## Primary Metrics

| Metric | Baseline | Measured | Delta | Method |
|--------|----------|----------|-------|--------|
| [Name] | [Before] | [After]  | [Δ]   | [How]  |

Step 3: Collect Secondary Metrics

Supporting context, early warning indicators, sanity checks.

## Secondary Metrics

| Metric | Value | Relevance |
|--------|-------|-----------|
| [Name] | [Value] | [Why we tracked this] |

Step 4: Capture Qualitative Data

Observations, feedback, anomalies, surprises.

## Qualitative Observations

**What We Observed:**
- [Observation 1]
- [Observation 2]

**Feedback Received:**
- [Source]: "[Quote]"

**Anomalies:**
- [Unexpected event 1]
- [Unexpected event 2]

**Surprises:**
- [What we didn't expect]

Step 5: Validate Data Quality

## Data Quality Check

**Completeness:**
- [ ] All planned data points collected
- [ ] No significant gaps in collection

**Accuracy:**
- [ ] Collection methods worked correctly
- [ ] No obvious measurement errors

**Outliers:**
- [List any outliers and whether they're valid or errors]

**Issues:**
- [Any problems with data collection]

---

Measurement Timing

During Experiment

• Monitor progress (don't just set and forget)
• Note anomalies in real-time
• Don't peek at aggregate results prematurely (avoid bias)

At Completion

• Collect final measurements
• Verify all planned data captured
• Document any deviations from plan

Post-Collection

• Archive raw data
• Create derived metrics
• Document methodology for reproducibility

---

Domain-Specific Measurement

Code Experiments

## Code Measurement

**Test Results:**
| Branch | Tests Pass | Coverage | Execution Time |
|--------|-----------|----------|----------------|
| control | [Y/N] | [%] | [time] |
| treatment | [Y/N] | [%] | [time] |

**Performance Benchmarks:**
| Metric | Control | Treatment | Delta |
|--------|---------|-----------|-------|
| [Metric] | [Value] | [Value] | [Δ%] |

**Code Quality:**
| Metric | Control | Treatment | Delta |
|--------|---------|-----------|-------|
| Complexity | [Score] | [Score] | [Δ] |
| LOC | [Lines] | [Lines] | [Δ] |

Prompt Experiments

## Eval Results

**Accuracy:**
| Variant | N | Correct | Accuracy | 95% CI |
|---------|---|---------|----------|--------|
| baseline | [N] | [count] | [%] | [range] |
| variant-1 | [N] | [count] | [%] | [range] |

**Format Compliance:**
| Variant | Compliant | Non-Compliant | Rate |
|---------|-----------|---------------|------|
| baseline | [count] | [count] | [%] |
| variant-1 | [count] | [count] | [%] |

Feature Experiments

## A/B Test Results

**Primary Metric:**
| Group | N | Conversions | Rate | 95% CI |
|-------|---|-------------|------|--------|
| control | [N] | [count] | [%] | [range] |
| treatment | [N] | [count] | [%] | [range] |

**Lift:** [%] (p-value: [p])

**Secondary Metrics:**
| Metric | Control | Treatment | Delta |
|--------|---------|-----------|-------|
| [Metric] | [Value] | [Value] | [Δ%] |

Research Experiments

## Evidence Collected

**For Hypothesis:**
| Source | Finding | Quality | Date |
|--------|---------|---------|------|
| [Ref] | [What it says] | [H/M/L] | [When] |

**Against Hypothesis:**
| Source | Finding | Quality | Date |
|--------|---------|---------|------|
| [Ref] | [What it says] | [H/M/L] | [When] |

---

Template

# Results: [Experiment Name]

**Experiment Run:** [Date range]
**Data Collected By:** [Who]

## Primary Metrics
| Metric | Baseline | Result | Change |
|--------|----------|--------|--------|
| [Name] | [Value]  | [Value]| [Δ]    |

## Secondary Metrics
| Metric | Value | Notes |
|--------|-------|-------|
| [Name] | [Value] | [Context] |

## Qualitative Observations
- [Observation 1]
- [Observation 2]

## Anomalies
- [Unexpected event 1]

## Data Quality
- Completeness: [Good/Issues noted]
- Accuracy: [Verified/Concerns]
- Outliers: [None/Listed]

## Raw Data Location
[Path to preserved raw data]

---

Common Pitfalls

Pitfall 1: Confirmation Peeking

❌ Checking results early and stopping when they look good ✅ Completing planned data collection before analysis

Pitfall 2: Selective Reporting

❌ Only recording measurements that support the hypothesis ✅ Recording all measurements, especially unexpected ones

Pitfall 3: Lost Context

❌ Raw numbers without notes on conditions ✅ Every measurement includes relevant context

Pitfall 4: Destroyed Raw Data

❌ Only keeping summarized/derived metrics ✅ Preserving original data files unmodified

Pitfall 5: Measurement Interference

❌ The act of measuring changes the outcome ✅ Using non-intrusive measurement methods

---

Integration with Next Phase

Measurements feed directly into AnalyzeResults.

Well-collected data enables:

• Objective comparison to success criteria
• Statistical analysis where appropriate
• Honest assessment of hypothesis status
• Reproducible conclusions

Quick Diagnosis Workflow

Level 1 Science - For problems under 15 minutes

This is Minimum Viable Science. When you've been stuck for 15+ minutes or intuition fails, STOP and run this lightweight diagnostic.

---

When to Use

• You've tried the obvious fix and it didn't work
• You're about to "try random stuff"
• You're pattern-matching without evidence ("I think it's probably...")
• The problem is taking longer than expected

Anti-Triggers (Don't Use When)

• The fix is obvious and takes 5 minutes
• You've solved this exact pattern 50+ times
• You're in creative/generative mode
• Cost of trying < cost of thinking

---

The Workflow (2-3 minutes)

Step 1: State Your Goal (30 seconds)

One sentence. What does "fixed" look like?

GOAL: [What will be true when this is solved?]

Examples:

• "User can log in without 500 error"
• "Build passes without TypeScript errors"
• "Component renders the updated data"

Step 2: Generate Hypotheses (60 seconds)

MINIMUM THREE. If you can't think of three, you haven't thought hard enough.

H1: [Most likely cause] - because [evidence/reasoning]
H2: [Second possibility] - because [evidence/reasoning]
H3: [Third possibility] - because [evidence/reasoning]

The Falsification Question: For each hypothesis, ask: "What would prove this WRONG?"

Step 3: Rank and Test (60 seconds)

Order by: Fastest to verify × Most likely

Test H[X] first because: [it takes 30 seconds to check]

Run the test. What did you observe?

Step 4: Update and Iterate (30 seconds)

Based on results:

• Hypothesis confirmed → Fix it, done
• Hypothesis refuted → Move to next hypothesis
• Inconclusive → Need more data, design better test

---

One-Liner Version

When you're really pressed for time:

"I think [X] because [Y]. If I'm wrong, [Z] would be true. Let me check [Z]."

This single sentence contains: hypothesis, rationale, falsification test.

---

Examples

Debugging a Bug

GOAL: API returns 200 instead of 500

H1: Database connection timeout - server logs show DB errors
H2: New code introduced regression - deployed yesterday
H3: Rate limiting kicked in - sudden traffic spike

Test H1 first (check logs):
→ Logs show "connection pool exhausted"
→ H1 CONFIRMED - increase pool size

Build Failure

GOAL: TypeScript build passes

H1: Type mismatch in new code - just edited that file
H2: Missing dependency types - added new package
H3: Config changed - someone touched tsconfig

Test H1 first (check error location):
→ Error is in file I edited, line I touched
→ H1 CONFIRMED - fix the type annotation

Content Not Rendering

GOAL: Component shows updated data

H1: API returning stale data - caching issue
H2: Component not re-rendering - state management
H3: Data transformation bug - shape mismatch

Test H1 first (check network tab):
→ API returns correct data
→ H1 REFUTED, test H2
→ React DevTools shows state not updating
→ H2 CONFIRMED - fix dependency array

---

The Anti-Pattern to Avoid

Random Flailing:

"Let me try this... nope. Let me try that... nope.
 Maybe if I restart everything... still broken.
 Let me Google random things..."

Structured Diagnosis:

"Let me form three hypotheses, test the fastest one first,
 and systematically eliminate until I find the cause."

---

Exit Criteria

You're done with Quick Diagnosis when:

• Problem is solved
• You need more information than available → escalate to StructuredInvestigation
• Problem is bigger than expected → escalate to StructuredInvestigation

Time limit: If 15 minutes of Quick Diagnosis doesn't solve it, escalate.

Structured Investigation Workflow

Level 2 Science - For problems taking hours to days

When Quick Diagnosis isn't enough, or the problem requires systematic exploration. This is the workhorse workflow for most non-trivial problems.

---

When to Use

• Quick Diagnosis escalated (15+ minutes without resolution)
• Problem involves multiple systems or unknowns
• Multiple people disagree on the cause
• High stakes - wrong answer costs significant time/resources
• You need to document your investigation for others

Anti-Triggers (Don't Use When)

• Problem can be solved in 5 minutes
• You're doing creative/generative work
• The answer is obvious to domain experts
• Formal process would take longer than just doing the work

---

The Workflow

Phase 1: Define the Goal (5 minutes)

Be specific. Vague goals = vague results.

## Goal Definition

**Outcome:** [What will exist/change when we succeed?]

**Success Criteria:**
- [ ] [Measurable criterion 1]
- [ ] [Measurable criterion 2]
- [ ] [Measurable criterion 3]

**Constraints:**
- Time: [Deadline or time budget]
- Resources: [What we have available]
- Quality: [Standards that must be maintained]

**Anti-Goals (what we're NOT trying to do):**
- [Explicitly out of scope item 1]
- [Explicitly out of scope item 2]

Quality Check: Can someone else read this and know exactly when we've succeeded?

Phase 2: Gather Context (15-30 minutes)

Understand before you hypothesize.

## Current State Observation

**What we know:**
- [Fact 1 - with source/evidence]
- [Fact 2 - with source/evidence]
- [Fact 3 - with source/evidence]

**What we don't know:**
- [Gap 1 - what information would help?]
- [Gap 2 - what information would help?]

**What has been tried:**
- [Previous attempt 1] - Result: [outcome]
- [Previous attempt 2] - Result: [outcome]

**Baseline measurements:**
- [Metric 1]: [current value]
- [Metric 2]: [current value]

Quality Check: Did we look for disconfirming evidence, not just confirming evidence?

Phase 3: Generate Hypotheses (15 minutes)

Minimum 3 hypotheses. Better: 5-10 for important problems.

## Hypotheses

### H1: [Name]
**Claim:** [What we believe is happening/will work]
**Rationale:** [Why we think this]
**Evidence For:** [What supports this]
**Evidence Against:** [What challenges this]
**Falsification Test:** This is WRONG if [observable outcome]
**Cost to Test:** [Time/resources needed]
**Confidence:** [Low/Medium/High]

### H2: [Name]
[Same structure]

### H3: [Name]
[Same structure]

The Plurality Requirement: Single-hypothesis thinking leads to confirmation bias. Force yourself to consider alternatives.

Quality Check: For each hypothesis, can you articulate what would prove it wrong?

Phase 4: Design Experiments (15 minutes)

Minimum Viable Experiments - the smallest test that gives meaningful data.

## Experiment Design

### Experiment for H1

**Method:**
1. [Step 1]
2. [Step 2]
3. [Step 3]

**Measuring:**
- Primary metric: [What confirms/refutes]
- Secondary metrics: [Supporting data]

**Success Criteria:**
- CONFIRMED if: [condition]
- REFUTED if: [condition]
- INCONCLUSIVE if: [condition]

**Duration:** [How long to run]
**Resources:** [What's needed]

Prioritization: Order experiments by (likelihood of success) × (cost to test). Cheap, high-likelihood first.

Phase 5: Execute Experiments

Run experiments. Collect data. Don't peek prematurely.

## Experiment Execution

### Experiment 1: [Name]
**Started:** [timestamp]
**Status:** [Running/Complete]
**Raw Observations:**
- [Observation 1]
- [Observation 2]
**Anomalies:** [Anything unexpected]

Parallel Execution: When experiments are independent, run them simultaneously.

Phase 6: Measure Results

Collect data related to success criteria.

## Results

### Experiment 1 Results
| Metric | Expected | Actual | Delta |
|--------|----------|--------|-------|
| [Name] | [Value]  | [Value]| [+/-] |

**Qualitative Observations:**
- [Notable event 1]
- [Notable event 2]

**Data Quality Notes:**
- [Any issues with collection]

Phase 7: Analyze Against Goal

Compare results to success criteria. Be honest.

## Analysis

### Goal Comparison
| Success Criterion | Required | Achieved | Status |
|-------------------|----------|----------|--------|
| [Criterion 1]     | [Value]  | [Value]  | ✅/❌  |
| [Criterion 2]     | [Value]  | [Value]  | ✅/❌  |

### Hypothesis Status
- H1: [CONFIRMED/REFUTED/INCONCLUSIVE] - because [evidence]
- H2: [CONFIRMED/REFUTED/INCONCLUSIVE] - because [evidence]
- H3: [CONFIRMED/REFUTED/INCONCLUSIVE] - because [evidence]

### Key Learnings
1. [What we learned that changes our understanding]
2. [Surprises - what we didn't expect]
3. [New questions that emerged]

### Implications
- [How this changes our approach]
- [What to do next]

Phase 8: Iterate

Based on analysis, decide next action.

Options: 1. Goal achieved → Document learnings, done 2. Promising hypothesis → Deeper investigation of that path 3. All hypotheses failed → Generate new hypotheses, new cycle 4. Need more data → Design additional experiments 5. Problem reframed → Return to Phase 1 with new understanding

---

Documentation Artifact

After completing, produce a summary:

## Investigation Summary: [Problem Name]

**Date:** [When]
**Duration:** [How long]

**Goal:** [What we were trying to achieve]

**Key Finding:** [One-sentence summary]

**Hypotheses Tested:**
- H1: [CONFIRMED/REFUTED]
- H2: [CONFIRMED/REFUTED]
- H3: [CONFIRMED/REFUTED]

**Resolution:** [What we did / what we learned]

**Learnings for Future:**
- [Generalizable insight 1]
- [Generalizable insight 2]

---

Escalation to Full Cycle

Escalate to FullCycle workflow when:

• Investigation spans multiple days/weeks
• Multiple stakeholders need to review methodology
• Results will be published or shared externally
• Statistical rigor is required
• The problem is at macro scale (product strategy, architecture)