Brainstorm Experiments

Experiment Design Expert

Overview

Design fast, low-cost experiments to validate product hypotheses before committing to full development. This skill applies Alberto Savoia's pretotyping philosophy ("Make sure you are building The Right It before you build It right") alongside lean experimentation methods for both new and existing products.

When to Use

• You have a product idea or feature hypothesis and need to validate it cheaply.
• You want to test willingness to pay or genuine user interest, not just stated preference.
• You need to choose the right experiment method for your context (new vs. existing product).

Core Principles

1. XYZ Hypothesis Format

Every experiment starts with a falsifiable hypothesis:

"At least X% of Y will do Z."

A good XYZ hypothesis is specific, measurable, and has a clear pass/fail threshold set before the experiment runs.

2. Skin-in-the-Game (SITG)

Stated interest is unreliable. Valid experiments measure actions that require commitment:

• Money -- Pre-orders, deposits, paid waitlists.
• Time -- Signing up, completing a multi-step flow, scheduling a demo.
• Reputation -- Sharing with colleagues, posting publicly.

Always prefer SITG signals over surveys, likes, or verbal feedback.

3. Your Own Data (YODA)

Do not rely on market reports, competitor benchmarks, or industry averages. Run your own experiment with your own audience to get Your Own Data. Others' data reflects their context, not yours.

Experiment Types

For New Products

For Existing Products

Methodology

Step 1: Write the XYZ Hypothesis

Start with the assumption you need to test. Convert it into XYZ format.

Weak: "Users will like the new dashboard." Strong: "At least 30% of active users who see the new dashboard will set it as their default view within 5 days."

Step 2: Select the Experiment Method

Choose based on:

• Product type (new vs. existing)
• What you are testing (demand, usability, willingness to pay, engagement)
• Available effort (team capacity and timeline)
• Required confidence (directional signal vs. statistically significant result)

Step 3: Define the Metric and Threshold

Step 4: Run the Experiment

• Set a timebox. Every experiment has a fixed end date.
• Do not peek. Avoid checking results daily and making early calls.
• Document everything. Record setup, audience, duration, and any anomalies.

Step 5: Evaluate Results

Python Tool: experiment_designer.py

Design experiments from hypotheses using the CLI tool:

# Run with demo data
python3 scripts/experiment_designer.py --demo

# Run with custom input
python3 scripts/experiment_designer.py input.json

# Output as JSON
python3 scripts/experiment_designer.py input.json --format json

Input Format

{
  "hypotheses": [
    {
      "hypothesis_text": "At least 20% of trial users will click Upgrade within 7 days",
      "target_segment": "trial users on free plan",
      "product_type": "existing"
    }
  ]
}

Output

For each hypothesis, the tool suggests 2-3 experiment designs with method, metric, success threshold, effort level, and duration estimate.

See scripts/experiment_designer.py for full documentation.

Output Template

Use assets/experiment_plan_template.md to document each experiment:

• Experiment card with hypothesis, method, metric, threshold, owner, timeline
• Experiment tracker for managing multiple concurrent experiments
• Results documentation for recording outcomes and decisions

Integration with Other Discovery Skills

• Use brainstorm-ideas/ to generate ideas that become hypotheses.
• Use identify-assumptions/ to find the riskiest assumptions to test.
• After experiments, use pre-mortem/ before committing to full build.

Troubleshooting

Success Criteria

• Every product hypothesis has a falsifiable XYZ statement before experiment design begins
• Experiments measure Skin-in-the-Game (SITG) signals, not stated preferences
• Pass/fail thresholds are defined before the experiment runs, not after
• Experiment duration does not exceed 4 weeks for any single hypothesis
• At least 70% of experiments produce a clear pass or fail verdict (not inconclusive)
• Results directly feed the build/pivot/abandon decision within 1 week of experiment completion
• Your Own Data (YODA) principle is followed -- no reliance on industry benchmarks for go/no-go decisions

Scope & Limitations

In Scope:

• XYZ hypothesis formulation and validation for product ideas
• Experiment method selection for both new products (landing page, pre-order, concierge, explainer video) and existing products (fake door, feature stub, A/B test, Wizard of Oz, in-app survey)
• Automated experiment design suggestions based on hypothesis keyword analysis
• Metric selection, success threshold definition, and effort/duration estimation

Out of Scope:

• Statistical power analysis or sample size calculation (use dedicated A/B test platforms)
• Experiment infrastructure setup (feature flags, analytics instrumentation)
• Running the actual experiment (this skill designs experiments, not executes them)
• Long-term product strategy or roadmap decisions (see execution/outcome-roadmap/)

Important Caveats:

• Pretotyping is for validating demand and value, not for measuring usability or performance.
• In-app surveys are the weakest SITG signal. Use them only when behavioral experiments are impractical.
• The tool's keyword-to-signal matching is heuristic-based. Review suggested experiments and override when domain knowledge dictates a better method.

Integration Points

Tool Reference

experiment_designer.py

Suggests 2-3 experiment designs for each product hypothesis based on keyword signal analysis.

References

• Alberto Savoia, The Right It (2019)
• Eric Ries, The Lean Startup (2011)
• Jeff Gothelf & Josh Seiden, Lean UX (2013)
• Teresa Torres, Continuous Discovery Habits (2021)

Experiment Plan Template

Experiment Overview

---

Experiment Card

Hypothesis (XYZ Format)

At least X% of Y will Z.

Method

Metrics

Success Criteria

• Pass: Primary metric >= threshold (X%)
• Fail: Primary metric < threshold
• Inconclusive: Insufficient sample size or metric within margin of error

---

Experiment Tracker

Use this table to manage multiple concurrent experiments.

Status values: Planned | Running | Complete (Pass) | Complete (Fail) | Complete (Inconclusive) | Abandoned

---

Results Documentation

Complete this section when the experiment ends.

Raw Results

Analysis

What happened?

Why do we think this happened?

Unexpected observations:

Decision

Next Steps

---

Notes

Capture any context, anomalies, or learnings not covered above.

Example: Acme Analytics — Lean Experiment: "Will Users Pay for AI Summarization?"

Real-world scenario showing how to apply this skill end-to-end.

Context

Acme Analytics (Series-B B2B data platform, ~220 employees, ~3,200 paying accounts) is considering an AI summarization feature: each dashboard would get a one-paragraph plain-English narrative auto-generated daily ("Revenue is up 8% week-over-week, driven by enterprise renewals. Three accounts are flagged at risk."). Engineering effort to ship a full version: ~5 engineer-months. Inference costs at scale: ~$80K/year.

The PM, Priya, hears strong stated interest from sales and customer success — "every customer wants this." But the PM has been burned before by stated preference. Before committing engineering, Priya wants to test the harder question: will customers actually pay for it? The brainstorm-experiments skill is being applied to design a lean validation in 3 weeks.

Inputs

• Stated interest: high (sales calls, CS escalations)
• Cost to build full version: ~5 engineer-months + $80K/year inference
• Time budget for experiment: 3 weeks max
• Product surface: existing dashboard product, 3,200 accounts, ~9,000 weekly active users
• Pricing context: current tiers are $99 / $499 / $2,499 per month; AI summarization is being considered as a paid add-on or a tier-3-only feature
• Internal politics: CTO is skeptical of the cost; CRO is pushing hard; this experiment must produce a defensible signal one way or the other

Applying the skill

1. Wrote the XYZ hypothesis. The instinct of a weaker PM would be "users will love AI summaries." Priya restated it as: "At least 8% of trialing tier-2 customers who see an AI summary option in their dashboard will click 'enable for $20/month' within 14 days." 2. Demanded skin-in-the-game. Not a survey. Not a "would you pay?" question. The signal is a paid upgrade click. SITG = money. 3. Picked a fake door + concierge hybrid. Fake door measures intent (click-to-pay). Concierge MVP validates that the manually-generated summary actually creates value for users who do convert. 4. Used YODA, not industry benchmarks. Did not look at OpenAI press claims or competitor case studies. Acme's data, Acme's customers. 5. Pre-set kill criteria. If conversion is below 4%, kill the feature. Between 4% and 12%, run a follow-up experiment. Above 12%, ship. 6. Excluded the "every customer wants this" signal. The experiment specifically does not measure stated interest. Conversations after the experiment will not change the kill/keep decision.

The artifact

================================================================
  EXPERIMENT BRIEF — AI SUMMARIZATION PAID ADD-ON
  Owner:   Priya Rao (PM, Activation)
  Drafted: 2026-05-22
  Status:  Approved, starts 2026-05-26
================================================================


PART 1 — HYPOTHESIS

XYZ Format:

  "At least 8% of tier-2 trial customers (Y) who see the AI
   Summary upsell in their dashboard (Y context) will click
   'enable for $20/month' within 14 days (Z), at a willingness-
   to-pay threshold of $20/month (X% threshold)."

  X (threshold) :  8%
  Y (population):  Tier-2 trial customers (currently 380 active
                   trials, ~270 visit dashboard in any given
                   14-day window)
  Z (action)    :  Click "Enable for $20/month" CTA

The hypothesis is FALSIFIABLE — if fewer than 8% click,
we have a clear signal the feature does not earn its cost.

Counter-hypothesis (the null we are trying to reject):
  "Stated interest will not convert to paid upgrade clicks
   above 4%."

Pre-experiment Bayesian prior:
  Priya's prior: 6% conversion (between sales optimism and
                 historical add-on pricing data).
  CTO's prior:   2% conversion.
  CRO's prior:   25% conversion.

Both priors are 3-4x apart from each other. That spread is
itself evidence the team should run the experiment.


PART 2 — METHOD

This experiment is a FAKE DOOR + CONCIERGE MVP hybrid.

Why two methods together:
  - Fake door measures intent (click-to-pay)
  - Concierge MVP validates that manually-generated summaries
    actually create value for users who DO convert

If we only did fake door, we'd test demand but not whether
the value lands. If we only did concierge, we'd test value
but not whether anyone would pay. Together they answer both.


PART 3 — DESIGN

[A] FAKE DOOR

  Where:    Tier-2 trial customer dashboard, top-right of the
            main dashboard view
  What:     A clearly visible card:
            "NEW: AI-generated summary of your dashboard,
             delivered daily.
             Enable for $20/month"
            [Enable] button
  Tracking: enable_click event in analytics; cohort tagged
            "ai_summary_v1"

  On click:
    Show a modal:
      "Thanks for your interest! AI Summary is in invitation-
       only beta. We will reach out within 2 business days to
       set up your trial."
    Capture the click as the primary metric.
    Trigger a Slack alert to Priya.

[B] CONCIERGE MVP FOR CLICKERS

  When someone clicks "Enable":
    Within 24 hours, Priya (or a contractor analyst) manually
    writes a plain-English summary of that customer's
    dashboard. Delivers via email by 9am customer-local-time.
    Continues daily for up to 14 days.

  Quality bar: a 60-90 word paragraph that names the top
    metric, week-over-week movement, and one notable account
    or segment change. NOT a copy-paste of dashboard numbers.

  Goal:    Validate whether customers who convert get value.
  Capacity: Up to 30 customers (Priya + 1 contractor analyst
           at 4 hours/day each)


PART 4 — METRICS

Primary metric:
  Click-to-enable conversion rate = clicks / dashboard views
  Pre-set thresholds:
    < 4%   KILL
    4-12%  RUN FOLLOW-UP (pricing test, value-prop variant)
    > 12%  SHIP (commit engineering investment)

Secondary metrics (for the concierge phase only):
  - Email open rate (target: >60%)
  - "Continue using after week 1" survey question (target: >70%)
  - Manual unsubscribe before day 14 (target: <15%)
  - Direct "I would pay for this" qualitative quotes (sample
    of 5 customers will get a 15-min follow-up call)

GUARDRAIL metrics (catch unintended damage):
  - Tier-2 trial conversion to paid (do not drop)
  - Support ticket volume from the experiment cohort (must
    stay within 2 std dev of control)


PART 5 — POPULATION & DURATION

Population:
  - Tier-2 trial customers active in the last 30 days
  - Estimated ~270 will see the card during the experiment
    window
  - NOT shown to: existing paid customers, tier-3 customers,
    or customers in active CS escalation

Duration:
  14 days for the fake door
  14 days of concierge delivery for clickers

Control:
  No A/B split needed for the fake-door test — we are
  measuring an absolute click rate against a fixed threshold,
  not comparing variants. (If conversion lands in the 4-12%
  band, the follow-up experiment will be properly A/B tested.)


PART 6 — RISKS & MITIGATIONS

R1  Customers feel deceived when "Enable" leads to a beta
    waitlist modal.
    Mitigation: The modal language is honest ("invitation-only
    beta"). Customers who reach out get the concierge service
    within 24 hours — they DO get the feature. The "door" is
    not fake from the customer's perspective.

R2  Concierge capacity overwhelmed if clicks exceed 30.
    Mitigation: If clicks pass 30 in days 1-3, we cap fresh
    enrollments and add a "next available slot" message for
    the rest of the experiment.

R3  Internal pressure to call the result early.
    Mitigation: Pre-committed to 14-day duration. No internal
    party may change the kill/keep thresholds mid-experiment.

R4  Sales team accidentally promotes the feature to customers
    not in the experiment cohort.
    Mitigation: Sales briefed pre-launch. The button does not
    render outside the cohort.

R5  Concierge summaries leak data (e.g., wrong customer's data
    in another customer's email).
    Mitigation: Per-customer separate Google Doc; 2-person
    review (Priya + contractor) before send for the first 5
    customers, then 1-person review after the process is
    proven.


PART 7 — TIMELINE

  Week -1 (May 19-25)
    [x] Design review with CTO, CRO
    [x] Engineering scoping: 1 day to ship the fake door
        (just a feature flag + tracking event)
    [x] Concierge process documented; sample summaries
        written for 3 internal accounts to calibrate quality
    [x] Slack alert wired

  Week 0 (May 26-Jun 1)  -- EXPERIMENT STARTS
    Day 1: Soft launch to 30 customers
    Day 2: Validate alerting works; review first summaries
           with the contractor
    Days 3-7: Full cohort

  Week 1 (Jun 2-8)
    Days 8-14: Continue, hold capacity if past 30 clicks

  Week 2 (Jun 9-15)
    Day 15: Wrap-up. Run the 5 qualitative follow-up calls.

  Week 3 (Jun 16-22)
    Readout. Decision meeting with CTO, CRO, VP Product.


PART 8 — DECISION CRITERIA (PRE-COMMITTED)

After 14 days, take the path indicated by the click rate.
We commit to following the rule we wrote BEFORE the data
came in.

  CLICK RATE                ACTION
  ------------------------  -------------------------------
  <4%                       KILL. Document in failed-
                            experiments register. Send a
                            short note to CRO with the data.
                            Re-visit in 12 months only with
                            new evidence.
  4% - 8%                   RUN FOLLOW-UP. Test variants
                            (price $10/$20/$40, value prop
                            language, free with tier-3).
  8% - 12%                  CONDITIONAL SHIP. Validate that
                            secondary metrics (email open
                            rate, week-1 retention) are
                            healthy. If yes, ship as $20/mo
                            add-on. If no, treat as 4-8%.
  >12%                      SHIP. Move to engineering scoping
                            of the production version.

The decision committee is:
  Priya (PM, owns the rec)
  CTO (must approve build cost)
  VP Product (final tie-breaker)

Notably absent from the committee:
  Sales leadership (their input is in the priors, not the
  decision). This is intentional — the experiment exists
  precisely because stated interest is not a buying signal.


PART 9 — READOUT TEMPLATE (TO BE FILLED AFTER EXPERIMENT)

  Click rate observed:     ___% (vs 8% threshold)
  Total clicks:            ___ of ~270 dashboard views
  Concierge enrolled:      ___ customers
  Email open rate:         ___%
  Week-1 retention:        ___%
  Direct paid-conversion:  ___ (any customer who said "stop
                            beta, just charge me"?)
  Qualitative themes:      Top 3 quotes from the 5 follow-up
                            calls

  Recommendation:          [KILL | FOLLOW-UP | SHIP]
  Rationale:               ___
  Next step:               ___


PART 10 — WHAT THIS EXPERIMENT IS NOT

  - Not a usability test (we have one already for the
    summary format)
  - Not a pricing test (it tests one price; pricing optimization
    is the follow-up if we land in the 4-8% band)
  - Not a competitive analysis (YODA principle — we are not
    comparing to vendors)
  - Not a measure of stated interest (we already know that
    is high)

The experiment is exactly one thing: does stated interest
convert into a paid-upgrade click at our test threshold?

Why this works

• Killed the "every customer wants this" signal up front. Sales leadership is excluded from the decision committee because their input is in the prior, not the data. This is the discipline that separates real validation from theater.
• XYZ hypothesis is falsifiable with a clear threshold. "8% of tier-2 trials will click 'enable' in 14 days." Not "users will love this." A weaker PM writes a hypothesis they cannot lose.
• SITG = paid click, not survey response. The whole point of the experiment is to test the difference between stated interest and behavioral commitment.
• Fake door + concierge hybrid covers both demand and value delivery. If only one method, only half the question is answered.
• Kill criteria pre-committed. If conversion is <4%, the feature dies. The decision rule was written before the data exists. This is the single biggest defense against post-hoc rationalization.
• CTO and CRO priors captured. The 2% vs 25% prior spread is itself the strongest evidence for why this experiment must be run. When the data lands, both leaders have publicly committed to a number.
• Capacity-bounded concierge. Up to 30 concierge customers. Not infinite. Real lean validation experiments name the limit and respect it.
• Honest "fake door." The modal does not lie — it says "invitation-only beta" and customers who click DO get the feature manually within 24 hours. The skill warns against deceptive fake doors; this avoids the anti-pattern.

What's next

• If the experiment lands in the 4-12% band, the follow-up pricing experiment uses `../../execution/pricing-prd/` for the pricing variants.
• If shipped, the production version requires an AI-specific PRD; see `../../execution/ai-feature-prd/` for eval, guardrails, model selection, and cost framing.
• Pre-build assumption mapping for the broader feature should use `../identify-assumptions/`.
• A pre-mortem for the production version uses `../pre-mortem/` to surface Tiger/Paper Tiger/Elephant risks.
• The 5 qualitative follow-up calls feed `../interview-synthesis/` for an opportunity solution tree.
• If killed, document in the experiments register linked from `../../confluence-expert/` and revisit only with new evidence.

Experiment Methods Reference

Pretotyping (Alberto Savoia)

Pretotyping is the practice of testing the initial appeal and actual usage of a potential new product by simulating its core experience with the smallest possible investment of time and money.

The Pretotyping Manifesto

1. Make sure you are building The Right It before you build It right. 2. Most new products fail -- even when well executed -- because they are the wrong product. 3. The biggest risk is not "Can we build it?" but "Should we build it?" 4. Your opinion, although interesting, is irrelevant. Test it. 5. Data beats opinions. Your Own Data beats others' data.

Pretotype Types

---

Lean Experiment Types Catalog

Discovery Experiments (Testing Demand)

Landing Page Test

• Setup: Single page with value proposition, benefits, and a CTA.
• CTA options: Sign up for waitlist, pre-order, request demo.
• Traffic source: Paid ads, social media, email to target segment.
• Key metric: Conversion rate (visitors to CTA).
• Duration: 1-2 weeks.
• Cost: Low (domain + hosting + ad spend).

Explainer Video Test

• Setup: 60-90 second video explaining the product concept.
• Distribution: Landing page, social media, email.
• Key metric: Watch completion rate + CTA conversion.
• Duration: 1-2 weeks.
• Tips: Keep production simple. Screen recordings or animations work fine.

Pre-Order / Crowdfunding

• Setup: Accept payment (or a deposit) for a product that does not exist yet.
• Key metric: Number of orders or total revenue.
• Duration: 2-4 weeks.
• Strongest SITG signal: People commit real money.

Concierge MVP

• Setup: Deliver the service manually to a small number of users.
• Key metric: Retention, satisfaction, willingness to pay after manual service.
• Duration: 2-4 weeks.
• Tips: Do not tell users it is manual. Observe whether they value the outcome regardless of delivery method.

Validation Experiments (Testing Solutions)

Fake Door Test

• Setup: Add a button, menu item, or link for a feature that does not exist. When clicked, show a message: "Coming soon! Sign up to be notified."
• Key metric: Click-through rate on the fake door element.
• Duration: 1-2 weeks.
• Ethical note: Be transparent. Show a message immediately; do not leave users confused.

Feature Stub

• Setup: Build a minimal, mostly static version of the feature behind a feature flag.
• Key metric: Engagement rate, completion rate, or adoption rate.
• Duration: 1-2 weeks.
• Tips: The stub can be a mockup, a simplified version, or a wizard-style flow.

A/B Test

• Setup: Randomly assign users to control (current experience) or variant (new experience). Measure a primary metric.
• Key metric: Depends on what you are testing (conversion rate, retention, engagement).
• Duration: 2-4 weeks (until statistical significance).
• Requirements: Sufficient traffic volume, proper randomization, no peeking.

Wizard of Oz

• Setup: The feature appears fully automated to the user, but a human operates it behind the scenes.
• Key metric: Same as if the feature were real (engagement, satisfaction, conversion).
• Duration: 2-4 weeks.
• Tips: Good for testing AI/ML features before building the model.

In-App Survey

• Setup: Trigger a survey for users who match specific behavioral criteria.
• Key metric: Response rate + stated preference distribution.
• Duration: 1 week.
• Limitations: Low SITG. Use only when behavioral experiments are impractical.

---

Metric Selection Guide

Choosing the Right Primary Metric

Setting the Success Threshold

The threshold must be set before the experiment starts.

Framework for setting thresholds:

1. Baseline: What is the current metric value (if applicable)? 2. Minimum viable: What is the minimum result that justifies further investment? 3. Aspiration: What result would signal strong product-market fit?

Example: Current trial-to-paid conversion is 8%. A new onboarding flow experiment sets:

• Threshold (pass): 10% (25% improvement over baseline)
• Aspiration: 14%+
• Fail: Below 10%

---

Sample Size Considerations

When Sample Size Matters

• A/B tests: Require statistical significance. Use a sample size calculator.
• Landing page tests: Directional signal is often sufficient with 200-500 visitors.
• Fake door tests: 1,000+ impressions of the element for reliable click-through rates.
• Concierge MVP: 5-15 users is sufficient for qualitative signal.

Quick Sample Size Rules of Thumb

For precise calculations, use an online sample size calculator with your baseline rate and minimum detectable effect.

---

Common Pitfalls

1. Testing the Solution Before the Problem

Run problem-validation experiments (interviews, surveys) before solution-validation experiments (prototypes, A/B tests).

2. No Clear Hypothesis

"Let's see what happens" is not an experiment. Every experiment needs an XYZ hypothesis with a pre-set threshold.

3. Peeking at Results

Checking results daily and stopping early when they look good inflates false positive rates. Set a duration and stick to it.

4. Confusing Correlation with Causation

Only randomized experiments (A/B tests) establish causation. Observational data (analytics) shows correlation.

5. Testing Too Many Variables

Change one thing at a time. Multi-variable tests require exponentially more sample size.

6. Ignoring Guardrail Metrics

A variant that improves sign-ups but degrades retention is not a win. Always monitor guardrails.

7. Survivorship Bias in Concierge Tests

Concierge users who stick around may not represent the broader population. Be cautious generalizing from small, self-selected samples.

8. Over-Reliance on Surveys

Surveys measure stated preference, not revealed preference. People say they want features they will never use. Prefer behavioral experiments.

Red Flags: Brainstorm Experiments

Common ways this skill's output goes wrong — concrete examples, why they're bad, and how to fix them. Pair with the SKILL.md and Troubleshooting table.

How to use this document

When you have just designed an experiment, scan the red flags below before running it. Each red flag shows a bad and good version of an experiment specification.

---

Red Flag 1: Confirmation-Biased Experiment Design

Symptom. The experiment is structured so that any plausible outcome will be interpreted as success. The hypothesis is unfalsifiable.

Why it's bad. Savoia's pretotyping insists on falsifiable XYZ hypotheses ("At least X% of Y will do Z") because the only useful experiment is one that can fail. Confirmation-biased designs produce false confidence: the team believes they validated the idea, but actually validated nothing.

Bad example:

Hypothesis: "Customers will be interested in our new AI assistant." Measurement: "Survey users; if any positive sentiment, we proceed."

Good example:

Hypothesis: "At least 25% of trial users who see the AI-assistant prompt will click 'Try It' within their first session." Threshold: 25%. Below 15% = kill. 15-25% = iterate on prompt. Above 25% = proceed to build."

How to catch it. State your kill criterion before running. If you cannot describe a result that would make you stop, the experiment is confirmation theater.

---

Red Flag 2: Asking Opinions Instead of Observing Behavior

Symptom. Experiment is a survey: "Would you use a feature that does X?" Results show 78% say yes; team builds; nobody uses it.

Why it's bad. Stated preference is a notoriously bad predictor of actual behavior (Fitzpatrick's Mom Test). People say they would exercise, would pay for premium, would use a productivity tool. They do not, do not, and do not. Pretotyping requires behavioral signal, not stated.

Bad example:

Survey: "Would you pay $19/month for an AI writing assistant?" Result: 67% yes. Decision: build.

Good example:

Smoke test: landing page with $19/month price and "Buy Now" button (routes to "coming soon, get notified" form requiring email + credit-card-on-file). Result: 4.2% of 5,000 visitors completed form. Decision: above 3% threshold, proceed."

How to catch it. Does the result rely on what people say, or what they do? If the former, redesign.

---

Red Flag 3: Sample Size Too Small to Mean Anything

Symptom. Experiment ran with 8 users. 5 of them clicked the button. Team declares "62.5% conversion" and proceeds.

Why it's bad. Small samples produce noisy results. A 5-of-8 result could easily be 2-of-8 or 7-of-8 in a different sample. Decisions made on small samples are coin flips with confidence theater.

Bad example:

"8 users tested. 5 clicked Upgrade. Conversion = 62.5%. Decision: ship."

Good example:

"Target: 200 users in landing-page test. Stopping rule: 95% confidence band must exclude 0%-or-baseline. Pre-registered threshold: 4% conversion. Actual: 247 users, 11 conversions (4.5%), confidence interval [2.5%, 7.1%]. Decision: proceed (lower bound > baseline)."

How to catch it. State your minimum sample size before running. Do not stop early just because the result looks good.

---

Red Flag 4: Mixing Variables (Can't Tell What Caused What)

Symptom. Experiment changes the headline, the button color, the price, and the call-to-action all at once. Some metric moves. Nobody knows which change drove it.

Why it's bad. Confounded experiments produce uninterpretable results. Multi-variable tests have a place (multivariate testing) but require either much larger samples or sequential isolation. Default to one variable per experiment.

Bad example:

A/B test compares "Old page" vs "New page" where new page has new headline, new image, new price, and new CTA. Conversion +12%. Decision: ship new page.

Good example:

Sequential single-variable tests: (1) headline test — flat. (2) image test — +3%. (3) price test — +8%. (4) CTA test — +1%. Total understanding: price was the driver. New experiment: explore price elasticity further."

How to catch it. How many variables changed in the variant? If more than 1, you have a confounded design.

---

Red Flag 5: Pretotype That Costs Too Much to Be a Pretotype

Symptom. "Quick smoke test" took 6 weeks and 3 engineers. By the time it ran, the team was already committed to the idea.

Why it's bad. The whole point of pretotyping is speed and cheapness. A pretotype that costs 6 weeks loses its purpose — by the time it returns data, the sunk-cost fallacy ensures the team will ship regardless. Pretotypes should cost hours, not weeks.

Bad example:

"Pretotype: built a working prototype with backend, login, and 3 features. 6 weeks engineering effort."

Good example:

"Pretotype: landing page with mockup screenshots and a 'Get Early Access' form. Built in 2 hours using Carrd. Live for 7 days, drove traffic via $200 LinkedIn ads. 11% form completion. Decision: above 7% threshold, build MVP."

How to catch it. Time from idea to live experiment. If above 1 week, it is not a pretotype.

---

Red Flag 6: No Counter-Metric (Only Looking at What Goes Up)

Symptom. Experiment measures "clicks on Upgrade button" but ignores whether users who clicked actually paid, retained, or were happy.

Why it's bad. Optimizing one metric without watching counter-metrics is how teams ship growth hacks that damage NPS. The experiment "succeeds" on the local metric and harms the global outcome.

Bad example:

Test: aggressive upsell modal. Result: Upgrade clicks +280%. Decision: ship.

Good example:

Test: aggressive upsell modal. Primary: Upgrade clicks +280%. Counter-metrics: 30-day refund rate +45%, NPS for upgraders -12 points, support tickets +60%. Decision: kill — net negative."

How to catch it. Did you pre-declare a counter-metric? If not, you are not measuring damage.

---

Red Flag 7: Hypothesis Without "Why" (No Causal Theory)

Symptom. "Let's try adding social proof to the landing page." No theory of why it should work, no prediction of magnitude.

Why it's bad. Experiments without causal theory teach you nothing transferable. Even if the result is positive, you do not know why, so you cannot generalize. The "X% of Y will do Z" format requires the why to be explicit.

Bad example:

"Add social proof. See if conversion goes up."

Good example:

"Hypothesis: SMB buyers (Y) on our landing page are blocked by trust uncertainty. Adding logos of recognizable customers (treatment) will increase signup conversion by at least 15% (X = 15pp lift from current 6% baseline). If true, trust-signaling is a real lever and we should test pricing-page logos next."

How to catch it. Can you state the causal theory in one sentence? If not, the experiment will not generalize.

---

Red Flag 8: Running Experiments Without Pre-Registration

Symptom. Test runs for a week. Looks at results. Defines "what counts as success" after seeing the numbers. Reports it as a win.

Why it's bad. Post-hoc threshold-setting is p-hacking. Any random data set will have something that looks like a win when you choose the metric after looking. Pre-registration (writing down what you'll measure and what threshold counts) eliminates this.

Bad example:

"We ran the test for 10 days. Conversion was flat, but session length went up 8%. We're calling it a win."

Good example:

"Pre-registered: primary = conversion to signup, threshold +15%. Secondary tracked but not decision-making: session length, page-view depth. Result: conversion -2% (not significant), session length +8%. Pre-registered decision: not a win, do not ship."

How to catch it. Is the metric + threshold + decision rule written down before the test started? If not, you are p-hacking.

---

Red Flag 9: Treating Statistical Significance as Practical Significance

Symptom. A/B test result: variant +0.4% conversion at p<0.05 (statistically significant). Team ships and celebrates.

Why it's bad. Statistical significance just means "this result is unlikely under the null." It does not mean the effect is large enough to be worth the engineering cost. A 0.4% lift in conversion may not pay for the maintenance burden of the new code path.

Bad example:

"+0.4%, p<0.05. Ship it."

Good example:

"+0.4%, p<0.05 (statistically significant). Practical significance: 0.4% of 1M monthly users at $40 ARPU = $192K ARR. Engineering cost to ship + maintain: ~$50K/year. Net positive. Ship. But: do not pursue similar 0.4%-lift experiments without first checking expected ROI — most won't clear the bar."

How to catch it. Did you compare effect size to maintenance cost? If only the p-value is in the report, the decision is incomplete.

---

Red Flag 10: Selection Bias in the Test Audience

Symptom. Experiment runs on power users (logged in, 30-day-active) and projects results to all users.

Why it's bad. Power users behave differently from average users. A feature that wins among power users may fail among casual users. Selection bias produces over-optimistic projections that fail in production rollout.

Bad example:

"Ran beta with our 50 most engaged customers. 80% loved it. Projecting full rollout will see strong adoption."

Good example:

"Ran beta with stratified sample: 20 power users (30-day-active), 20 casual (7-30 day active), 20 lapsed (30-90 day inactive). Adoption: 80%, 35%, 8% respectively. Full rollout will see ~25% blended adoption — adjust expectations and rollout plan."

How to catch it. Does your test audience match your target audience? If you tested on a slice, do not project to the whole.

---

Red Flag 11: Experiment That Cannot Be Stopped (No Kill Switch)

Symptom. Test runs hot — engagement drops 15%, error rate climbs — but there is no clear mechanism to halt it. Discussion drags on for days while damage continues.

Why it's bad. Every experiment must have a kill switch and a kill criterion. Experiments that cannot be stopped quickly become outages.

Bad example:

A/B test deployed via code. Rollback requires a new deploy. Discussion: "do we kill it?" takes 3 days while error rate stays elevated.

Good example:

A/B test deployed via feature flag with kill criteria: error rate +20% triggers automatic flag-off; manual override available 24/7. Documented kill switch + on-call ownership. Test rolled back within 15 minutes when criterion hit on day 4."

How to catch it. Can you turn off this experiment in under 15 minutes without a deploy? If not, you have no kill switch.

---

Red Flag 12: Treating One Successful Experiment as Strategic Validation

Symptom. One A/B test confirms a smaller change works. Team uses this as evidence to commit to a 6-month strategic bet.

Why it's bad. Tactical experiments validate tactical hypotheses. They do not validate strategic theses. A button-color win does not validate the product direction. The team is over-claiming the scope of validation.

Bad example:

"Our 'Try Free' button A/B test showed +12% conversion. This validates our self-serve strategy."

Good example:

"Our 'Try Free' button A/B test showed +12% conversion. This validates one tactical change on one page. The self-serve strategy requires separate validation: PMF signal in the self-serve segment, retention, expansion, support load. Separate experiments planned."

How to catch it. What is the scope of your validation? If it is a tactical change, do not claim strategic validation.

---

Red Flag Quick Reference

Related Reading

• SKILL.md Troubleshooting section (for symptom -> root cause -> resolution)
• references/xyz-hypothesis.md (for the XYZ format, if present)
• identify-assumptions/references/red-flags.md (for assumption-prioritization patterns)
• activation-funnel/references/red-flags.md (for metric counter-metric patterns)