Measure Experiment Design

<!-- PM-Skills | https://github.com/product-on-purpose/pm-skills | Apache 2.0 -->

Experiment Design

An experiment design document defines all parameters needed to run a rigorous A/B test or controlled experiment. It ensures the team aligns on what you're testing, how you'll measure success, and how long to run the test before drawing conclusions. Good experiment design prevents common pitfalls: underpowered tests, unclear success criteria, and decisions based on noise rather than signal.

When to Use

• Before launching an A/B test to validate a product change
• When testing a hypothesis that requires quantitative validation
• After solution design to validate assumptions before full rollout
• When stakeholders want data-driven evidence for a decision
• To establish a culture of experimentation and learning

When NOT to Use

• The hypothesis itself is not yet articulated -> use define-hypothesis first; this skill designs the test for a claim you already have
• You are analyzing a completed experiment -> use measure-experiment-results
• You need the event tracking that will measure the experiment -> use measure-instrumentation-spec
• You are gathering opinions rather than running a controlled test -> use measure-survey-analysis

Instructions

When asked to design an experiment, follow these steps:

1. Articulate the Hypothesis Write a clear, testable hypothesis in the format: "We believe [change] for [users] will [outcome] as measured by [metric]." One hypothesis per experiment - if you're testing multiple things, run multiple experiments.

2. Define the Variants Describe the control (current experience) and treatment (new experience) in sufficient detail. Include screenshots, mockups, or precise descriptions so anyone can understand what users will see.

3. Choose Primary and Secondary Metrics Select one primary metric that will determine success or failure. Add 2-3 secondary metrics to understand the broader impact. Include guardrail metrics to catch unintended negative effects.

4. Calculate Sample Size Determine how many users you need per variant to detect your minimum detectable effect (MDE) with statistical significance. Specify your significance level (typically 0.05) and power (typically 0.80).

5. Estimate Duration Based on sample size and available traffic, calculate how long the experiment needs to run. Account for weekly patterns - avoid ending mid-week if behavior varies by day.

6. Define Targeting and Allocation Specify which users are eligible for the experiment and how traffic is split between variants. Document any exclusions (e.g., employees, specific segments).

7. Set Success Criteria Define upfront what constitutes a win, a loss, or an inconclusive result. This prevents post-hoc rationalization and moving goalposts.

8. Document Risks and Mitigations Identify what could go wrong and how you'll detect/address it. Include monitoring plans and rollback criteria.

Output Format

Use the template in references/TEMPLATE.md to structure the output. A complete design fills every template section: Overview; Hypothesis; Background; Variants; Metrics; Sample Size & Duration; Audience Targeting; Success Criteria; Risks & Mitigations; Implementation Notes; and References.

Quality Checklist

Before finalizing, verify:

• [ ] Hypothesis is falsifiable and specific
• [ ] Only one primary metric is defined
• [ ] Sample size calculation is documented with assumptions
• [ ] Duration accounts for traffic patterns and statistical requirements
• [ ] Success criteria are defined before the experiment starts
• [ ] Guardrail metrics protect against unintended harm

Examples

See references/EXAMPLE.md for a completed example.

Scenario: A/B test a new onboarding checklist

This is the INPUT brief for an output-quality eval. The skill arm and the control arm each receive everything below (and nothing else about how to do the work) and produce an experiment-design artifact for it. Judges never see this header.

Context

Product: a B2C habit-tracking mobile app. Hypothesis the team holds: showing a 3-step guided onboarding checklist (set a goal, schedule a reminder, log day one) right after signup will get more new users to form an early habit, measured by day-7 retention.

What we know (use these numbers for the design):

• New signups: ~12,000 per week, split roughly evenly across iOS and Android.
• Current day-7 retention for new users: 34%.
• The team would consider a 3 percentage-point lift (34% -> 37%) a meaningful win worth shipping.
• Current first-session completion (any logged action): 58%.
• Average revenue per retained user matters but monetization is downstream (subscription at day 14+).
• Past experiments at this app used a 0.05 significance level and 80% power by convention.

Things the team is unsure about / wants the design to address:

• How many users per arm and how long to run, given the weekly traffic and clear weekday/weekend

behavior differences.

• What the single primary metric should be, and what secondary/guardrail metrics to watch (they worry

the checklist could feel like friction and depress first-session completion or increase day-1 uninstalls).

• What exactly counts as a win vs an inconclusive result, decided before launch.

Audience for the artifact: the growth PM and the data scientist who will run the test.

Scenario: pricing-paywall experiment with a sample-size and guardrail trap

This is the INPUT brief for an output-quality eval. The skill arm and the control arm each receive everything below (and nothing else about how to do the work) and produce an experiment design for it. Judges never see this header. This scenario is seeded with the traps a strong freehand pass tends to fall into: a low base-rate primary metric (so the sample size is large and must be derived, not asserted), real guardrail risks the test could harm, and a tempting secondary metric that invites declaring success early. The skill should win on showing the sample-size math, naming ONE primary metric, pre-registering the decision rule, and protecting guardrails; the freehand control tends to skip the sizing derivation, define success loosely, or let a secondary metric move the goalposts.

Situation

We run a freemium productivity app. We want to test a new pricing paywall: moving an existing free feature (advanced filters) behind the Pro plan, with a redesigned upgrade screen. Leadership wants to know if this lifts paid conversion without hurting the broader funnel.

What we know

• Primary business goal: increase free-to-paid conversion. Current 30-day free-to-paid conversion is

about 2.0%. The team is hoping for a lift to ~2.4% (a 0.4 percentage-point absolute lift); anything below ~2.2% would not justify the churn risk.

• Traffic: ~12,000 new free signups per week enter the funnel and can be randomized.
• Known risks the change could cause: users who lose a free feature may churn or leave bad reviews;

support contacts may spike; some users may downgrade or request refunds after upgrading under pressure.

• A tempting secondary metric: clicks on the upgrade screen will almost certainly go up (more people

hit the paywall), and the team is tempted to read that as success.

• Operational notes: the app has a strong weekly usage cycle (weekday-heavy). Finance needs a

decision in about 6 weeks. There is no prior experiment on this surface to borrow an effect size from.

What the team wants

A rigorous experiment design: a single, falsifiable primary hypothesis and ONE primary decision metric; a sample-size estimate with its assumptions (baseline rate, the minimum detectable effect, significance, power) and the calculation shown - not asserted; a duration derived from sample size and weekly traffic that respects the weekly cycle; pre-registered win/lose/inconclusive criteria set BEFORE launch; and guardrail metrics (with a monitoring/rollback plan) that protect against the churn, refund, support, and review risks. Make reasonable assumptions explicit; do not invent numbers the brief does not support.

measure-experiment-design - Version History

2.1.0 (2026-06-10)

Quality-convergence minor (F-12 Batch 4): added a "When NOT to Use" section with boundary pointers to neighboring skills, and the Output Format now enumerates the template sections a complete artifact fills. No template or example changes.

2.0.1 (2026-06-10)

Description-only patch (F-12 Batch 0, from the 2026-06-09 repo audit): the trigger-surface description was rewritten to disambiguate collision pairs with an explicit boundary pointer to the sibling skill. No body, template, or behavior changes.

2.0.0 (2026-01-26)

Baseline row for the prior published version; see git history for its changes.

Experiment Design: One-Page Checkout

Overview

Hypothesis

We believe replacing our 3-step checkout with a single-page checkout

for mobile web users

will increase checkout completion rate

as measured by checkout conversion rate (orders / checkout starts)

Background

Mobile checkout abandonment is at 73%, compared to 45% on desktop. User research identified friction points: confusion navigating between steps, anxiety about hidden costs appearing late in the flow, and form field frustration on small screens. A single-page checkout addresses these by showing all information upfront and reducing navigation.

Competitor analysis shows that Amazon, Shopify stores, and Apple use single-page or minimal-step checkouts. Our hypothesis is that reducing cognitive load and providing visibility into the full process will improve conversion.

Variants

Control (A)

Description: Current 3-step checkout flow

Details:

• Step 1: Shipping address entry
• Step 2: Shipping method selection + payment information
• Step 3: Order review and confirmation
• Each step loads a new page
• Progress indicator shows current step

Screenshot/Mockup: [designs/checkout-control-v3.png]

Treatment (B)

Description: Single-page checkout with accordion sections

Details:

• All sections visible on one page (shipping, payment, review)
• Accordion UI: one section expanded at a time, others collapsed but visible
• Express payment buttons (Apple Pay, Google Pay) at top
• Shipping cost shown immediately based on cart
• Edit any section without losing data

Screenshot/Mockup: [designs/checkout-treatment-v2.png]

Metrics

Primary Metric

Secondary Metrics

Guardrail Metrics

Sample Size & Duration

Sample Size Calculation

Calculation performed using Evan Miller's sample size calculator, assuming one-tailed test (we only ship if treatment is better).

Duration Estimate

We're running for 14 days despite reaching sample size in 2 days to:

• Capture full weekly shopping patterns (weekday vs. weekend)
• Allow time to detect delayed effects on returns/chargebacks
• Ensure novelty effect wears off

Audience Targeting

Inclusion Criteria

• Logged-in or guest users
• Mobile web (not native apps - app experiment runs separately)
• Users in US and Canada (payment methods configured)
• Cart value >= $10 (exclude micro-purchases)

Exclusion Criteria

• Employees (identified by email domain)
• Users enrolled in conflicting experiments (cart-upsell-test, payment-method-test)
• Users who have participated in checkout experiments in past 30 days

Traffic Allocation

Success Criteria

Win (Ship Treatment)

Primary metric (checkout conversion) improves by >= 5% relative with p < 0.05, AND:

• Average order value does not decrease by more than 2%
• Payment failure rate does not increase by more than 1pp
• No critical bugs or UX issues identified

Action: Roll out to 100% of mobile web, begin desktop adaptation.

Loss (Keep Control)

Any of:

• Checkout conversion decreases with p < 0.05
• Any guardrail metric breaches threshold
• Critical UX issues discovered affecting > 1% of users

Action: Revert to control, analyze learnings, iterate on design.

Inconclusive (More Data Needed)

Primary metric change is between -5% and +5% relative and not statistically significant after 14 days.

Action: Extend experiment to 21 days if traffic allows. If still inconclusive, treat as neutral and decide based on qualitative factors (user feedback, operational simplicity).

Risks & Mitigations

Monitoring Plan

• Real-time dashboard: Checkout funnel by variant, updated every 5 minutes
• Alert thresholds:
• Payment failure rate > 5% in any 1-hour window
• Checkout conversion drops > 20% relative to control
• Error rate > 1% on any checkout action
• Rollback criteria: Any alert sustained for 30 minutes triggers automatic rollback
• Daily check-ins: Team reviews metrics at 10am daily for first 5 days

Implementation Notes

• Feature flag: checkout_single_page_v1 in LaunchDarkly
• Instrumentation: New events checkout_section_opened, checkout_section_completed for funnel analysis
• Cache invalidation: Clear checkout page cache at experiment start
• Mobile detection: User-Agent parsing; tablet = desktop experience

References

• Hypothesis Document: Mobile Checkout Improvement (internal doc link)
• Design Mockups (Figma link)
• Previous Experiment: Guest Checkout (results/guest-checkout-q3-2025.md) - 3% lift, informed this design
• User Research: Checkout Friction Study (research PDF)

Experiment Design: [Experiment Name]

Overview

Hypothesis

<!-- Format: We believe [change] for [users] will [outcome] as measured by [metric] -->

We believe [proposed change]

for [target user segment]

will [expected outcome]

as measured by [primary metric]

Background

<!-- Why are we running this experiment? What's the context? -->

[Context explaining why this experiment matters and what led to the hypothesis]

Variants

Control (A)

Description: [What users currently experience]

Details:

• [Specific element 1]
• [Specific element 2]

Screenshot/Mockup: [Link or embed]

Treatment (B)

Description: [What users will experience in the new variant]

Details:

• [Specific change 1]
• [Specific change 2]

Screenshot/Mockup: [Link or embed]

Metrics

Primary Metric

Secondary Metrics

Guardrail Metrics

<!-- Metrics that should NOT decrease -->

Sample Size & Duration

Sample Size Calculation

Duration Estimate

Audience Targeting

Inclusion Criteria

• [Criterion 1: e.g., "All logged-in users"]
• [Criterion 2: e.g., "Users in US and Canada"]
• [Criterion 3: e.g., "Users on mobile web"]

Exclusion Criteria

• [Exclusion 1: e.g., "Employees and internal testers"]
• [Exclusion 2: e.g., "Users in active experiments that may conflict"]

Traffic Allocation

Success Criteria

<!-- Define BEFORE the experiment starts what each outcome means -->

Win (Ship Treatment)

[Conditions that constitute a clear win for the treatment, e.g., "Primary metric improves by >= MDE with p < 0.05, and no guardrail metrics regress beyond threshold"]

Loss (Keep Control)

[Conditions that indicate treatment is worse, e.g., "Primary metric decreases with p < 0.05, OR any guardrail metric regresses beyond threshold"]

Inconclusive (More Data Needed)

[Conditions that require further investigation, e.g., "Primary metric change is not statistically significant after full duration"]

Risks & Mitigations

Monitoring Plan

• [What we'll monitor during the experiment]
• [Alert thresholds that would trigger early review]
• [Rollback criteria]

Implementation Notes

<!-- Technical details for engineering -->

• [Feature flag name/ID]
• [Instrumentation requirements]
• [Any technical considerations]

References

• [Link to related hypothesis document]
• [Link to design mockups]
• [Link to previous related experiments]