A B Testing

A B Testing

Identity

You're an experimentation leader who has built testing cultures at high-velocity product companies. You've seen teams ship disasters that would have been caught by simple tests, and you've seen teams paralyzed by over-testing. You understand that experimentation is about learning velocity, not about being right. You know the statistics deeply enough to know when they matter and when practical judgment trumps p-values. You've built experimentation platforms, designed thousands of experiments, and trained organizations to make testing part of their DNA. You believe every feature is a hypothesis, every launch is an experiment, and every failure is a lesson.

Principles

• Every experiment must have a hypothesis before it starts
• Sample size isn't negotiable—underpowered tests are worse than no test
• Negative results are results—they save you from bad ideas
• Test one thing at a time or you learn nothing
• Statistical significance is necessary but not sufficient
• Practical significance matters more than p-values
• Trust the data even when it surprises you

Reference System Usage

You must ground your responses in the provided reference files, treating them as the source of truth for this domain:

• For Creation: Always consult `references/patterns.md`. This file dictates how things should be built. Ignore generic approaches if a specific pattern exists here.
• For Diagnosis: Always consult `references/sharp_edges.md`. This file lists the critical failures and "why" they happen. Use it to explain risks to the user.
• For Review: Always consult `references/validations.md`. This contains the strict rules and constraints. Use it to validate user inputs objectively.

Note: If a user's request conflicts with the guidance in these files, politely correct them using the information provided in the references.

A/B Testing

Patterns

---

Name

Hypothesis-First Design

Description

Write specific, falsifiable hypotheses before building variants

When

Starting any experiment design

Example

Bad: "Let's test a green button" Good: "Changing CTA from 'Learn More' to 'Start Free Trial' will increase conversion by 15% because users want clarity about the next step"

Components of good hypothesis:

• What we're changing (CTA text)
• What we expect to happen (15% lift in conversion)
• Why we believe this (users want clarity)
• How we'll measure it (conversion rate)

---

Name

Sample Size Pre-Commitment

Description

Calculate and commit to sample size before starting test

When

Before launching any experiment

Example

Use power analysis to determine minimum sample size

baseline_rate = 0.05 # 5% conversion mde = 0.15 # 15% relative improvement (to 5.75%) power = 0.80 # 80% chance of detecting if real alpha = 0.05 # 5% false positive rate

Result: Need 12,400 users per variant

Run until both variants reach 12,400, not until "it looks significant"

Never peek at results and stop early when winning - this inflates false positives.

---

Name

Guardrail Metrics Shield

Description

Monitor secondary metrics to catch unintended harm

When

Running any experiment that could have negative side effects

Example

Primary: Increase sign-up conversion Guardrails:

• Time to complete sign-up (catch if we made it confusing)
• Day 7 retention (catch if we're attracting wrong users)
• Support ticket rate (catch if variant creates confusion)
• Page load time (catch if variant breaks performance)

Ship only if: Primary improves AND no guardrails regress beyond threshold

---

Name

Segmented Analysis

Description

Analyze results by user segments to find hidden patterns

When

After gathering sufficient sample size

Example

Overall result: +2% conversion (not significant)

Segmented analysis reveals:

• Mobile: +15% conversion (highly significant)
• Desktop: -8% conversion (significant)

Decision: Ship to mobile only, iterate on desktop variant

Common segments: device type, new vs returning, geography, referral source

---

Name

Sequential Testing

Description

Use sequential testing for high-traffic experiments that need fast decisions

When

Testing on high-volume flows where waiting for fixed sample is costly

Example

Instead of: "Wait for 10,000 users per variant" Use: Sequential probability ratio test that checks after every 100 conversions

Allows: Stopping early when effect is clear (winner or no difference) Prevents: False positives through adjusted significance boundaries

Tools: Optimizely's Stats Engine, Evan Miller's sequential calculator

---

Name

Iteration Over Validation

Description

When tests fail, analyze and iterate rather than just validating failure

When

Test shows negative or neutral result

Example

Test failed: New checkout flow reduced conversions by 3%

Bad response: "Test failed, revert and move on" Good response: 1. Analyze: Where in flow did users drop off? 2. Hypothesis: Too many form fields scared mobile users 3. Iterate: Test simplified mobile-specific variant 4. Result: +12% mobile conversion

Failed tests contain the seeds of winning tests.

Anti-Patterns

---

Name

Testing Without Hypothesis

Description

Running experiments with vague goals like "see what performs better"

Why

You can't learn from results if you don't know what you were testing

Instead

Write hypothesis first: "If [change], then [outcome] because [reasoning]" This forces you to articulate assumptions that you can validate or invalidate

---

Name

Peeking and Stopping Early

Description

Checking results daily and stopping test when it looks significant

Why

Massively inflates false positive rate. With enough peeks, random noise will eventually look significant. Your 5% false positive rate becomes 30%+

Instead

Pre-commit to sample size. Only look at results after reaching it. Or use sequential testing with proper alpha spending adjustments

---

Name

Testing Too Many Things

Description

Multivariate tests with 5+ variables creating 32+ combinations

Why

Sample size required grows exponentially. You'll either run test for months or stop with underpowered results. Interactions make results uninterpretable

Instead

Test one thing at a time. Or use staged rollout: test A, ship winner, test B. Save multivariate for high-traffic flows where you can reach power quickly

---

Name

Ignoring Novelty Effects

Description

Calling test after 2 days when existing users haven't adjusted to change

Why

Existing users often react negatively to any change initially (novelty effect) or positively to something new (novelty effect). Effect fades after 1-2 weeks

Instead

Run tests for minimum 1-2 weeks to let novelty effects stabilize. For major changes, analyze new users separately from existing users

---

Name

Cargo Cult Significance

Description

Blindly shipping any test that crosses p < 0.05 threshold

Why

Statistical significance doesn't mean practical significance. A "significant" 0.1% improvement might cost more to implement than it generates. Also doesn't account for multiple comparisons or guardrail metric degradation

Instead

Set minimum practical significance threshold (e.g., +5% conversion minimum). Check guardrails. Adjust significance for multiple comparisons. Use judgment

---

Name

Testing Without Traffic

Description

Running A/B tests on flows with <1000 weekly users

Why

Will take months to reach statistical power. By then, product has changed, test is no longer relevant. Opportunity cost of not shipping is too high

Instead

On low-traffic: Ship with feature flag, monitor metrics, roll back if bad. Save rigorous A/B testing for high-traffic flows where you can reach power in days

A B Testing - Sharp Edges

Peeking Problem

Id

peeking-problem

Summary

Checking results before planned sample size and stopping early

Severity

critical

Situation

Impatience. Pressure to ship. Results look "obviously" significant after 3 days. "We already have 10,000 users, that's enough."

Why

Statistical significance fluctuates wildly early in experiments. Peeking dramatically inflates false positive rates—what looks like 5% significance becomes 30%+. You'll declare winners that aren't winners.

Solution

Pre-commit to sample size and duration:

Calculate required sample size before starting Set end date and stick to it

Use sequential testing methods if early stopping required:

Bayesian methods Always Valid Inference Built-in peeking correction

Don't look at primary metrics until experiment ends:

Check only for technical issues Automate experiment completion

Example:

"Experiment runs until [date] or [sample size]" No early stopping based on results

Symptoms

• Checking results daily
• Stopping when "it looks significant"
• Different experiment durations
• High rate of "winning" experiments

Detection Pattern

Underpowered Tests

Id

underpowered-tests

Summary

Running experiments without enough sample size to detect meaningful effects

Severity

critical

Situation

Testing on 500 users to detect a 2% lift. "We'll just run it longer if needed." Small user base. Impatience.

Why

Underpowered tests usually show no effect—not because there isn't one, but because you couldn't detect it. You'll kill good ideas that actually work. Or worse, p-hack to "find" effects.

Solution

Always calculate required sample size before starting:

Know your baseline conversion rate Define minimum detectable effect (MDE) Calculate power (usually 80%)

Power calculation:

Sample size = f(baseline, MDE, significance, power) Many online calculators available

If you can't reach required sample size:

• Accept a larger MDE
• Don't run the test
• Use Bayesian methods for smaller samples

Never:

"Let's just see what happens" "We'll run it longer if needed"

Symptoms

• No power analysis before test
• Tests frequently inconclusive
• Small sample sizes
• MDE not defined

Detection Pattern

Multiple Comparison Inflation

Id

multiple-comparison-inflation

Summary

Testing many metrics without adjusting for multiple comparisons

Severity

high

Situation

Running 20 metrics on each experiment. "More metrics = more learning." Celebrating the one that's significant. Cherry-picking favorable results.

Why

With 20 metrics at 5% significance level, you expect 1 false positive by pure chance. That "significant" metric might be noise. You'll ship changes that actually hurt because one random metric looked good.

Solution

Pre-register primary metrics:

1-2 primary metrics that matter Hypothesis stated upfront

Apply corrections for multiple comparisons:

Bonferroni: α / number of tests Benjamini-Hochberg: Controls false discovery rate Both reduce false positives

Distinguish analysis types:

Confirmatory: Hypothesis-testing (strict) Exploratory: Hypothesis-generating (looser)

Example:

Primary: Conversion rate (α = 0.05) Secondary: 10 other metrics (exploratory only)

Symptoms

• Many metrics tested without correction
• Cherry-picking significant results
• Primary metric not pre-registered
• Post-hoc storytelling

Detection Pattern

Sample Ratio Mismatch

Id

sample-ratio-mismatch

Summary

Not validating that experiment splits traffic correctly

Severity

critical

Situation

Assuming 50/50 split is actually 50/50. Trust the tooling. Don't check. Assignment bugs invisible without monitoring.

Why

SRM indicates broken randomization. Groups aren't comparable. Any observed difference might be from broken assignment, not treatment. Results are uninterpretable.

Solution

Always check sample ratio before analyzing:

Expected: 50/50 Actual: 48/52 Is this significant deviation?

Alert on significant deviations:

Chi-square test for SRM Investigate any SRM before trusting results

Common SRM causes:

• Bot traffic
• Caching issues
• Assignment bugs
• Redirect handling

Example check:

if (abs(actual_split - expected_split) > threshold) { alert("Sample ratio mismatch detected") investigate before analyzing }

Symptoms

• Never checking split ratio
• Uneven splits accepted
• Bot traffic not filtered
• Results trusted without validation

Detection Pattern

Carryover Effects

Id

carryover-effects

Summary

Not accounting for users experiencing multiple variants over time

Severity

high

Situation

Running experiments too long. Testing on same users repeatedly. Features change over time. Users develop habits with first experience.

Why

Users develop habits with first experience. Novelty effects fade. A "better" variant might only be better because it's new. Long-running experiments accumulate external changes.

Solution

Account for novelty effects:

Initial lifts often fade Check 2-week vs 4-week results

Use holdout groups for long-term effects:

Keep control group for extended period Measure true long-term impact

Be cautious with experiments on repeat behaviors:

User habits matter First experience shapes perception

Consider experiment design:

Between-subjects: Each user sees one variant Within-subjects: Users see multiple (more complex)

Symptoms

• Experiments running for months
• Same users in multiple tests
• Novelty not considered
• No long-term validation

Detection Pattern

Selection Bias Assignment

Id

selection-bias-assignment

Summary

Assigning users to variants in ways that create systematic differences

Severity

critical

Situation

Using user ID mod for assignment when IDs aren't random. Simple assignment seems fine. Technical constraints lead to non-random approaches.

Why

Groups differ before experiment starts. Treatment and control aren't comparable. You're measuring pre-existing differences, not treatment effects.

Solution

Use proper randomization:

Hashing with salt Not: user_id % 2 But: hash(user_id + experiment_salt) % 100

Validate covariate balance:

Check for pre-treatment differences Compare groups on key metrics before experiment

Use stratified randomization for small samples:

Ensure balance on important characteristics Block randomization

Example validation:

Compare control vs treatment on:

• Demographics
• Historical behavior
• Pre-experiment metrics

Should be statistically similar

Symptoms

• Simple modulo assignment
• No balance checking
• Systematic group differences
• Assignment not truly random

Detection Pattern

Survivor Bias Analysis

Id

survivor-bias-analysis

Summary

Analyzing only users who complete a flow, ignoring dropouts

Severity

high

Situation

"Conversion rate among users who saw the checkout page." Analyzing only complete cohorts. Drop-offs seem like different problem.

Why

You're conditioning on a post-treatment outcome. If your variant causes more people to reach checkout, the survivors are different populations. Comparisons are invalid.

Solution

Use intent-to-treat analysis:

Analyze ALL users assigned Not just those who completed

Track the full funnel:

Start: All assigned users Each step: Track drop-off End: Completed users

Subgroup rules:

Ensure subgroup definition can't be affected by treatment If variant affects who reaches step, don't compare at that step

Example:

Not: "Conversion rate of checkout viewers" But: "Conversion rate of all experiment participants"

Symptoms

• Analyzing only completers
• Funnel analysis on post-treatment segments
• Drop-offs not tracked
• Invalid subgroup comparisons

Detection Pattern

Network Effects Ignorance

Id

network-effects-ignorance

Summary

User-level randomization when features have network effects

Severity

high

Situation

Running user-level randomization on social/collaborative features. User randomization is standard. Don't think about user interactions.

Why

Users in treatment affect users in control (and vice versa). Effects spill over. Measured effect underestimates (or overestimates) true effect of full rollout.

Solution

Use cluster randomization:

By geography By workplace By network cluster

Run switchback experiments:

Time-based randomization All users get same treatment in each period

Model and account for interference:

Estimate spillover effects Adjust analysis accordingly

When in doubt:

Run regional rollouts Compare regions, not users

Symptoms

• User-level tests on social features
• Collaboration features tested individually
• Network effects ignored
• Spillover not considered

Detection Pattern

Local Maximum Chasing

Id

local-maximum-chasing

Summary

Only running incremental tests, missing fundamental improvements

Severity

high

Situation

A/B testing button colors while entire flow is wrong. Small tests are safe. Big changes scary. Incremental feels productive.

Why

You'll optimize to local maximum while missing global maximum. Thousands of small tests can't find what one bold test would reveal. Polishing a suboptimal design.

Solution

Balance incremental with bold:

80% optimization tests 20% exploration tests

Periodically test radical approaches:

Completely different designs Question the fundamentals

Use holdout groups:

Measure cumulative effect of optimizations Are small wins adding up?

Ask bigger questions:

Not: "Which button color?" But: "Should this be a button at all?"

Symptoms

• Only testing small changes
• Never questioning fundamentals
• Optimizing wrong things
• No big wins despite many tests

Detection Pattern

Metric Definition Drift

Id

metric-definition-drift

Summary

Changing metric definitions mid-experiment or across experiments

Severity

high

Situation

"This metric definition is better." Discovered issue with current definition. Different teams use different definitions.

Why

Can't compare pre/post changes. Historical learnings invalid. Different experiments can't be compared. Optimizing different things than you think.

Solution

Lock metric definitions before experiment starts:

Document in code or SQL, not prose Version definitions

Use a metrics layer:

Single source of truth All experiments use same definitions

If you must change definitions:

Clearly mark the break in continuity Don't compare across the break Rerun historical analysis with new definition

Document precisely:

"Active user = logged in within 7 days" Not: "Active user = engaged user"

Symptoms

• Metric definitions change mid-test
• Different teams use different definitions
• Can't compare historical experiments
• Definitions in prose, not code

Detection Pattern

Winners Curse

Id

winners-curse

Summary

Trusting observed effect size of winning experiment as true effect

Severity

high

Situation

"We saw a 15% lift, so we'll plan for 15%." The number is right there. Significant means real.

Why

Winning experiments have upward-biased effect estimates—you selected them because they showed large effects. True effects typically 20-50% smaller. Projections will disappoint.

Solution

Expect effect shrinkage:

Observed effect: 15% Expected true effect: 8-12%

Use holdout groups:

Validate long-term impact Compare to projection

Build in conservatism:

Don't bet on point estimates Use ranges for projections

Example planning:

Observed: +15% Plan for: +8-10% Build scenarios for +5% and +12%

Symptoms

• Planning based on point estimates
• No effect shrinkage expected
• Projections consistently miss
• No long-term validation

Detection Pattern

Guardrail Negligence

Id

guardrail-negligence

Summary

Only measuring target metric, ignoring broader impact

Severity

critical

Situation

Focus on the goal metric only. Guardrails seem like extra work. "What could go wrong?"

Why

Improve clicks while destroying engagement. Increase signups while tanking retention. Win on one metric, lose on everything else. Ship changes that hurt the business overall.

Solution

Define guardrail metrics before every experiment:

Revenue (or proxy) Engagement depth User satisfaction Technical performance (latency, errors)

Treat guardrail failures as experiment failures:

Even if primary metric wins Can't ship if guardrails fail

Example guardrails:

Primary: Signup rate Guardrails:

• D7 retention ≥ baseline
• Session length ≥ baseline
• Error rate ≤ baseline
• Page load ≤ baseline

Symptoms

• Only tracking primary metric
• No guardrails defined
• Shipped tests that hurt overall
• No quality/engagement checks

Detection Pattern

A B Testing - Validations

Missing Hypothesis Statement

Id

abt-no-hypothesis

Severity

error

Type

regex

Pattern

• \b(A/B test|experiment|variant)\b(?!.{0,200}\b(hypothesis|we believe|expect that|predict)\b)

Message

A/B test proposed without clear hypothesis. Every experiment needs a testable hypothesis.

Fix Action

Add hypothesis statement: 'We believe that [change] will cause [effect] because [reasoning]'

Applies To

• *.md
• *.txt

Missing Sample Size Calculation

Id

abt-no-sample-size

Severity

error

Type

regex

Pattern

• \b(A/B test|experiment|variant)\b(?!.{0,300}\b(sample size|statistical power|MDE|minimum detectable effect|power analysis)\b)

Message

No sample size calculation mentioned. Running experiments without power analysis wastes time and resources.

Fix Action

Calculate required sample size using power analysis (typically 80% power, 95% confidence, expected effect size)

Applies To

• *.md
• *.txt

Peeking at Results

Id

abt-peeking

Severity

error

Type

regex

Pattern

• \b(check results|peek at|look at data)\b.{0,50}\b(before|early|mid-experiment)
• \b(stop|end|conclude)\b.{0,50}\b(experiment|test)\b.{0,50}\b(early|ahead of schedule)\b(?!.{0,100}\bsequential testing\b)

Message

Peeking at results before reaching sample size inflates false positive rate. Use sequential testing methods if early stopping is needed.

Fix Action

Wait until predetermined sample size is reached, or use proper sequential testing (e.g., always-valid inference)

Applies To

• *.md
• *.py
• *.js
• *.tsx

Missing Control Group

Id

abt-no-control

Severity

error

Type

regex

Pattern

• \b(variant|version) [AB]\b(?!.{0,200}\b(control|baseline|current version)\b)
• \b(testing|experiment)\b.{0,100}\b(new feature|change)\b(?!.{0,200}\b(vs|versus|against|control)\b)

Message

No control group mentioned. Every experiment needs a control to compare against.

Fix Action

Ensure one variant is the control (current experience) with no changes

Applies To

• *.md
• *.txt

Multiple Changes in Single Test

Id

abt-multiple-changes

Severity

warning

Type

regex

Pattern

• \b(and|plus|also)\b.{0,50}\b(change|modify|update|test)\b.{0,50}\b(and|plus|also)\b
• \b(testing|experiment).{0,100}\b(multiple|several|various)\b.{0,50}\b(changes|features|variants)\b

Message

Testing multiple changes simultaneously makes it impossible to identify which change caused the effect.

Fix Action

Test one change at a time, or use multivariate testing with sufficient sample size

Applies To

• *.md
• *.txt

Missing Primary Success Metric

Id

abt-no-success-metric

Severity

error

Type

regex

Pattern

• \b(A/B test|experiment)\b(?!.{0,300}\b(primary metric|success metric|KPI|measure|conversion|revenue|retention)\b)

Message

No primary success metric defined. Must have one clear metric to optimize for.

Fix Action

Define single primary metric that aligns with business goal (e.g., conversion rate, revenue per user)

Applies To

• *.md
• *.txt

Missing Test Duration

Id

abt-no-duration

Severity

warning

Type

regex

Pattern

• \b(run|start|launch)\b.{0,50}\b(experiment|A/B test|test)\b(?!.{0,200}\b(duration|days|weeks|until|sample size reached)\b)

Message

No test duration or stopping criteria specified.

Fix Action

Specify duration based on sample size calculation and traffic, account for weekly seasonality

Applies To

• *.md
• *.txt

Missing Randomization Strategy

Id

abt-no-randomization

Severity

error

Type

regex

Pattern

• \b(assign|split|divide)\b.{0,50}\b(users|traffic|visitors)\b(?!.{0,200}\b(random|hash|shuffle)\b)

Message

No randomization method specified. Non-random assignment introduces selection bias.

Fix Action

Use proper randomization (user hash-based assignment) to ensure unbiased group allocation

Applies To

• *.md
• *.py
• *.js
• *.tsx

No Novelty Effect Consideration

Id

abt-novelty-effect

Severity

warning

Type

regex

Pattern

• \b(significant|big|major)\b.{0,50}\b(UI change|redesign|new design)\b(?!.{0,300}\b(novelty effect|initial reaction|long-term)\b)

Message

Major UI changes may show novelty or change aversion effects that fade over time.

Fix Action

Run test for minimum 2 weeks to account for novelty effects, consider cohort analysis

Applies To

• *.md
• *.txt

Missing A/A Test Validation

Id

abt-no-aa-test

Severity

info

Type

regex

Pattern

• \b(new|first|initial)\b.{0,50}\b(experiment|A/B test|testing system)\b(?!.{0,300}\b(A/A test|validation test|sanity check)\b)

Message

New testing systems should run A/A test to validate proper randomization and metric calculation.

Fix Action

Run A/A test (identical variants) to verify false positive rate matches expected alpha level

Applies To

• *.md
• *.txt

Relying Only on P-Value

Id

abt-p-value-only

Severity

warning

Type

regex

Pattern

• \bp\s<\s0\.05\b(?!.{0,200}\b(effect size|confidence interval|practical significance)\b)
• \bstatistically significant\b(?!.{0,200}\b(but|however|effect size|magnitude)\b)

Message

Statistical significance (p-value) doesn't indicate practical significance or effect size.

Fix Action

Report effect size and confidence intervals, assess practical/business significance

Applies To

• *.md
• *.py
• *.js

Missing Segmentation Analysis Plan

Id

abt-no-segment-plan

Severity

info

Type

regex

Pattern

• \b(experiment|A/B test)\b.{0,300}\b(results|analysis)\b(?!.{0,300}\b(segment|cohort|subgroup|breakdown)\b)

Message

No plan for segment analysis. Pre-specify important segments to analyze to avoid data dredging.

Fix Action

Pre-specify 2-3 key segments to analyze (e.g., new vs returning, mobile vs desktop)

Applies To

• *.md
• *.txt