Ab Testing Framework

A/B Testing Framework

Design, run, and analyze conversion experiments with statistical rigor.

Install

git clone https://github.com/thatrebeccarae/claude-marketing.git && cp -r claude-marketing/skills/ab-testing-framework ~/.claude/skills/

Test Design Process

Step 1: Hypothesis

Template: If we [change X], then [metric Y] will [increase/decrease] by [Z%] because [reason].

Good hypothesis: "If we change the CTA from Get Started to Start Free Trial, then signup rate will increase by 15% because it reduces uncertainty about cost."

Bad hypothesis: "If we change the button color, conversions will improve." (No reasoning, no expected magnitude.)

Step 2: Sample Size Calculation

To determine how long to run a test:

Required sample per variation = 16 * (p * (1-p)) / (MDE^2)

Where:
  p = baseline conversion rate (as decimal)
  MDE = minimum detectable effect (as decimal)

Minimum test duration: 2 full business weeks (to capture day-of-week effects), even if sample size is reached sooner.

Step 3: Test Execution Rules

1. Random assignment — visitors must be randomly assigned to control/variant 2. No peeking — do not check results before reaching sample size 3. No mid-test changes — do not modify variants during the test 4. Even traffic split — 50/50 for A/B, even splits for multivariate 5. Single variable — change only one thing per test (unless multivariate) 6. Full duration — run for the pre-calculated duration, not until significance

Step 4: Statistical Analysis

Frequentist Approach

Z-test for proportions:

Z = (p1 - p2) / sqrt(p_pooled * (1 - p_pooled) * (1/n1 + 1/n2))

Where:
  p1, p2 = conversion rates of control and variant
  p_pooled = (x1 + x2) / (n1 + n2)
  n1, n2 = sample sizes

p-value interpretation:

• p < 0.05: Statistically significant (95% confidence)
• p < 0.01: Highly significant (99% confidence)
• p >= 0.05: Not significant — do not declare a winner

Bayesian Approach

When to use Bayesian:

• Low traffic (small sample sizes)
• Need to make decisions faster
• Want probability of each variant being best (not just "significant or not")

Interpretation: "There is a 94% probability that Variant B is better than Control" vs frequentist "We reject the null hypothesis at 95% confidence."

Step 5: Decision Framework

Common Testing Pitfalls

1. Peeking — checking results early inflates false positive rate from 5% to 26%+ 2. Stopping early — reaching significance != reaching required sample size 3. Testing too many variants — each variant needs full sample size 4. Ignoring segments — overall winner may be loser for key segments 5. Too small an effect — testing for 2% lift needs enormous sample sizes 6. Not accounting for seasonality — run full weeks, avoid holidays 7. Multiple metrics — primary metric must be pre-declared; secondary are directional 8. Survivorship bias — only measuring users who complete, not those who abandon 9. Simpson paradox — segment-level winners can reverse at aggregate level 10. Novelty effect — new designs get temporary lift; re-test after 2-4 weeks

What to Test (Prioritized by Impact)

High Impact

• Value proposition / headline
• CTA text and placement
• Pricing and offer structure
• Form length (fields removed)
• Page layout (single column vs multi)
• Social proof presence and placement

Medium Impact

• Image/video vs static
• Testimonial format (text vs video)
• Navigation presence on landing pages
• Trust badges and security signals
• Urgency elements (countdown, stock)

Low Impact (Usually Not Worth Testing)

• Button color (unless extreme contrast issue)
• Font changes
• Minor copy tweaks
• Icon styles
• Footer content

Integration with Other Skills

• cro-auditor — CRO audit generates test hypotheses; this skill designs the experiments
• google-analytics — GA4 for experiment data and segment analysis
• copywriting-frameworks — Generate variant copy using proven frameworks

A/B Testing Framework — Examples

Example 1: Landing Page Headline Test

Prompt

Design an A/B test for our SaaS landing page headline. Current conversion rate is 3.2% with 5,000 visitors/week. We want to test a benefit-focused headline vs our current feature-focused one.

What the skill does

1. Formulates hypothesis: "Changing headline from feature-focused to benefit-focused will increase signup rate by 20% because benefits resonate more with non-technical buyers" 2. Calculates sample size: baseline 3.2%, MDE 20% → ~14,400 per variation → ~29,000 total → ~6 weeks at 5K/week 3. Designs test: Control ("AI-Powered Analytics Platform") vs Variant ("See What Is Costing You Customers — in 60 Seconds") 4. Specifies: 50/50 split, primary metric = signup rate, secondary = bounce rate 5. Sets decision criteria and pre-registers the analysis plan

---

Example 2: Analyze Test Results

Prompt

Our A/B test finished. Control: 412 conversions from 15,200 visitors. Variant: 478 conversions from 15,100 visitors. Is this significant?

What the skill does

1. Calculates conversion rates: Control 2.71%, Variant 3.17% 2. Runs Z-test: Z = 2.41, p = 0.016 (statistically significant at 95%) 3. Calculates relative lift: +17.0% 4. Computes confidence interval: 3.1% to 31.8% relative lift 5. Recommendation: Implement variant. 98.4% probability variant is better than control.

---

Example 3: Low-Traffic Test Strategy

Prompt

We only get 800 visitors/month to our pricing page. Can we still A/B test?

What the skill does

1. Calculates: at 800/month, testing for 20% lift on a 5% conversion rate would take 24+ months — not feasible 2. Recommends alternatives: test for larger effects (50%+ lift), use Bayesian approach for faster reads, test higher-traffic pages first 3. Suggests qualitative research instead: user testing, session recordings, survey 4. Proposes "big swing" tests that could show 50%+ lift (restructure pricing page entirely vs minor copy tweak)

Ab Testing Framework Reference

Reference data, benchmarks, and platform-specific details for the Ab Testing Framework skill.

<!-- TODO: Add platform API schemas, benchmark data, and domain-specific reference tables -->