Content Experimentation Best Practices

# Content Experimentation Best Practices

Principles and patterns for running effective content experiments to improve conversion rates, engagement, and user experience.

## When to Apply

Reference these guidelines when:
- Setting up A/B or multivariate testing infrastructure
- Designing experiments for content changes
- Analyzing and interpreting test results
- Building CMS integrations for experimentation
- Deciding what to test and how

## Core Concepts

### A/B Testing
Comparing two variants (A vs B) to determine which performs better.

### Multivariate Testing
Testing multiple variables simultaneously to find optimal combinations.

### Statistical Significance
The confidence level that results aren't due to random chance.

### Experimentation Culture
Making decisions based on data rather than opinions (HiPPO avoidance).

## References

Start with the reference that matches the current problem, such as design, statistics, CMS integration, or pitfalls. See `references/` for detailed guidance:
- `references/experiment-design.md` — Hypothesis framework, metrics, sample size, and what to test
- `references/statistical-foundations.md` — p-values, confidence intervals, power analysis, Bayesian methods
- `references/cms-integration.md` — CMS-managed variants, field-level variants, external platforms
- `references/common-pitfalls.md` — 17 common mistakes across statistics, design, execution, and interpretation


# Common Experimentation Pitfalls

Avoid these mistakes that invalidate results or lead to wrong conclusions.

## Statistical Mistakes

### 1. Stopping Early (Peeking)

**The problem:** Checking results daily and stopping when you see significance.

**Why it's wrong:** Statistical significance fluctuates. At any point during a test, you might see "significance" that disappears with more data. This is called the "peeking problem" or "repeated significance testing."

**The fix:**
- Pre-calculate required sample size
- Commit to running until you reach it
- If you must peek, use sequential testing methods that account for multiple looks

### 2. Underpowered Tests

**The problem:** Running tests without enough traffic to detect realistic effect sizes.

**Why it's wrong:** You'll conclude "no difference" when there actually is one—you just couldn't detect it.

**The fix:**
- Calculate required sample size before starting
- Be realistic about minimum detectable effect (can you act on a 0.5% improvement?)
- If traffic is low, test bigger changes

### 3. Multiple Comparisons

**The problem:** Testing many variants or metrics and celebrating any that reach significance.

**Why it's wrong:** With 20 metrics, you expect 1 false positive at 95% confidence—by chance alone.

**The fix:**
- Define ONE primary metric before starting
- Use Bonferroni correction or similar for multiple comparisons
- Treat secondary metrics as directional, not conclusive

### 4. Ignoring Segments

**The problem:** Only looking at aggregate results.

**Why it's wrong:** Simpson's Paradox—overall winner might be loser for your key segments.

**The fix:**
- Always segment by device, traffic source, user type
- Check if results are consistent across segments
- If segments differ dramatically, investigate why

## Design Mistakes

### 5. Testing Too Many Things

**The problem:** Changing headline, image, CTA, and layout simultaneously.

**Why it's wrong:** You won't know which change caused the result. And each variable multiplies required sample size.

**The fix:**
- Test one variable at a time (A/B testing)
- If testing multi