Ab Testing Engineer

A/B Testing Engineer

When to Use

• Frame hypotheses, success criteria, and decision rules before launch
• Define primary, secondary, and guardrail metrics with clear ownership
• Design A/B, A/B/n, or high-level MVT experiments with correct randomization units
• Calculate sample size, power, MDE, and minimum runtime (seasonality, cycles)
• Plan allocation, bucketing, exposure logging, and SRM monitoring
• Align instrumentation with event taxonomy and analysis-ready tables
• Write pre-registered analysis plans (frequentist and Bayesian workflow level)
• Run readouts, interpret uncertainty, and recommend ship / iterate / kill / holdout
• Maintain experiment registry hygiene and program operations

When NOT to Use

• Build or operate warehouse pipelines, dbt models, or BI stack → analytics-data-engineer, data-warehouse-engineer
• Executive dashboards and KPI storytelling without experiment design → bi-analyst
• General ML modeling, causal inference beyond experiments, MLOps → data-scientist
• Model training/eval A/B for ML systems (offline/online model comparison) → ml-research-engineer-safeguards, data-scientist
• Marketing copy, campaigns, or channel strategy → communication-lead, cmo-advisor (if installed)
• Product roadmap, portfolio prioritization without measurement design → product-management-monetization, cpo-advisor (if installed)
• Weekly metrics review cadence without new experiment → metrics-review (if installed), bi-analyst
• Legal privacy, consent, or regulatory sign-off for tracking → compliance-engineer, privacy-research-engineer-safeguards
• Growth experiment ideation without statistical design → growth-marketer (if installed), marketing-analyst (if installed)

Related skills

Core Workflows

1. Hypothesis and metric contract

1. State problem, user segment, and expected mechanism 2. Lock one primary metric for the decision (pre-register) 3. List secondary (learning) and guardrail (safety) metrics 4. Define non-goals and segments that are out of scope for decision 5. Agree rollout criteria: win threshold, guardrail breach, neutral band

See `references/hypothesis_metrics_and_design.md`.

2. Design and power

1. Choose design type: A/B, A/B/n, factorial/MVT (high level), holdout 2. Set MDE from business impact, not statistical convenience 3. Compute per-variant sample size (α, power, one- vs two-sided) 4. Estimate duration: traffic, allocation %, seasonality, full cycles 5. Document exclusions, mutual exclusion with other tests, and cooldown rules

See `references/sample_size_power_and_duration.md`.

3. Randomization, instrumentation, and SRM

1. Pick randomization unit (user, account, device, session—justify match to analysis) 2. Define bucketing hash, sticky assignment, and cross-device policy 3. Map exposure events to assignment logs; verify join keys 4. Pre-launch QA: assignment distribution, latency, fallback behavior 5. Monitor SRM and invariant checks during the run

See `references/randomization_instrumentation_and_srm.md`.

4. Analysis, readout, and decision

1. Follow pre-registered analysis plan; intent-to-treat by default 2. Run primary analysis with CIs; pre-defined segment slices only 3. Apply multiplicity policy if multiple variants or metrics drive decisions 4. Compare Bayesian vs frequentist readout needs with stakeholders 5. Produce readout: effect, uncertainty, guardrails, recommendation, risks

See `references/analysis_readout_and_decision.md`.

5. Governance and program ops

1. Register experiment in catalog (owner, dates, links, status) 2. Enforce ethical constraints: no p-hacking playbooks, no optional peeking without correction 3. Archive artifacts: config, power calc, analysis notebook, decision log 4. Plan ramp, rollback, and long-term holdout if shipping

See `references/governance_ethics_and_program_ops.md`.

When to load references

A/B Testing Engineer — Scope

Table of contents

1. Role definition 2. In scope 3. Out of scope 4. Deliverables 5. Stakeholder interfaces 6. Quality bar

Role definition

The A/B Testing Engineer owns experimentation engineering: turning product or growth questions into testable, measurable, and decision-ready experiments. The role sits between product/growth intent and data implementation—ensuring designs are statistically sound, instrumented correctly, and interpreted without procedural bias.

This is not a data platform role, not a marketing copy role, and not a general data science role—though it partners closely with each.

In scope

Out of scope

Deliverables

Typical artifacts the skill helps produce:

1. Experiment brief — hypothesis, population, design, metrics, decision rules 2. Power / sample size memo — assumptions, per-variant N, runtime estimate 3. Instrumentation spec — events, properties, assignment join path 4. Pre-registered analysis plan — primary test, segments, multiplicity, stopping 5. Launch checklist — QA, SRM monitor, comms, kill switch 6. Readout doc — results, guardrails, recommendation, follow-ups 7. Registry entry — metadata, links, final status and learnings

Stakeholder interfaces

Quality bar

An experiment is ready to launch when:

• [ ] One primary metric is named for the decision
• [ ] MDE is justified in business terms, not reverse-engineered from traffic
• [ ] Randomization unit matches how users experience treatment
• [ ] Assignment and exposure are logged and joinable
• [ ] SRM and invariant checks are defined with owners
• [ ] Analysis plan exists before peeking at results
• [ ] Guardrails have explicit breach actions
• [ ] Experiment is registered with owner and dates

An experiment is ready to decide when:

• [ ] Planned runtime or sequential stopping criteria met
• [ ] SRM resolved or documented
• [ ] Primary analysis matches pre-registration (or deviations explained)
• [ ] Guardrails reviewed with explicit pass/fail
• [ ] Recommendation states ship, iterate, kill, or extend with risks

Analysis, Readout, and Decision

Table of contents

1. Pre-registered analysis plan 2. Frequentist workflow 3. Bayesian workflow (high level) 4. Segments and multiplicity 5. Stopping rules 6. Readout template 7. Ship, kill, iterate, holdout 8. Anti-patterns

Pre-registered analysis plan

Publish before viewing results:

1. Primary metric and test (z-test, t-test, bootstrap, regression) 2. Population (ITT vs exposed) and filters 3. Covariates / CUPED (if any)—specified upfront 4. Segment list (max 3–5 pre-defined) 5. Multiplicity correction for secondary claims 6. Handling of outliers and winsorization 7. Missing data policy

Deviations require documented reason in readout.

Frequentist workflow

Standard steps:

1. Validate data — assignment counts, SRM, exposure rate 2. Estimate effect — absolute and relative lift on primary 3. Uncertainty — 95% CI (or pre-specified level) 4. p-value — report with CI; do not decide on p alone 5. Guardrails — same procedure with veto thresholds 6. Sensitivity — ITT vs exposed; with/without outliers if pre-specified

Test selection (quick reference)

Always check independence assumption at chosen unit.

Bayesian workflow (high level)

Use when stakeholders prefer probability of lift language:

1. Choose priors (weakly informative default; document if informative) 2. Compute posterior for treatment − control 3. Report P(lift > 0), P(lift > MDE), expected loss 4. Decision rule: e.g., ship if P(lift > 0) > 0.95 and guardrail posteriors acceptable

Caution: Peeking changes posterior interpretability unless using sequential Bayesian methods pre-specified.

Partner with data-scientist for prior sensitivity analysis on high-stakes tests.

Segments and multiplicity

Never promote a segment that "saved" a flat global result without correction.

Stopping rules

If stopping early for harm (guardrail), document as safety stop separate from efficacy peeking.

Readout template

# Experiment readout: [Name] ([ID])

## Summary
- **Decision:** Ship / No ship / Iterate / Extend
- **Primary metric:** [+X%] (95% CI: [a, b]); [met / did not meet] MDE
- **Guardrails:** [All pass / FAIL: metric]

## Context
- Hypothesis, variants, dates, population
- Runtime, allocation, incidents

## Data quality
- SRM: pass/fail
- Assignment N per variant; exposure rate
- Notes on exclusions

## Results
- Primary (ITT): effect, CI, p if frequentist
- Secondary metrics (labeled exploratory if applicable)
- Pre-defined segments

## Recommendation
- Action and ramp plan
- Risks and monitoring post-ship
- Follow-up experiments

## Appendix
- Link to analysis notebook, registry, config hash

Ship, kill, iterate, holdout

Ramp plans: 5% → 25% → 100% with rollback triggers on guardrails.

Anti-patterns

Sound practice only: report uncertainty, document process, prefer pre-registration.

Governance, Ethics, and Program Operations

Table of contents

1. Experiment registry 2. Lifecycle states 3. Ethical and statistical standards 4. Privacy and compliance hooks 5. Mutual exclusion and traffic budgeting 6. Archival and learning 7. Program metrics 8. RACI (lightweight)

Experiment registry

Maintain a single catalog (spreadsheet, Notion, internal tool) with:

Hygiene rules:

• No unregistered production experiments
• One registry row per experiment_id; new IDs for restarts after SRM bugs

Lifecycle states

Draft → Review → Ready → Running → Paused → Analyzed → Decision → Archived

Ethical and statistical standards

This skill teaches sound practice only. Do not provide guidance to:

• Manipulate metrics, drop outliers selectively post hoc, or re-run until significant
• Hide negative results from registry
• Cherry-pick segments without multiplicity control
• Recommend optional peeking without sequential methods

Do provide:

• Pre-registration encouragement
• Transparent reporting of null results
• Guardrails for user harm and business risk
• Clear "we do not know" when underpowered

Privacy and compliance hooks

Coordinate with compliance-engineer and privacy-research-engineer-safeguards when:

• New tracking events or PII properties added
• Regional restrictions (GDPR, COPPA, sector rules)
• Email/push experiments with consent requirements
• Sensitive populations (minors, health, financial)

Document lawful basis and data retention for experiment logs—not legal advice; escalate to counsel.

Mutual exclusion and traffic budgeting

Define max concurrent experiments per surface to protect power and UX coherence.

Archival and learning

After decision, archive:

• Final config / flag state
• Power calculation inputs and outputs
• Analysis notebook (frozen hash)
• Readout PDF or wiki page
• Post-ship monitoring plan (2–4 weeks)

Learning library: tag outcomes (win, loss, inconclusive) for meta-analysis quarterly.

Program metrics

Track program health (not individual test p-values):

RACI (lightweight)

R = responsible, A = accountable, C = consulted, I = informed.

Hypothesis, Metrics, and Experiment Design

Table of contents

1. Hypothesis framing 2. Metric hierarchy 3. Metric contract template 4. Design types 5. Population and eligibility 6. Decision rules 7. Common pitfalls

Hypothesis framing

Use a structure stakeholders can challenge before build:

Because [observation/data],
we believe [change]
will cause [measurable outcome]
for [population]
because [mechanism].
We will know this is true when [primary metric + direction + threshold].

Good hypotheses are falsifiable, name a mechanism, and tie to one decision metric.

Weak hypotheses ("improve UX") without a metric or population—refine before sizing.

Metric hierarchy

Guardrail examples

• Revenue per user, refund rate, crash rate, p95 latency
• Unsubscribe rate, support tickets per active user
• Fraud or abuse flags (if applicable)

Define breach thresholds up front (e.g., "stop if guardrail CI excludes 0 and effect < -1%").

Metric contract template

For each metric document:

Align names with event taxonomy (analytics-data-engineer) before launch.

Design types

A/B (two variants)

• Control vs treatment; default when one clear change exists.
• Simplest power and interpretation.

A/B/n (multiple treatments)

• One control + multiple treatments (e.g., copy A, B, C).
• Increases multiplicity—pre-define whether any variant wins vs best variant.
• Allocate traffic evenly unless powered for unequal splits.

Multivariate (MVT) / factorial (high level)

• Test multiple factors simultaneously (e.g., headline × CTA).
• Requires more traffic; interactions are hard to power.
• Prefer fractional factorial or sequential tests when traffic is limited.
• Document which interactions are in scope for decisions.

Holdout

• Long-running control cell excluded from launches.
• Measures cumulative lift; not a substitute for pre-launch A/B on a specific change.

Switchback / geo (awareness)

• When user-level randomization is infeasible, time or geo designs may apply.
• Different assumptions (autocorrelation, spillover)—flag as advanced; partner with data-scientist for causal design.

Population and eligibility

Define:

• Targeting: new vs returning, platform, locale, plan tier
• Trigger: exposure on page view vs action vs assignment at login
• Exclusions: bots, employees, accounts in other experiments (mutual exclusion)
• Cooldown: users recently in conflicting tests

Document analysis population: all assigned (ITT) vs exposed only (justify if used).

Decision rules

Pre-register:

Avoid "win if p < 0.05" without effect size and guardrail context.

Common pitfalls

Randomization, Instrumentation, and SRM

Table of contents

1. Randomization unit 2. Bucketing and assignment 3. Exposure vs assignment 4. Instrumentation and event taxonomy 5. Pre-launch QA 6. Sample ratio mismatch (SRM) 7. Invariant metrics 8. Operational playbooks

Randomization unit

The randomization unit is the entity hashed into variants. It must match the unit of analysis for the primary metric when possible.

Rule: If treatment is applied per user but analysis is per session without adjustment, expect bias.

Document sticky assignment duration (forever vs 30 days vs session).

Bucketing and assignment

Standard pattern:

variant = hash(experiment_id + stable_id + salt) % bucket_map

Requirements:

• Stable across sessions for sticky tests
• Independent salt per experiment to reduce correlation across tests
• Deterministic given ID (reproducible QA)
• Logged at assignment time with timestamp and experiment version

Feature-flag systems should log:

• experiment_id, variant_id, assignment_timestamp
• assignment_reason (eligible, excluded, override)
• experiment_version / config hash

Exposure vs assignment

Intent-to-treat (ITT): analyze all assigned users—recommended default.

Per-protocol / exposed-only: analyze exposed users only when exposure is well measured; risks bias if exposure differs by variant. Pre-register and report both when stakeholders require.

Log events:

• experiment_assigned
• experiment_exposed (with surface, locale, platform)
• experiment_exposure_failed (optional, for debugging)

Instrumentation and event taxonomy

Minimum schema alignment with analytics engineering:

Property standards:

• snake_case names; versioned event specs
• No PII in experiment properties unless approved (compliance-engineer)
• Include app_version, platform, locale for debugging splits

Partner with analytics-data-engineer for idempotent daily assignment tables.

Pre-launch QA

Checklist before enabling traffic:

• [ ] Hash distribution uniform in staging (chi-square on synthetic IDs)
• [ ] Assignment logged on all code paths (including errors, mobile, web)
• [ ] Exposure fires only when treatment visible
• [ ] Mutual exclusion rules enforced in code
• [ ] Holdout and internal users excluded
• [ ] Backward compatibility when flag off
• [ ] Join query: assignment → exposure → primary metric returns sensible counts

Run shadow mode (assign but do not change UX) when platform is new.

Sample ratio mismatch (SRM)

SRM occurs when observed assignment counts deviate significantly from expected allocation (e.g., 50/50 becomes 48/52 at scale).

Response:

1. Pause experiment if SRM is severe and unexplained 2. Query assignment logs at source vs warehouse 3. Fix bug; restart with new experiment_id if assignment corrupted 4. Do not treat results as valid until SRM resolved

Use chi-square test on assignment counts; platforms (Optimizely, internal) often automate SRM alerts—tune false positive rate.

Invariant metrics

Metrics that should not differ by variant if randomization worked:

• Device type mix, OS, country (if not targeted)
• New vs returning mix (unless pre-period imbalance expected)
• Baseline activity pre-assignment (AA test)

Run A/A or pre-period balance on key covariates when traffic allows.

Operational playbooks

Sample Size, Power, and Duration

Table of contents

1. Core parameters 2. MDE and business alignment 3. Sample size (proportions) 4. Sample size (continuous) 5. Multiple variants 6. Duration and seasonality 7. Allocation and traffic 8. Worked example checklist

Core parameters

Document all assumptions in the experiment brief; revisit if baseline shifts during run.

MDE and business alignment

MDE should answer: "If we only detect a change this large, is the experiment still worth it?"

Expected annual impact ≈ (detectable lift) × (eligible users/year) × (value per user)

If required N exceeds available traffic within acceptable calendar time:

• Increase MDE (accept smaller detectable effect only if business agrees)
• Lengthen runtime (seasonality permitting)
• Narrow population
• Use variance reduction (CUPED, stratification)—coordinate with data-scientist

Do not silently lower power below 0.8 without stakeholder sign-off.

Sample size (proportions)

For conversion-style metrics (Bernoulli):

• Inputs: baseline rate p, absolute or relative MDE, α, power
• Per-group N increases as p approaches 0.5 and as MDE shrinks

Relative vs absolute MDE:

• Relative: 5% lift on 10% baseline → 10.5% absolute
• State which definition was used in the memo

Use a standard calculator or library (statsmodels, Evan Miller calculator) and paste inputs into registry.

Sample size (continuous)

For means (revenue per user, session length):

• Need baseline mean and standard deviation (or variance of user-level metric)
• Heavy-tailed metrics (revenue): consider winsorization policy in analysis plan
• Log transform or bootstrap analysis—pre-specify

Ratio metrics (clicks/views): treat carefully; consult statistician or data-scientist for delta-method/bootstrap plan.

Multiple variants

For k treatment arms + control:

• Total sample often scales with number of comparisons of interest
• Pre-specify: test each vs control (k tests) vs best vs control (different multiplicity)
• Apply multiplicity adjustment in analysis plan (Bonferroni, Holm, FDR)

Power all comparisons you will report as wins.

Duration and seasonality

Minimum runtime should cover:

Runtime formula (approximate):

days ≈ (required sample per variant × number of variants) / (daily eligible assignments × allocation fraction)

Add buffer (10–20%) for data loss, SRM exclusions, and incident pauses.

Allocation and traffic

Document mutual exclusion with other experiments consuming same population.

Worked example checklist

• [ ] Baseline estimated from last 14–28 days (stable)
• [ ] MDE signed by PM/finance for primary metric
• [ ] α, power, one/two-sided documented
• [ ] Per-variant N and total N calculated
• [ ] Calendar days estimated with allocation %
• [ ] Seasonality and conflicting launches checked
• [ ] Max runtime and extension criteria pre-defined
• [ ] Results pasted into experiment registry