Ab Testing Ecommerce

A/B Testing for E-commerce

Overview

A/B testing (split testing) runs controlled experiments where a random subset of visitors sees a variant while the rest see the control. Statistical analysis then determines whether any difference is real or due to chance. Good testing disciplines — calculating required sample size before starting, running tests for at least two full weeks, and never stopping early — separate genuine insights from noise.

This skill guides you through running A/B tests on your specific platform, choosing the right tools, and interpreting results correctly.

When to Use This Skill

• When making product page, checkout, or pricing changes and wanting data-driven validation
• When migrating from a client-side A/B testing tool to server-side assignment for accuracy
• When needing statistical power calculations before starting an experiment
• When analyzing experiment results and determining when to ship or kill a variant
• When running a pricing test and needing to ensure consistent pricing per customer
• When wanting to understand what sample size is needed before a test is meaningful

Core Instructions

Step 1: Determine your platform and choose the right testing tool

Step 2: Calculate required sample size before launching

Never launch a test without knowing how many visitors each variant needs. Running a test without a pre-determined stopping rule leads to peeking and false positives.

Use the free calculator at https://www.evanmiller.org/ab-testing/sample-size.html or follow this guide:

• Baseline conversion rate: Pull your current CVR from your platform analytics (last 30 days)
• Minimum detectable effect: The smallest lift you care about detecting (typically 0.3–1 percentage point)
• Statistical power: 80% is standard
• Significance level: 95% confidence (alpha = 0.05)

Example: A Shopify store with 2.5% CVR wanting to detect a 0.3pp lift needs approximately 8,600 sessions per variant. At 500 sessions/day, that is 17 days per variant minimum.

Write down the required sample size before the test starts. This is your mandatory stopping rule.

Step 3: Set up the experiment on your platform

---

Shopify

Option A: Theme-based tests with Convert.com

1. Install Convert.com and add the tracking script via Online Store → Themes → Edit code → theme.liquid 2. In Convert.com, go to Experiences → Create Experience → A/B Test 3. Use the visual editor to create your variant (change button color, headline, layout) 4. Set goals: Add to Cart or Purchase (Convert tracks Shopify purchase events automatically) 5. Set traffic allocation (50/50 for most tests) 6. Set the minimum sample size you calculated as the stopping condition

Option B: Pricing tests with Intelligems

1. Install Intelligems from the Shopify App Store 2. Go to Intelligems → Price Tests → New Test 3. Select the product(s) to test and set variant prices 4. Intelligems handles sticky assignment server-side — the same customer always sees the same price 5. Set the test duration to your pre-calculated sample size 6. Review results in Intelligems' dashboard: it shows revenue per visitor (not just CVR) as the primary metric

For Shopify checkout tests (Shopify Plus only):

• Use Checkout Extensibility or Shopify Functions to create checkout variants
• Shopify's built-in A/B testing via Checkout profiles is available on Plus

---

WooCommerce

Using Nelio A/B Testing (recommended)

1. Install Nelio A/B Testing from the WordPress plugin directory 2. Go to Nelio A/B Testing → Add New Test 3. Choose the test type:

• Page Test: Test different landing page or product page variants
• WooCommerce Test: Test product pricing, descriptions, or images
• Headline Test: Test page titles or CTAs

4. Set your goal to WooCommerce Order (conversion event) 5. Nelio tracks statistical significance in real time — do not stop early just because significance is reached; wait for your pre-calculated sample size 6. View results at Nelio A/B Testing → Results

Alternative: Google Optimize (free, requires Google Analytics 4) 1. Create a Google Optimize account and link it to your GA4 property 2. Add the Optimize container ID to your WordPress site via MonsterInsights plugin (simplest method) or manually in the <head> 3. Create an A/B test in Optimize pointing to your WooCommerce product or checkout URLs 4. Set objectives using GA4 events (e.g., purchase)

---

BigCommerce

1. Go to Storefront → Script Manager → Create a Script 2. Add your A/B testing tool script (Convert.com, VWO, or Optimizely) with placement Head and All pages 3. In your testing tool, create an experiment targeting your BigCommerce product or category page URL 4. Set the conversion goal to track the order confirmation page URL (/order-confirmation) 5. BigCommerce also has built-in Multivariate Testing under Marketing → Banner Manager for banner-level tests (limited to visual banner content)

---

Custom / Headless

For headless storefronts, use server-side assignment to avoid flickering and to support pricing tests:

Using GrowthBook (open source, recommended)

1. Install GrowthBook: npm install @growthbook/growthbook 2. Initialize on the server side with your user ID for sticky assignment:

import { GrowthBook } from "@growthbook/growthbook";

const gb = new GrowthBook({
  apiHost: "https://cdn.growthbook.io",
  clientKey: process.env.GROWTHBOOK_CLIENT_KEY,
  attributes: {
    id: userId, // stable user ID for consistent assignment
    loggedIn: !!customerId,
  },
});

await gb.loadFeatures();

// Assign variant — deterministic for the same userId
const checkoutButtonVariant = gb.getFeatureValue("checkout-button-color", "blue");

3. Track exposures and conversions back to GrowthBook:

gb.setTrackingCallback((experiment, result) => {
  analytics.track("Experiment Viewed", {
    experimentId: experiment.key,
    variationId: result.key,
  });
});

// On order completion:
analytics.track("Purchase", { revenue: order.total });

4. View statistical results in the GrowthBook UI — it runs Bayesian or frequentist significance tests on your data

Step 4: Interpret results correctly

When reviewing results:

1. Wait for the pre-calculated sample size — do not stop because it "looks significant" 2. Check revenue per visitor, not just CVR — a checkout test might increase CVR but decrease AOV; measure both 3. Run for at least 2 full weeks — day-of-week effects distort 7-day tests 4. Look at guardrail metrics — even if your primary metric improved, check return rates and customer service ticket volume

Best Practices

• Calculate sample size before starting — running until it "looks significant" is p-hacking; use the pre-calculated size as your stopping rule
• Use server-side assignment for pricing tests — client-side tools create flickering and can show different prices on page reload, which is a legal and UX risk
• Never run more than 3–4 experiments on the same page simultaneously — interaction effects between experiments contaminate all results
• Exclude internal team traffic — add your office IP to an exclusion list in your testing tool to prevent internal browsing from polluting results
• Document the hypothesis before starting — write down what you expect to happen and why; post-hoc hypothesis generation leads to confirmation bias
• Run experiments for at least 2 full business weeks — account for day-of-week and weekend shopping pattern differences

Common Pitfalls

Related Skills

• @conversion-rate-optimization
• @product-analytics
• @customer-analytics
• @sales-reporting-dashboard
• @attribution-modeling

Experiment Analytics: Tracking and Results Analysis

Problem/Feature Description

A fashion retailer's engineering team has an A/B test running on their product detail page: one variant shows a redesigned "Add to Cart" button with social proof messaging. The experiment assignment system is already in place, but the team has no tracking or analysis layer. Results are currently being estimated by eyeballing order counts in a spreadsheet, which the data team says is completely unreliable.

The analytics lead wants a proper tracking system that records when users are actually exposed to each variant, and a results API that computes statistical significance so the team can confidently decide whether to ship the new button. She specifically flagged two past mistakes to avoid: (1) a previous system double-counted exposures when users visited the same page multiple times, inflating exposure counts and making the test appear more powered than it was; (2) a separate system counted conversions for users who had never actually seen the experiment variant, because someone placed an order through a direct link.

The experiment is for a checkout event called order_placed and has a control variant and a treatment variant. The team wants to be able to call a results endpoint to see the current state of the experiment.

Output Specification

Produce a TypeScript file experiment-tracking.ts containing:

1. A trackExposure(experimentId, variantId, userId) function that safely records experiment exposure. 2. A trackConversion(experimentId, userId, event, value?) function that records conversion events. 3. A calculateSignificance(results) function that takes control and variant exposure/conversion counts and returns p-value, significance flag, and relative lift. 4. A getExperimentResults(req, res) Express handler for GET /api/experiments/:id/results that assembles the full results response.

Also produce a analysis-notes.md explaining the statistical method used and any important caveats about when results should and should not be trusted.

Pricing Experimentation: Safe Variant Delivery and Experiment Health Monitoring

Problem/Feature Description

A subscription software company wants to run a pricing test on their plans page: they want to show a subset of visitors a different monthly price for their Pro tier and measure whether it improves revenue. Legal has flagged a concern — a customer complained about seeing different prices during the same browsing session, which raised consumer protection questions. The engineering team needs a pricing variant delivery system that guarantees price consistency per user across visits.

Additionally, the experiment operations team has noticed that the checkout funnel currently has four overlapping experiments running. A data scientist warned in Slack that the interactions between these experiments are making all of them unreliable. Going forward, there should be a validation step that prevents more than a safe number of experiments from running simultaneously on the same page surface.

The team has also learned the hard way that improving conversion rate doesn't always mean success — a previous checkout test improved CVR but tanked average order value. They want an automated guardrail system that monitors key health metrics for running experiments and pauses them automatically when something looks wrong.

Output Specification

Produce a TypeScript file pricing-experiment.ts containing:

1. A getProductPrice(productId, userId) function that retrieves the correct price for a user, using the user's assigned experiment variant when a pricing test is active for that product. 2. A validateExperimentCount(pageId) function (or equivalent) that checks whether adding a new experiment to a page would exceed safe limits, and returns an error or warning if it would. 3. A checkExperimentGuardrails(experimentId) function that checks key health metrics and pauses the experiment with a reason code if a guardrail is breached.

Also produce a design-decisions.md file explaining the pricing consistency strategy, the rationale for the concurrent experiment limit, and what metrics are being monitored as guardrails and why.

Experiment Framework: Core Assignment Engine

Problem/Feature Description

The growth team at a mid-size e-commerce company wants to start running controlled experiments on their Node.js/Express storefront but has no experimentation infrastructure in place. They've been making product changes based on gut feel and want to move to data-driven decisions. The engineering lead has asked you to build the foundational experiment engine that will power all future tests.

The team's primary concern is accuracy: in the past, third-party A/B tools caused pricing "flicker" on the page and were blocked by some customers' ad blockers, which skewed results. They need a TypeScript implementation that lives entirely in their backend.

Before any test goes live, the team lead requires a way to know how many users are needed to run the experiment for meaningful results, given the current checkout conversion rate of 3.1% and a desire to detect improvements of at least 0.4 percentage points. The team runs tests at 80% statistical power and a 5% significance level.

Output Specification

Produce a single TypeScript file called experiment-engine.ts that contains:

1. A calculateRequiredSampleSize function that takes baseline conversion rate, minimum detectable effect, statistical power, and significance level as parameters and returns the per-variant sample size. 2. An assignVariant function that deterministically assigns a user to a variant given an experiment ID and user ID. The assignment must be stable — the same user must always get the same variant for a given experiment. 3. An Express middleware function that reads active experiments and attaches each experiment's variant assignment to the request context, using the customer's stable identity. 4. A brief README.md explaining the design decisions and how to use the two key functions, including an example call to calculateRequiredSampleSize with the parameters described above and its output.