Ab Testing Pricing

A/B Testing Pricing

Overview

Price A/B testing lets you run controlled experiments — showing one price to a segment of visitors and a different price to another — before committing to a permanent change. This is one of the highest-leverage optimizations available, but also one of the riskiest: a poorly executed test destroys customer trust and produces misleading data. Most platforms do not have native price A/B testing, so this requires either a dedicated app or a custom implementation. This skill walks you through setting up price experiments correctly on each major platform.

When to Use This Skill

• When you need data-driven evidence for a price change before rolling it out site-wide
• When testing price sensitivity across different customer segments or product categories
• When evaluating the revenue impact of a new pricing model (e.g., switching from $49.99 to $45.00)
• When running multiple concurrent price experiments on different products without interference
• When regulatory or ethical requirements demand that price tests are documented, time-limited, and reversible

Core Instructions

Step 1: Determine the merchant's platform and choose the right approach

Step 2: Design the experiment

Before setting up any tool, define these parameters:

1. Hypothesis — "Reducing the price from $49.99 to $44.99 will increase revenue per visitor by more than 5%" 2. Primary metric — Revenue per visitor (RPV), not just conversion rate. A lower price converts better but may earn less per order 3. Traffic split — 50/50 is standard; never run more than 2 variants simultaneously on the same product 4. Minimum duration — At least 7 days to capture weekday/weekend variation; ideally 14 days 5. Minimum sample size — Calculate using a significance calculator (use abtestguide.com). Aim for at least 100 conversions per variant before declaring a winner 6. Exclusion rules — Existing customers who paid the old price should not be shown the new price mid-experiment

Step 3: Platform-specific setup

---

Shopify

Option A: Intelligems (recommended for most merchants)

Intelligems is the leading Shopify app for price testing. It handles session stickiness, statistical significance, and integrates with Shopify's checkout natively.

1. Install Intelligems from the Shopify App Store 2. Go to Intelligems → Tests → New Price Test 3. Select the product(s) to test 4. Set your variant prices (e.g., $49.99 vs $44.99) 5. Set traffic split (50/50 recommended) 6. Set a start date and estimated end date based on your sample size calculator 7. Intelligems assigns visitors to variants via a persistent cookie and reports RPV, conversion rate, and statistical significance in their dashboard

Key settings to configure:

• Sticky bucketing — ensure the same visitor always sees the same price (enabled by default)
• Returning customer exclusion — under Advanced Settings, exclude customers who have previously purchased the product at the original price
• Minimum detectable effect — set to the minimum revenue change that would justify the price change (e.g., 5%)

Option B: Neat A/B Testing

A lighter-weight alternative if you only need simple price split tests: 1. Install Neat A/B Testing from the App Store 2. Create a test targeting a specific product or collection 3. Set the price variant and traffic split 4. Monitor results in the Neat dashboard

Option C: Shopify Plus — Shopify Functions (advanced)

For Plus merchants who need full control: 1. Create a Shopify Function (Discount Function type) 2. The function receives cart context and customer ID 3. Use deterministic hashing on the customer/session ID to assign to a variant 4. Apply a fixed-amount or percentage discount equal to the price difference 5. Track conversions via Shopify's order webhook + a custom analytics store

---

WooCommerce

Option A: Nelio A/B Testing plugin

1. Install Nelio A/B Testing from the WordPress plugin directory 2. Go to Nelio A/B Testing → Add New Experiment → WooCommerce Product Experiment 3. Select the product and set the alternative price 4. Configure the conversion goal (purchase of that product) 5. The plugin tracks sessions and reports conversion rates with statistical significance

Option B: Manual with Google Optimize (sunsetted) replacements

Since Google Optimize was discontinued in 2023, consider:

• VWO (Visual Website Optimizer) — supports WooCommerce price testing via JavaScript
• AB Tasty — enterprise-grade with WooCommerce native integration
• Convert.com — WooCommerce-compatible with server-side testing support

Important WooCommerce note: Client-side price changes (JavaScript-based) are cosmetic only — they do not change the actual checkout price. Always ensure your A/B testing tool changes the price at the WooCommerce product level or through the woocommerce_get_price filter, not just the displayed number.

---

BigCommerce

1. Use BigCommerce's Price Lists feature (available on Plus and above):

• Go to Products → Price Lists → Add Price List
• Create a price list for your "treatment" price
• Assign the price list to a specific customer group

2. Split customers into control/treatment groups by assigning them to the customer group 3. Track conversions using BigCommerce's built-in analytics or Google Analytics with custom dimensions

Alternatively, use Intelligems for BigCommerce which manages the visitor bucketing automatically.

---

Custom / Headless

For headless storefronts, you need to implement session bucketing, price resolution, and conversion tracking:

import crypto from 'crypto';

// Deterministic variant assignment by session ID
function assignVariant(
  experimentId: string,
  sessionId: string,
  variants: { id: string; weight: number }[]  // weights must sum to 100
): string {
  const hash = parseInt(
    crypto.createHash('sha256')
      .update(`${experimentId}:${sessionId}`)
      .digest('hex')
      .slice(0, 8),
    16
  ) % 100;

  let cumulative = 0;
  for (const variant of variants) {
    cumulative += variant.weight;
    if (hash < cumulative) return variant.id;
  }
  return variants[variants.length - 1].id;
}

// Usage: get the price for a visitor
const variantId = assignVariant('exp_price_widget_pro', sessionId, [
  { id: 'control',   weight: 50 },  // $49.99
  { id: 'treatment', weight: 50 },  // $44.99
]);

const prices = { control: 4999, treatment: 4499 };  // cents
const priceForVisitor = prices[variantId];

Track conversions and calculate statistical significance:

// Two-proportion z-test for statistical significance
function zTest(
  controlConversions: number,
  controlVisitors: number,
  treatmentConversions: number,
  treatmentVisitors: number
): { pValue: number; significant: boolean } {
  const p1 = controlConversions / controlVisitors;
  const p2 = treatmentConversions / treatmentVisitors;
  const pPooled = (controlConversions + treatmentConversions) / (controlVisitors + treatmentVisitors);
  const se = Math.sqrt(pPooled * (1 - pPooled) * (1 / controlVisitors + 1 / treatmentVisitors));
  if (se === 0) return { pValue: 1, significant: false };
  const z = Math.abs((p2 - p1) / se);
  // Approximate p-value (two-tailed)
  const pValue = 2 * (1 - normalCDF(z));
  return { pValue, significant: pValue < 0.05 };
}

Use your platform's pricing API (Shopify Admin API, BigCommerce Catalog API) to push price updates once a winner is declared.

Step 4: Monitor and conclude the experiment

Track these metrics during the experiment:

Declare a winner only when:

• You have reached the pre-calculated minimum sample size (not before)
• The p-value is below 0.05
• The experiment has run for at least 7 days

After declaring a winner, update the product price permanently and end the experiment.

Best Practices

• Use Revenue Per Visitor (RPV), not just conversion rate — a lower price may convert better but generate less revenue; RPV captures the full picture
• Require sticky bucketing — once a session is assigned to a variant, always show the same price; inconsistent prices destroy trust and distort results
• Pre-register your minimum sample size — decide before the experiment starts how many conversions you need; stopping as soon as p < 0.05 (peeking) inflates false positive rates
• Exclude existing customers — showing an existing customer a price lower than what they paid triggers complaints and churn
• Run only one experiment per product at a time — overlapping experiments on the same product create interaction effects that make results uninterpretable
• Pause experiments during promotions — site-wide sales or marketing campaigns confound price experiment results
• Log the variant assignment on every order — store the variant ID in the order metadata so you can later audit revenue attribution

Common Pitfalls

Related Skills

• @dynamic-pricing
• @price-rules-engine
• @coupon-management
• @discount-engine
• @demand-forecasting

Price Experiment Event Tracking Integration

Problem/Feature Description

A retail engineering team has built a basic price experimentation system but hasn't yet wired up the analytics side. Currently, the system assigns shoppers to price variants but there is no record of what happens next — no tracking of which shoppers add products to their cart, no recording of completed purchases, and no way to reconcile the revenue back to the experiment that influenced the sale.

The head of data has raised a specific concern: they want to be able to run post-hoc revenue analysis months later, which means every order record must carry enough metadata to identify which experiment and variant influenced that sale. The team also needs to understand shopping funnel behaviour at the variant level, not just final conversion.

Output Specification

Produce the following files:

1. eventTracking.ts — A TypeScript module exporting:

• A function to record a conversion event (add to cart or purchase) for a given experiment variant, including any relevant revenue amount.
• A function to persist a completed order in the database along with any relevant experiment context needed for later attribution.

2. checkoutIntegration.ts — A TypeScript module showing how to integrate the event tracking into a typical checkout flow. It should demonstrate the full sequence from showing a price, through add-to-cart, to completing a purchase.

3. INTEGRATION_NOTES.md — A brief document (under 300 words) summarising the rules the team must follow when integrating the experiment tracking into the checkout flow. Focus on data integrity and correctness concerns.

Assume a db object is available with methods including db.raw, db.orders.insert, and the methods from the experiment system.

Price Experiment Infrastructure Setup

Problem/Feature Description

A mid-size e-commerce company is launching a new initiative to run controlled price tests before committing to any pricing changes across their catalog. The engineering lead has asked you to build the foundational layer: the database schema and the core TypeScript function that decides which price variant a shopper sees.

The system must handle tens of thousands of concurrent sessions reliably. A shopper who loads a product page twice in the same session should always see the same price — price inconsistency within a session breaks trust and skews data. The schema must also support partial traffic rollouts (not all shoppers need to see the experiment), a clean lifecycle for experiments, and the ability to run independent experiments on different products.

Output Specification

Produce the following files:

1. schema.sql — the complete SQL DDL to create all necessary tables for the experiment system 2. assignVariant.ts — a TypeScript function assignVariant(experimentId: string, sessionId: string) that returns the assigned variant (or null if the session is not in the experiment). Include any helper types needed. 3. getExperimentPrice.ts — a TypeScript function getExperimentPrice(productId: string, sessionId: string, defaultPrice: number) that looks up any active experiment for the product and returns the price to show along with tracking metadata.

Assume a db object is available with methods: db.priceExperiments.findById, db.priceExperiments.findOne, db.priceExperimentVariants.findByExperiment, db.priceExperimentAssignments.findOne, db.priceExperimentAssignments.insert, db.priceExperimentVariants.findById, db.raw.

Price Experiment Results Analyzer

Problem/Feature Description

A growth engineering team has been running a series of price tests over the past quarter and needs a standalone TypeScript module to analyze the accumulated data. The results dashboard currently just shows raw counts but the team needs statistically rigorous analysis so they can tell whether one price variant is genuinely outperforming another, rather than just looking better by chance.

The analytics lead has stressed that the module must enforce proper statistical discipline — the team has been burned before by acting on early results that later reversed. There is also a concern that a variant with a higher conversion rate might not actually be the better business outcome, so the analysis needs to surface more than just conversion numbers.

Output Specification

Produce the following files:

1. analysis.ts — A TypeScript module exporting:

• A function calculateSignificance(controlConversions, controlImpressions, treatmentConversions, treatmentImpressions) that returns a statistical test result with at least zScore, pValue, and significant fields.
• A function getExperimentResults(experimentId: string) that retrieves variant data and returns enriched results per variant.
• A function isReadyToConclude(variant: object, pValue: number) that returns true only when the experiment meets minimum requirements to be acted upon.
• A helper function implementing an approximation for the standard normal CDF (do not use an external statistics library).

2. analysis.test.ts — A set of TypeScript test cases (plain assertions or any test framework) that demonstrate the significance calculation working correctly on some sample data. Include at least one case that should be significant and one that should not.

Assume a db object is available with db.priceExperimentVariants.findByExperiment(experimentId).