Ab Test Analysis
- 2k installs
- 24.9k repo stars
- Updated July 3, 2026
- phuryn/pm-skills
Given experiment data (control/variant metrics, sample sizes, duration), calculate statistical significance, interpret practical impact, and recommend ship/extend/stop/investigate actions.
About
A/B Test Analysis evaluates experiment results through statistical significance testing, sample size validation, and confidence interval calculations. It translates raw test data into actionable product decisions by checking primary metrics, guardrail metrics, and practical business impact. The skill validates test setup (sample power, duration, randomization), calculates relative lift and p-values, interprets results against significance thresholds, and provides structured recommendations: Ship (positive significant lift), Extend (positive trend, underpowered), Stop (flat/insignificant), or Investigate (guardrail trade-offs). Designed for product managers and data analysts evaluating split tests, conversion experiments, and feature rollout decisions.
- Statistical significance testing with p-values and 95% confidence intervals
- Sample size validation using power analysis and minimum detectable effect (MDE)
- Guardrail metric monitoring to catch unintended negative side effects
- Structured ship/extend/stop/investigate decision framework
- Python script generation for direct data analysis from CSV/Excel exports
Ab Test Analysis by the numbers
- 1,983 all-time installs (skills.sh)
- +79 installs in the week ending Aug 5, 2026 (Skillselion tracking)
- Ranked #242 of 3,282 Productivity & Planning skills by installs in the Skillselion catalog
- Security screen: LOW risk (skills.sh audit)
- Data as of Aug 5, 2026 (Skillselion catalog sync)
npx skills add https://github.com/phuryn/pm-skills --skill ab-test-analysisAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 2k |
|---|---|
| repo stars | ★ 24.9k |
| Security audit | 3 / 3 scanners passed |
| Last updated | July 3, 2026 |
| Repository | phuryn/pm-skills ↗ |
What it does
Validate experiment results with statistical rigor to decide ship, extend, or stop product variants.
Who is it for?
Product managers, data analysts, and engineering teams validating feature changes, conversion optimizations, pricing experiments, and user experience variants before full rollout.
Skip if: Real-time ad campaign optimization, high-frequency trading, or single-sample observational studies lacking randomization controls.
When should I use this skill?
After an A/B test completes or reaches interim power checkpoint; before deciding to ship, extend, or stop an experiment; when interpreting conflicting metric signals.
What you get
Teams confidently evaluate A/B tests using validated sample sizes, significance thresholds, and guardrail checks, reducing rollout risk and improving decision velocity.
- Significance analysis report
- Ship/extend/stop recommendation
- Python analysis script
Files
A/B Test Analysis
Evaluate A/B test results with statistical rigor and translate findings into clear product decisions.
Context
You are analyzing A/B test results for $ARGUMENTS.
If the user provides data files (CSV, Excel, or analytics exports), read and analyze them directly. Generate Python scripts for statistical calculations when needed.
Instructions
1. Understand the experiment:
- What was the hypothesis?
- What was changed (the variant)?
- What is the primary metric? Any guardrail metrics?
- How long did the test run?
- What is the traffic split?
2. Validate the test setup:
- Sample size: Is the sample large enough for the expected effect size?
- Use the formula: n = (Z²α/2 × 2 × p × (1-p)) / MDE²
- Flag if the test is underpowered (<80% power)
- Duration: Did the test run for at least 1-2 full business cycles?
- Randomization: Any evidence of sample ratio mismatch (SRM)?
- Novelty/primacy effects: Was there enough time to wash out initial behavior changes?
3. Calculate statistical significance:
- Conversion rate for control and variant
- Relative lift: (variant - control) / control × 100
- p-value: Using a two-tailed z-test or chi-squared test
- Confidence interval: 95% CI for the difference
- Statistical significance: Is p < 0.05?
- Practical significance: Is the lift meaningful for the business?
If the user provides raw data, generate and run a Python script to calculate these.
4. Check guardrail metrics:
- Did any guardrail metrics (revenue, engagement, page load time) degrade?
- A winning primary metric with degraded guardrails may not be a true win
5. Interpret results:
| Outcome | Recommendation |
|---|---|
| Significant positive lift, no guardrail issues | Ship it — roll out to 100% |
| Significant positive lift, guardrail concerns | Investigate — understand trade-offs before shipping |
| Not significant, positive trend | Extend the test — need more data or larger effect |
| Not significant, flat | Stop the test — no meaningful difference detected |
| Significant negative lift | Don't ship — revert to control, analyze why |
6. Provide the analysis summary:
## A/B Test Results: [Test Name]
**Hypothesis**: [What we expected]
**Duration**: [X days] | **Sample**: [N control / M variant]
| Metric | Control | Variant | Lift | p-value | Significant? |
|---|---|---|---|---|---|
| [Primary] | X% | Y% | +Z% | 0.0X | Yes/No |
| [Guardrail] | ... | ... | ... | ... | ... |
**Recommendation**: [Ship / Extend / Stop / Investigate]
**Reasoning**: [Why]
**Next steps**: [What to do]Think step by step. Save as markdown. Generate Python scripts for calculations if raw data is provided.
---
Further Reading
Related skills
Grill MeHave an agent relentlessly pressure-test their plan or design until every decision branch is resolved and shared understanding is reached.691k196kGrill With DocsPressure-test architecture decisions against their live codebase and living documentation before writing production code.585k196k
Lark CalendarLet their coding agents read, create, and manage events inside Lark (Feishu) Calendar directly from natural language instructions.493kLark ImLet their coding agent send instant notifications, post updates, and receive commands directly inside Lark (Feishu) workspaces.493kLark ContactLet their AI coding agents read and write contacts, send messages, and manage user data inside Lark (Feishu).493kLark AttendanceAutomatically fetch and report daily attendance data from Lark (Feishu) into their own tools or agents.493kHow it compares
Choose ab-test-analysis over generic data-analysis skills when the goal is experiment decision-making with significance testing rather than open-ended exploration.
FAQ
What decisions does ab-test-analysis produce?
ab-test-analysis evaluates A/B test data for statistical significance and sample size validity, then recommends whether to ship the winning variant, extend the test, or stop the experiment.
What data formats does ab-test-analysis accept?
ab-test-analysis reads CSV, Excel, and analytics export files directly. The skill generates Python scripts for statistical calculations when automated analysis is required.
Is Ab Test Analysis safe to install?
skills.sh reports 3 of 3 security scanners passed. Review the Security Audits panel on this page before installing in production.