Trustworthy Experiments
- 305 installs
- 43 repo stars
- Updated March 7, 2026
- pmprompt/claude-plugin-product-management
trustworthy-experiments is a Claude skill that designs rigorous A/B tests and holdouts with hypotheses, sample sizing, guardrails, and analysis plans for developers who need statistically trustworthy product experiment r
About
trustworthy-experiments is a product-management Claude skill grounded in Ronny Kohavi's Trustworthy Online Controlled Experiments framework. The skill walks developers and PMs through designing controlled experiments with clear hypotheses, sample-size planning, guardrail metrics, and pre-registered analysis plans before shipping pricing, onboarding, or feature changes. It activates on prompts about A/B tests, statistical significance, false positives, and experiment guardrails. Developers reach for trustworthy-experiments when they need to structure an experiment correctly rather than ship a change and debate noisy metrics afterward.
- Hypothesis and metric guardrails
- Sample size and duration guidance
- Bias and peeking risk checks
- Analysis plan before launch
- Decision rules for ship or kill
Trustworthy Experiments by the numbers
- 305 all-time installs (skills.sh)
- +12 installs in the week ending Aug 4, 2026 (Skillselion tracking)
- Ranked #586 of 2,064 Data Science & ML skills by installs in the Skillselion catalog
- Data as of Aug 4, 2026 (Skillselion catalog sync)
npx skills add https://github.com/pmprompt/claude-plugin-product-management --skill trustworthy-experimentsAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 305 |
|---|---|
| repo stars | ★ 43 |
| Last updated | March 7, 2026 |
| Repository | pmprompt/claude-plugin-product-management ↗ |
How do you design statistically trustworthy A/B tests?
Design rigorous A/B tests and holdouts with clear hypotheses, sample sizing, guardrails, and analysis plans before shipping pricing, onboarding, or feature experiments.
Who is it for?
Product engineers and PMs planning pricing, onboarding, or feature experiments who need guardrails against false positives.
Skip if: Teams that only need SQL queries or dashboard charts without designing the experiment methodology first.
When should I use this skill?
A developer asks to run an A/B test, check statistical significance, set experiment guardrails, or avoid false positives in product metrics.
What you get
Experiment design document with hypothesis, sample sizing, guardrail metrics, holdout plan, and analysis protocol
- experiment design document
- analysis plan
- guardrail metric list
Files
Domain Context
This skill implements a proven product management framework. The approach combines best practices from industry leaders and is designed for practical application in day-to-day PM work.
Input Requirements
- Context about your product, feature, or problem
- Relevant data, research, or constraints (recommended but optional)
- Clear articulation of what you're trying to achieve
Trustworthy Experiments
What It Is
Trustworthy Experiments is a framework for running controlled experiments (A/B tests) that produce reliable, actionable results. The core insight: most experiments fail, and many "successful" results are actually false positives.
The key shift: Move from "Did the experiment show a positive result?" to "Can I trust this result enough to act on it?"
Ronny Kohavi, who built experimentation platforms at Microsoft, Amazon, and Airbnb, found that:
- 66-92% of experiments fail to improve the target metric
- 8% of experiments have invalid results due to sample ratio mismatch alone
- When the base success rate is 8%, a P-value of 0.05 still means 26% false positive risk
When to Use It
Use Trustworthy Experiments when you need to:
- Design an A/B test that will produce valid, actionable results
- Determine sample size and runtime for statistical power
- Validate experiment results before making ship/no-ship decisions
- Build an experimentation culture at your company
- Choose metrics (OEC) that balance short-term gains with long-term value
- Diagnose why results look suspicious (Twyman's Law)
- Speed up experimentation without sacrificing validity
When Not to Use It
Don't use controlled experiments when:
- You don't have enough users — Need tens of thousands minimum
- The decision is one-time — Can't A/B test mergers or acquisitions
- There's no real user choice — Employer-mandated software
- You need immediate decisions — Experiments need time
- The metric can't be measured — No experiment without observable outcomes
Resources
Book:
- Trustworthy Online Controlled Experiments by Ronny Kohavi, Diane Tang, and Ya Xu
Related skills
Microsoft FoundryDeploy, evaluate, and continuously improve Microsoft Foundry agents from a single agent interface.490k1.3k
Ai Research ReproductionOrchestrate trustworthy, auditable reproduction of deep learning repositories directly from their READMEs.176k511Run TrainSafely execute selected deep learning training commands with standardized evidence capture.176k511Explore RunSafely run isolated exploratory experiments with clear recording and conservative selection before committing changes.176k511Paper Context ResolverFetch precise reproduction-critical details like dataset splits, preprocessing steps, or evaluation protocols from the original academic paper when the repo README leav141k511Repo Intake And PlanScan unfamiliar AI research repositories and receive a minimal, trustworthy reproduction target before investing significant time.140k511How it compares
Pick trustworthy-experiments when you need experiment methodology and statistical rigor, not when you only need to implement tracking pixels or write SQL for existing dashboards.
FAQ
What framework does trustworthy-experiments use?
trustworthy-experiments implements Ronny Kohavi's Trustworthy Online Controlled Experiments framework. The skill structures hypotheses, holdouts, guardrails, and analysis plans so product experiments produce interpretable, statistically defensible results.
When should I use trustworthy-experiments?
Use trustworthy-experiments before shipping pricing, onboarding, or feature changes that need A/B validation. The skill designs sample sizing, guardrail metrics, and analysis protocols upfront instead of retrofitting statistics after launch.