Statistical Power

Statistical Power & Sample Size

Overview

Power analysis answers one of the most consequential questions in study planning: how large a sample do you need to reliably detect an effect of a given size, and what could you detect with the sample you can afford? An underpowered study wastes resources and produces inconclusive or irreproducible results; an overpowered one wastes participants, money, and (in clinical work) exposes more people to risk than necessary. Getting this right before data collection is the single highest-leverage statistical decision in a project.

Four quantities are locked together for any given test: sample size (n), effect size, significance level (α), and power (1 − β). Fix any three and the fourth is determined. Every calculation in this skill is some rearrangement of that relationship.

This skill covers the two ways to do power analysis:

• Closed-form formulas (fast, exact for standard tests) — see references/closed_form_recipes.md.
• Simulation / Monte Carlo (works for any design or model you can simulate and analyze) — see references/simulation_based_power.md.

For choosing and converting effect sizes — usually the hardest part — see references/effect_sizes.md.

When to Use This Skill

• Determining required sample size before collecting data (a priori power analysis)
• Finding the minimum detectable effect (MDE) for a fixed, already-determined sample size
• Producing power curves (power vs. n, or power vs. effect size) for a grant or protocol
• Justifying a sample size for an IRB submission, grant, or pre-registration
• Powering designs with unequal group sizes or non-1:1 allocation
• Powering anything without a textbook formula (mixed models, logistic/Poisson regression, cluster-randomized trials, survival analysis, mediation, interactions) via simulation
• Accounting for multiple comparisons, attrition/dropout, or clustering in the sample-size estimate

Installation

Use uv. Pin versions in production; unpinned is fine for exploration.

uv pip install "statsmodels>=0.14.6" "scipy>=1.11" "pingouin>=0.6" "numpy>=1.26" matplotlib pandas
# For simulation-based power of advanced models (optional, add as needed):
uv pip install lifelines            # survival
# mixed models and GLMs come with statsmodels

Compatibility note: use statsmodels>=0.14.6 with scipy>=1.11 to avoid _lazywhere import errors on SciPy 1.16+. Pingouin 0.5+ renamed power-function arguments to match the names used below.

---

The one decision that drives everything: the effect size

Power calculations are only as trustworthy as the effect size you feed them. Do not invent a number. Use, in rough order of preference:

1. A minimally important effect — the smallest effect that would actually change a decision or matter scientifically/clinically (the "smallest effect size of interest", SESOI). This is the most defensible basis: you power to detect what matters, not what you hope to see. 2. A pilot or prior-study estimate, but shrink it — published and pilot effects are inflated by publication bias and the winner's curse. Powering on a raw pilot estimate routinely underpowers the real study. 3. A convention (Cohen's small/medium/large) only as a last resort, and say so explicitly.

Whatever you pick, run a sensitivity analysis: report how required n changes across a plausible range of effect sizes, not a single point. A power analysis presented as one number hides its biggest source of uncertainty. See references/effect_sizes.md for benchmarks and conversions between d, f, r, η², odds ratios, and Cohen's h/w.

Avoid post-hoc ("observed") power. Computing power from the effect size you just estimated is circular: it is a deterministic function of the p-value and tells you nothing new. If a study is already done and you want to know what it could have detected, report a sensitivity analysis (MDE at the achieved n) or, better, the confidence interval around the observed effect. This is a common reviewer complaint — do not produce observed power even if asked without flagging the issue.

---

Quick recipes (closed-form)

The bundled scripts/power.py wraps statsmodels into one consistent interface so you don't have to remember which solver belongs to which test. Run from the skill directory or add scripts/ to sys.path.

from power import sample_size, power, mde, power_curve

# 1. How many per group to detect Cohen's d = 0.5, two-sided, 80% power?
sample_size(test="t_ind", effect_size=0.5, power=0.80, alpha=0.05)
# -> required n per group

# 2. Two groups, 3:1 allocation (e.g. more controls than cases)
sample_size(test="t_ind", effect_size=0.5, power=0.80, ratio=3.0)

# 3. Fixed n=30/group — what's the minimum detectable d at 80% power?
mde(test="t_ind", nobs1=30, power=0.80, alpha=0.05)

# 4. One-way ANOVA, 4 groups, detect Cohen's f = 0.25
sample_size(test="anova", effect_size=0.25, k_groups=4, power=0.80)

# 5. Two proportions: 0.40 vs 0.55 (auto-converts to Cohen's h)
sample_size(test="two_proportions", prop1=0.40, prop2=0.55, power=0.80)

# 6. Correlation: detect r = 0.30
sample_size(test="correlation", effect_size=0.30, power=0.80)

# 7. Power curve for the grant figure
power_curve(test="t_ind", effect_size=0.5, n_range=range(10, 120, 5),
            save="power_curve.png")

Supported test= values: t_ind (two independent means), t_paired/t_one (paired or one-sample mean), anova (one-way), two_proportions, one_proportion, correlation, chi2 (goodness-of-fit / contingency via effect size w), linear_regression (R² increment / f²). Full argument tables and the underlying statsmodels calls are in references/closed_form_recipes.md.

---

When there is no formula: simulate

Closed-form power exists only for a handful of simple tests. For logistic/Poisson regression, mixed-effects / repeated-measures models, cluster-randomized trials, survival analysis, mediation, multi-way interactions, or any non-standard analysis, the right tool is simulation. The logic is always the same three steps:

1. Simulate a dataset from your assumed truth (the effect you want to detect, plus realistic noise, baseline rates, cluster structure, etc.). 2. Analyze it with the exact test/model you plan to use on the real data. 3. Repeat many times (≥1,000; 5,000–10,000 for a stable estimate near 80%). Power is the fraction of replicates in which the test is significant.

scripts/simulate_power.py provides a reusable harness plus worked examples (two-group difference, logistic regression, cluster-randomized trial with an ICC, and a linear mixed model). The core is just:

from simulate_power import simulate_power

def gen_and_test(n, rng):
    # build a dataset of size n under the assumed effect, run the planned test,
    # return True if the result is significant
    ...

est = simulate_power(gen_and_test, n=200, n_sims=2000, alpha=0.05)
print(f"Power at n=200: {est.power:.3f} (95% CI {est.ci_low:.3f}-{est.ci_high:.3f})")

Report the Monte Carlo confidence interval on the estimate (the harness returns it) so the reader knows whether 0.81 vs. 0.79 is signal or simulation noise. See references/simulation_based_power.md for the full patterns, including how to search for the n that hits target power and how to model dropout and clustering.

---

Adjustments people forget

These routinely make the difference between an adequately powered study and an underpowered one. Apply them explicitly and state that you did.

• Multiple comparisons. If the analysis tests m hypotheses with a Bonferroni-style correction, power each test at the corrected α (e.g. α/m), which raises n. Better: power on the family-wise or FDR-controlled procedure directly via simulation. Ignoring this silently underpowers every secondary endpoint.
• Attrition / dropout / unusable samples. Power gives the n you need analyzed. Inflate the enrolled n: n_enroll = ceil(n_analyzed / (1 − dropout_rate)). A 20% dropout rate means enrolling 25% more than the formula returns.
• Clustering (design effect). When observations are nested (patients within clinics, cells within animals, repeated measures within subject), the effective sample size is smaller than the raw count. Inflate by the design effect DEFF = 1 + (m − 1)·ICC, where m is cluster size and ICC the intraclass correlation. Treating clustered data as independent is pseudoreplication and badly overstates power — for cluster-randomized designs, simulate instead.
• One- vs. two-sided. Two-sided is the default and almost always the right choice; a one-sided test buys power only by refusing to detect an effect in the unexpected direction. Justify any one-sided test.
• Unequal allocation. Equal groups are most efficient for a fixed total n. If allocation is fixed by design (e.g. 2:1 treatment:control), pass ratio= so the calculation reflects it.

---

Workflow

1. State the design and the planned analysis. The test you will run determines the power method. If the analysis is a mixed model or GLM, go straight to simulation. 2. Choose the effect size on a defensible basis (SESOI > shrunk pilot > convention) and write down the justification. 3. Set α and target power. Conventional defaults are α = 0.05 (two-sided) and power = 0.80; 0.90 is common for confirmatory/clinical work. State them. 4. Compute with scripts/power.py (closed-form) or scripts/simulate_power.py (simulation). 5. Sensitivity analysis. Recompute across a range of plausible effect sizes and produce a power curve. This is the deliverable, not a single number. 6. Apply adjustments for dropout, clustering, and multiplicity. 7. Report following the template below.

---

Reporting template

A defensible power statement contains every input, so a reader could reproduce it. Adapt:

A priori power analysis was conducted to determine the sample size needed to detect
a [between-group difference of Cohen's d = 0.50], which we considered the smallest
effect of clinical interest. With α = .05 (two-sided) and power = .80, a two-sample
t-test requires n = 64 per group (128 total; computed with statsmodels 0.14).
Allowing for 20% attrition, we will enrol 160 participants. A sensitivity analysis
showed required n ranges from 45 to 105 per group across plausible effects
d = 0.40–0.60 (Figure X).

For simulation: also state the data-generating assumptions (baseline rate, residual SD, ICC, cluster sizes), the number of simulations, and the Monte Carlo CI.

---

Common pitfalls

1. Inventing the effect size or copying an inflated pilot estimate — the most common way power analyses go wrong. 2. Reporting a single n instead of a sensitivity range / power curve. 3. Post-hoc / observed power — circular and uninformative; use sensitivity analysis or the effect-size CI instead. 4. Ignoring clustering (pseudoreplication) — counting cells/measurements as if they were independent subjects. 5. Forgetting dropout — powering the analyzed n but enrolling the same number. 6. Confusing α with power, or one-sided with two-sided. 7. Powering only the primary endpoint while reporting secondary/interaction tests that need far larger n. 8. Using a t-test formula for a model you won't actually fit (e.g. planning a logistic regression with a means-based calculation) — match the power method to the planned analysis.

---

Resources

Scripts

• scripts/power.py — unified closed-form interface (sample_size, power, mde, power_curve) over statsmodels/pingouin for all standard tests.
• scripts/simulate_power.py — Monte Carlo power harness with simulate_power() and find_sample_size(), plus worked examples (two-group, logistic regression, cluster-randomized, linear mixed model).

References

• references/closed_form_recipes.md — per-test argument tables and exact statsmodels/pingouin calls, including proportions, chi-square, and regression.
• references/simulation_based_power.md — full simulation patterns for GLMs, mixed models, cluster designs, survival, and dropout.
• references/effect_sizes.md — choosing effect sizes (SESOI), Cohen's benchmarks, and conversions between d, f, r, η²/f², OR, h, and w.

Related skills

• experimental-design — once you know n, lay out the actual study (randomization, blocking, factorial/DOE, crossover, sequential designs).
• statistical-analysis — assumption checks, running the test, effect sizes, and APA reporting after data collection.
• statsmodels / pymc — fitting the models referenced here.

Key references

• Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
• Lakens, D. (2022). Sample Size Justification. Collabra: Psychology, 8(1).
• Arnold, B. F. et al. (2011). Simulation methods to estimate design power. BMC Medical Research Methodology, 11:94.

Closed-Form Power Recipes

Exact argument tables and the underlying statsmodels/scipy calls for every test the scripts/power.py helper supports. Use this when you need to call statsmodels directly, understand an argument, or handle a case the wrapper doesn't cover.

The four solver quantities — effect_size, sample size, alpha, power — obey one identity: pass three, set the fourth to None, and solve_power returns it.

Table of contents

• Two independent means (t-test)
• Paired / one-sample mean
• One-way ANOVA
• Two proportions
• One proportion
• Correlation
• Chi-square (goodness-of-fit / contingency)
• Multiple regression (R² increment)
• Effect-size argument cheat sheet

---

Two independent means

Effect size = Cohen's d = (μ₁ − μ₂) / σ_pooled.

from statsmodels.stats.power import TTestIndPower
analysis = TTestIndPower()

# n per group for d=0.5, 80% power, two-sided
n1 = analysis.solve_power(effect_size=0.5, alpha=0.05, power=0.80,
                          ratio=1.0, alternative="two-sided")

# achieved power at n1=64 per group
pw = analysis.solve_power(effect_size=0.5, nobs1=64, alpha=0.05,
                          ratio=1.0, alternative="two-sided")

# minimum detectable d at n1=30 per group, 80% power
d_min = analysis.solve_power(nobs1=30, alpha=0.05, power=0.80, ratio=1.0,
                             alternative="two-sided")

ratio = nobs2 / nobs1. For 2:1 allocation set ratio=2.0; the returned nobs1 is the smaller group. alternative ∈ "two-sided", "larger", "smaller".

Paired / one-sample mean

Effect size = Cohen's dz for paired (mean difference / SD of the differences), or d for one-sample. Use TTestPower (single-sample solver); nobs is the number of pairs / observations.

from statsmodels.stats.power import TTestPower
TTestPower().solve_power(effect_size=0.4, alpha=0.05, power=0.80,
                         alternative="two-sided")  # -> number of pairs

Note: for paired designs dz depends on the within-pair correlation ρ: dz = d_raw / sqrt(2(1−ρ)). Higher ρ ⇒ larger dz ⇒ smaller n. If you only know the raw mean difference and SDs, estimate ρ or simulate.

One-way ANOVA

Effect size = Cohen's f = sqrt(η² / (1 − η²)). nobs here is total n across all groups; divide by k_groups for per-group n.

from statsmodels.stats.power import FTestAnovaPower
total_n = FTestAnovaPower().solve_power(effect_size=0.25, k_groups=4,
                                        alpha=0.05, power=0.80)
per_group = total_n / 4

Conversions: f = 0.10 (small), 0.25 (medium), 0.40 (large). From η²: f = sqrt(eta2/(1-eta2)). From R²: same formula with R².

Two proportions

Effect size = Cohen's h = 2·asin(√p₁) − 2·asin(√p₂). Convert proportions to h, then use the normal approximation NormalIndPower.

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize
h = proportion_effectsize(0.40, 0.55)
n1 = NormalIndPower().solve_power(effect_size=h, alpha=0.05, power=0.80,
                                  ratio=1.0, alternative="two-sided")

Alternative (exact-ish, gives per-group n directly, handles unequal n via ratio):

from statsmodels.stats.proportion import samplesize_proportions_2indep_onetail
# one-sided; double alpha intent by passing alpha/... per your convention

For small samples or rare events, prefer simulation with the exact test you'll run (Fisher's exact, or a chi-square with continuity correction).

One proportion

Test p against a fixed reference p₀. Convert both to the arcsine scale via Cohen's h and treat the reference group as infinite (ratio=0).

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize
h = proportion_effectsize(0.60, 0.50)
n = NormalIndPower().solve_power(effect_size=h, alpha=0.05, power=0.80, ratio=0.0)

For exact binomial planning use statsmodels.stats.proportion.proportion_effectsize with the exact-test power via simulation if the sample is small.

Correlation

Effect size = Pearson r. No statsmodels solver; use the Fisher z transform (implemented in power.py). Required n for r at power 1−β, two-sided:

z_r = arctanh(r)
n   = ((z_{1-α/2} + z_{1-β}) / z_r)^2 + 3

pingouin.power_corr(r=0.3, power=0.8, alternative="two-sided") gives the same answer if you prefer a library call.

Chi-square

Effect size = Cohen's w = sqrt(Σ (p_i − p0_i)² / p0_i). For a contingency table, w = sqrt(χ²/N) and equals Cramér's V·sqrt(min(r−1, c−1)). Degrees of freedom: goodness-of-fit dof = k − 1; contingency dof = (r−1)(c−1). n_bins = dof + 1.

from statsmodels.stats.power import GofChisquarePower
n = GofChisquarePower().solve_power(effect_size=0.3, n_bins=5, alpha=0.05, power=0.80)

w benchmarks: 0.10 (small), 0.30 (medium), 0.50 (large).

Multiple regression

Effect size = Cohen's f² = R²/(1−R²) for the overall model, or ΔR²/(1−R²_full) for a set of added predictors. power.py solves this directly via the noncentral F (noncentrality λ = f²·n), which is more reliable than statsmodels' FTestPower for sample-size search.

from power import sample_size, power
# detect f^2 = 0.15 from 3 tested predictors (3 total in the model)
sample_size("linear_regression", effect_size=0.15, df_num=3, k_total=3, power=0.80)

• df_num = number of predictors being tested (the numerator df).
• k_total = total predictors in the model (including controls). df_denom = n − k_total − 1.

f² benchmarks: 0.02 (small), 0.15 (medium), 0.35 (large).

Effect-size units per test

Benchmarks are last-resort conventions — prefer a smallest-effect-of-interest. See effect_sizes.md.

Choosing and Converting Effect Sizes

The effect size is the input that makes or breaks a power analysis, and it is the one people most often get wrong. Power computed from a guessed or inflated effect is worse than no power analysis, because it carries false authority. This file covers how to pick a defensible value and how to convert between the metrics different tests use.

How to choose (in order of preference)

1. Smallest effect size of interest (SESOI) — best

Power to detect the smallest effect that would actually change a decision or matter scientifically/clinically, not the effect you hope or expect to see. Ways to set it:

• Anchor-based: the smallest difference patients/users can perceive or that

crosses a clinical threshold (e.g. a 5-point change on a validated scale).

• Resource/decision-based: the smallest effect that would justify adopting the

intervention given its cost.

• Benchmark-based: an effect smaller than which you'd treat the result as

practically null.

Powering on the SESOI is the most defensible choice: if the true effect is larger, you're even better powered; if it's smaller, you've decided it doesn't matter.

2. Prior estimate — but shrink it

Pilot studies and published effects are biased upward (publication bias, the "winner's curse," and the fact that significant pilots are the ones that get followed up). Powering on a raw pilot d routinely underpowers the real study. If you must use a prior estimate:

• Use the lower bound of its confidence interval, or
• Apply a shrinkage / safeguard (e.g. Perugini et al.'s safeguard power uses the

CI lower limit), and

• Never rely on a single small pilot (n < ~50) for a point estimate of the effect.

3. Convention — last resort, and say so

Cohen's small/medium/large are arbitrary and field-blind. They were never meant as substitutes for domain knowledge. Use them only when nothing better exists, state explicitly that you did, and prefer "small" unless you have a reason — most real effects in many fields are small.

Always do a sensitivity analysis

Whatever you pick, report how required n varies across a plausible range of effects (e.g. a power curve, or a small table of n at d = 0.3, 0.4, 0.5). A single n hides the dominant source of uncertainty. This is the actual deliverable of a good power analysis.

Benchmark table (Cohen's conventions)

(OR benchmarks are very context-dependent and depend on the base rate — treat as rough only.)

Conversions

d ↔ r (two-group comparison ↔ point-biserial)

r = d / sqrt(d^2 + 4)            # equal groups
d = 2r / sqrt(1 - r^2)

d ↔ Cohen's f (k groups; for two equal groups f = d/2)

f = d / 2                        # two groups

f ↔ η²

f   = sqrt(eta2 / (1 - eta2))
eta2 = f^2 / (1 + f^2)

f² ↔ R² (regression)

f2 = R2 / (1 - R2)               # whole model
f2 = dR2 / (1 - R2_full)         # increment from added predictors

proportions → Cohen's h

h = 2*asin(sqrt(p1)) - 2*asin(sqrt(p2))

In Python: statsmodels.stats.proportion.proportion_effectsize(p1, p2).

proportions → Cohen's w (for chi-square, against expected p0_i)

w = sqrt( sum( (p_i - p0_i)^2 / p0_i ) )

For a 2×2 table, w = sqrt(chi2 / N), and w = V * sqrt(min(r-1, c-1)) where V is Cramér's V.

odds ratio → log-odds (for logistic-regression power by simulation)

beta = log(OR)                   # coefficient to plug into the simulated linear predictor

standardized → raw A standardized effect is only as good as the SD you divide by. If you know the raw difference and the SD, work in raw units and convert at the end: d = (mean1 - mean2) / sd_pooled. For paired designs, dz uses the SD of the differences, which depends on the within-pair correlation — see closed_form_recipes.md.

Common mistakes

• Using the observed/expected effect instead of the SESOI — you end up powered

for your hopes, not for what matters.

• Copying a published d without shrinking — inflated by publication bias.
• Mixing up d and f, or η² and f² — they differ by the conversions above; a

factor-of-2 error in d quadruples or quarters the required n.

• Reporting one number — always show the sensitivity range.
• Treating Cohen's benchmarks as truth — they're conventions, not measurements.

Simulation-Based (Monte Carlo) Power

Closed-form power exists for a handful of standard tests. For everything else, simulate. This is not a second-best approximation — for complex designs it is the correct method, and it has one big advantage: the power estimate uses the exact analysis you will run on the real data, so there is no mismatch between the planning model and the analysis model.

The recipe (always the same)

1. Simulate a dataset of size n from your assumed truth: the effect you want to detect, plus realistic structure (baseline rates, residual SD, cluster random effects, dropout, covariate distributions). 2. Analyze it with the exact model/test planned for the real study. 3. Repeat R times. Power = fraction of replicates where the test is significant. Use R ≥ 1,000; use 5,000–10,000 for a stable estimate near the 80% decision point.

Always report the Monte Carlo confidence interval on the estimate (the scripts/simulate_power.py harness returns a Wilson interval). With R = 1,000 the ±2 SE width near p = 0.8 is roughly ±0.025, so don't over-interpret 0.81 vs 0.79.

Using the harness

scripts/simulate_power.py gives you simulate_power() and find_sample_size(). You supply a function gen_and_test(n, rng) -> bool that builds one dataset, runs the analysis, and returns whether it was significant. The rng is a seeded numpy.random.Generator so runs are reproducible and replicates are independent.

from simulate_power import simulate_power, find_sample_size

def gen_and_test(n, rng):
    # ... simulate n observations under the assumed effect ...
    # ... fit the planned model ...
    return pvalue < 0.05

# power at a fixed n
print(simulate_power(gen_and_test, n=200, n_sims=2000))

# search for the n that hits 80% power
n, est = find_sample_size(gen_and_test, target_power=0.80, n_sims=2000)

The file ships four adaptable examples: two-group difference (a sanity check against the closed-form t-test), logistic regression, a cluster-randomized trial with an ICC, and a repeated-measures linear mixed model. Copy the closest one and edit the data-generating block.

When you must simulate

Key correctness points

• Analyze exactly as planned. If the real analysis adjusts for covariates,

include them in the simulation. If it uses a robust SE or a specific correction, apply it in gen_and_test. The whole value of simulation is fidelity to the plan.

• Handle estimation failures. GLMs and mixed models can fail to converge or hit

perfect separation. Wrap the fit in try/except and count a failure as not significant (conservative). If failures are common, that itself is a warning about the design or sample size.

• Watch Type I error too. As a check, simulate under the null (effect = 0)

and confirm the rejection rate ≈ α. If it's inflated (common with small-cluster mixed models or naive cluster SEs), your planned analysis is anticonservative and the power number is meaningless until you fix the analysis.

• Seed it. A fixed seed makes the search reproducible and stops find_sample_size

from chasing simulation noise around the boundary.

Modeling realistic complications

• Dropout. Either simulate missingness directly (drop rows / occasions under

the assumed mechanism, then analyze the reduced data — captures the real power loss including any bias), or compute the analyzed n and inflate the enrolled n by 1/(1−dropout).

• Clustering / ICC. Split total variance into between-cluster (τ²) and residual

(σ²) with ICC = τ²/(τ²+σ²), draw a cluster random effect ~ N(0, τ), add it to every member of the cluster. See example_cluster_randomized.

• Unequal allocation / stratification. Generate the exact group sizes and strata

the design will produce; don't assume balance the design won't deliver.

• Repeated measures. Subject random intercept (and slope, if relevant) plus a

within-subject residual; the within-subject correlation is τ²/(τ²+σ²).

Reporting a simulation-based power analysis

State enough that someone could rerun it:

Power was estimated by simulation (5,000 replicates per sample size). Data were
generated assuming a baseline event rate of 20%, a treatment log-odds of 0.8, and
analyzed with logistic regression adjusting for age and site, matching the planned
analysis. A sample of n = 150 per arm yielded 82% power (95% Monte Carlo CI
80.5-83.5%) at α = .05 (two-sided). Code is available at [link].