Peer Review

# Common Methodological and Statistical Issues in Scientific Manuscripts

This document catalogs frequent issues encountered during peer review, organized by category. Use this as a reference to identify potential problems and provide constructive feedback.

## Statistical Issues

### 1. P-Value Misuse and Misinterpretation

**Common Problems:**
- P-hacking (selective reporting of significant results)
- Multiple testing without correction (familywise error rate inflation)
- Interpreting non-significance as proof of no effect
- Focusing exclusively on p-values without effect sizes
- Dichotomizing continuous p-values at arbitrary thresholds (p=0.049 vs p=0.051)
- Confusing statistical significance with biological/clinical significance

**How to Identify:**
- Suspiciously high proportion of p-values just below 0.05
- Many tests performed but no correction mentioned
- Statements like "no difference was found" from non-significant results
- No effect sizes or confidence intervals reported
- Language suggesting p-values indicate strength of effect

**What to Recommend:**
- Report effect sizes with confidence intervals
- Apply appropriate multiple testing corrections (Bonferroni, FDR, Holm-Bonferroni)
- Interpret non-significance cautiously (lack of evidence ≠ evidence of lack)
- Pre-register analyses to avoid p-hacking
- Consider equivalence testing for "no difference" claims

### 2. Inappropriate Statistical Tests

**Common Problems:**
- Using parametric tests when assumptions are violated (non-normal data, unequal variances)
- Analyzing paired data with unpaired tests
- Using t-tests for multiple groups instead of ANOVA with post-hoc tests
- Treating ordinal data as continuous
- Ignoring repeated measures structure
- Using correlation when regression is more appropriate

**How to Identify:**
- No mention of assumption checking
- Small sample sizes with parametric tests
- Multiple pairwise t-tests instead of ANOVA
- Likert scales analyzed with t-tests
- Time-series data analyzed without accounting for repeated measures

**What to Recommend:**
- Check assumptions explicitly (normality tests, Q-Q plots)
- Use non-parametric alternatives when appropriate
- Apply proper corrections for multiple comparisons after ANOVA
- Use mixed-effects models for repeated measures
- Consider ordinal regression for ordinal outcomes

### 3. Sample Size and Power Issues

**Common Problems:**
- No sample size justification or power calculation
- Underpowered studies claiming "no effect"
- Post-hoc power calculations (which are uninformative)
- Stopping rules not pre-specified
- Unequal group sizes without justification

**How to Identify:**
- Small sample sizes (n<30 per group for typical designs)
- No mention of power analysis in methods
- Statements about post-hoc power
- Wide confidence intervals suggesting imprecision
- Claims of "no effect" with large p-values and small n

**What to Recommend:**
- Conduct a priori power analysis based on expected effect size
- Report achieved power or precision (confidence interval width)
- Acknowledge when studies are underpowered
- Consider effect sizes and confidence intervals for interpretation
- Pre-register sample size and stopping rules

### 4. Missing Data Problems

**Common Problems:**
- Complete case analysis without justification (listwise deletion)
- Not reporting extent or pattern of missingness
- Assuming data are missing completely at random (MCAR) without testing
- Inappropriate imputation methods
- Not performing sensitivity analyses

**How to Identify:**
- Different n values across analyses without explanation
- No discussion of missing data
- Participants "excluded from analysis"
- Simple mean imputation used
- No sensitivity analyses comparing complete vs. imputed data

**What to Recommend:**
- Report extent and patterns of missingness
- Test MCAR assumption (Little's test)
- Use appropriate methods (multiple imputation, maximum likelihood)
- Perform sensitivity analyses
- Consider intention-to-t