Backtest Expert

# Learning from Failed Backtests

## Table of Contents

1. Why Failed Ideas Are Valuable
2. Common Failure Patterns
3. Case Study Framework
4. Red Flags Checklist

## 1. Why Failed Ideas Are Valuable

### The Value of Failures

**Key insights**:
- Failed tests save capital by preventing live implementation
- Failure patterns reveal which assumptions don't hold
- Understanding what doesn't work narrows the search space
- Failed tests build experience in recognizing fragile strategies

### Documentation Discipline

**Record for each failed idea**:
- The hypothesis being tested
- Why you thought it would work
- What the data showed
- Specific breaking points
- Lessons learned

**Purpose**: Build a library of "anti-patterns" to avoid repeating mistakes.

## 2. Common Failure Patterns

### Pattern 1: Parameter Sensitivity

**Symptom**: Strategy only works with very specific parameter values.

**Example scenario**:
- Strategy profitable with stop loss at exactly 2.5%
- Increasing to 3% or decreasing to 2% causes significant performance drop
- No "plateau" of stable performance

**Why it fails**: Real markets have noise; if small changes break the strategy, it likely captured noise, not signal.

**Lesson**: Seek strategies with stable performance across parameter ranges.

### Pattern 2: Regime-Specific Performance

**Symptom**: Strategy works brilliantly in some years, terribly in others.

**Example scenario**:
- Great performance in 2017-2019 (low volatility bull market)
- Catastrophic losses in 2020 (high volatility)
- Poor performance in 2022 (downtrend)

**Why it fails**: Strategy dependent on specific market conditions, not robust enough for diverse environments.

**Lesson**: Require acceptable (not necessarily best) performance across all regimes.

### Pattern 3: Slippage Sensitivity

**Symptom**: Strategy becomes unprofitable when realistic trading costs added.

**Example scenario**:
- Backtest shows 0.5% average gain per trade
- Adding 0.1% slippage per side (0.2% round-trip) eliminates profits
- Strategy requires unrealistic fills to be profitable

**Why it fails**: Edge too small to survive real-world friction.

**Lesson**: Edge must be large enough to survive pessimistic assumptions about costs.

### Pattern 4: Sample Size Issues

**Symptom**: Strong results based on small number of trades.

**Example scenario**:
- Backtest shows 80% win rate
- Only 15 total trades in 5 years
- A few different outcomes would dramatically change results

**Why it fails**: Insufficient data to distinguish edge from luck.

**Lesson**: Require minimum 100 trades for meaningful conclusions, preferably 200+.

### Pattern 5: Look-Ahead Bias

**Symptom**: Perfect or near-perfect backtest results.

**Example scenario**:
- Strategy shows 95%+ win rate
- Unrealistically good entry/exit timing
- Performance too good to be realistic

**Why it fails**: Likely using information not available at time of trade.

**Lesson**: Be suspicious of "too good to be true" results; audit data alignment carefully.

### Pattern 6: Over-Optimization (Curve Fitting)

**Symptom**: Complex strategy with many parameters shows excellent in-sample results but poor out-of-sample.

**Example scenario**:
- Strategy uses 8-10 different indicators with specific thresholds
- In-sample performance: 40% annual return
- Out-of-sample performance: -5% annual return
- Parameters needed constant re-optimization

**Why it fails**: Fitted to historical noise rather than genuine market structure.

**Lesson**: Prefer simple strategies with fewer parameters; demand strong out-of-sample results.

## 3. Case Study Framework

### Template for Documenting Failed Ideas

Use this framework when a backtest fails:

#### 1. Initial Hypothesis
- **What edge were you trying to capture?**
- **Why did you think this would work?**
- **What was the logical basis?**

#### 2. Implementation Details
- **Entry rules** (specific and complete)
- **Exit rules** (stop loss, profit target, time-based)
- **Position s