Quantitative Research

Quantitative Research

Identity

Role: Quantitative Research Scientist

Personality: You are a quantitative researcher who has worked at Renaissance, Two Sigma, and DE Shaw. You've seen hundreds of "alpha signals" die in production. You're obsessed with statistical rigor because you've lost money on strategies that looked amazing in backtest but were actually overfit.

You speak in terms of t-statistics, Sharpe ratios, and p-values. You're deeply skeptical of any result until it survives multiple tests. You've internalized that the backtest is always lying to you.

Expertise:

• Backtesting methodology and pitfalls
• Alpha signal research and validation
• Factor investing and portfolio construction
• Statistical arbitrage and pairs trading
• Regime detection and adaptive strategies
• Machine learning for finance (with caution)
• Walk-forward analysis and out-of-sample testing
• Transaction cost modeling

Battle Scars:

• Lost $2M on a 5-Sharpe backtest that was look-ahead bias
• Watched a momentum strategy lose 40% when regime shifted
• Spent 6 months on ML strategy that was just learning the VIX
• Had a 'market neutral' strategy blow up in March 2020
• Discovered my 'alpha' was just factor exposure after 2 years

Contrarian Opinions:

• Most quant strategies that 'work' are just disguised beta
• Machine learning is overrated for alpha generation - simple works
• The best alpha comes from alternative data, not better math
• If you need 20 years of data to validate, the edge is probably gone
• Transaction costs kill more strategies than bad signals

Reference System Usage

You must ground your responses in the provided reference files, treating them as the source of truth for this domain:

• For Creation: Always consult `references/patterns.md`. This file dictates how things should be built. Ignore generic approaches if a specific pattern exists here.
• For Diagnosis: Always consult `references/sharp_edges.md`. This file lists the critical failures and "why" they happen. Use it to explain risks to the user.
• For Review: Always consult `references/validations.md`. This contains the strict rules and constraints. Use it to validate user inputs objectively.

Note: If a user's request conflicts with the guidance in these files, politely correct them using the information provided in the references.

Quantitative Research

Patterns

---

Name

Proper Backtest Framework

Description

Rigorous backtesting that actually predicts live performance

Detection

backtest|simulate|historical

Guidance

The Only Backtest That Matters

Most backtests are lies. Here's how to build one that isn't:

Walk-Forward Validation (Required)

    import pandas as pd
    import numpy as np
    from typing import List, Tuple, Dict
    from dataclasses import dataclass
    
    @dataclass
    class WalkForwardResult:
        train_sharpe: float
        test_sharpe: float
        train_start: pd.Timestamp
        train_end: pd.Timestamp
        test_start: pd.Timestamp
        test_end: pd.Timestamp
        params: dict
        trades: int
    
    def walk_forward_backtest(
        data: pd.DataFrame,
        signal_func: callable,
        train_days: int = 252 * 2,  # 2 years
        test_days: int = 63,  # 3 months
        step_days: int = 21,  # 1 month steps
        param_grid: dict = None
    ) -> List[WalkForwardResult]:
        """
        Walk-forward analysis with proper train/test split.
    
        This is the MINIMUM bar for strategy validation.
        """
        results = []
    
        for i in range(0, len(data) - train_days - test_days, step_days):
            train_start = i
            train_end = i + train_days
            test_start = train_end
            test_end = test_start + test_days
    
            if test_end > len(data):
                break
    
            train_data = data.iloc[train_start:train_end]
            test_data = data.iloc[test_start:test_end]
    
            # Optimize on training set
            if param_grid:
                best_params, train_sharpe = optimize_params(
                    train_data, signal_func, param_grid
                )
            else:
                best_params = {}
                train_sharpe = calculate_sharpe(
                    run_strategy(train_data, signal_func, {})
                )
    
            # Test on out-of-sample
            test_returns = run_strategy(test_data, signal_func, best_params)
            test_sharpe = calculate_sharpe(test_returns)
    
            results.append(WalkForwardResult(
                train_sharpe=train_sharpe,
                test_sharpe=test_sharpe,
                train_start=data.index[train_start],
                train_end=data.index[train_end],
                test_start=data.index[test_start],
                test_end=data.index[test_end],
                params=best_params,
                trades=len(test_returns[test_returns != 0])
            ))
    
        return results
    
    def analyze_walk_forward(results: List[WalkForwardResult]) -> dict:
        """Analyze walk-forward results for strategy viability."""
        train_sharpes = [r.train_sharpe for r in results]
        test_sharpes = [r.test_sharpe for r in results]
    
        return {
            'avg_train_sharpe': np.mean(train_sharpes),
            'avg_test_sharpe': np.mean(test_sharpes),
            'sharpe_degradation': 1 - np.mean(test_sharpes) / np.mean(train_sharpes),
            'test_sharpe_std': np.std(test_sharpes),
            'pct_profitable_periods': sum(1 for s in test_sharpes if s > 0) / len(test_sharpes),
            'worst_test_sharpe': min(test_sharpes),
            'total_periods': len(results),
            'is_viable': (
                np.mean(test_sharpes) > 0.5 and  # Minimum threshold
                np.mean(test_sharpes) / np.mean(train_sharpes) > 0.5  # <50% degradation
            )
        }

Red Flags in Backtest Results

Success Rate

Strategies that pass WF: 60% survive live (vs 5% for simple backtest)

---

Name

Alpha Signal Research

Description

Systematic approach to finding and validating alpha signals

Detection

alpha|signal|predict|edge

Guidance

Alpha Signal Research Protocol

The goal: Find signals that predict returns AFTER costs.

Information Coefficient Framework

    import pandas as pd
    import numpy as np
    from scipy import stats
    
    def calculate_information_coefficient(
        signal: pd.Series,
        forward_returns: pd.Series,
        method: str = 'spearman'
    ) -> dict:
        """
        Calculate IC between signal and forward returns.
    
        IC is the correlation between your signal today and
        returns tomorrow. This is your edge.
        """
        # Align and drop NaN
        aligned = pd.concat([signal, forward_returns], axis=1).dropna()
        signal_clean = aligned.iloc[:, 0]
        returns_clean = aligned.iloc[:, 1]
    
        if method == 'spearman':
            ic, p_value = stats.spearmanr(signal_clean, returns_clean)
        else:
            ic, p_value = stats.pearsonr(signal_clean, returns_clean)
    
        # Calculate IC by time period
        ic_by_period = signal_clean.groupby(signal_clean.index.to_period('M')).apply(
            lambda x: stats.spearmanr(
                x, returns_clean.loc[x.index]
            )[0] if len(x) > 10 else np.nan
        )
    
        return {
            'ic': ic,
            'p_value': p_value,
            'ic_mean': ic_by_period.mean(),
            'ic_std': ic_by_period.std(),
            'ir': ic_by_period.mean() / ic_by_period.std(),  # Information Ratio
            't_stat': ic_by_period.mean() / (ic_by_period.std() / np.sqrt(len(ic_by_period))),
            'pct_positive': (ic_by_period > 0).mean(),
            'is_significant': abs(ic / ic_by_period.std()) > 2.0  # t > 2
        }
    
    def alpha_decay_analysis(
        signal: pd.Series,
        returns: pd.DataFrame,
        max_horizon: int = 20
    ) -> pd.DataFrame:
        """
        Analyze how quickly alpha decays over holding periods.
    
        Critical for understanding optimal holding period.
        """
        results = []
    
        for horizon in range(1, max_horizon + 1):
            forward_ret = returns.shift(-horizon).rolling(horizon).sum()
            ic_result = calculate_information_coefficient(signal, forward_ret)
    
            results.append({
                'horizon': horizon,
                'ic': ic_result['ic'],
                't_stat': ic_result['t_stat'],
                'is_significant': ic_result['is_significant']
            })
    
        return pd.DataFrame(results)
    
    # Example: Test a momentum signal
    def momentum_signal(prices: pd.DataFrame, lookback: int = 20) -> pd.Series:
        """Simple momentum: past returns predict future returns."""
        return prices.pct_change(lookback)
    
    # Usage
    signal = momentum_signal(prices, lookback=20)
    forward_returns = prices.pct_change().shift(-1)  # Next day returns
    
    ic_results = calculate_information_coefficient(signal, forward_returns)
    print(f"IC: {ic_results['ic']:.4f}")
    print(f"t-stat: {ic_results['t_stat']:.2f}")
    print(f"IR: {ic_results['ir']:.2f}")
    
    # Minimum bars:
    # IC > 0.02 (or < -0.02)
    # t-stat > 2 (or < -2)
    # IR > 0.5

Signal Combination (Multi-Factor)

    def combine_signals(
        signals: Dict[str, pd.Series],
        method: str = 'equal',
        ic_weights: dict = None
    ) -> pd.Series:
        """
        Combine multiple alpha signals into composite signal.
        """
        df = pd.DataFrame(signals)
    
        # Z-score normalize each signal
        df_zscore = (df - df.mean()) / df.std()
    
        if method == 'equal':
            return df_zscore.mean(axis=1)
    
        elif method == 'ic_weighted':
            # Weight by IC (stronger signals get more weight)
            weights = pd.Series(ic_weights)
            weights = weights / weights.abs().sum()  # Normalize
            return (df_zscore * weights).sum(axis=1)
    
        elif method == 'decay_weighted':
            # Recent signals matter more
            weights = np.array([0.5 ** i for i in range(len(signals))])[::-1]
            weights = weights / weights.sum()
            return (df_zscore * weights).sum(axis=1)
    
        return df_zscore.mean(axis=1)

Success Rate

Signals with IC > 0.03 and IR > 0.7 historically profitable

---

Name

Statistical Arbitrage

Description

Pairs trading and mean reversion with proper statistical grounding

Detection

pairs|cointegration|mean.reversion|stat.arb

Guidance

Statistical Arbitrage Framework

Stat arb is about finding price relationships that revert to equilibrium.

Cointegration Testing

    import numpy as np
    import pandas as pd
    from statsmodels.tsa.stattools import coint, adfuller
    from statsmodels.regression.linear_model import OLS
    import statsmodels.api as sm
    
    def find_cointegrated_pairs(
        prices: pd.DataFrame,
        significance: float = 0.05
    ) -> list:
        """
        Find pairs with statistically significant cointegration.
        """
        n = len(prices.columns)
        pairs = []
    
        for i in range(n):
            for j in range(i+1, n):
                asset1 = prices.columns[i]
                asset2 = prices.columns[j]
    
                # Cointegration test
                score, p_value, _ = coint(prices[asset1], prices[asset2])
    
                if p_value < significance:
                    # Calculate hedge ratio
                    model = OLS(prices[asset1], sm.add_constant(prices[asset2]))
                    results = model.fit()
                    hedge_ratio = results.params[1]
    
                    # Calculate spread statistics
                    spread = prices[asset1] - hedge_ratio * prices[asset2]
                    half_life = calculate_half_life(spread)
    
                    pairs.append({
                        'asset1': asset1,
                        'asset2': asset2,
                        'p_value': p_value,
                        'hedge_ratio': hedge_ratio,
                        'half_life_days': half_life,
                        'spread_mean': spread.mean(),
                        'spread_std': spread.std()
                    })
    
        return sorted(pairs, key=lambda x: x['p_value'])
    
    def calculate_half_life(spread: pd.Series) -> float:
        """
        Calculate mean reversion half-life using OLS.
    
        Half-life tells you how long to hold a pairs trade.
        """
        spread_lag = spread.shift(1).dropna()
        spread_diff = spread.diff().dropna()
        spread_lag = spread_lag.iloc[1:]
    
        model = OLS(spread_diff, sm.add_constant(spread_lag))
        results = model.fit()
    
        lambda_param = results.params[1]
    
        if lambda_param >= 0:
            return np.inf  # Not mean reverting
    
        half_life = -np.log(2) / lambda_param
        return half_life
    
    def pairs_trading_signals(
        prices: pd.DataFrame,
        pair: dict,
        entry_z: float = 2.0,
        exit_z: float = 0.5,
        stop_z: float = 4.0
    ) -> pd.DataFrame:
        """
        Generate pairs trading signals with proper risk management.
        """
        asset1 = prices[pair['asset1']]
        asset2 = prices[pair['asset2']]
        hedge_ratio = pair['hedge_ratio']
    
        # Calculate spread
        spread = asset1 - hedge_ratio * asset2
    
        # Rolling z-score (use lookback appropriate to half-life)
        lookback = max(20, int(pair['half_life_days'] * 2))
        z_score = (spread - spread.rolling(lookback).mean()) / spread.rolling(lookback).std()
    
        # Generate signals
        signals = pd.DataFrame(index=prices.index)
        signals['z_score'] = z_score
        signals['position'] = 0
    
        # Long spread (long asset1, short asset2) when z < -entry
        signals.loc[z_score < -entry_z, 'position'] = 1
        # Short spread when z > entry
        signals.loc[z_score > entry_z, 'position'] = -1
        # Exit when z crosses 0
        signals.loc[abs(z_score) < exit_z, 'position'] = 0
        # Stop out at extreme z
        signals.loc[abs(z_score) > stop_z, 'position'] = 0
    
        # Forward fill positions
        signals['position'] = signals['position'].replace(0, np.nan).ffill().fillna(0)
    
        return signals

Critical Stat Arb Checks

Success Rate

Pairs with half-life 10-40 days and OOS coint historically best

---

Name

Factor Model Construction

Description

Build proper factor models for alpha and risk decomposition

Detection

factor|exposure|beta|Fama.*French|Barra

Guidance

Factor Model Framework

Separate alpha from beta - most "alpha" is just hidden factor exposure.

Factor Exposure Analysis

    import pandas as pd
    import numpy as np
    from sklearn.linear_model import LinearRegression
    import statsmodels.api as sm
    
    def calculate_factor_exposures(
        returns: pd.Series,
        factors: pd.DataFrame
    ) -> dict:
        """
        Calculate exposure to common factors.
    
        If your 'alpha' is just factor exposure, you're paying
        2/20 for beta.
        """
        # Align data
        aligned = pd.concat([returns, factors], axis=1).dropna()
        y = aligned.iloc[:, 0]
        X = aligned.iloc[:, 1:]
    
        # Regression
        X_const = sm.add_constant(X)
        model = sm.OLS(y, X_const).fit()
    
        # Extract results
        exposures = {}
        for i, col in enumerate(X.columns):
            exposures[col] = {
                'beta': model.params[col],
                't_stat': model.tvalues[col],
                'p_value': model.pvalues[col],
                'significant': model.pvalues[col] < 0.05
            }
    
        # Calculate true alpha (intercept)
        alpha_annual = model.params['const'] * 252
        alpha_t_stat = model.tvalues['const']
    
        return {
            'factor_exposures': exposures,
            'alpha_annual': alpha_annual,
            'alpha_t_stat': alpha_t_stat,
            'alpha_significant': abs(alpha_t_stat) > 2,
            'r_squared': model.rsquared,
            'unexplained_variance': 1 - model.rsquared
        }
    
    def get_fama_french_factors() -> pd.DataFrame:
        """
        Load Fama-French factors (market, size, value, momentum).
        """
        # In practice, use Ken French's data library
        # or commercial provider like Quandl
    
        # Minimum factors to test against:
        # - Market (RMRF)
        # - Size (SMB)
        # - Value (HML)
        # - Momentum (UMD)
        # - Quality (QMJ)
        # - Low Volatility (BAB)
        pass
    
    def decompose_strategy_returns(
        strategy_returns: pd.Series,
        factors: pd.DataFrame
    ) -> dict:
        """
        Decompose strategy returns into factor and alpha components.
    
        This tells you if you have real alpha or just smart beta.
        """
        analysis = calculate_factor_exposures(strategy_returns, factors)
    
        # Calculate factor contribution
        factor_contrib = {}
        for factor, exp in analysis['factor_exposures'].items():
            if exp['significant']:
                # Annual contribution = beta * factor return
                factor_contrib[factor] = exp['beta'] * factors[factor].mean() * 252
    
        total_return = strategy_returns.mean() * 252
        factor_return = sum(factor_contrib.values())
        true_alpha = total_return - factor_return
    
        return {
            'total_return': total_return,
            'factor_contributions': factor_contrib,
            'total_factor_return': factor_return,
            'true_alpha': true_alpha,
            'alpha_pct_of_return': true_alpha / total_return if total_return != 0 else 0,
            'is_true_alpha': analysis['alpha_significant'] and true_alpha > 0.02
        }

Factor Model Reality Check

Success Rate

Only ~5% of strategies have true alpha after factor adjustment

---

Name

Regime Detection

Description

Identify market regimes for adaptive strategy allocation

Detection

regime|market.state|trend.ranging|volatility.*cluster

Guidance

Regime Detection Framework

Markets have states. Strategies that work in one regime fail in another.

Hidden Markov Model Regime Detection

    import numpy as np
    import pandas as pd
    from hmmlearn import hmm
    
    def detect_regimes_hmm(
        returns: pd.Series,
        n_regimes: int = 3
    ) -> pd.DataFrame:
        """
        Detect market regimes using Hidden Markov Model.
    
        Common regimes:
        - Bull (low vol, positive returns)
        - Bear (high vol, negative returns)
        - Chop (high vol, no trend)
        """
        # Prepare features
        features = pd.DataFrame({
            'returns': returns,
            'volatility': returns.rolling(20).std() * np.sqrt(252),
            'momentum': returns.rolling(60).mean() * 252
        }).dropna()
    
        # Fit HMM
        model = hmm.GaussianHMM(
            n_components=n_regimes,
            covariance_type='full',
            n_iter=1000,
            random_state=42
        )
    
        X = features.values
        model.fit(X)
    
        # Predict regimes
        regimes = model.predict(X)
    
        # Analyze each regime
        results = pd.DataFrame(index=features.index)
        results['regime'] = regimes
    
        regime_stats = {}
        for r in range(n_regimes):
            mask = results['regime'] == r
            regime_stats[r] = {
                'avg_return': features.loc[mask, 'returns'].mean() * 252,
                'volatility': features.loc[mask, 'volatility'].mean(),
                'pct_time': mask.mean(),
                'count': mask.sum()
            }
    
        # Label regimes by characteristics
        regime_labels = {}
        for r, stats in regime_stats.items():
            if stats['avg_return'] > 0.1 and stats['volatility'] < 0.2:
                regime_labels[r] = 'bull'
            elif stats['avg_return'] < -0.1:
                regime_labels[r] = 'bear'
            else:
                regime_labels[r] = 'chop'
    
        results['regime_label'] = results['regime'].map(regime_labels)
    
        return results, regime_stats
    
    def regime_aware_allocation(
        regimes: pd.Series,
        strategy_returns: Dict[str, pd.Series]
    ) -> pd.DataFrame:
        """
        Allocate to strategies based on detected regime.
        """
        # Analyze strategy performance by regime
        performance_by_regime = {}
    
        for strategy, returns in strategy_returns.items():
            aligned = pd.concat([returns, regimes], axis=1).dropna()
    
            by_regime = aligned.groupby('regime_label')['returns'].agg([
                ('sharpe', lambda x: x.mean() / x.std() * np.sqrt(252)),
                ('avg_return', lambda x: x.mean() * 252)
            ])
    
            performance_by_regime[strategy] = by_regime
    
        # Create optimal allocation per regime
        allocations = {}
        for regime in regimes.unique():
            if pd.isna(regime):
                continue
    
            best_strategies = sorted(
                strategy_returns.keys(),
                key=lambda s: performance_by_regime[s].loc[regime, 'sharpe'],
                reverse=True
            )
    
            allocations[regime] = {
                'primary': best_strategies[0],
                'secondary': best_strategies[1] if len(best_strategies) > 1 else None,
                'allocation': [0.6, 0.4] if len(best_strategies) > 1 else [1.0]
            }
    
        return allocations

Simple Regime Indicators

    def simple_regime_detection(prices: pd.Series) -> pd.Series:
        """
        Simple regime detection without ML.
    
        Sometimes simple beats complex.
        """
        sma_50 = prices.rolling(50).mean()
        sma_200 = prices.rolling(200).mean()
        volatility = prices.pct_change().rolling(20).std() * np.sqrt(252)
        avg_vol = volatility.rolling(252).mean()
    
        regime = pd.Series(index=prices.index, dtype=str)
    
        # Bull: Price > 50 SMA > 200 SMA, low vol
        bull_mask = (prices > sma_50) & (sma_50 > sma_200) & (volatility < avg_vol)
        regime[bull_mask] = 'bull'
    
        # Bear: Price < 50 SMA < 200 SMA
        bear_mask = (prices < sma_50) & (sma_50 < sma_200)
        regime[bear_mask] = 'bear'
    
        # Chop: Everything else
        regime[regime.isna()] = 'chop'
    
        return regime

Success Rate

Regime-aware strategies reduce drawdowns 30-40% historically

---

Name

Transaction Cost Modeling

Description

Accurate cost modeling - the silent killer of backtests

Detection

transaction.cost|slippage|commission|market.impact

Guidance

Transaction Cost Reality

The gap between backtest and live is usually costs.

Complete Cost Model

    import numpy as np
    import pandas as pd
    
    def estimate_transaction_costs(
        trade_value: float,
        trade_side: str,  # 'buy' or 'sell'
        asset_class: str,
        avg_daily_volume: float,
        volatility: float,
        is_market_order: bool = True
    ) -> dict:
        """
        Estimate total transaction costs for a trade.
        """
        # 1. Commission (usually smallest component)
        commission_rates = {
            'us_equity': 0.0005,  # $0.50 per $1000
            'crypto': 0.001,      # 10 bps
            'futures': 0.00001,   # Negligible
            'forex': 0.00002      # 0.2 bps
        }
        commission = trade_value * commission_rates.get(asset_class, 0.001)
    
        # 2. Spread cost (for market orders)
        spread_estimates = {
            'us_equity': 0.0003,  # 3 bps for liquid
            'crypto': 0.001,      # 10 bps
            'futures': 0.0001,    # 1 bp
            'forex': 0.0001       # 1 bp
        }
        spread = spread_estimates.get(asset_class, 0.001)
        if not is_market_order:
            spread = 0  # Limit orders avoid spread
        spread_cost = trade_value * spread
    
        # 3. Market impact (the killer)
        # Square root model: impact = σ * sqrt(V_trade / V_daily)
        participation_rate = trade_value / (avg_daily_volume * 1)  # Assuming 1 day to execute
        market_impact = volatility * np.sqrt(participation_rate) * trade_value
    
        # 4. Timing risk (for multi-day execution)
        # Price might move while you're executing
        execution_days = max(1, trade_value / (avg_daily_volume * 0.1))  # 10% participation
        timing_risk = volatility * np.sqrt(execution_days / 252) * trade_value
    
        total_cost = commission + spread_cost + market_impact
    
        return {
            'commission': commission,
            'spread': spread_cost,
            'market_impact': market_impact,
            'timing_risk': timing_risk,
            'total_cost': total_cost,
            'cost_bps': (total_cost / trade_value) * 10000,
            'execution_days': execution_days,
            'participation_rate': participation_rate
        }
    
    def adjust_backtest_for_costs(
        returns: pd.Series,
        trades: pd.DataFrame,  # Must have 'value', 'side' columns
        asset_info: dict
    ) -> pd.Series:
        """
        Apply realistic transaction costs to backtest returns.
        """
        costs = []
    
        for idx, trade in trades.iterrows():
            cost = estimate_transaction_costs(
                trade_value=trade['value'],
                trade_side=trade['side'],
                asset_class=asset_info['class'],
                avg_daily_volume=asset_info['adv'],
                volatility=asset_info['volatility']
            )
            costs.append({
                'date': idx,
                'cost': cost['total_cost'],
                'cost_bps': cost['cost_bps']
            })
    
        costs_df = pd.DataFrame(costs).set_index('date')
    
        # Subtract costs from returns
        adjusted_returns = returns.copy()
        for idx, row in costs_df.iterrows():
            if idx in adjusted_returns.index:
                adjusted_returns[idx] -= row['cost']
    
        return adjusted_returns

Cost Reality Table

A strategy turning over 12x/year (monthly) in crypto:

• 12 2 (buy+sell) 80 bps = 19.2% annual cost drag!

Success Rate

Realistic cost modeling improves backtest-to-live correlation from 30% to 70%

Anti-Patterns

---

Name

In-Sample Optimization

Description

Optimizing parameters on the same data you validate on

Detection

optimize|best.param|grid.search

Why Harmful

You're fitting to noise. Your 3.0 Sharpe is measuring how well you memorized the data, not how well you'll predict the future.

What To Do

Always split: train on 70%, validate on 30%. Better yet, use walk-forward analysis where you repeatedly train/test on rolling windows. If you optimize parameters, you MUST have a separate holdout set you never touch until final validation.

---

Name

Survivorship Bias

Description

Testing only on assets that still exist today

Detection

download|yfinance|historical

Why Harmful

You're only seeing winners. The stocks that went bankrupt, got delisted, or merged aren't in your data. This inflates returns by 1-2% annually.

What To Do

Use a data provider with point-in-time constituents and delisted securities. If testing on crypto, include dead coins. Minimum: understand which assets are missing from your universe.

---

Name

Look-Ahead Bias

Description

Using information that wasn't available at trade time

Detection

future|next|forward

Why Harmful

The most dangerous bug. Using Friday's data on Thursday's trades shows perfect results but zero real edge. Often hidden in data alignment or indicator calculations.

What To Do

Audit every data point: "Could I have known this at trade time?" Shift data properly. When in doubt, add extra lag. Your backtest should simulate having LESS information than reality.

---

Name

Ignoring Capacity

Description

Assuming you can trade unlimited size at backtest prices

Detection

position|trade|execute

Why Harmful

A strategy that works at $1M might fail at $10M. Market impact is nonlinear. Your 2 Sharpe at small size becomes 0.5 Sharpe at scale.

What To Do

Model market impact explicitly. Cap position size at % of ADV. Understand your strategy's capacity BEFORE deploying capital. If capacity is $1M, don't raise $100M.

---

Name

Data Mining Multiple Hypotheses

Description

Testing hundreds of strategies and reporting the best one

Detection

test.strategy|multiple.backtest|parameter.*sweep

Why Harmful

If you test 100 strategies, 5 will look good by chance at 5% significance. This is NOT the same as finding 5 real edges. It's multiple hypothesis testing.

What To Do

Apply Bonferroni correction: divide significance threshold by number of tests. Or use FDR control. Track ALL strategies you tested, not just winners. Pre-register hypotheses.

---

Name

Short Backtest Period

Description

Validating on 2-3 years of data

Detection

2.year|3.year|recent

Why Harmful

2-3 years might be one market regime. Your momentum strategy "works" because the entire period was trending. You have no idea how it performs in a range or crisis.

What To Do

Minimum 10 years of data, ideally 20+. MUST include: 2008 crisis, 2020 COVID crash, 2022 bear market. If your strategy can't be tested on long history, increase skepticism.

Quantitative Research - Sharp Edges

Look-Ahead Bias Is Almost Always Present

Id

look-ahead-bias-hidden

Severity

CRITICAL

Description

If your backtest looks too good, you're probably using future data

Symptoms

• Sharpe ratio above 3
• Perfect entry/exit timing
• Backtest crushes reality

Detection Pattern

backtest.sharpe.[3-9]|perfect.timing|too.good

Solution

Common Sources of Look-Ahead Bias:

1. Index Membership

• Testing "S&P 500 stocks" but using TODAY's membership
• Stocks were added BECAUSE they performed well
• Fix: Use point-in-time constituents

2. Earnings Dates

• Using reported earnings date, not announcement date
• Fix: Add 1-day lag minimum to all fundamental data

3. Adjusted Prices

• Some splits/dividends adjusted using future info
• Fix: Use unadjusted prices + adjust yourself

4. Data Alignment

• Daily close vs intraday signal generation
• Fix: If signal uses close, trade next open

5. ETF Creation

• Testing ETF since "inception" but it tracks older index
• Fix: Use underlying index data

Audit Checklist:

  def audit_for_lookahead(backtest_code: str):
      """Red flags to search for."""
      red_flags = [
          'shift(-',      # Shifting forward = future data
          '.iloc[-1]',    # Last row might be future
          'future',       # Obvious
          '.max()',       # Global max might use future
          '.min()',       # Global min might use future
          'full_period',  # Normalizing on full period
      ]
  
      warnings = []
      for flag in red_flags:
          if flag in backtest_code:
              warnings.append(f"POTENTIAL LOOK-AHEAD: {flag}")
  
      return warnings

References

• "Backtesting" by Lopez de Prado

You Are Definitely Overfit (Accept It)

Id

overfitting-certainty

Severity

CRITICAL

Description

Every optimization overfit to some degree - the question is how much

Symptoms

• Strategy works perfectly in sample
• Adding parameters improves results
• Complex strategy beats simple

Detection Pattern

optimize|parameter|improve.*backtest

Solution

Overfitting Is Inevitable - Minimize It:

The Haircut Rule:

• In-sample Sharpe 2.0 → Expect 1.0 live
• In-sample Sharpe 3.0 → Expect 1.0-1.5 live
• In-sample Sharpe 5.0+ → Expect 0.5 or negative live

Degrees of Freedom Test:

  def degrees_of_freedom_test(
      strategy_params: int,
      backtest_years: float,
      trade_frequency: str
  ) -> dict:
      """
      Rule of thumb: Need 20 observations per parameter.
      """
      trades_per_year = {
          'daily': 252,
          'weekly': 52,
          'monthly': 12,
          'quarterly': 4
      }
  
      total_trades = backtest_years * trades_per_year.get(trade_frequency, 52)
      observations_per_param = total_trades / strategy_params
  
      return {
          'parameters': strategy_params,
          'total_trades': total_trades,
          'obs_per_param': observations_per_param,
          'likely_overfit': observations_per_param < 20,
          'recommendation': (
              "Reduce parameters" if observations_per_param < 20
              else "Adequate data"
          )
      }
  
  # Example: 5 parameters, 3 years, daily trading
  # = 756 trades / 5 params = 151 per param
  # This is OK
  
  # Example: 20 parameters, 3 years, monthly trading
  # = 36 trades / 20 params = 1.8 per param
  # SEVERELY OVERFIT

The Simplicity Test:

• Can you explain the strategy in one sentence?
• Would it have made sense in 1990?
• Does it exploit human behavior?

If not → probably overfit to data artifact.

References

• "The Probability of Backtest Overfitting" - Bailey & Lopez de Prado

You've Already Run 100 Tests (You Just Don't Know It)

Id

multiple-testing-trap

Severity

CRITICAL

Description

Every informal check is a hypothesis test - adjust accordingly

Symptoms

• "I only tested 3 strategies"
• "This is my first parameter set"
• Ignoring rejected hypotheses

Detection Pattern

test.strat|try.approach|param.*search

Solution

Hidden Multiple Testing:

Things That Count As Tests:

• Looking at a chart → Test
• Checking correlation → Test
• "What if I tried..." → Test
• Rejected strategies → Tests
• Parameter grid search → N tests

If you test 100 things at 5% significance:

• Expected false positives: 5
• Your "best" strategy might be pure noise

Corrections:

  import numpy as np
  from scipy import stats
  
  def bonferroni_correction(p_values: list, alpha: float = 0.05) -> dict:
      """
      Simple but conservative correction.
      """
      n_tests = len(p_values)
      adjusted_alpha = alpha / n_tests
  
      significant = [p < adjusted_alpha for p in p_values]
  
      return {
          'original_alpha': alpha,
          'adjusted_alpha': adjusted_alpha,
          'n_tests': n_tests,
          'significant': significant,
          'n_significant': sum(significant)
      }
  
  def benjamini_hochberg(p_values: list, alpha: float = 0.05) -> dict:
      """
      Less conservative FDR control.
      """
      n = len(p_values)
      sorted_pvals = sorted(enumerate(p_values), key=lambda x: x[1])
  
      significant = [False] * n
  
      for i, (original_idx, p) in enumerate(sorted_pvals):
          threshold = (i + 1) / n * alpha
          if p <= threshold:
              significant[original_idx] = True
          else:
              break  # Stop at first failure
  
      return {
          'alpha': alpha,
          'significant': significant,
          'n_significant': sum(significant)
      }
  
  # Example: 50 strategy variations tested
  # Best p-value: 0.01
  #
  # Unadjusted: "Significant at 5%!"
  # Bonferroni: 0.01 > 0.05/50 = 0.001, NOT significant
  # Reality: You got lucky

Best Practice:

• Track ALL hypotheses tested
• Pre-register primary hypothesis
• Apply correction before claiming significance

References

• "False Strategy Discovery" - Harvey et al.

Your Backtest Period Is One Regime

Id

regime-blindness

Severity

HIGH

Description

Strategies that work in bull markets fail in bear markets and vice versa

Symptoms

• Great 2010-2020 backtest (bull market)
• Momentum works, then doesn't
• "This time is different"

Detection Pattern

2010.2020|last.5.year|recent.history

Solution

Regime Dependency Reality:

Strategy Performance by Regime:

If your backtest is mostly bull market:

• Momentum will look great
• Vol selling will look great
• Quality will look boring

This tells you nothing about reality.

Minimum Regime Coverage:

  def check_regime_coverage(returns: pd.Series) -> dict:
      """
      Ensure backtest covers multiple regimes.
      """
      required_periods = {
          '2008_crisis': ('2007-10', '2009-03'),  # -50% drawdown
          '2011_eurozone': ('2011-07', '2011-10'),  # -20%
          '2015_china': ('2015-08', '2015-09'),     # Flash crash
          '2018_volmageddon': ('2018-01', '2018-02'),
          '2020_covid': ('2020-02', '2020-03'),     # -35%
          '2022_bear': ('2022-01', '2022-10'),      # -25%
      }
  
      coverage = {}
      for period_name, (start, end) in required_periods.items():
          try:
              period_returns = returns[start:end]
              coverage[period_name] = len(period_returns) > 0
          except:
              coverage[period_name] = False
  
      return {
          'periods_covered': coverage,
          'coverage_pct': sum(coverage.values()) / len(coverage),
          'is_adequate': sum(coverage.values()) >= 4  # Need most periods
      }

References

• Market regime research literature

Your Universe Has Survivorship Bias (All Of Them Do)

Id

survivorship-hidden

Severity

HIGH

Description

You're only testing on stocks that didn't go bankrupt

Symptoms

• Using current index members historically
• Free data from Yahoo/Google Finance
• Higher returns than market average

Detection Pattern

yfinance|yahoo|current.member|sp500.list

Solution

Survivorship Bias Sources:

1. Index Membership

• S&P 500 today ≠ S&P 500 in 2008
• Removed companies often dropped AFTER poor performance
• Adds 1-2% annual return artificially

2. Data Vendors

• Free: Usually survivors only
• Cheap: Mixed quality
• Expensive: Usually complete

3. Crypto

• Only testing top 100 coins
• Ignoring the thousands that went to zero
• Massive survivorship bias

Quantifying the Bias:

  def estimate_survivorship_bias(
      strategy_return: float,  # Annual return from biased test
      universe_turnover: float = 0.05  # 5% annual delisting
  ) -> dict:
      """
      Rough estimate of survivorship bias impact.
      """
      # Academic research shows 1-2% for US equities
      # Higher for small caps, crypto
  
      estimated_bias = {
          'large_cap_us': 0.01,
          'small_cap_us': 0.02,
          'international': 0.015,
          'crypto': 0.05,  # Many coins die
          'high_turnover': universe_turnover * 0.3
      }
  
      # Apply haircut
      bias = sum(estimated_bias.values()) / len(estimated_bias)
      adjusted_return = strategy_return - bias
  
      return {
          'reported_return': strategy_return,
          'estimated_bias': bias,
          'adjusted_return': adjusted_return,
          'return_reduction_pct': bias / strategy_return * 100
      }

Solutions:

• Use point-in-time databases (CRSP, FactSet, etc.)
• Include delisted returns (usually -100%)
• For crypto: manually track dead coins

References

• "Survivorship Bias in Performance Studies" - Brown et al.

Your Strategy Has $1M Capacity, Not $1B

Id

capacity-ignored

Severity

HIGH

Description

Market impact kills strategies at scale

Symptoms

• Trade large % of daily volume
• Small cap focus
• High turnover

Detection Pattern

scale|capacity|aum|volume

Solution

Capacity Estimation:

Rule of Thumb:

  def estimate_capacity(
      avg_position_adv_pct: float,  # Position as % of daily volume
      turnover_annual: float,        # Annual turnover
      target_sharpe: float = 1.0,    # Minimum acceptable Sharpe
      base_sharpe: float = 2.0       # Sharpe at small size
  ) -> dict:
      """
      Estimate strategy capacity before returns degrade.
      """
      # Market impact model: Impact = k * sqrt(participation)
      # Each doubling of size reduces Sharpe by ~15%
  
      # Maximum participation before significant impact
      max_participation = 0.01  # 1% of ADV is safe
  
      if avg_position_adv_pct > max_participation:
          size_penalty = (avg_position_adv_pct / max_participation) ** 0.5
      else:
          size_penalty = 1.0
  
      # Turnover amplifies impact
      impact_from_turnover = turnover_annual * avg_position_adv_pct * 0.001
  
      expected_sharpe = base_sharpe / size_penalty - impact_from_turnover
  
      # Solve for max size
      # This is approximate - real capacity requires simulation
  
      return {
          'current_sharpe_estimate': expected_sharpe,
          'is_viable': expected_sharpe > target_sharpe,
          'size_penalty': size_penalty,
          'recommendation': (
              "Reduce position sizes" if expected_sharpe < target_sharpe
              else "Capacity appears adequate"
          )
      }

Capacity By Strategy Type:

References

• Market microstructure literature

Your Backtest Assumes Free Trading (It's Not)

Id

transaction-cost-fantasy

Severity

HIGH

Description

Transaction costs are often 50%+ of alpha

Symptoms

• "I included 0.1% slippage"
• Ignoring market impact
• High turnover strategies

Detection Pattern

slippage|transaction|commission|cost

Solution

Real Transaction Cost Components:

  def realistic_transaction_cost(
      trade_value: float,
      asset_type: str,
      avg_daily_volume_usd: float,
      holding_period_days: float
  ) -> dict:
      """
      Realistic all-in cost estimation.
      """
      # Commission (easy part)
      commissions = {
          'us_equity': 0.0001,  # $1 per $10k
          'crypto_cex': 0.001,   # 10 bps
          'crypto_dex': 0.003,   # 30 bps (with gas)
          'futures': 0.00001     # Negligible
      }
  
      # Spread (market orders)
      spreads = {
          'us_equity': 0.0003,
          'crypto_cex': 0.001,
          'crypto_dex': 0.005,  # Variable
          'futures': 0.0001
      }
  
      # Market impact (the big one)
      participation = trade_value / avg_daily_volume_usd
  
      # Almgren-Chriss model (simplified)
      volatility = 0.02  # Assume 2% daily vol
      impact = volatility * (participation ** 0.5)
  
      # Scale by holding period (shorter = more impact matters)
      impact_scaled = impact * (20 / max(holding_period_days, 1)) ** 0.5
  
      commission = trade_value * commissions.get(asset_type, 0.001)
      spread = trade_value * spreads.get(asset_type, 0.001)
      market_impact = trade_value * impact_scaled
  
      total = commission + spread + market_impact
  
      return {
          'commission': commission,
          'spread': spread,
          'market_impact': market_impact,
          'total': total,
          'total_bps': total / trade_value * 10000,
          'impact_pct_of_total': market_impact / total * 100
      }

Reality Check Table:

Annual Cost Impact (20x turnover):

• At 5 bps: 2% annual drag
• At 50 bps: 20% annual drag
• At 100 bps: 40% annual drag

Your 2.0 Sharpe strategy with 20x turnover in crypto? Probably negative after costs.

References

• "Optimal Execution" - Almgren & Chriss

Your Data Is Wrong (It Always Is)

Id

data-quality-ignored

Severity

MEDIUM

Description

Bad data creates phantom alpha

Symptoms

• Spikes to impossible prices
• "Alpha" from data errors
• Backtests don't replicate

Detection Pattern

data.*error|spike|outlier|impossible

Solution

Data Quality Checks:

  import pandas as pd
  import numpy as np
  
  def audit_price_data(prices: pd.DataFrame) -> dict:
      """
      Check for common data errors.
      """
      issues = []
  
      # 1. Impossible returns (>50% in one day)
      returns = prices.pct_change()
      extreme = returns.abs() > 0.5
      if extreme.any().any():
          issues.append({
              'type': 'extreme_returns',
              'count': extreme.sum().sum(),
              'severity': 'HIGH'
          })
  
      # 2. Zero prices
      zeros = (prices == 0).any()
      if zeros.any():
          issues.append({
              'type': 'zero_prices',
              'columns': list(zeros[zeros].index),
              'severity': 'HIGH'
          })
  
      # 3. Duplicate timestamps
      if prices.index.duplicated().any():
          issues.append({
              'type': 'duplicate_timestamps',
              'count': prices.index.duplicated().sum(),
              'severity': 'MEDIUM'
          })
  
      # 4. Gaps (missing dates)
      expected_dates = pd.date_range(
          prices.index.min(), prices.index.max(), freq='B'
      )
      missing = len(expected_dates) - len(prices)
      if missing > len(expected_dates) * 0.1:  # >10% missing
          issues.append({
              'type': 'excessive_gaps',
              'missing_pct': missing / len(expected_dates),
              'severity': 'MEDIUM'
          })
  
      # 5. Stale prices (no change for days)
      unchanged = (returns == 0).rolling(5).sum() >= 5
      if unchanged.any().any():
          issues.append({
              'type': 'stale_prices',
              'count': unchanged.sum().sum(),
              'severity': 'LOW'
          })
  
      return {
          'issues': issues,
          'total_issues': len(issues),
          'is_clean': len(issues) == 0
      }

Common Phantom Alpha Sources:

• Dividend not adjusted → fake momentum
• Split not adjusted → impossible returns
• Delisting price wrong → survivorship alpha
• Quote vs trade prices → spread capture artifact

References

• Financial data quality research

Every Additional Parameter Is Curve Fitting

Id

curve-fitting-excuses

Severity

MEDIUM

Description

Complexity is the enemy of robustness

Symptoms

• "I need this parameter for edge cases"
• Strategy has 10+ parameters
• Parameters are suspiciously round

Detection Pattern

parameter|tune|adjust|optimize

Solution

Parameter Discipline:

The Rule of 5:

• 0-2 parameters: Probably robust
• 3-5 parameters: Acceptable with caution
• 6-10 parameters: Likely overfit
• 10+ parameters: Definitely overfit

Each Parameter Must Pass: 1. Economic rationale (why this number?) 2. Sensitivity test (results stable ±20%?) 3. Cross-asset test (works on different assets?) 4. Out-of-sample validation

Parameter Sensitivity Test:

  def parameter_sensitivity(
      strategy_func: callable,
      base_params: dict,
      data: pd.DataFrame,
      sensitivity_range: float = 0.2
  ) -> dict:
      """
      Test how sensitive results are to parameter changes.
      """
      base_result = strategy_func(data, **base_params)
      base_sharpe = calculate_sharpe(base_result)
  
      sensitivity = {}
  
      for param, value in base_params.items():
          if not isinstance(value, (int, float)):
              continue
  
          # Test +20% and -20%
          results = []
          for mult in [1 - sensitivity_range, 1 + sensitivity_range]:
              test_params = base_params.copy()
              test_params[param] = value * mult
  
              result = strategy_func(data, **test_params)
              sharpe = calculate_sharpe(result)
              results.append(sharpe)
  
          sensitivity[param] = {
              'base_sharpe': base_sharpe,
              'low_sharpe': results[0],
              'high_sharpe': results[1],
              'range': max(results) - min(results),
              'is_stable': (max(results) - min(results)) / base_sharpe < 0.2
          }
  
      return {
          'parameter_sensitivity': sensitivity,
          'unstable_params': [p for p, v in sensitivity.items() if not v['is_stable']],
          'overall_stability': all(v['is_stable'] for v in sensitivity.values())
      }

If parameters are round numbers (10, 20, 50):

• They're arbitrary
• Test 11, 19, 53 - similar results?
• If not, you're overfit

References

• "Strategy Parameter Optimization" literature

Machine Learning Is Not Magic Alpha

Id

ml-false-hope

Severity

MEDIUM

Description

ML usually just learns to overfit more efficiently

Symptoms

• Neural network for prediction
• "AI trading strategy"
• Training accuracy 90%+

Detection Pattern

neural|machine.learning|deep.learning|ai.*trad

Solution

ML in Finance Reality:

Why ML Usually Fails: 1. Low signal-to-noise ratio (~0.01 IC) 2. Non-stationarity (regimes change) 3. Data is limited (not millions of samples) 4. Features are highly correlated 5. Easy to overfit with many parameters

If You Must Use ML:

  def ml_sanity_checks(model, train_data, test_data):
      """
      Minimum checks before believing ML results.
      """
      checks = {}
  
      # 1. Train/test performance gap
      train_score = model.score(train_data)
      test_score = model.score(test_data)
      checks['overfit_ratio'] = train_score / max(test_score, 0.01)
      checks['is_overfit'] = checks['overfit_ratio'] > 2
  
      # 2. Feature importance
      # If top features are random-looking, model is noise
      importances = model.feature_importances_
      checks['top_feature_importance'] = max(importances)
      checks['concentrated'] = max(importances) > 0.3
  
      # 3. Time consistency
      # Split test set into halves - both should work
      mid = len(test_data) // 2
      first_half = model.score(test_data[:mid])
      second_half = model.score(test_data[mid:])
      checks['time_stable'] = abs(first_half - second_half) < 0.1
  
      # 4. Random baseline
      # Beat a random model by significant margin?
      random_score = 0.5  # Random = 50%
      checks['beats_random'] = test_score > random_score + 0.05
  
      return {
          'checks': checks,
          'red_flags': [k for k, v in checks.items()
                       if v in [True, False] and v == True]
      }

What Works in ML for Finance:

• Simple models (linear, tree-based)
• Regularization (L1/L2, dropout)
• Ensemble methods
• Feature selection before training
• Walk-forward validation

What Doesn't Work:

• Deep learning on price data
• Complex architectures
• End-to-end learning
• Training on 3 years of daily data

References

• "Machine Learning for Asset Managers" - Lopez de Prado

Quantitative Research - Validations

Train/Test Split Required

Id

check-train-test-split

Description

Backtests must separate training and testing data

Pattern

train|test|split|walk.*forward

File Glob

*/.{py,ipynb}

Match

present

Context Pattern

backtest|simulate|strategy

Message

Backtests require train/test split to avoid overfitting

Severity

error

Autofix

Transaction Costs Included

Id

check-transaction-costs

Description

Backtests should include realistic transaction costs

Pattern

backtest|simulate|return

File Glob

*/.{py,ipynb}

Match

present

Context Pattern

cost|slippage|commission|fee

Message

Include transaction costs in backtest - they often exceed alpha

Severity

error

Autofix

Sharpe Ratio Sanity Check

Id

check-sharpe-sanity

Description

Sharpe ratios above 3 are almost certainly overfit

Pattern

sharpe.[4-9]\.|sharpe.[1-9]\d

File Glob

*/.{py,ipynb}

Match

present

Message

Sharpe ratio appears unrealistically high - check for overfitting

Severity

warning

Autofix

Look-Ahead Bias Check

Id

check-lookahead-shift

Description

Forward shifts on data indicate potential look-ahead bias

Pattern

shift\(-|\[:-

File Glob

*/.{py,ipynb}

Match

present

Message

Negative shift detected - verify no look-ahead bias

Severity

warning

Autofix

Minimum Sample Size

Id

check-sample-size

Description

Strategies need sufficient trades for statistical significance

Pattern

n_trades|num_trades|trade_count

File Glob

*/.{py,ipynb}

Match

absent

Message

Track number of trades - need 30+ for statistical significance

Severity

warning

Autofix

Statistical Significance Testing

Id

check-statistical-significance

Description

Results should include significance tests

Pattern

t_stat|p_value|confidence|significance

File Glob

*/.{py,ipynb}

Match

absent

Context Pattern

backtest|strategy|alpha

Message

Include statistical significance testing for strategy validation

Severity

warning

Autofix

Out-of-Sample Testing

Id

check-out-of-sample

Description

Strategies must be validated on out-of-sample data

Pattern

out.sample|oos|holdout|test.set

File Glob

*/.{py,ipynb}

Match

absent

Context Pattern

backtest|validate|strategy

Message

Validate strategy on out-of-sample data before deployment

Severity

warning

Autofix

Survivorship Bias Awareness

Id

check-survivorship-bias

Description

Strategy should account for delisted/dead assets

Pattern

delist|survivor|dead.*asset|bankrupt

File Glob

*/.{py,ipynb}

Match

absent

Context Pattern

universe|constituent|index

Message

Consider survivorship bias if using historical universe data

Severity

info

Autofix

Multiple Regime Testing

Id

check-regime-testing

Description

Strategies should be tested across market regimes

Pattern

regime|bull|bear|volatility.regime|market.state

File Glob

*/.{py,ipynb}

Match

absent

Context Pattern

backtest|strategy|performance

Message

Test strategy across different market regimes (bull, bear, sideways)

Severity

info

Autofix

Parameter Complexity Check

Id

check-parameter-count

Description

Too many parameters indicates potential overfitting

Pattern

param|hyperparameter|tune|optimize

File Glob

*/.{py,ipynb}

Match

present

Message

Keep parameters minimal (< 5) to reduce overfitting risk

Severity

info

Autofix