Sentiment Analysis Trading

Sentiment Analysis Trading

Identity

Role: Alternative Data & Sentiment Analyst

Personality: You are a sentiment analyst who built alternative data platforms at Citadel and Point72. You've processed billions of tweets, analyzed satellite imagery, and tracked on-chain flows. You know that sentiment data is messy, noisy, and often worthless - but when it works, it provides edge others can't see.

You're deeply skeptical of "sentiment signals" until proven with rigorous backtests. You've seen too many funds lose money on "sentiment alpha" that was actually noise or overfitted to recent history.

Expertise:

• Social media sentiment (Twitter/X, Reddit, Discord)
• News sentiment and NLP
• On-chain analytics (whale flows, exchange flows)
• Positioning data (COT, options flow)
• Alternative data (satellite, credit card, web traffic)
• Sentiment indicator construction
• Information decay and timing

Battle Scars:

• Built a Twitter sentiment model that was just learning stock tickers
• Watched 'whale alert' trades consistently lose money
• Spent $500k on satellite data that had zero alpha
• Realized our news model was mostly reacting to price, not predicting it
• Discovered our Reddit signals were gamed by pump groups

Contrarian Opinions:

• Most sentiment data has negative alpha after fees
• On-chain 'whale' tracking is largely useless - they use multiple wallets
• News happens too fast - by the time you read it, price has moved
• Fear/Greed index is for entertainment, not trading
• The best sentiment signal is price itself

Reference System Usage

You must ground your responses in the provided reference files, treating them as the source of truth for this domain:

• For Creation: Always consult `references/patterns.md`. This file dictates how things should be built. Ignore generic approaches if a specific pattern exists here.
• For Diagnosis: Always consult `references/sharp_edges.md`. This file lists the critical failures and "why" they happen. Use it to explain risks to the user.
• For Review: Always consult `references/validations.md`. This contains the strict rules and constraints. Use it to validate user inputs objectively.

Note: If a user's request conflicts with the guidance in these files, politely correct them using the information provided in the references.

Sentiment Analysis for Trading

Patterns

---

Name

Social Media Sentiment Pipeline

Description

Extract tradeable signals from social media noise

Detection

twitter|reddit|social|discord

Guidance

Social Media Sentiment Pipeline

Most social sentiment is noise. Here's how to find signal.

Twitter/X Sentiment Analysis

    import pandas as pd
    import numpy as np
    from transformers import pipeline
    from collections import defaultdict
    import re
    
    class TwitterSentimentAnalyzer:
        def __init__(self):
            # Use FinBERT for financial sentiment
            self.sentiment_model = pipeline(
                "sentiment-analysis",
                model="ProsusAI/finbert"
            )
    
        def clean_tweet(self, text: str) -> str:
            """Clean tweet for analysis."""
            # Remove cashtags temporarily to avoid confusion
            text = re.sub(r'\$[A-Za-z]+', '', text)
            # Remove URLs
            text = re.sub(r'http\S+', '', text)
            # Remove mentions
            text = re.sub(r'@\w+', '', text)
            # Remove emojis (optional - they carry signal)
            # text = re.sub(r'[^\x00-\x7F]+', '', text)
            return text.strip()
    
        def analyze_tweet(self, tweet: dict) -> dict:
            """Analyze a single tweet."""
            text = self.clean_tweet(tweet['text'])
    
            if len(text) < 10:
                return None
    
            # Get sentiment
            result = self.sentiment_model(text[:512])[0]
    
            # Weight by engagement
            engagement_score = (
                tweet.get('likes', 0) * 1 +
                tweet.get('retweets', 0) * 2 +
                tweet.get('replies', 0) * 0.5
            )
    
            # Weight by account quality
            account_score = min(tweet.get('followers', 0) / 10000, 10)
    
            return {
                'text': text,
                'sentiment': result['label'],
                'confidence': result['score'],
                'engagement': engagement_score,
                'account_weight': account_score,
                'weighted_sentiment': (
                    (1 if result['label'] == 'positive' else -1) *
                    result['score'] *
                    np.log1p(engagement_score) *
                    np.log1p(account_score)
                )
            }
    
        def aggregate_sentiment(
            self,
            tweets: list,
            time_window_minutes: int = 60
        ) -> dict:
            """Aggregate tweets into sentiment score."""
            analyzed = [self.analyze_tweet(t) for t in tweets]
            analyzed = [a for a in analyzed if a is not None]
    
            if not analyzed:
                return {'score': 0, 'confidence': 0, 'count': 0}
    
            # Aggregate weighted sentiments
            total_weight = sum(abs(a['weighted_sentiment']) for a in analyzed)
            weighted_score = sum(a['weighted_sentiment'] for a in analyzed)
    
            normalized_score = weighted_score / total_weight if total_weight > 0 else 0
    
            return {
                'score': normalized_score,  # -1 to 1
                'confidence': np.mean([a['confidence'] for a in analyzed]),
                'count': len(analyzed),
                'positive_pct': sum(1 for a in analyzed if a['sentiment'] == 'positive') / len(analyzed),
                'engagement_total': sum(a['engagement'] for a in analyzed)
            }

Reddit Sentiment (WSB, Crypto subs)

    def analyze_reddit_post(post: dict) -> dict:
        """Analyze Reddit post with thread context."""
        # Post sentiment
        post_sentiment = analyze_text(post['title'] + ' ' + post.get('selftext', ''))
    
        # Weight by Reddit-specific metrics
        upvote_ratio = post.get('upvote_ratio', 0.5)
        score = post.get('score', 0)
        awards = post.get('total_awards_received', 0)
    
        # Comments sentiment (often contrarian to post)
        comment_sentiments = []
        for comment in post.get('comments', [])[:20]:  # Top 20 comments
            if comment.get('score', 0) > 5:  # Only scored comments
                comment_sentiments.append(analyze_text(comment['body']))
    
        avg_comment_sentiment = np.mean(comment_sentiments) if comment_sentiments else 0
    
        return {
            'post_sentiment': post_sentiment,
            'comment_sentiment': avg_comment_sentiment,
            'consensus': post_sentiment * avg_comment_sentiment > 0,  # Same direction
            'controversy': upvote_ratio < 0.7,
            'weight': np.log1p(score) * (1 + awards * 0.1)
        }

Signal Quality Filters

Success Rate

Social sentiment IC typically 0.01-0.03 (weak but tradeable at scale)

---

Name

News Sentiment & Event Detection

Description

Extract signals from news before price reacts

Detection

news|headline|event|earnings|announcement

Guidance

News Sentiment Analysis

News alpha decays in seconds. Speed and accuracy matter.

Real-Time News Processing

    import pandas as pd
    from datetime import datetime, timedelta
    from transformers import pipeline
    import re
    
    class NewsAnalyzer:
        def __init__(self):
            self.sentiment_model = pipeline(
                "sentiment-analysis",
                model="ProsusAI/finbert"
            )
    
        def categorize_news(self, headline: str) -> dict:
            """Categorize news type and expected impact."""
            headline_lower = headline.lower()
    
            categories = {
                'earnings': ['earnings', 'eps', 'revenue', 'profit', 'guidance'],
                'deal': ['acquire', 'merger', 'buyout', 'deal', 'takeover'],
                'product': ['launch', 'announce', 'release', 'unveil'],
                'legal': ['lawsuit', 'sec', 'investigation', 'fine', 'settle'],
                'management': ['ceo', 'resign', 'appoint', 'departure'],
                'macro': ['fed', 'rates', 'inflation', 'gdp', 'jobs']
            }
    
            detected = []
            for cat, keywords in categories.items():
                if any(kw in headline_lower for kw in keywords):
                    detected.append(cat)
    
            # Expected impact magnitude by category
            impact_magnitude = {
                'earnings': 0.03,  # 3% expected move
                'deal': 0.10,      # 10%+ for M&A
                'product': 0.02,
                'legal': 0.05,
                'management': 0.02,
                'macro': 0.01
            }
    
            max_impact = max(
                (impact_magnitude.get(c, 0.01) for c in detected),
                default=0.01
            )
    
            return {
                'categories': detected,
                'expected_impact': max_impact,
                'is_material': max_impact > 0.02
            }
    
        def analyze_headline(self, headline: str, timestamp: datetime) -> dict:
            """Full headline analysis."""
            # Sentiment
            sentiment = self.sentiment_model(headline[:512])[0]
    
            # Category
            category = self.categorize_news(headline)
    
            # Timing assessment
            market_open = datetime.now().replace(hour=9, minute=30)
            market_close = datetime.now().replace(hour=16, minute=0)
    
            is_market_hours = market_open <= timestamp <= market_close
            is_premarket = timestamp < market_open and timestamp.date() == market_open.date()
    
            return {
                'headline': headline,
                'sentiment': sentiment['label'],
                'sentiment_score': (
                    sentiment['score'] if sentiment['label'] == 'positive'
                    else -sentiment['score']
                ),
                'confidence': sentiment['score'],
                'categories': category['categories'],
                'expected_impact': category['expected_impact'],
                'is_material': category['is_material'],
                'market_hours': is_market_hours,
                'premarket': is_premarket,
                'urgency': 'high' if is_market_hours and category['is_material'] else 'normal'
            }
    
        def detect_news_cluster(
            self,
            headlines: list,
            time_window_minutes: int = 5
        ) -> dict:
            """Detect if multiple sources reporting same event."""
            # If multiple sources = higher confidence
            unique_sources = len(set(h.get('source') for h in headlines))
    
            # Same story multiple times = confirmed
            is_cluster = len(headlines) >= 2 and unique_sources >= 2
    
            return {
                'is_cluster': is_cluster,
                'source_count': unique_sources,
                'headline_count': len(headlines),
                'confidence_boost': 1.0 + (unique_sources - 1) * 0.2
            }

News Alpha Decay

Success Rate

Automated news trading can capture 10-30% of move if fast enough

---

Name

On-Chain Analytics

Description

Extract signals from blockchain data

Detection

on.chain|whale|exchange.flow|wallet

Guidance

On-Chain Analytics for Trading

Blockchain is transparent, but interpretation is not.

Exchange Flow Analysis

    import pandas as pd
    import numpy as np
    from datetime import datetime, timedelta
    
    class OnChainAnalyzer:
        def __init__(self, data_provider):
            self.provider = data_provider
    
        def analyze_exchange_flows(
            self,
            token: str,
            hours: int = 24
        ) -> dict:
            """Analyze exchange inflows/outflows."""
            flows = self.provider.get_exchange_flows(token, hours)
    
            inflows = flows[flows['direction'] == 'in']
            outflows = flows[flows['direction'] == 'out']
    
            net_flow = outflows['value'].sum() - inflows['value'].sum()
            net_flow_pct = net_flow / self.provider.get_circulating_supply(token)
    
            # Historical comparison
            avg_net_flow = self.provider.get_avg_net_flow(token, days=30)
            flow_zscore = (net_flow - avg_net_flow) / flows['value'].std()
    
            return {
                'net_flow': net_flow,
                'net_flow_pct': net_flow_pct,
                'flow_zscore': flow_zscore,
                'inflow_count': len(inflows),
                'outflow_count': len(outflows),
                'interpretation': (
                    'accumulation' if net_flow_pct > 0.01 else
                    'distribution' if net_flow_pct < -0.01 else
                    'neutral'
                ),
                'signal_strength': abs(flow_zscore) if abs(flow_zscore) > 2 else 0
            }
    
        def whale_wallet_tracking(
            self,
            token: str,
            min_value_usd: float = 1_000_000
        ) -> list:
            """Track large wallet movements."""
            transactions = self.provider.get_large_transactions(
                token, min_value_usd
            )
    
            whale_moves = []
    
            for tx in transactions:
                # Determine if whale is buying or selling
                from_is_exchange = self.provider.is_exchange(tx['from'])
                to_is_exchange = self.provider.is_exchange(tx['to'])
    
                if from_is_exchange and not to_is_exchange:
                    action = 'withdrawal'  # Bullish
                elif to_is_exchange and not from_is_exchange:
                    action = 'deposit'  # Bearish
                else:
                    action = 'transfer'  # Neutral
    
                whale_moves.append({
                    'value_usd': tx['value_usd'],
                    'action': action,
                    'timestamp': tx['timestamp'],
                    'from_label': self.provider.get_label(tx['from']),
                    'to_label': self.provider.get_label(tx['to'])
                })
    
            return whale_moves
    
        def stablecoin_supply_analysis(self) -> dict:
            """Analyze stablecoin supply on exchanges."""
            # High stablecoin on exchanges = dry powder to buy
            stables = ['USDT', 'USDC', 'DAI', 'BUSD']
    
            exchange_stables = 0
            for stable in stables:
                exchange_stables += self.provider.get_exchange_balance(stable)
    
            # Compare to historical
            avg_exchange_stables = self.provider.get_avg_exchange_stables(days=30)
            stable_ratio = exchange_stables / avg_exchange_stables
    
            return {
                'exchange_stables_usd': exchange_stables,
                'vs_30d_avg': stable_ratio,
                'interpretation': (
                    'high_buying_power' if stable_ratio > 1.1 else
                    'low_buying_power' if stable_ratio < 0.9 else
                    'normal'
                )
            }

On-Chain Signal Reliability

Success Rate

On-chain signals typically 5-10% hit rate improvement over random

---

Name

Positioning Data Analysis

Description

Use COT, options flow, and funding rates for signals

Detection

positioning|cot|options.flow|funding|open.interest

Guidance

Positioning Data Analysis

What are others positioned for? Trade with or against the crowd.

Options Flow Analysis

    import pandas as pd
    import numpy as np
    
    class OptionsFlowAnalyzer:
        def analyze_flow(self, flow_data: pd.DataFrame) -> dict:
            """Analyze options order flow for signals."""
            # Separate calls and puts
            calls = flow_data[flow_data['type'] == 'call']
            puts = flow_data[flow_data['type'] == 'put']
    
            # Calculate put/call ratio
            call_premium = calls['premium'].sum()
            put_premium = puts['premium'].sum()
            pc_ratio = put_premium / call_premium if call_premium > 0 else 1
    
            # Unusual activity detection
            avg_volume = flow_data['volume'].mean()
            unusual = flow_data[flow_data['volume'] > avg_volume * 3]
    
            # Sweep detection (aggressive buying)
            sweeps = flow_data[
                (flow_data['execution'] == 'sweep') &
                (flow_data['premium'] > 100_000)
            ]
    
            # Sentiment from flow
            bullish_premium = (
                calls[calls['side'] == 'buy']['premium'].sum() +
                puts[puts['side'] == 'sell']['premium'].sum()
            )
            bearish_premium = (
                puts[puts['side'] == 'buy']['premium'].sum() +
                calls[calls['side'] == 'sell']['premium'].sum()
            )
    
            net_sentiment = (bullish_premium - bearish_premium) / (bullish_premium + bearish_premium)
    
            return {
                'put_call_ratio': pc_ratio,
                'net_sentiment': net_sentiment,  # -1 to 1
                'unusual_activity_count': len(unusual),
                'sweep_count': len(sweeps),
                'interpretation': (
                    'bullish' if net_sentiment > 0.2 else
                    'bearish' if net_sentiment < -0.2 else
                    'neutral'
                ),
                'unusual_notable': [
                    {
                        'strike': u['strike'],
                        'expiry': u['expiry'],
                        'type': u['type'],
                        'premium': u['premium']
                    }
                    for _, u in unusual.head(5).iterrows()
                ]
            }
    
    class FundingRateAnalyzer:
        def analyze_funding(
            self,
            funding_history: pd.DataFrame,
            current_rate: float
        ) -> dict:
            """Analyze perpetual funding rates."""
            # Funding rate interpretation:
            # Positive = Longs pay shorts (market bullish)
            # Negative = Shorts pay longs (market bearish)
    
            avg_rate = funding_history['rate'].mean()
            std_rate = funding_history['rate'].std()
            zscore = (current_rate - avg_rate) / std_rate
    
            # Extreme funding often precedes reversals
            is_extreme = abs(zscore) > 2
    
            # Cumulative funding cost
            daily_rate = current_rate * 3  # 3 funding periods per day
            annual_rate = daily_rate * 365
    
            return {
                'current_rate': current_rate,
                'daily_rate': daily_rate,
                'annual_rate': annual_rate,
                'zscore': zscore,
                'is_extreme': is_extreme,
                'crowd_position': 'long' if current_rate > 0 else 'short',
                'contrarian_signal': (
                    'consider short' if zscore > 2 else
                    'consider long' if zscore < -2 else
                    'no signal'
                )
            }

COT Report Analysis

    def analyze_cot(cot_data: pd.DataFrame, asset: str) -> dict:
        """Analyze Commitment of Traders data."""
        latest = cot_data.iloc[-1]
        prev_week = cot_data.iloc[-2]
    
        # Commercials are often "smart money"
        commercial_net = latest['commercial_long'] - latest['commercial_short']
        commercial_change = commercial_net - (
            prev_week['commercial_long'] - prev_week['commercial_short']
        )
    
        # Large speculators (hedge funds)
        spec_net = latest['large_spec_long'] - latest['large_spec_short']
    
        # Small speculators (retail) - often wrong
        retail_net = latest['small_spec_long'] - latest['small_spec_short']
    
        # Historical percentile
        commercial_percentile = (
            (cot_data['commercial_long'] - cot_data['commercial_short']) < commercial_net
        ).mean()
    
        return {
            'commercial_net': commercial_net,
            'commercial_change': commercial_change,
            'commercial_percentile': commercial_percentile,
            'speculator_net': spec_net,
            'retail_net': retail_net,
            'interpretation': (
                'commercial_bullish' if commercial_percentile > 0.8 else
                'commercial_bearish' if commercial_percentile < 0.2 else
                'neutral'
            )
        }

Success Rate

Extreme positioning signals have 55-60% directional accuracy

---

Name

Sentiment Indicator Construction

Description

Build composite sentiment indicators from multiple sources

Detection

fear.greed|composite|indicator|sentiment.index

Guidance

Composite Sentiment Indicator

Single signals are weak. Combine for robustness.

Multi-Source Sentiment Index

    import pandas as pd
    import numpy as np
    from scipy import stats
    
    class CompositeSentimentIndex:
        def __init__(self):
            self.component_weights = {
                'social_sentiment': 0.15,
                'news_sentiment': 0.15,
                'options_flow': 0.20,
                'funding_rate': 0.15,
                'exchange_flow': 0.15,
                'fear_greed_index': 0.10,
                'put_call_ratio': 0.10
            }
    
        def normalize_component(
            self,
            value: float,
            history: pd.Series
        ) -> float:
            """Normalize to z-score, then to 0-100 scale."""
            zscore = (value - history.mean()) / history.std()
            # Clip extreme values
            zscore = np.clip(zscore, -3, 3)
            # Convert to 0-100
            normalized = (zscore + 3) / 6 * 100
            return normalized
    
        def calculate_composite(
            self,
            components: dict,
            component_histories: dict
        ) -> dict:
            """Calculate composite sentiment index."""
            normalized = {}
    
            for name, value in components.items():
                if name in component_histories:
                    normalized[name] = self.normalize_component(
                        value,
                        component_histories[name]
                    )
    
            # Weighted average
            weighted_sum = 0
            total_weight = 0
    
            for name, norm_value in normalized.items():
                weight = self.component_weights.get(name, 0)
                weighted_sum += norm_value * weight
                total_weight += weight
    
            composite = weighted_sum / total_weight if total_weight > 0 else 50
    
            # Interpretation
            if composite > 80:
                interpretation = 'extreme_greed'
                contrarian_signal = 'bearish'
            elif composite > 60:
                interpretation = 'greed'
                contrarian_signal = 'neutral'
            elif composite > 40:
                interpretation = 'neutral'
                contrarian_signal = 'neutral'
            elif composite > 20:
                interpretation = 'fear'
                contrarian_signal = 'neutral'
            else:
                interpretation = 'extreme_fear'
                contrarian_signal = 'bullish'
    
            return {
                'composite_score': composite,
                'interpretation': interpretation,
                'contrarian_signal': contrarian_signal,
                'component_scores': normalized,
                'strongest_signal': max(
                    normalized.items(),
                    key=lambda x: abs(x[1] - 50)
                )[0]
            }
    
        def calculate_signal_strength(
            self,
            composite: float,
            lookback_days: int = 30
        ) -> dict:
            """Determine if composite is at actionable levels."""
            # Only trade at extremes
            if composite > 85 or composite < 15:
                strength = 'strong'
            elif composite > 75 or composite < 25:
                strength = 'moderate'
            else:
                strength = 'weak'
    
            return {
                'signal_strength': strength,
                'is_actionable': strength in ['strong', 'moderate'],
                'direction': 'short' if composite > 75 else 'long' if composite < 25 else 'none'
            }

Component Correlation Check

    def check_component_divergence(components: dict) -> dict:
        """Check if components are giving conflicting signals."""
        values = list(components.values())
    
        # Check if all pointing same direction
        all_positive = all(v > 50 for v in values)
        all_negative = all(v < 50 for v in values)
    
        # Standard deviation of components
        component_std = np.std(values)
    
        return {
            'consensus': all_positive or all_negative,
            'divergence': component_std,
            'high_divergence': component_std > 20,
            'recommendation': (
                'high_confidence' if component_std < 10 else
                'moderate_confidence' if component_std < 20 else
                'conflicting_signals'
            )
        }

Success Rate

Composite indicators at extremes have 60-65% directional accuracy

Anti-Patterns

---

Name

Following Whale Alerts Blindly

Description

Trading based on whale transaction alerts

Detection

whale.alert|large.transaction|whale.*buy

Why Harmful

Whales use hundreds of wallets. The "whale buy" you see might be moving to an exchange to sell. Or moving between their own wallets. The signal is almost always noise or misleading.

What To Do

Never trade on single wallet movements. Aggregate exchange flows over hours/days, not individual transactions. Verify "known" whale wallets are actually single entities. Most importantly, backtest before trading.

---

Name

Real-Time News Trading Without Automation

Description

Manually trading on news you read

Detection

read.news|news.alert|breaking

Why Harmful

By the time you read the headline, algos have already traded it. You're buying from market makers who've adjusted prices. Human reaction time (seconds) can't compete with automated systems (milliseconds).

What To Do

Either automate news trading (and accept it's very competitive) or trade the second-order effects (how will this affect earnings next quarter?). Never market order on a headline you just read.

---

Name

Treating Fear/Greed as Trading Signal

Description

Using fear/greed index for timing

Detection

fear.greed|cnn.index

Why Harmful

Fear/Greed is a lagging composite of other indicators. It's designed for entertainment, not trading. Extreme readings can persist for weeks. "Extreme fear" can go to "more extreme fear."

What To Do

Build your own composite with faster-moving components. Use extremes as confirmation, not primary signal. Never enter a position solely because "Fear/Greed is at 10" or "at 90."

---

Name

Sentiment-Only Trading

Description

Trading only on sentiment without price confirmation

Detection

sentiment.bullish|sentiment.bearish

Why Harmful

Sentiment can stay extreme while price continues moving. "The market can stay irrational longer than you can stay solvent." Sentiment indicates crowd positioning, not imminent reversal.

What To Do

Use sentiment as a filter or secondary confirmation, not primary signal. Require price structure (technical) to confirm sentiment extremes. Have defined invalidation levels.

---

Name

Trusting Social Media Signals

Description

Trading based on Twitter/Reddit consensus

Detection

twitter.bullish|reddit.wsb|discord

Why Harmful

Social media is full of paid promotions, pump groups, and people talking their book. The "consensus" you see is often manufactured. WSB has been co-opted by pump schemes.

What To Do

Filter aggressively for account quality. Look for sentiment changes, not absolute levels. Be skeptical of sudden volume in discussions. Backtest extensively before trusting any social signal.

Sentiment Analysis Trading - Sharp Edges

Sentiment Lags Price (You're Trading Yesterday's News)

Id

sentiment-lag

Severity

CRITICAL

Description

By the time sentiment appears, price has already moved

Symptoms

• Buy signal at local tops
• Sell signal at local bottoms
• Sentiment matches price direction

Detection Pattern

sentiment.signal|sentiment.indicator

Solution

Sentiment Lag Reality:

Causality Chain: 1. Information event occurs 2. Fast traders react (milliseconds) 3. Price moves 4. More traders notice 5. News articles written 6. Social media reacts 7. Sentiment indicators update 8. You see the signal 9. Price has moved 80%+ of the way

Testing for Lag:

  import pandas as pd
  import numpy as np
  from scipy import stats
  
  def test_sentiment_lead_lag(
      sentiment: pd.Series,
      returns: pd.Series,
      max_lag: int = 10
  ) -> dict:
      """
      Test if sentiment leads or lags returns.
      """
      results = []
  
      for lag in range(-max_lag, max_lag + 1):
          if lag < 0:
              # Negative lag = sentiment leads
              corr = stats.pearsonr(
                  sentiment.iloc[:lag],
                  returns.iloc[-lag:]
              )[0]
          elif lag > 0:
              # Positive lag = sentiment lags (BAD)
              corr = stats.pearsonr(
                  sentiment.iloc[lag:],
                  returns.iloc[:-lag]
              )[0]
          else:
              corr = stats.pearsonr(sentiment, returns)[0]
  
          results.append({'lag': lag, 'correlation': corr})
  
      df = pd.DataFrame(results)
      max_corr_lag = df.loc[df['correlation'].abs().idxmax(), 'lag']
  
      return {
          'correlations_by_lag': df,
          'peak_correlation_lag': max_corr_lag,
          'is_lagging': max_corr_lag > 0,
          'interpretation': (
              'USELESS: Sentiment lags price' if max_corr_lag > 0 else
              'USEFUL: Sentiment leads price' if max_corr_lag < 0 else
              'CONCURRENT: No lead/lag'
          )
      }

If Your Sentiment Lags:

• Don't trade on it directly
• Use as contrarian indicator at extremes
• Combine with faster signals
• Look for sentiment CHANGES, not levels

References

• Sentiment analysis and market timing research

Your Sentiment Data Is Being Manipulated

Id

sentiment-data-gamed

Severity

CRITICAL

Description

Bad actors actively game sentiment sources for profit

Symptoms

• Sudden sentiment spikes with no real news
• Coordinated campaigns detected
• Pump and dump patterns

Detection Pattern

twitter|reddit|social|discord

Solution

How Sentiment Gets Gamed:

1. Bot Networks

• Thousands of fake accounts
• Coordinate to post about asset
• Create fake "consensus"
• Cost: $50-500 to run a campaign

2. Paid Promoters

• Real accounts, but paid to shill
• Don't disclose sponsorship
• Mix real content with promotion

3. Pump Groups

• Private Discord/Telegram groups
• Coordinate buys, then dump
• Create social buzz during pump

4. Sentiment Provider Gaming

• If traders use sentiment provider X
• Manipulate X's data sources
• Create fake signals

Detection Methods:

  def detect_coordination(
      posts: list,
      time_window_minutes: int = 30
  ) -> dict:
      """Detect coordinated posting campaigns."""
      # Time clustering
      timestamps = [p['timestamp'] for p in posts]
      gaps = [(timestamps[i+1] - timestamps[i]).total_seconds()
              for i in range(len(timestamps)-1)]
  
      avg_gap = np.mean(gaps)
      min_gap = np.min(gaps)
  
      # Account analysis
      accounts = [p['account'] for p in posts]
      unique_accounts = len(set(accounts))
      repeat_posters = len(accounts) - unique_accounts
  
      # Content similarity
      texts = [p['text'].lower() for p in posts]
      unique_phrases = len(set(texts))
  
      # Red flags
      is_coordinated = (
          min_gap < 5 or  # Posts within 5 seconds
          unique_accounts < len(accounts) * 0.5 or  # Many repeats
          unique_phrases < len(texts) * 0.3  # Same content
      )
  
      return {
          'is_coordinated': is_coordinated,
          'unique_account_ratio': unique_accounts / len(accounts),
          'content_diversity': unique_phrases / len(texts),
          'min_post_gap_seconds': min_gap,
          'red_flags': [
              'rapid_posting' if min_gap < 5 else None,
              'repeat_accounts' if repeat_posters > 5 else None,
              'similar_content' if unique_phrases < 10 else None
          ]
      }

Protection:

• Filter for account age > 30 days
• Require engagement history
• Detect sudden volume spikes
• Cross-reference multiple platforms

References

• Social media manipulation research

On-Chain Data Doesn't Mean What You Think

Id

on-chain-data-misinterpreted

Severity

HIGH

Description

Blockchain data is transparent but easily misinterpreted

Symptoms

• "Whale accumulation" trades fail
• Exchange flows don't predict price
• On-chain signals are consistently wrong

Detection Pattern

on.chain|whale|exchange.flow|wallet

Solution

On-Chain Interpretation Errors:

1. Wallet Fragmentation

• Whales use dozens of wallets
• "Whale buy" might be one of 50 wallets
• True position unknown

2. Exchange Cold Wallet Confusion

• Exchange moves to cold storage = "outflow"
• Not user withdrawals
• Bullish interpretation wrong

3. OTC Trades

• Large trades happen off-chain
• On-chain settlement later
• Timing mismatch with price

4. Smart Contract Interactions

• DeFi deposits ≠ Buying
• LP provision is neutral
• Staking is lock-up, not sentiment

Reality Check:

  def validate_onchain_signal(
      signal_type: str,
      historical_accuracy: pd.DataFrame
  ) -> dict:
      """Check if on-chain signal has predictive value."""
      # Calculate hit rate
      correct = (historical_accuracy['signal_direction'] ==
                 historical_accuracy['actual_direction']).mean()
  
      # Calculate average return following signal
      avg_return = historical_accuracy['subsequent_return'].mean()
  
      # Is it better than random?
      is_useful = (
          correct > 0.52 and  # Better than coin flip
          avg_return > 0.001   # Positive expected value
      )
  
      return {
          'signal_type': signal_type,
          'historical_accuracy': correct,
          'avg_return': avg_return,
          'is_useful': is_useful,
          'recommendation': (
              'usable' if is_useful else
              'ignore - not better than random'
          )
      }

Reliable vs Unreliable:

References

• On-chain analytics research

News Is Priced In Before You Read It

Id

news-already-priced

Severity

HIGH

Description

By the time you see news, algos have traded it

Symptoms

• Buy the news, price drops
• Sell the news, price rises
• Surprise announcements don't move price

Detection Pattern

news|headline|announcement|report

Solution

News Pricing Speed:

Timeline of News Event:

• T+0ms: Event occurs
• T+10ms: Direct feeds receive data
• T+50ms: Algo traders execute
• T+500ms: Bloomberg/Reuters update
• T+2000ms: Aggregators receive
• T+5000ms: Your phone notification
• T+10000ms: You read it
• T+15000ms: You decide to trade
• T+20000ms: Your order executes

By T+20 seconds, you're trading against people who traded at T+50ms. 99% of the move is done.

What Actually Works:

  def news_trading_strategy(news_type: str) -> dict:
      """Determine viable strategy for news type."""
      strategies = {
          'scheduled_announcement': {
              # Earnings, FOMC, etc.
              'strategy': 'trade_before_or_not_at_all',
              'why': 'Price moves in first second',
              'alternative': 'Trade implied volatility (options)'
          },
          'breaking_unscheduled': {
              'strategy': 'only_if_automated',
              'why': 'Milliseconds matter',
              'alternative': 'Look for second-order effects'
          },
          'developing_story': {
              # Multi-day events (geopolitical, legal)
              'strategy': 'position_for_resolution',
              'why': 'Information unfolds over time',
              'alternative': 'Trade thesis, not headlines'
          },
          'thematic': {
              # Industry trends, macro themes
              'strategy': 'position_over_weeks',
              'why': 'Slow information incorporation',
              'alternative': 'Build thesis from many data points'
          }
      }
  
      return strategies.get(news_type, {
          'strategy': 'skip',
          'why': 'Unknown news type',
          'alternative': 'Do nothing'
      })

If You Must Trade News:

• Trade the expected vs actual (earnings surprise)
• Trade second-order effects (supplier implications)
• Position before announcement (risky)
• Fade the initial overreaction (contrarian)

References

• Market efficiency and news incorporation research

Extreme Sentiment Can Persist for Months

Id

sentiment-extreme-persistence

Severity

HIGH

Description

Markets can stay irrational longer than you can stay solvent

Symptoms

• "Fear is extreme" but market keeps falling
• "Greed is extreme" but market keeps rising
• Contrarian signals lose money

Detection Pattern

extreme|overbought|oversold|fear|greed

Solution

Extreme Sentiment Persistence:

Historical Examples:

• 1999-2000: Greed extreme for 18 months
• 2008: Fear extreme for 6 months
• 2020 COVID: Fear extreme for 3 weeks only
• 2021 Crypto: Greed extreme for 8 months
• 2022 Crypto: Fear extreme for 10 months

"Extreme fear" is a buy signal...eventually. But "eventually" can destroy your account.

Better Approach:

  def sentiment_extreme_strategy(
      sentiment_score: float,
      duration_at_extreme_days: int,
      price_action: str  # 'trending', 'consolidating', 'reversing'
  ) -> dict:
      """
      Strategy for sentiment extremes that accounts for persistence.
      """
      is_extreme_fear = sentiment_score < 20
      is_extreme_greed = sentiment_score > 80
  
      # Phase analysis
      if duration_at_extreme_days < 7:
          phase = 'early_extreme'
          action = 'wait'
          reason = 'Extremes often persist - too early'
  
      elif duration_at_extreme_days < 30:
          phase = 'established_extreme'
          if price_action == 'reversing':
              action = 'scale_in'
              reason = 'Price confirming reversal'
          else:
              action = 'wait'
              reason = 'No price confirmation yet'
  
      else:  # > 30 days
          phase = 'extended_extreme'
          action = 'contrarian_scale_in'
          reason = 'Duration suggests exhaustion'
  
      return {
          'phase': phase,
          'action': action,
          'reason': reason,
          'required_confirmation': 'price_structure_break',
          'max_position': 0.25 if phase == 'early_extreme' else 0.5
      }

Rules for Extremes: 1. Never full size at first extreme 2. Require price confirmation 3. Scale in, don't go all in 4. Have defined invalidation 5. Expect to be early

References

• Behavioral finance and market timing

95% of Alternative Data Has Zero Alpha

Id

alternative-data-garbage

Severity

MEDIUM

Description

Most alt data is expensive garbage that looks good in backtests

Symptoms

• Expensive data subscription
• "Works" in backtest, fails live
• High correlation with price (lagging indicator)

Detection Pattern

alternative.data|satellite|credit.card|web.*traffic

Solution

Alternative Data Reality:

Why Most Alt Data Fails: 1. Survivorship bias in data sets 2. Look-ahead bias in vendor backtests 3. Data decay (everyone has it now) 4. High correlation with price (lagging) 5. Overfit vendor models

Due Diligence Checklist:

  def evaluate_alt_data(
      data_name: str,
      cost_per_month: float,
      claimed_ic: float,
      years_available: int,
      uniqueness_score: float  # 0-1, how many others have it
  ) -> dict:
      """Evaluate if alternative data is worth the cost."""
      # IC decay assumption
      # More users = less alpha
      adjusted_ic = claimed_ic * (1 - uniqueness_score * 0.5)
  
      # Backtest IC is always optimistic
      realistic_ic = adjusted_ic * 0.5
  
      # Minimum IC for profitability
      # Need ~0.02 IC to be useful after costs
      is_potentially_useful = realistic_ic > 0.02
  
      # Cost-benefit
      # Need $X in alpha to justify $Y in data cost
      min_aum_for_breakeven = cost_per_month * 12 / (realistic_ic * 0.1)
  
      return {
          'claimed_ic': claimed_ic,
          'realistic_ic': realistic_ic,
          'cost_per_month': cost_per_month,
          'is_potentially_useful': is_potentially_useful,
          'min_aum_for_breakeven': min_aum_for_breakeven,
          'recommendation': (
              'worth_investigating' if is_potentially_useful and
              years_available >= 5 else
              'likely_garbage'
          )
      }

Red Flags in Alt Data:

• Vendor won't share raw data (only signals)
• Backtest starts at exact low/high
• "Proprietary methodology" (can't validate)
• Only works on specific assets
• Less than 3 years history

References

• Alternative data and alpha decay research

All Sentiment Signals Correlate in Crisis

Id

sentiment-correlation-crisis

Severity

MEDIUM

Description

In crisis, everything says "fear" and nothing helps you time the bottom

Symptoms

• All signals say sell at the bottom
• Sentiment composite maxes out
• No divergence to trade

Detection Pattern

crisis|crash|fear|extreme

Solution

Crisis Sentiment Behavior:

In Normal Markets:

• Social sentiment: +0.2
• Options flow: -0.1
• Exchange flows: +0.3
• Funding rates: +0.1
• Different signals, some useful

In Crisis (March 2020, for example):

• Social sentiment: -0.9
• Options flow: -0.8
• Exchange flows: -0.9
• Funding rates: -0.7
• ALL signals say the same thing

When everyone says "sell," who's left to sell?

Crisis Strategy:

  def crisis_sentiment_strategy(
      sentiment_composite: float,
      sentiment_component_std: float,
      vix_level: float,
      days_in_crisis: int
  ) -> dict:
      """Strategy when all sentiment signals align negatively."""
      # Low component std = all signals agree = crisis mode
      is_crisis = (
          sentiment_composite < 15 and
          sentiment_component_std < 10 and
          vix_level > 40
      )
  
      if not is_crisis:
          return {'mode': 'normal', 'strategy': 'use_signals'}
  
      # Crisis mode - signals less useful
      if days_in_crisis < 5:
          return {
              'mode': 'crisis_early',
              'strategy': 'wait',
              'reason': 'Capitulation may not be complete'
          }
      elif days_in_crisis < 20:
          return {
              'mode': 'crisis_middle',
              'strategy': 'scale_in_slowly',
              'reason': 'Sentiment maxed, watch for divergence'
          }
      else:
          return {
              'mode': 'crisis_extended',
              'strategy': 'contrarian_accumulate',
              'reason': 'Extended fear often near bottom'
          }

In Crisis, Look For:

• Sentiment staying extreme but price stabilizing
• One component diverging (e.g., smart money buying)
• VIX starting to decline
• Funding rates normalizing first

References

• Crisis behavior and capitulation patterns

Sentiment Analysis Trading - Validations

Sentiment Signal Backtested

Id

check-sentiment-backtested

Description

Sentiment signals must be backtested before use

Pattern

sentiment|social|twitter|reddit|news

File Glob

*/.{py,js,ts}

Match

present

Context Pattern

backtest|test|validate|ic|correlation

Message

Backtest sentiment signals before using in production

Severity

error

Autofix

Lead/Lag Analysis Required

Id

check-lag-analysis

Description

Must analyze if sentiment leads or lags price

Pattern

sentiment.signal|sentiment.trade

File Glob

*/.{py,js,ts}

Match

present

Context Pattern

lag|lead|granger|causality

Message

Analyze lead/lag relationship - sentiment often lags price

Severity

warning

Autofix

Data Quality Filtering

Id

check-data-quality-filter

Description

Social data needs quality filters (bots, spam)

Pattern

twitter|reddit|social|discord

File Glob

*/.{py,js,ts}

Match

present

Context Pattern

filter|quality|bot|spam|account.*age

Message

Filter for data quality - bots and spam are prevalent

Severity

warning

Autofix

Multiple Data Sources

Id

check-multiple-sources

Description

Don't rely on single sentiment source

Pattern

sentiment.index|composite.sentiment

File Glob

*/.{py,js,ts}

Match

absent

Context Pattern

single.source|only.twitter

Message

Use multiple sentiment sources for robustness

Severity

info

Autofix

Manipulation Detection

Id

check-manipulation-detection

Description

Check for coordinated manipulation campaigns

Pattern

twitter|reddit|social

File Glob

*/.{py,js,ts}

Match

present

Context Pattern

coordinate|manipulate|campaign|bot.*detect

Message

Implement manipulation detection for social signals

Severity

info

Autofix

News Latency Awareness

Id

check-news-latency

Description

Account for news processing latency

Pattern

news.*trade|headline|breaking

File Glob

*/.{py,js,ts}

Match

present

Context Pattern

latency|delay|speed|fast

Message

Account for news latency - by the time you read it, price has moved

Severity

warning

Autofix

On-Chain Data Verification

Id

check-onchain-verification

Description

Verify on-chain data interpretation

Pattern

on.chain|whale|exchange.flow

File Glob

*/.{py,js,ts}

Match

present

Context Pattern

verify|label|known|exchange.*wallet

Message

Verify on-chain data labels and interpretations

Severity

info

Autofix

Extreme Sentiment Persistence

Id

check-extreme-persistence

Description

Account for persistent extremes in strategy

Pattern

extreme|contrarian|fear|greed

File Glob

*/.{py,js,ts}

Match

present

Context Pattern

duration|persist|confirm|wait

Message

Account for sentiment extreme persistence in contrarian trades

Severity

info

Autofix