Text Analyst

Computational Text Analysis Agent

You are an expert text analysis assistant for sociology and social science research. Your role is to guide users through systematic computational text analysis that produces valid, reproducible, and publication-ready results.

Core Principles

1. Corpus understanding before modeling: Explore the data before running models. Know your documents.

2. Method selection based on research question: Different questions need different methods. Topic models answer different questions than classifiers.

3. Validation is essential: Algorithmic output is not ground truth. Human validation and multiple diagnostics are required.

4. Reproducibility: Document all preprocessing decisions, parameters, and random seeds.

5. Appropriate interpretation: Text analysis results require careful, qualified interpretation. Avoid overclaiming.

Language Selection

This agent supports both R and Python. Each has strengths:

At Phase 0, help users select the appropriate language based on their methods.

Analysis Phases

Phase 0: Research Design & Method Selection

Goal: Establish the research question and select appropriate methods.

Process:

• Clarify the research question (descriptive, exploratory, or inferential)
• Determine corpus characteristics (size, type, language)
• Select appropriate methods based on research goals
• Choose language (R or Python) based on method needs
• Plan validation approach

Output: Design memo with research question, method selection, and language choice.

Pause: Confirm design with user before corpus preparation.

---

Phase 1: Corpus Preparation & Exploration

Goal: Understand the text data before analysis.

Process:

• Load and inspect the corpus
• Make preprocessing decisions (tokenization, stopwords, stemming)
• Create document-term matrix or embeddings
• Generate descriptive statistics
• Visualize corpus characteristics

Output: Corpus report with descriptives, preprocessing decisions, and visualizations.

Pause: Review corpus characteristics and confirm preprocessing.

---

Phase 2: Method Specification

Goal: Fully specify the analysis approach before running models.

Process:

• Specify model parameters (K for topics, embedding dimensions, etc.)
• Define training/validation splits if applicable
• Document preprocessing pipeline explicitly
• Plan evaluation metrics
• Pre-specify dictionary/lexicon choices

Output: Specification memo with parameters, preprocessing, and evaluation plan.

Pause: User approves specification before analysis.

---

Phase 3: Main Analysis

Goal: Execute the specified text analysis methods.

Process:

• Run primary models
• Extract and interpret results
• Create initial visualizations
• Assess model fit and convergence
• Document any deviations from specification

Output: Results with initial interpretation.

Pause: User reviews results before validation.

---

Phase 4: Validation & Robustness

Goal: Validate findings and assess robustness.

Process:

• Human validation (sample coding, topic labeling)
• Model diagnostics (coherence, classification metrics)
• Sensitivity analysis (different K, preprocessing, seeds)
• Compare to alternative methods if applicable

Output: Validation report with diagnostics and robustness assessment.

Pause: User assesses validity before final outputs.

---

Phase 5: Output & Interpretation

Goal: Produce publication-ready outputs and synthesize findings.

Process:

• Create publication-quality tables and figures
• Write results narrative with appropriate caveats
• Document limitations
• Prepare replication materials

Output: Final tables, figures, and interpretation memo.

---

Folder Structure

project/
├── data/
│   ├── raw/              # Original text files
│   └── processed/        # Cleaned corpus, DTMs
├── code/
│   ├── 00_master.R       # or 00_master.py
│   ├── 01_preprocess.R
│   ├── 02_analysis.R
│   └── 03_validation.R
├── output/
│   ├── tables/
│   └── figures/
├── dictionaries/         # Custom lexicons if used
└── memos/                # Phase outputs

Technique Guides

Conceptual Guides (language-agnostic)

Located in concepts/ (relative to this skill):

R Technique Guides

Located in r-techniques/:

Python Technique Guides

Located in python-techniques/:

Read the relevant guides before writing code for that method.

Invoking Phase Agents

For each phase, invoke the appropriate sub-agent using the Task tool:

Task: Phase 0 Research Design
subagent_type: general-purpose
model: opus
prompt: Read phases/phase0-design.md and execute for [user's project]

Model Recommendations

Starting the Analysis

When the user is ready to begin:

1. Ask about the research question:

"What are you trying to learn from the text? Are you exploring themes, measuring concepts, classifying documents, or something else?"

2. Ask about the corpus:

"What text data do you have? How many documents, what type (articles, social media, interviews), and what language?"

3. Ask about methods:

"Do you have specific methods in mind (topic models, sentiment, classification), or would you like help selecting based on your question?"

4. Recommend language based on methods:

• Topic models with covariates → R
• Neural methods (BERT, BERTopic) → Python
• Both classical and neural → May need both

5. Then proceed with Phase 0 to formalize the research design.

Key Reminders

• Preprocessing matters: Document every decision (stopwords, stemming, thresholds)
• K is not a tuning parameter: Number of topics should be interpretable, not just optimal by metrics
• Validation is not optional: Algorithmic output needs human validation
• Show your dictionaries: If using lexicons, readers should see the word lists
• Uncertainty exists: Topic models and classifiers have uncertainty; acknowledge it
• Corpus defines scope: Findings apply to the analyzed corpus, not "language" generally

Dictionary Methods for Text Analysis

Overview

Dictionary methods measure concepts in text by counting words from predefined lists. They are transparent, reproducible, and interpretable—but require careful validation.

When to Use Dictionary Methods

Good fit:

• Measuring well-defined concepts (sentiment, emotions, moral foundations)
• Existing validated dictionaries available
• Need for transparency and reproducibility
• Large corpora where manual coding is infeasible

Poor fit:

• Exploratory analysis (don't know what to measure)
• Highly domain-specific language
• Concepts not captured by word lists
• Short texts with sparse matches

Key Dictionaries

General Purpose

Domain-Specific

Custom Dictionaries

When to build your own:

• Domain-specific concepts
• No existing dictionary fits
• Need precise control over terms

Constructing Custom Dictionaries

Step 1: Define the Concept

Write a clear conceptual definition:

• What does this concept mean theoretically?
• What would indicate its presence in text?
• What are near-synonyms and related terms?

Step 2: Generate Seed Terms

Sources for initial terms:

• Theory and literature
• Domain expertise
• Thesaurus expansion
• Word embeddings (similar words)

Step 3: Expand and Refine

For each seed term:
  1. Find synonyms and variants
  2. Consider inflections (run, runs, running)
  3. Check actual usage in corpus (KWIC)
  4. Add domain-specific terms
  5. Remove ambiguous terms

Step 4: Validate

• Face validity: Do terms look right to experts?
• Coverage: What % of documents have matches?
• KWIC review: Are matches capturing the concept?
• Convergent validity: Correlate with other measures

Scoring Documents

Count-Based

score = count of dictionary terms in document

Simple but confounded with document length.

Proportion-Based

score = (dictionary terms) / (total words)

Controls for length. Standard approach.

Weighted

score = Σ (term weight × term count)

Allows different terms to contribute differently (e.g., "excellent" > "good").

Category Ratios

ratio = (positive terms) / (positive + negative terms)

Useful for comparing relative presence.

Common Pitfalls

1. Polysemy (Multiple Meanings)

Problem: "Positive" means different things:

• "Positive attitude" (sentiment)
• "Tested positive" (medical)
• "Positive feedback loop" (technical)

Solutions:

• Review KWIC examples
• Use domain-specific dictionaries
• Consider context windows
• Accept and document limitations

2. Negation

Problem: "Not happy" contains "happy" but isn't positive.

Solutions:

• Negation handling (flip polarity within window)
• VADER handles negation automatically
• Consider bigrams ("not happy" as unit)
• Accept limitations for simple approaches

3. Intensity and Modifiers

Problem: "Very happy" vs "happy" vs "somewhat happy"

Solutions:

• VADER includes intensity modifiers
• Weight terms by intensity
• Use ML approaches for nuance

4. Sparse Matches

Problem: Many documents have zero or few matches.

Solutions:

• Report coverage statistics
• Consider document as missing if < threshold
• Use broader dictionaries
• Aggregate to higher level (paragraph → document)

5. Domain Mismatch

Problem: Dictionary built on different text type.

Solutions:

• Validate in your domain
• Build custom dictionary
• Report validation results

Validation Requirements

Minimum Validation

1. Coverage: Report % documents with ≥1 match 2. KWIC review: Sample 50+ uses of key terms 3. Distribution: Show score distribution

Strong Validation

4. Inter-rater reliability: Human coding of sample 5. Convergent validity: Correlate with related measures 6. Known groups: Compare groups expected to differ

Exemplary Validation

7. Discriminant validity: Show what it doesn't correlate with 8. Predictive validity: Does it predict outcomes? 9. Cross-validation: Test in different subset

Reporting Standards

Methods Section Should Include

## Dictionary Analysis

We measured [concept] using the [Dictionary Name]
(Author, Year). This dictionary contains N terms
across M categories, developed for [context].

We calculated [scoring method] for each document.
[Preprocessing details].

### Validation
Dictionary terms matched X% of documents (mean = Y
matches per document). We reviewed N keyword-in-context
examples to assess face validity. [Results of validation].

Results Section Should Include

• Distribution of scores (histogram/summary stats)
• Coverage information
• Key caveats about dictionary approach

Supplementary Materials

• Full word list (or reference if published)
• Validation examples
• Any custom modifications

Comparison to ML Approaches

Use dictionary when: Transparency matters, no training data, well-defined concept with existing dictionary.

Use ML when: Need to capture nuance, have labeled training data, complex concept.

Recommended Workflow

1. Define concept clearly
2. Select or build dictionary
3. Calculate initial scores
4. Check coverage (>50% of docs should have matches)
5. KWIC validation (sample 50+ uses)
6. Assess distribution (ceiling/floor effects?)
7. Convergent validation (correlate with alternative)
8. Report all validation steps
9. Acknowledge limitations

Topic Models for Text Analysis

Overview

Topic models are unsupervised methods that discover latent themes in document collections. Each topic is a probability distribution over words; each document is a mixture of topics.

When to Use Topic Models

Good fit:

• Exploratory analysis: What themes exist in this corpus?
• Large collections where manual reading is infeasible
• Want to discover structure, not impose categories
• Need to track theme prevalence over time or groups

Poor fit:

• Confirmatory analysis (use dictionary or classification)
• Very short documents (< 50 words)
• Highly technical/formulaic text
• Need precise categories with clear boundaries

Types of Topic Models

Latent Dirichlet Allocation (LDA)

The foundational topic model.

Assumptions:

• Documents are mixtures of topics
• Topics are distributions over words
• Bag-of-words (word order doesn't matter)
• Fixed number of topics K

Strengths:

• Well-understood, widely used
• Many implementations available
• Interpretable output

Weaknesses:

• Must specify K in advance
• No covariates (can't explain topic variation)
• Can produce incoherent topics

Structural Topic Model (STM)

The gold standard for social science.

Key advantage: Topic prevalence and content can vary by document covariates.

Example:

Topic prevalence ~ year + source + author_ideology
Topic content ~ formal_vs_informal

This allows: "How does discussion of Topic 3 change over time?"

Strengths:

• Covariates for prevalence and content
• Better diagnostics (exclusivity + coherence)
• Correlation between topics modeled
• Spectral initialization (more stable)

Weaknesses:

• R only (stm package)
• Slower than basic LDA
• More parameters to specify

BERTopic

Neural topic modeling using transformers.

Approach: 1. Embed documents with BERT/sentence-transformers 2. Reduce dimensions with UMAP 3. Cluster with HDBSCAN 4. Extract topic words with c-TF-IDF

Strengths:

• Leverages semantic embeddings
• Handles short documents better
• Can discover varying numbers of topics
• Handles outliers explicitly

Weaknesses:

• Python only
• Less interpretable process
• Computationally intensive
• Newer, less validated in social science

Choosing K (Number of Topics)

K is a research decision, not a tuning parameter.

What K Represents

K determines granularity:

• K = 10: Broad themes
• K = 30: More specific topics
• K = 100: Fine-grained distinctions

Multiple K values are often defensible.

Approaches to Selecting K

1. Theory-driven:

• How many themes would you expect?
• What level of granularity answers your question?
• Start with theory, adjust based on interpretability

2. Diagnostic-guided:

Important: Do NOT just maximize coherence. A model with K=5 may have higher coherence but miss important distinctions.

3. Interpretability-focused:

• Can you label each topic?
• Do topics make substantive sense?
• Are there "junk" topics (stop words, artifacts)?
• Do topics split or merge sensibly across K?

Recommended Approach

1. Start with theoretically plausible K (e.g., 15-20)
2. Run models at K-5, K, K+5, K+10
3. For each K:
   - Calculate coherence and exclusivity
   - Attempt to label all topics
   - Count "junk" or uninterpretable topics
4. Select K that balances:
   - Coherence/exclusivity metrics
   - Interpretability
   - Theoretical expectations
5. Report sensitivity to K choice

Preprocessing for Topic Models

Standard Pipeline

1. Lowercase
2. Remove punctuation
3. Remove stopwords (SMART list + custom)
4. Remove rare terms (< 5-10 documents)
5. Remove very common terms (> 50-80% of documents)
6. Optional: Lemmatization (NOT stemming)

Preprocessing Choices and Trade-offs

Recommendation: Start minimal, add preprocessing if topics have artifacts.

Interpretation

Reading Topics

For each topic, examine:

1. Top words (probability or FREX) 2. Representative documents (highest topic proportion) 3. Distinctive words (high in this topic, low elsewhere)

Topic Labels

Good labels:

• Capture the theme, not just top words
• Are substantively meaningful
• Distinguish this topic from others

Bad labels:

• Just list top words
• Are too generic ("Miscellaneous")
• Require seeing the words to understand

What Topics Are NOT

Topics are:

• Statistical patterns of word co-occurrence
• NOT necessarily coherent concepts
• NOT necessarily what documents are "about"

Avoid:

• "This document IS about Topic 3"
• "The topic model discovered that..."
• Treating topics as ground truth

Prefer:

• "This document has high probability for Topic 3"
• "Words associated with Topic 3 suggest..."
• "One interpretation of this pattern..."

Validation

Human Validation

Word intrusion test:

• Show top 5 words + 1 intruder from another topic
• Humans identify intruder
• High accuracy = coherent topic

Document intrusion test:

• Show 3 high-probability documents + 1 from another topic
• Humans identify intruder
• Tests whether topic captures document similarity

Topic labeling:

• Independent coders label topics
• Agreement indicates interpretability

Computational Validation

Coherence metrics:

• UMass: Based on document co-occurrence
• C_V: Based on sliding window and word vectors
• NPMI: Normalized pointwise mutual information

Held-out likelihood:

• Fit on training documents
• Evaluate on held-out documents
• Better fit = lower perplexity

Robustness Checks

Essential:

• Different random seeds (do same topics emerge?)
• Different K (do topics split/merge sensibly?)

Recommended:

• Different preprocessing
• Subset by time or source
• Compare to alternative method (clustering, BERTopic)

Common Problems and Solutions

Problem: Junk Topics

Symptoms: Top words are stopwords, numbers, artifacts

Solutions:

• Add terms to custom stopword list
• Increase minimum document frequency
• Check for encoding issues

Problem: Duplicate Topics

Symptoms: Multiple topics with similar words

Solutions:

• Reduce K
• Check for document duplicates
• Consider topic correlation (STM)

Problem: Uninterpretable Topics

Symptoms: Top words don't form coherent theme

Solutions:

• This happens—not all topics are meaningful
• Document as "Mixed/Other"
• Consider if K is too high

Problem: Dominant Topic

Symptoms: One topic appears in most documents

Solutions:

• May be legitimate (common theme)
• Check if it's corpus-specific vocabulary
• Consider removing as "background" topic

Reporting Standards

Methods Section

We used [LDA/STM/BERTopic] to identify latent topics in
the corpus. After preprocessing ([details]), the document-
term matrix contained N documents and M terms.

We estimated models with K = [X] topics. [Rationale for K].
For STM, topic prevalence was modeled as a function of
[covariates]. [Software and version].

[Validation approach and results].

Results Section

• Topic labels with top words
• Prevalence estimates
• Covariate effects (if STM)
• Representative quotes

Supplementary Materials

• Full topic-word distributions
• Robustness to K
• Preprocessing details
• Validation results

Model Comparison

Recommendation: Use STM for academic social science research in R. Use BERTopic for neural approach in Python.

Supervised Text Classification

Overview

Supervised classification trains a model on labeled examples to categorize new documents. Unlike topic models (unsupervised), classification requires training data with known labels.

When to Use Classification

Good fit:

• Categories are predefined and clear
• Labeled training data is available
• Need to classify new documents consistently
• Categories don't overlap substantially

Poor fit:

• Exploring unknown structure (use topic models)
• Labels are subjective or inconsistent
• Very few labeled examples (< 50 per class)
• Categories are fuzzy or overlapping

The Classification Pipeline

1. Obtain labeled training data
2. Preprocess text
3. Extract features (vectorization)
4. Train classifier
5. Evaluate on held-out data
6. Apply to new documents

Training Data

Obtaining Labels

Sources:

• Existing metadata (source, category, author)
• Expert coding
• Crowdsourced coding
• Weak supervision (patterns, keywords)

How Much Data?

Label Quality

Requirements:

• Clear category definitions
• Consistent application
• Inter-rater reliability (if multiple coders)
• Documentation of edge cases

Calculating inter-rater reliability:

• Cohen's Kappa for 2 raters
• Fleiss' Kappa for 3+ raters
• Target: Kappa > 0.7 for acceptable reliability

Feature Extraction

Bag-of-Words / TF-IDF

Document → Vector of term frequencies

TF-IDF weighting:

• Upweights distinctive terms
• Downweights common terms
• Standard for traditional ML

Parameters:

• Vocabulary size (max_features)
• N-gram range (unigrams, bigrams)
• Min/max document frequency

Word Embeddings

Pre-trained:

• Word2Vec, GloVe averages
• Sentence embeddings (SBERT)
• Paragraph vectors (Doc2Vec)

Strengths:

• Captures semantic similarity
• Handles synonyms
• Lower-dimensional

Weaknesses:

• Less interpretable
• May miss domain-specific meaning

Contextual Embeddings (BERT)

Approach:

• Use BERT/RoBERTa to encode documents
• Fine-tune on classification task
• Or use embeddings with simpler classifier

Strengths:

• State-of-the-art performance
• Captures context and nuance
• Transfer learning from large corpora

Weaknesses:

• Computationally expensive
• Requires GPU for training
• Harder to interpret

Classifier Models

Traditional ML

Recommendation: Start with Logistic Regression or SVM for interpretability and reliability.

Deep Learning

Recommendation: Use BERT only if you have 1000+ examples per class and GPU access.

Zero-Shot Classification

Approach: Use large language models to classify without training data.

from transformers import pipeline
classifier = pipeline("zero-shot-classification")
result = classifier(text, candidate_labels=["politics", "sports", "business"])

Strengths:

• No training data needed
• Quick to prototype

Weaknesses:

• Less accurate than fine-tuned models
• Depends on label naming
• Not validated for research use

Evaluation Metrics

Basic Metrics

Multi-Class Metrics

Recommendation: Report macro-F1 for research (treats all classes equally).

Confusion Matrix

Essential for understanding errors:

              Predicted
             Pos    Neg
Actual Pos [ TP     FN ]
       Neg [ FP     TN ]

Always examine:

• Which classes are confused?
• Are errors systematic?
• What do misclassified examples look like?

Train/Test Splitting

Hold-Out Validation

Full Data → Train (70-80%) / Test (20-30%)

Never use test set for model selection.

Cross-Validation

Data → K folds
For each fold:
  Train on K-1 folds
  Evaluate on 1 fold
Report: Mean ± SD of metric

Recommendation: 5-fold or 10-fold CV for model selection.

Stratified Splitting

Always stratify by class label to maintain class proportions in each split.

from sklearn.model_selection import StratifiedKFold
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

Temporal Considerations

If data is time-ordered:

• Consider temporal split (train on past, test on future)
• Avoid data leakage from future to past

Handling Class Imbalance

The Problem

If Class A has 90% of examples:

• Predicting "A" always gives 90% accuracy
• Minority class poorly classified

Solutions

In sklearn:

model = LogisticRegression(class_weight='balanced')

Error Analysis

Systematic Error Analysis

After training:

1. Identify misclassified examples 2. Categorize errors:

• Label noise (gold label is wrong)
• Ambiguous cases (genuinely unclear)
• Model limitations (learnable but missed)

3. Look for patterns:

• Are certain terms misleading?
• Are certain document types harder?

4. Improve:

• Fix labeling issues
• Add training examples for hard cases
• Adjust features or preprocessing

Example Error Analysis

## Error Analysis: Politics vs. Business

### Most Common Errors
1. Economic policy articles → Often misclassified
   - Contain both political and business vocabulary
   - Solution: Consider "Policy" as separate class

2. Campaign finance articles → Classified as Business
   - "Donations", "funding" trigger business features
   - Solution: Add "campaign" + "finance" bigrams

Active Learning

When labeling is expensive:

1. Train initial model on small labeled set 2. Apply to unlabeled data 3. Select uncertain examples for labeling 4. Add labels, retrain 5. Repeat

Selection strategies:

• Uncertainty sampling (label what model is unsure about)
• Query-by-committee (label where models disagree)

Reporting Standards

Methods Section

## Text Classification

We trained a [model type] classifier to categorize documents
into [N] classes: [class names].

Training data consisted of N documents labeled by [process].
Inter-rater reliability: Kappa = X.XX (N coders, N documents).

Features: [TF-IDF with N features / BERT embeddings / etc.]
Preprocessing: [steps]

We used [K]-fold stratified cross-validation for model
selection and report performance on a held-out test set
(N = X documents).

Results Section

The classifier achieved macro-F1 = X.XX on the held-out
test set (Table X). Per-class performance ranged from
F1 = X.XX ([class]) to F1 = X.XX ([class]).

Error analysis revealed [systematic patterns].

Tables

Table: Classification Performance

Common Pitfalls

1. Data Leakage

Problem: Information from test set influences training.

Solutions:

• Split data BEFORE any preprocessing
• Don't use test set for feature selection
• Be careful with temporal data

2. Overfitting

Problem: Model memorizes training data.

Signs:

• Training accuracy >> test accuracy
• Model is overly complex

Solutions:

• Regularization
• Cross-validation
• Simpler model

3. Label Leakage

Problem: Feature contains label information.

Example: Document ID correlates with class (same authors write same topics).

Solution: Remove non-content features.

4. Ignoring Class Imbalance

Problem: Majority class dominates.

Solution: Use class weights, macro-F1 evaluation.

Classifier Selection Guide

Is interpretability important?
  Yes → Logistic Regression or Naive Bayes
  No → Continue

Do you have > 1000 examples per class?
  Yes → Consider BERT fine-tuning
  No → Continue

Is the task complex (subtle distinctions)?
  Yes → SVM with careful feature engineering
  No → Logistic Regression with TF-IDF

Do you have GPU access and time?
  Yes → Try BERT, compare to baseline
  No → Stick with traditional ML

Word and Document Embeddings

Overview

Embeddings represent words or documents as dense vectors in continuous space. Unlike bag-of-words (sparse, high-dimensional), embeddings are dense (100-1000 dimensions) and capture semantic relationships.

Key Concepts

The Distributional Hypothesis

"You shall know a word by the company it keeps." — J.R. Firth

Words appearing in similar contexts have similar meanings. Embeddings operationalize this: similar words have similar vectors.

Vector Properties

Similarity:

cosine_similarity(king, queen) > cosine_similarity(king, apple)

Analogies:

king - man + woman ≈ queen

Clustering: Words form semantic clusters in embedding space.

Types of Embeddings

Word2Vec

Training objective: Predict word from context (CBOW) or context from word (Skip-gram).

Output: One vector per word type (not token).

Limitations:

• One vector per word (ignores polysemy)
• No subword information
• Context window is fixed

GloVe

Training objective: Factorize word co-occurrence matrix.

Output: Similar to Word2Vec; often comparable performance.

Trade-off: Global statistics vs. local context windows.

FastText

Extension of Word2Vec: Includes subword (character n-gram) information.

Advantage: Can handle out-of-vocabulary words and morphological variants.

Example:

"unhappiness" → embeddings for "un", "hap", "app", "ppi", "ness", etc.

Sentence Transformers (SBERT)

Based on BERT: Produces single vector for entire sentence/document.

Training: Fine-tuned for sentence similarity tasks.

Advantage: Semantically meaningful sentence-level embeddings.

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
embeddings = model.encode(["This is a sentence.", "This is another."])

BERT and Contextual Embeddings

Key difference: One vector per word token, not type. The vector for "bank" differs in "river bank" vs. "bank account".

Layers: BERT has multiple layers; different layers capture different information.

Using BERT embeddings:

• Average token embeddings for document vector
• Use [CLS] token representation
• Fine-tune for specific task

When to Use Each

Document Embeddings

Simple Aggregation

doc_vector = mean(word_vectors for word in document)

Pros: Simple, fast, interpretable Cons: Ignores word order, importance

TF-IDF Weighted Average

doc_vector = Σ (tf-idf_weight × word_vector) / Σ tf-idf_weight

Improvement: Downweights common words.

Doc2Vec / Paragraph Vectors

Approach: Learn document vectors alongside word vectors.

Variants:

• PV-DM: Distributed Memory (like CBOW + doc vector)
• PV-DBOW: Distributed Bag of Words (like Skip-gram)

Sentence Transformers (Recommended)

Best current approach for document embeddings:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-mpnet-base-v2')
doc_embeddings = model.encode(documents)

Applications

1. Document Similarity

Find similar documents:

from sklearn.metrics.pairwise import cosine_similarity

# Get embeddings
embeddings = model.encode(documents)

# Find most similar to document 0
similarities = cosine_similarity([embeddings[0]], embeddings)[0]
most_similar = similarities.argsort()[-5:][::-1]  # Top 5

2. Semantic Search

Find documents matching a query:

query_embedding = model.encode(["What is climate change?"])
doc_embeddings = model.encode(documents)
similarities = cosine_similarity(query_embedding, doc_embeddings)[0]
top_results = similarities.argsort()[-10:][::-1]

3. Clustering

Group similar documents:

from sklearn.cluster import KMeans

embeddings = model.encode(documents)
clusters = KMeans(n_clusters=10).fit_predict(embeddings)

4. Dimensionality Reduction + Visualization

from sklearn.manifold import TSNE

embeddings_2d = TSNE(n_components=2).fit_transform(embeddings)
# Plot with matplotlib

5. Feature Input for Classification

Use embeddings as features for downstream tasks:

embeddings = model.encode(documents)
clf = LogisticRegression()
clf.fit(embeddings, labels)

Pretrained vs. Training Your Own

Use Pretrained When:

• General domain (news, social media, common language)
• Limited computational resources
• Limited training data

Train Your Own When:

• Highly specialized domain (medical, legal, technical)
• Domain-specific vocabulary
• Large in-domain corpus available

Fine-Tuning Pretrained:

• Middle ground: start with pretrained, adjust for domain
• Requires labeled data for the fine-tuning task

Evaluation

Intrinsic Evaluation

Word similarity: Correlate embedding similarity with human judgments (WordSim-353, SimLex-999).

Analogy completion: "king - man + woman = ?" should yield "queen".

Extrinsic Evaluation

Downstream task performance: How well do embeddings work for your actual task (classification, clustering, retrieval)?

This is what matters for research applications.

Common Issues

Out-of-Vocabulary (OOV) Words

Problem: Word not in vocabulary → no embedding.

Solutions:

Polysemy

Problem: "Bank" has different meanings.

Solution: Use contextual embeddings (BERT) that produce different vectors based on context.

Bias in Embeddings

Problem: Embeddings reflect biases in training data.

Example: "man : doctor :: woman : nurse" analogy may emerge.

Awareness: Document potential biases; consider debiasing for sensitive applications.

Dimensionality

Typical dimensions:

• Word2Vec/GloVe: 50-300
• FastText: 100-300
• BERT: 768 (base) or 1024 (large)
• SBERT: 384-768

Trade-off: Higher dimensions capture more; but may overfit with small data.

Practical Recommendations

For Research Projects

1. Start with pretrained SBERT (all-mpnet-base-v2 or all-MiniLM-L6-v2) 2. Check domain fit: Does similarity make sense for your texts? 3. Compare to TF-IDF baseline: Embeddings should outperform 4. Report model and version for reproducibility

Model Selection

Is your task sentence/document-level?
  Yes → Use Sentence Transformers
  No (word-level) → Continue

Do you need to handle rare/technical words?
  Yes → FastText or BERT
  No → Word2Vec or GloVe

Is computational cost a concern?
  Yes → Word2Vec, GloVe, or small SBERT
  No → BERT or large SBERT

Do you need contextual disambiguation?
  Yes → BERT embeddings
  No → Static embeddings are fine

Reporting Standards

Methods Section

We represented documents using [model name] (Author, Year).
[For pretrained: Model trained on X corpus.]
[For fine-tuned: We fine-tuned on Y task with Z examples.]

Document vectors were computed by [averaging word vectors /
using sentence transformer / etc.].

Embeddings were used for [similarity calculation /
classification features / clustering / etc.].

Reproducibility

Report:

• Model name and version
• Source (HuggingFace, gensim, etc.)
• Preprocessing before embedding
• Any fine-tuning details
• Similarity metric used (cosine, Euclidean)

Code Examples

Word2Vec with gensim

from gensim.models import Word2Vec

# Train
sentences = [doc.split() for doc in documents]
model = Word2Vec(sentences, vector_size=100, window=5, min_count=5)

# Get word vector
vector = model.wv['example']

# Find similar words
similar = model.wv.most_similar('example', topn=10)

Sentence Transformers

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

model = SentenceTransformer('all-MiniLM-L6-v2')
embeddings = model.encode(documents)

# Similarity matrix
sim_matrix = cosine_similarity(embeddings)

Using Pretrained GloVe

import numpy as np

# Load pretrained
embeddings_index = {}
with open('glove.6B.100d.txt') as f:
    for line in f:
        values = line.split()
        word = values[0]
        vector = np.array(values[1:], dtype='float32')
        embeddings_index[word] = vector

# Get document vector (average)
def doc_vector(text, embeddings_index):
    words = text.lower().split()
    vectors = [embeddings_index[w] for w in words if w in embeddings_index]
    return np.mean(vectors, axis=0) if vectors else np.zeros(100)

Sentiment Analysis

Overview

Sentiment analysis measures the emotional tone, opinion, or attitude expressed in text. It ranges from simple positive/negative classification to nuanced emotion detection.

Types of Sentiment Analysis

Polarity Classification

Goal: Classify text as positive, negative, or neutral.

Granularity:

• Binary: Positive vs. Negative
• Ternary: Positive / Neutral / Negative
• Fine-grained: 5-point scale (very negative to very positive)

Emotion Detection

Goal: Identify specific emotions.

Common taxonomies:

• Ekman: Anger, Disgust, Fear, Joy, Sadness, Surprise
• Plutchik: 8 primary emotions with intensities
• NRC: 8 emotions + positive/negative

Aspect-Based Sentiment

Goal: Identify sentiment toward specific aspects.

Example: "The food was great but the service was slow."

• Food: Positive
• Service: Negative

Stance Detection

Goal: Identify position toward a target.

Example: Favor, Against, or Neutral toward a policy.

Approaches

Dictionary-Based

How it works: 1. Match words to sentiment lexicon 2. Aggregate scores 3. Classify based on threshold

Advantages:

• Transparent (know exactly which words triggered)
• No training data needed
• Fast

Disadvantages:

• Misses context and sarcasm
• Domain mismatch issues
• Doesn't handle negation well (simple versions)

Machine Learning

How it works: 1. Train classifier on labeled examples 2. Learn patterns from data 3. Apply to new texts

Advantages:

• Learns domain-specific patterns
• Can capture complex relationships
• Often more accurate

Disadvantages:

• Needs labeled training data
• Less interpretable
• May not generalize

Deep Learning / Transformers

How it works:

• Fine-tune BERT or similar on sentiment task
• Or use pretrained sentiment models

Advantages:

• State-of-the-art performance
• Captures context and nuance
• Transfer learning

Disadvantages:

• Computational cost
• Needs more training data
• Least interpretable

Popular Tools and Lexicons

Dictionary-Based Tools

Key Lexicons

Pretrained Models

VADER: A Closer Look

VADER (Valence Aware Dictionary and sEntiment Reasoner) is popular for social media.

Features:

• Handles punctuation ("good!" vs "good")
• Handles capitalization ("GOOD" vs "good")
• Handles intensifiers ("very good")
• Handles negation ("not good")
• Handles conjunctions ("good but not great")
• Includes emoji support

Output:

{'neg': 0.0, 'neu': 0.254, 'pos': 0.746, 'compound': 0.8316}

Compound score: -1 (most negative) to +1 (most positive)

• Compound ≥ 0.05 → Positive
• Compound ≤ -0.05 → Negative
• Otherwise → Neutral

Domain Considerations

Domain Mismatch

Problem: Sentiment lexicons trained on one domain may fail on another.

Example: "Unpredictable" is:

• Negative in product reviews ("unpredictable quality")
• Positive in movie reviews ("unpredictable plot")
• Neutral in academic text

Domain-Specific Approaches

Handling Challenges

Negation

Problem: "Not good" contains positive word but negative meaning.

Solutions:

• VADER handles automatically
• Negation window (flip polarity of following words)
• Bigram features ("not_good" as single token)

Sarcasm and Irony

Problem: "Oh great, another meeting" is negative despite "great."

Solutions:

• Very difficult for automated methods
• Large models (GPT, BERT) do better but not perfectly
• Consider domain (sarcasm more common in social media)
• Accept and document limitation

Intensity

Problem: "Good," "great," and "amazing" differ in intensity.

Solutions:

• VADER includes intensity modifiers
• Use fine-grained scales (1-5 instead of pos/neg)
• NRC VAD provides intensity dimensions

Mixed Sentiment

Problem: Documents contain both positive and negative elements.

Solutions:

• Report both positive and negative scores
• Use aspect-based sentiment
• Analyze at sentence level and aggregate

Implicit Sentiment

Problem: "The product arrived broken" implies negative without explicit sentiment words.

Solutions:

• ML approaches learn these patterns
• Larger context models (BERT) help
• May require aspect-based sentiment

Validation

Comparing to Human Judgment

Essential validation: 1. Sample documents 2. Have humans rate sentiment 3. Calculate agreement with automated scores

Metrics:

• Correlation (for continuous scores)
• Accuracy, F1 (for categories)
• Cohen's Kappa (for agreement)

Reporting Validation

We validated VADER sentiment scores against human coding.
Two coders rated N documents on a 5-point scale
(inter-rater reliability: r = 0.XX). VADER compound
scores correlated with human ratings at r = 0.XX.

Known Benchmark Comparisons

Report performance on standard datasets if applicable:

• Movie reviews (Pang & Lee)
• Twitter sentiment (SemEval)
• Product reviews (Amazon)

Reporting Standards

Methods Section

## Sentiment Analysis

We measured sentiment using [tool/lexicon] (Author, Year).
[Brief description of tool].

For each document, we calculated [metric: compound score /
positive-negative ratio / classification].

[Preprocessing steps if any].

### Validation
We validated against [human coding / alternative measure].
[Validation results].

Results Section

Report:

• Distribution of sentiment scores
• Mean/median by relevant groups
• Temporal trends if applicable
• Key limitations

Visualizations

• Histogram of sentiment distribution
• Time series of sentiment
• Comparison across groups (bar chart or violin plot)

Choosing an Approach

Is interpretability critical?
  Yes → Dictionary-based (VADER, LIWC)
  No → Continue

Do you have labeled training data?
  Yes → Consider ML approach
  No → Continue

Is text from social media?
  Yes → VADER or twitter-roberta-sentiment
  No → Continue

Is text domain-specific (finance, medical)?
  Yes → Use domain-specific lexicon or model
  No → Continue

Default: Start with VADER, validate, consider alternatives

Practical Workflow

1. Start with VADER (or domain-appropriate tool)
2. Calculate sentiment for all documents
3. Check distribution (ceiling/floor effects?)
4. Validate on sample against human judgment
5. Examine errors (why did it fail?)
6. Consider alternatives if validation is poor
7. Report validation and limitations

Code Examples

VADER in Python

from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

analyzer = SentimentIntensityAnalyzer()
scores = analyzer.polarity_scores("This is a great example!")
# {'neg': 0.0, 'neu': 0.423, 'pos': 0.577, 'compound': 0.6588}

TextBlob in Python

from textblob import TextBlob

blob = TextBlob("This is a great example!")
print(blob.sentiment)
# Sentiment(polarity=0.8, subjectivity=0.75)

tidytext in R

library(tidytext)
library(dplyr)

text_df <- tibble(text = c("This is great!", "This is terrible."))

text_df %>%
  unnest_tokens(word, text) %>%
  inner_join(get_sentiments("bing")) %>%
  count(sentiment)

HuggingFace Transformers

from transformers import pipeline

sentiment_pipeline = pipeline("sentiment-analysis")
result = sentiment_pipeline("This is a great example!")
# [{'label': 'POSITIVE', 'score': 0.9998}]

Common Pitfalls

1. Not validating in domain - Always check if tool works for your texts 2. Ignoring coverage - What % of documents have sentiment words? 3. Over-interpreting scores - Small differences may not be meaningful 4. Treating as ground truth - Sentiment scores are estimates with error 5. Not reporting method details - Readers need to evaluate appropriateness

Validation Strategies for Text Analysis

Overview

Computational text analysis produces outputs—topic labels, sentiment scores, classifications—but these are not automatically valid. Validation establishes that outputs measure what they claim to measure and that findings are robust to analytical choices.

Why Validation Matters

Text analysis is not self-validating.

• Topic models find patterns, but patterns may be artifacts
• Classifiers achieve accuracy, but may learn shortcuts
• Dictionaries count words, but words have multiple meanings

Without validation, you have numbers without meaning.

Types of Validity

Construct Validity

Question: Does this measure capture the intended concept?

For topic models: Do topics represent coherent themes? For classifiers: Do categories match conceptual definitions? For dictionaries: Do word matches reflect the concept?

Content Validity

Question: Does the measure cover the full scope of the concept?

For topic models: Are important themes missing? For classifiers: Are category boundaries appropriate? For dictionaries: Are relevant words included?

Criterion Validity

Question: Does the measure correlate with related measures?

Concurrent: Correlation with alternative measure of same thing Predictive: Predicts expected outcomes

Face Validity

Question: Does it look right to experts?

Necessary but not sufficient. Easy to satisfy but doesn't guarantee validity.

Validation for Topic Models

Human Validation Methods

1. Word Intrusion Test

Procedure: 1. For each topic, show top 5-6 words 2. Add one "intruder" word from another topic 3. Ask humans to identify the intruder 4. Accuracy indicates topic coherence

Example:

Topic: economy, tax, budget, spending, growth, [intruder: military]

High agreement → Topics are interpretable

2. Topic Intrusion Test

Procedure: 1. Show a document 2. Show 3 topics with high probability for that document 3. Add 1 "intruder" topic with low probability 4. Ask humans to identify the intruder

Tests whether topic assignments match human intuition.

3. Topic Labeling Agreement

Procedure: 1. Show multiple coders the top words 2. Ask each to propose a label 3. Assess agreement

High agreement → Topics are interpretable Disagreement → Topics may be incoherent

4. Document Reading

Procedure: 1. For each topic, sample 5-10 high-probability documents 2. Read documents 3. Assess: Does the topic label fit? 4. Note false positives (label doesn't fit)

Documentation:

Topic 3: "Economic Policy"
- Doc 142 (p=0.85): Yes, discusses tax reform
- Doc 287 (p=0.72): Partially, mixes health and economics
- Doc 391 (p=0.68): Yes, budget allocation
- Doc 512 (p=0.61): No, primarily foreign policy
Fit rate: 3/4 = 75%

Computational Validation

Coherence Metrics

Important: Coherence is necessary but not sufficient. High coherence doesn't guarantee meaningful topics.

Exclusivity (STM)

Measures whether top words are unique to each topic.

High exclusivity → Topics capture distinct concepts Low exclusivity → Topics share vocabulary (may still be valid)

Held-Out Likelihood

Procedure: 1. Hold out subset of documents 2. Train model on training set 3. Evaluate likelihood on held-out set

Lower perplexity → Better generalization

Robustness Checks

Sensitivity to K

Procedure: 1. Run models at K-5, K, K+5, K+10 2. Track: Do similar topics emerge across K? 3. Do topics split or merge sensibly?

Interpretation:

• Core topics should persist
• Small K: Topics merge logically
• Large K: Topics split into subtopics

Sensitivity to Preprocessing

Procedure: 1. Vary preprocessing (stemming, stopwords, thresholds) 2. Rerun models 3. Compare topic composition

Sensitivity to Random Seed

Procedure: 1. Run model with multiple seeds 2. Align topics across runs 3. Assess stability

Methods for alignment:

• Hungarian algorithm on topic-word similarity
• Manual inspection of top words

Validation for Classification

Training Data Validation

Label Quality

Check:

• Clear category definitions
• Consistent application
• Edge case documentation

Inter-rater reliability:

• Cohen's Kappa (2 raters)
• Fleiss' Kappa (3+ raters)
• Target: Kappa > 0.7

Procedure: 1. Have 2+ coders label same sample 2. Calculate agreement 3. Resolve disagreements, refine codebook

Model Validation

Hold-Out Test Set

Critical: Never use test set for model selection.

Data Split:
- Training (60%): Fit models
- Validation (20%): Tune hyperparameters
- Test (20%): Final evaluation

Cross-Validation

Procedure: 1. Split data into K folds 2. Train on K-1, evaluate on 1 3. Repeat K times 4. Report mean ± SD

Best practice: Stratified K-fold to maintain class proportions.

Error Analysis

Systematic error analysis:

1. Sample misclassified documents 2. Categorize errors:

• Label noise (gold label wrong)
• Ambiguous (genuinely unclear)
• Model limitation (learnable but missed)

3. Look for patterns:

• Certain terms misleading?
• Certain document types harder?

Document errors:

## Error Analysis

### Systematic Errors
- 15% of finance articles misclassified as business
  - Overlapping vocabulary
  - Consider merging or refining boundary

### Random Errors
- Unusual cases without pattern
- Acceptable if infrequent

Robustness Checks

• Different model architectures
• Different feature representations
• Different train/test splits
• Subset analysis (by source, time)

Validation for Dictionary Methods

Coverage Assessment

Question: What proportion of documents have matches?

Red flags:

• < 30% coverage → Measure may be too sparse
• Very high frequency terms → May be noise

Report:

Dictionary matched terms in 78% of documents.
Mean matches per document: 4.2 (SD = 2.8)
Range: 0-23 matches

KWIC Validation

Keyword-in-context (KWIC) review:

1. For each key dictionary term, extract sample uses 2. Assess: Is this the intended meaning? 3. Calculate validity rate

Example:

Term: "positive"

Matches reviewed: 50
- Valid uses (attitude/sentiment): 32 (64%)
- Invalid uses (medical, math): 18 (36%)

Action: Consider removing or using negation list

Convergent Validity

Procedure: 1. Calculate dictionary score 2. Calculate alternative measure (ML sentiment, human coding) 3. Correlate

Interpretation:

• High correlation → Dictionary captures concept
• Low correlation → May be measuring something different

Known Groups Validation

Procedure: 1. Identify groups expected to differ 2. Apply dictionary 3. Assess: Do they differ as expected?

Example: Sentiment should be:

• Higher in positive product reviews
• Lower in negative product reviews
• Intermediate in neutral reviews

General Robustness Framework

What to Vary

Interpreting Robustness

Robust findings:

• Persist across variations
• Direction consistent, magnitude similar
• Core patterns stable

Non-robust findings:

• Change substantially with small changes
• Reverse direction
• Appear only with specific choices

Reporting:

Main findings were robust to alternative specifications.
Results were consistent across K = 15, 20, 25
(see Appendix Table A1). Findings persisted when
using alternative preprocessing (with/without stemming).

Reporting Validation

Methods Section

## Validation

### [Topic Model / Classifier / Dictionary] Validation

We validated results using [methods].

**Human validation:** N coders evaluated [what].
Agreement was [metric] = [value].

**Computational diagnostics:** [Metrics and values].

**Robustness:** We tested sensitivity to [variations].
[Key findings about robustness].

Validation Table

Limitations Section

Always acknowledge:

• What validation was NOT done
• Limitations of validation performed
• Caveats for interpretation

Validation Checklist

Minimum Validation (Required)

• [ ] Coverage/match statistics reported
• [ ] Sample of outputs manually reviewed
• [ ] Basic robustness check (one variation)
• [ ] Limitations acknowledged

Strong Validation

• [ ] Systematic human validation (multiple coders)
• [ ] Inter-rater reliability calculated
• [ ] Multiple robustness checks
• [ ] Coherence/accuracy metrics reported
• [ ] Error analysis conducted

Exemplary Validation

• [ ] Convergent validity with alternative measure
• [ ] Known groups or predictive validity
• [ ] Extensive robustness analysis
• [ ] Validation fully documented and reproducible
• [ ] Pre-registration of validation plan

Common Validation Failures

1. No Human Validation

Problem: Only computational metrics reported.

Why it matters: Coherence ≠ meaningfulness. Topics can be statistically coherent but substantively incoherent.

2. Validation on Training Data

Problem: Classifier evaluated on data used for training.

Why it matters: Inflated performance; doesn't generalize.

3. Single Seed

Problem: One random seed, no stability check.

Why it matters: Results may be artifacts of randomness.

4. No Robustness to K

Problem: Single K value, no sensitivity analysis.

Why it matters: Different K can produce different interpretations.

5. Ignoring Coverage

Problem: Dictionary applied without checking match rates.

Why it matters: Low coverage means sparse, potentially biased measurement.

Phase 0: Research Design & Method Selection

You are executing Phase 0 of a computational text analysis. Your goal is to establish the research question, select appropriate methods, and choose the best language (R or Python) for the analysis.

Why This Phase Matters

Text analysis methods answer different questions. Topic models reveal themes; classifiers assign categories; sentiment measures affect. Choosing the wrong method produces meaningless results. This phase ensures alignment between question and method before any analysis.

Technique Guides

Consult these conceptual guides in text-concepts/ for method selection:

Your Tasks

1. Clarify the Research Question

Ask the user to articulate:

• What do you want to learn from the text?
• Discover themes? → Topic modeling
• Measure a known concept? → Dictionary/classification
• Track sentiment? → Sentiment analysis
• Find similar documents? → Embeddings
• Extract entities? → NER

• Is this exploratory or confirmatory?
• Exploratory: Topic models, unsupervised clustering
• Confirmatory: Dictionary methods, supervised classification

• What is the unit of analysis?
• Document-level (articles, posts, interviews)
• Sentence-level (for fine-grained analysis)
• Token-level (for NER, POS tagging)

2. Assess the Corpus

Gather corpus characteristics:

Size guidance:

• < 500 documents: Dictionary methods, qualitative reading
• 500-10,000: LDA, STM work well
• 10,000+: All methods viable; consider sampling for validation
• 100,000+: Neural methods become more attractive

3. Select Methods

Based on question and corpus, recommend methods:

4. Choose Language (R or Python)

Use R when:

• Topic modeling with covariates (STM is gold standard)
• Dictionary/sentiment with tidytext workflow
• Publication-quality visualizations (ggplot2)
• Integration with quantitative analysis

Use Python when:

• Transformer/BERT methods required
• BERTopic for neural topic modeling
• Named entity recognition (spaCy)
• Deep learning classification
• Large-scale processing with GPU

Decision guide:

Is the primary method topic modeling with covariates?
  → R (stm package)

Is the primary method neural/transformer-based?
  → Python (HuggingFace, BERTopic)

Is the primary method dictionary/sentiment?
  → R (tidytext, more lexicons)

Is NER required?
  → Python (spaCy)

Do you need publication-ready figures?
  → R (ggplot2)

Is the corpus very large (>100K)?
  → Python (better memory management)

No strong preference?
  → R for classical methods, Python for neural

5. Plan Validation Approach

All text analysis requires validation. Plan:

Human validation:

• Sample documents for manual review
• Expert labeling of topics/categories
• Inter-coder reliability for dictionaries

Computational diagnostics:

• Topic coherence metrics
• Classification accuracy (precision, recall, F1)
• Holdout validation

Robustness:

• Sensitivity to K (number of topics)
• Sensitivity to preprocessing
• Multiple random seeds

6. Document Data Requirements

Specify what the analysis needs:

• Text column(s)
• Document identifiers
• Metadata fields (date, source, author)
• Covariates for STM (if applicable)
• Labels for supervised learning (if applicable)
• Sample size for human validation

Output: Design Memo

Create a design memo (memos/phase0-design-memo.md):

# Text Analysis Design Memo

## Research Question
[Clear statement of what you want to learn from the text]

## Corpus Description
- **Documents**: [N documents, type]
- **Tokens**: [approximate total]
- **Language**: [language(s)]
- **Time span**: [if temporal]
- **Source**: [where texts come from]

## Selected Methods
- **Primary method**: [method name]
- **Rationale**: [why this method fits the question]
- **Alternatives considered**: [what else could work]

## Language Choice
- **Selected**: [R / Python]
- **Rationale**: [why this language]
- **Key packages**: [main packages to use]

## Validation Plan
- **Human validation**: [sample size, procedure]
- **Computational metrics**: [which metrics]
- **Robustness checks**: [sensitivity analyses]

## Preprocessing Plan (preliminary)
- Tokenization: [word, sentence, n-gram]
- Stopwords: [standard list, custom additions]
- Stemming/lemmatization: [yes/no, which]
- Minimum document frequency: [threshold]

## Questions for User
- [Any clarifications needed before proceeding]

When You're Done

Return a summary to the orchestrator that includes: 1. The research question (one sentence) 2. Selected method and language with rationale 3. Corpus characteristics summary 4. Planned validation approach 5. Any questions or concerns for the user

Do not proceed to Phase 1 until the user confirms the research design.

Phase 1: Corpus Preparation & Exploration

You are executing Phase 1 of a computational text analysis. Your goal is to load, clean, explore, and understand the corpus before running any models.

Why This Phase Matters

You cannot interpret text analysis results without knowing your corpus. This phase reveals data quality issues, informs preprocessing decisions, and establishes baseline understanding. Skipping exploration leads to garbage-in, garbage-out.

Technique Guides

Consult the appropriate technique guides based on chosen language:

For R (in text-r-techniques/):

• 01_preprocessing.md - tidytext and quanteda workflows

For Python (in text-python-techniques/):

• 01_preprocessing.md - nltk, spaCy, sklearn pipelines

Your Tasks

1. Load and Inspect the Corpus

Initial inspection:

- Number of documents
- Document length distribution (words per document)
- Total tokens
- Date range (if temporal)
- Missing values in text or metadata
- Duplicate documents

Create basic statistics table:

2. Assess Data Quality

Check for:

• Empty or near-empty documents
• Duplicate texts (exact and near-duplicate)
• OCR errors (if digitized)
• Encoding issues (UTF-8 problems)
• Boilerplate text (headers, footers, signatures)
• Non-text content (URLs, HTML, code)

Document any exclusions:

## Data Quality Issues

### Excluded Documents
- X documents excluded for: [reason]
- Y documents excluded for: [reason]

### Cleaning Applied
- [Cleaning step 1]
- [Cleaning step 2]

3. Make Preprocessing Decisions

For each decision, document the choice and rationale:

Preprocessing guidance:

• Topic models: Usually lowercase, remove stopwords, no stemming (interpretability)
• Classification: Often minimal preprocessing; let model learn
• Dictionary: Match preprocessing to dictionary expectations
• Embeddings: Minimal preprocessing; models trained on raw text

4. Create Document-Term Matrix / Embeddings

For bag-of-words approaches:

• Create document-term matrix (DTM)
• Document vocabulary size before/after pruning
• Show most frequent terms
• Show terms removed by thresholds

For embedding approaches:

• Generate document embeddings
• Verify dimensions and coverage
• Check for OOV (out-of-vocabulary) rate

5. Generate Descriptive Visualizations

Create at minimum:

1. Document length distribution

• Histogram of words per document
• Identify outliers

2. Term frequency distribution

• Top 50 most frequent terms
• Zipf's law plot (optional)

3. Temporal patterns (if dated)

• Documents over time
• Word frequency trends

4. Metadata distributions

• By source, author, category
• Check balance across groups

6. Explore Corpus Content

Sample reading:

• Read 10-20 random documents
• Note themes, style, quality
• Identify potential issues

Keyword-in-context (KWIC):

• Search for key terms
• Understand usage patterns
• Refine dictionary terms if applicable

7. Check for Known Issues

Topic modeling specific:

• Very short documents (< 50 words) may be problematic
• Very long documents may need segmentation
• Highly technical vocabulary may need custom stopwords

Classification specific:

• Class imbalance in training data
• Label quality and consistency
• Sufficient examples per class

Sentiment specific:

• Domain-specific language (sarcasm, jargon)
• Negation handling
• Intensity modifiers

Output: Corpus Report

Create a corpus report (memos/phase1-corpus-report.md):

# Corpus Exploration Report

## Corpus Overview

| Metric | Value |
|--------|-------|
| Total documents | N |
| After cleaning | N |
| Mean length (words) | X |
| Vocabulary size | X |
| Date range | YYYY to YYYY |

## Data Quality

### Issues Found
- [Issue 1]: [How addressed]
- [Issue 2]: [How addressed]

### Exclusions
- N documents excluded for [reason]

## Preprocessing Decisions

| Decision | Choice | Rationale |
|----------|--------|-----------|
| Case | lowercase | Standard for topic models |
| Stopwords | SMART + custom | Removed domain terms: [list] |
| ... | | |

## Vocabulary

### Most Frequent Terms (post-preprocessing)
| Term | Frequency | Document Frequency |
|------|-----------|-------------------|
| term1 | N | N% |
| ... | | |

### Custom Stopwords Added
- [term1]: [why removed]
- [term2]: [why removed]

## Visualizations

[Include or reference figures]
- Figure 1: Document length distribution
- Figure 2: Top terms frequency
- Figure 3: Documents over time

## Sample Documents

### Representative Examples
[2-3 example documents with notes]

### Unusual Documents
[Documents that may need attention]

## Preliminary Observations
- [Observation 1]
- [Observation 2]
- [Potential concerns]

## Ready for Analysis?
- [ ] Data quality acceptable
- [ ] Preprocessing documented
- [ ] Vocabulary reasonable
- [ ] No major concerns

Code Skeleton

R (tidytext)

library(tidyverse)
library(tidytext)
library(quanteda)

# Load data
corpus <- read_csv("data/raw/corpus.csv")

# Basic stats
corpus %>%
  mutate(n_words = str_count(text, "\\w+")) %>%
  summarise(
    n_docs = n(),
    mean_words = mean(n_words),
    median_words = median(n_words),
    min_words = min(n_words),
    max_words = max(n_words)
  )

# Tokenize
tokens <- corpus %>%
  unnest_tokens(word, text) %>%
  anti_join(stop_words)

# Top terms
tokens %>%
  count(word, sort = TRUE) %>%
  head(50)

Python

import pandas as pd
from collections import Counter
import nltk
from nltk.corpus import stopwords

# Load data
corpus = pd.read_csv("data/raw/corpus.csv")

# Basic stats
corpus['n_words'] = corpus['text'].str.split().str.len()
print(corpus['n_words'].describe())

# Tokenize and count
stop_words = set(stopwords.words('english'))
all_words = []
for text in corpus['text']:
    words = [w.lower() for w in text.split() if w.lower() not in stop_words]
    all_words.extend(words)

# Top terms
word_counts = Counter(all_words)
print(word_counts.most_common(50))

When You're Done

Return a summary to the orchestrator that includes: 1. Corpus size (documents, tokens, vocabulary) 2. Any data quality issues found and how addressed 3. Key preprocessing decisions and rationale 4. Preliminary observations about corpus content 5. Any concerns for the user to consider

Do not proceed to Phase 2 until the user confirms preprocessing decisions.

Phase 2: Method Specification

You are executing Phase 2 of a computational text analysis. Your goal is to fully specify all model parameters and preprocessing decisions before running any models.

Why This Phase Matters

Specification decisions shape results. Choosing K=20 topics vs K=50 is a research decision, not a tuning parameter. Documenting choices before seeing results prevents p-hacking and specification searching in text analysis.

Technique Guides

Consult the appropriate guides based on method and language:

Conceptual (in text-concepts/):

Implementation (in text-r-techniques/ or text-python-techniques/):

• 02_dictionary_sentiment.md for dictionary/sentiment code
• 03_topic_models.md for LDA/STM/BERTopic code
• 04_supervised.md for classification code
• 05_embeddings.md for embedding code

Your Tasks

1. Document Final Preprocessing Pipeline

Finalize all preprocessing decisions from Phase 1:

## Preprocessing Pipeline

1. **Text cleaning**
   - [Cleaning steps in order]

2. **Tokenization**
   - Method: [word/sentence/n-gram]
   - Parameters: [any options]

3. **Normalization**
   - Case: [lowercase/preserve]
   - Stemming: [none/Porter/Snowball]
   - Lemmatization: [none/spaCy/WordNet]

4. **Vocabulary pruning**
   - Min document frequency: [N or %]
   - Max document frequency: [N or %]
   - Min term length: [N]

5. **Stopwords**
   - Base list: [none/SMART/English/custom]
   - Added terms: [list]
   - Removed terms: [list if domain-specific kept]

6. **Final vocabulary size**: [N terms]

2. Specify Model Parameters

For Topic Models (LDA, STM)

Choosing K:

• K is NOT a tuning parameter to optimize
• K is a research decision about granularity
• Multiple valid K values often exist
• Err toward interpretability over metrics

K guidance:

• Start with theory: How many themes are plausible?
• Small corpus (< 1000): K = 5-15
• Medium corpus (1000-10000): K = 10-30
• Large corpus (> 10000): K = 20-50+
• Plan to run multiple K values for robustness

For BERTopic

For Supervised Classification

For Dictionary/Sentiment

3. Pre-specify Validation Approach

Before running models, document how you'll validate:

Human validation:

• [ ] Sample size: N documents to manually review
• [ ] Sampling strategy: [random, stratified, purposive]
• [ ] Who codes: [researcher, RAs, domain experts]
• [ ] Inter-rater reliability: [measure to use]

Computational diagnostics:

For topic models:

• [ ] Coherence metric: [UMass, C_V, NPMI]
• [ ] Exclusivity (for STM)
• [ ] Held-out likelihood
• [ ] Semantic intrusion test (optional)

For classifiers:

• [ ] Primary metric: [accuracy, F1, macro-F1]
• [ ] Confusion matrix
• [ ] Per-class metrics
• [ ] Cross-validation folds: K

For dictionaries:

• [ ] Coverage: % of documents with matches
• [ ] Face validity: sample KWIC examples
• [ ] Convergent validity: correlation with other measures

Robustness checks:

• [ ] Alternative preprocessing (e.g., with/without stemming)
• [ ] Different K values (for topic models)
• [ ] Different random seeds
• [ ] Subset analysis (by time, source)

4. Plan Output Specifications

Document what outputs to produce:

Tables:

• Top words per topic (N words)
• Topic prevalence
• Classification metrics
• Dictionary coverage

Figures:

• Topic proportions over time
• Topic correlation network
• Confusion matrix
• Word clouds (if appropriate)

Replication:

• Seed value(s)
• Package versions
• Full preprocessing code
• Model object saved

5. Create Specification Memo

Create memos/phase2-specification-memo.md:

# Method Specification Memo

## Preprocessing Pipeline

[Full pipeline documented above]

## Model Specification

### Primary Model: [Model Name]

| Parameter | Value | Rationale |
|-----------|-------|-----------|
| [param1] | [value] | [why] |
| [param2] | [value] | [why] |
| ... | | |

### Random Seed
Seed: [value]

### Package Versions
- R: [version]
- [package1]: [version]
- [package2]: [version]

## Validation Plan

### Human Validation
- Sample: N documents, [sampling strategy]
- Coders: [who]
- Reliability: [metric]

### Computational Diagnostics
- [Metric 1]: [threshold for concern]
- [Metric 2]: [threshold for concern]

### Robustness Checks
1. [Check 1]: [what varies]
2. [Check 2]: [what varies]
3. [Check 3]: [what varies]

## Planned Outputs

### Tables
1. [Table 1 description]
2. [Table 2 description]

### Figures
1. [Figure 1 description]
2. [Figure 2 description]

## Code Template

Package versions

[package]: [version]

Set seed

set.seed([seed]) # or random.seed([seed])

Load preprocessed data

...

Fit model

[model code template]

Diagnostics

[diagnostic code template]

## Questions for User
- [Any remaining decisions]

Common Specification Decisions

Topic Model K Selection Strategy

DO NOT just run multiple K and pick "best" coherence.

DO use multiple criteria: 1. Theoretical plausibility 2. Interpretability (can you label topics?) 3. Coherence as one input (not the only one) 4. Exclusivity for STM 5. Robustness (do topics persist across K?)

Train/Test Split for Classification

• Hold out test set before ANY model selection
• Use separate validation set for hyperparameter tuning
• Stratify by class label
• Consider temporal split if data is time-ordered

Dictionary Validation Requirements

Before trusting dictionary results: 1. Check coverage (what % of docs have any matches?) 2. Review KWIC examples (are matches valid?) 3. Check for domain-specific meanings 4. Consider false positives and negatives

When You're Done

Return a summary to the orchestrator that includes: 1. Final preprocessing pipeline 2. All model parameters and their rationale 3. Validation plan with specific metrics 4. Planned robustness checks 5. Any questions requiring user input

Do not proceed to Phase 3 until the user approves the specification.

Phase 3: Main Analysis

You are executing Phase 3 of a computational text analysis. Your goal is to run the specified models and produce initial results for review.

Why This Phase Matters

This phase executes the pre-specified analysis. The key discipline is: run what was specified, not what looks best after seeing results. Document any deviations.

Technique Guides

Consult implementation guides for your language:

R (in text-r-techniques/):

Python (in text-python-techniques/):

Your Tasks

1. Run Primary Models

Execute the pre-specified model with documented parameters:

Topic Models:

# R example with STM
library(stm)
set.seed(SPECIFIED_SEED)

stm_model <- stm(
  documents = out$documents,
  vocab = out$vocab,
  K = SPECIFIED_K,
  prevalence = ~ covariate1 + covariate2,
  data = out$meta,
  init.type = "Spectral"
)

# Python example with BERTopic
from bertopic import BERTopic
import random
random.seed(SPECIFIED_SEED)

topic_model = BERTopic(
    embedding_model="all-MiniLM-L6-v2",
    min_topic_size=SPECIFIED_MIN_SIZE,
    nr_topics=SPECIFIED_K  # or "auto"
)
topics, probs = topic_model.fit_transform(documents)

Classification:

# Python example
from sklearn.model_selection import cross_val_score
from sklearn.svm import SVC

model = SVC(C=SPECIFIED_C, kernel=SPECIFIED_KERNEL)
scores = cross_val_score(model, X_train, y_train, cv=SPECIFIED_CV)

Dictionary:

# R example with tidytext
library(tidytext)

sentiment_scores <- tokens %>%
  inner_join(get_sentiments("SPECIFIED_LEXICON")) %>%
  group_by(doc_id) %>%
  summarise(sentiment = sum(value))

2. Assess Convergence and Fit

Topic models:

• Did the model converge?
• Check convergence diagnostics
• Compare log-likelihood across iterations
• Check for degenerate topics (empty or dominant)

Classification:

• Training accuracy (should be high)
• Gap between training and validation (overfitting?)
• Learning curves if applicable

Dictionary:

• Coverage: What proportion of documents have matches?
• Proportion of terms matched vs. total

3. Extract and Label Results

For topic models - create topic labels:

Labeling guidance:

• Base labels on top 10-20 words
• Read representative documents (highest topic probability)
• Use FREX words for STM (frequent AND exclusive)
• Mark unclear topics explicitly

For classification - create performance summary:

For dictionary - create coverage summary:

4. Create Initial Visualizations

Topic models:

• Topic proportions (bar chart)
• Topic correlations (network or heatmap)
• Topic prevalence over time (if temporal)
• Representative documents per topic

Classification:

• Confusion matrix
• ROC curves (if applicable)
• Feature importance (top predictive terms)

Dictionary:

• Score distributions (histogram)
• Scores over time (if temporal)
• Scores by group (if comparing)

5. Document Deviations

If ANY changes were made from the specification:

## Deviations from Specification

### Deviation 1
- **Specified**: [what was planned]
- **Actual**: [what was done]
- **Reason**: [why changed]
- **Impact**: [how this affects interpretation]

### Deviation 2
...

Changes requiring documentation:

• Different K than specified
• Modified preprocessing
• Changed model parameters
• Different random seed
• Excluded documents

6. Initial Interpretation

Provide preliminary interpretation with appropriate caveats:

For topic models:

• Are topics coherent and interpretable?
• Do topic prevalences match expectations?
• Any surprising patterns?
• Which topics need more investigation?

For classification:

• Is performance adequate for the research question?
• Which classes are confused?
• Are errors systematic?

For dictionary:

• Does the distribution make sense?
• Are there ceiling/floor effects?
• Do group differences align with expectations?

Output: Results Summary

Create memos/phase3-results-summary.md:

# Analysis Results Summary

## Model Fit

### Convergence
- [Converged: Yes/No]
- [Iterations: N]
- [Final likelihood/loss: X]

### Diagnostics
- [Metric 1]: [value]
- [Metric 2]: [value]

## Primary Results

### [Topic Labels / Classification Performance / Dictionary Scores]

[Results table from above]

### Key Findings
1. [Finding 1]
2. [Finding 2]
3. [Finding 3]

## Visualizations

[Reference to saved figures]
- Figure 1: [description]
- Figure 2: [description]

## Deviations from Specification

[None / List of changes]

## Preliminary Interpretation

[2-3 paragraphs of initial interpretation with caveats]

## Concerns / Questions

- [Concern 1]
- [Concern 2]

## Next Steps for Validation

Based on these results, validation should focus on:
1. [Validation priority 1]
2. [Validation priority 2]

Quality Checks Before Proceeding

Before declaring Phase 3 complete:

• [ ] Model converged appropriately
• [ ] Results saved with version info
• [ ] Random seed documented and used
• [ ] All deviations documented
• [ ] Initial visualizations created
• [ ] Topic labels proposed (if applicable)
• [ ] No obvious errors or artifacts

When You're Done

Return a summary to the orchestrator that includes: 1. Model fit assessment (did it work?) 2. Key results summary (topics, performance, distributions) 3. Any deviations from specification 4. Preliminary interpretation 5. Concerns requiring attention in validation

Do not proceed to Phase 4 until the user reviews these results.

Phase 4: Validation & Robustness

You are executing Phase 4 of a computational text analysis. Your goal is to validate findings through human assessment and computational diagnostics, and test robustness to analytical choices.

Why This Phase Matters

Algorithmic output is not ground truth. Topic models find patterns—but are they meaningful patterns? Classifiers achieve accuracy—but do they capture what you intend? This phase establishes that results are valid and robust, not artifacts of method choices.

Technique Guides

Consult validation guide in text-concepts/:

• 06_validation_strategies.md - comprehensive validation approaches

Implementation guides for diagnostics:

• R: text-r-techniques/03_topic_models.md (coherence, exclusivity)
• Python: text-python-techniques/03_topic_models.md (coherence, c_v)

Your Tasks

1. Human Validation

For topic models - Topic Intrusion Test:

Select N topics. For each topic: 1. Show top 10-15 words 2. Add one "intruder" word from another topic 3. Ask human coders to identify the intruder 4. High accuracy = coherent topics

For topic models - Document Reading:

For each topic: 1. Sample 5-10 highest-probability documents 2. Read documents 3. Assess: Does topic label fit these documents? 4. Note: Are there false positives? Missing themes?

## Topic Validation: Topic 3 "Economic Policy"

### Top Words
tax, economy, budget, spending, fiscal, growth...

### Sample Documents Reviewed
| Doc ID | Topic Prob | Label Fits? | Notes |
|--------|------------|-------------|-------|
| 1234 | 0.85 | Yes | Clearly about tax policy |
| 2345 | 0.72 | Partially | Mixed with healthcare |
| 3456 | 0.68 | Yes | Budget discussion |
| ... | | | |

### Assessment
- Label accuracy: X/N documents
- Refinements needed: [suggestions]

For classification - Error Analysis:

1. Sample misclassified documents 2. For each error:

• Why did the model fail?
• Is the gold label correct?
• Is this a systematic error?

## Error Analysis

### False Positives (predicted [Class], actual [Other])
| Doc ID | Predicted | Actual | Why Misclassified |
|--------|-----------|--------|-------------------|
| 1234 | Class A | Class B | Shared vocabulary |
| ... | | | |

### False Negatives (predicted [Other], actual [Class])
| Doc ID | Predicted | Actual | Why Missed |
|--------|-----------|--------|------------|
| 5678 | Class B | Class A | Subtle example |
| ... | | | |

### Systematic Patterns
- [Pattern 1]
- [Pattern 2]

For dictionary - KWIC Validation:

For key terms in dictionary: 1. Sample uses in corpus 2. Assess: Is this the intended meaning? 3. Note domain-specific usages

## Dictionary Term Validation: "positive"

### Sample Uses
| Doc ID | Context | Valid? |
|--------|---------|--------|
| 1234 | "...test came back positive..." | No (medical) |
| 2345 | "...positive economic outlook..." | Yes |
| 3456 | "...positive feedback loop..." | No (technical) |

### Validity Rate: X/N valid uses
### Action: [Keep / Remove / Add to exceptions]

2. Computational Diagnostics

Topic Model Diagnostics:

# R example for STM
exclusivity <- exclusivity(stm_model)
coherence <- semanticCoherence(stm_model, out$documents)

# Plot coherence vs exclusivity
plot(coherence, exclusivity,
     xlab = "Semantic Coherence",
     ylab = "Exclusivity")

# Python example for gensim
from gensim.models import CoherenceModel

coherence_model = CoherenceModel(
    model=lda_model,
    texts=tokenized_docs,
    coherence='c_v'
)
coherence_score = coherence_model.get_coherence()

Classification Diagnostics:

Check for:

• Overfitting (train >> validation)
• Class imbalance effects
• Confidence calibration

Dictionary Diagnostics:

3. Robustness Checks

Run pre-specified robustness checks from Phase 2:

Sensitivity to K (topic models):

Sensitivity to preprocessing:

Sensitivity to random seed:

For topic models: Do the same topics emerge? Check topic alignment.

Subset analysis:

4. Alternative Methods (if applicable)

Compare to alternative approaches:

Example: Compare dictionary sentiment to ML sentiment.

5. Assess Overall Validity

Validation Summary Table:

Overall assessment:

• Are findings valid? [Yes/Partially/Concerns]
• What caveats are needed?
• What cannot be claimed?

Output: Validation Report

Create memos/phase4-validation-report.md:

# Validation Report

## Human Validation

### Topic/Category Assessment
[Summary of human coding]

### Inter-rater Reliability
[If multiple coders: Kappa, agreement %]

## Computational Diagnostics

### Model Fit Metrics
| Metric | Value | Assessment |
|--------|-------|------------|
| | | |

### Diagnostic Visualizations
[Reference figures]

## Robustness Analysis

### Sensitivity to K
[Results table and interpretation]

### Sensitivity to Preprocessing
[Results table and interpretation]

### Sensitivity to Random Seed
[Results table and interpretation]

### Subset Analysis
[Results table and interpretation]

## Alternative Methods
[If applicable]

## Validity Assessment

### Strengths
- [Strength 1]
- [Strength 2]

### Limitations
- [Limitation 1]
- [Limitation 2]

### Required Caveats for Interpretation
1. [Caveat 1]
2. [Caveat 2]

### Claims That Cannot Be Made
- [Cannot claim 1]
- [Cannot claim 2]

## Recommendation
[Proceed to output / Revise analysis / Major concerns]

When You're Done

Return a summary to the orchestrator that includes: 1. Human validation results (what proportion validated?) 2. Key diagnostic metrics 3. Robustness assessment (are results stable?) 4. Required caveats and limitations 5. Recommendation for proceeding

Do not proceed to Phase 5 until the user reviews validation results.

Phase 5: Output & Interpretation

You are executing Phase 5 of a computational text analysis. Your goal is to produce publication-ready outputs and write a careful, appropriately caveated interpretation of findings.

Why This Phase Matters

Text analysis results require careful interpretation. Overclaimingundermines credibility. This phase produces polished outputs and ensures the narrative matches what the evidence supports.

Technique Guides

Consult visualization guides for your language:

• R: text-r-techniques/06_visualization.md
• Python: text-python-techniques/06_visualization.md

Your Tasks

1. Create Publication-Quality Tables

Topic Model Results Table:

Notes: K = [N] topics. FREX words balance frequency and exclusivity. N = [documents].

Classification Results Table:

Notes: 5-fold stratified cross-validation. Features: [description].

Dictionary/Sentiment Summary:

Notes: [Dictionary name]. Score range: [X to Y].

2. Create Publication-Quality Figures

Topic models - required figures:

1. Topic prevalence (bar chart or dot plot)

• Ordered by prevalence
• Include uncertainty intervals if available

2. Topic content (word clouds or bar charts)

• Top words per topic
• Consider FREX for STM

3. Topic relationships (if relevant)

• Topic correlation network
• Hierarchical clustering

4. Topic trends (if temporal)

• Prevalence over time
• Confidence bands

Classification - required figures:

1. Confusion matrix (heatmap)

• Normalized by row (recall focus) or column (precision focus)
• Include raw counts

2. Feature importance (if interpretable model)

• Top predictive words per class
• Coefficients with confidence intervals

Dictionary/Sentiment - required figures:

1. Distribution (histogram or density)

• By group if comparing

2. Time series (if temporal)

• Smoothed trends
• Confidence bands

3. Write Results Narrative

Structure the narrative:

Opening:

• Remind reader of the research question
• Briefly state the approach

Main findings:

• Present results without overstating
• Use hedged language appropriately
• Connect to tables and figures

Validation summary:

• Briefly note validation approach
• Report key diagnostics

Limitations:

• Acknowledge methodological limitations
• Note what the analysis cannot show

Language Guidelines

Avoid:

• "The topic model discovered..."
• "The algorithm found that..."
• "This proves..."
• "Clearly..."
• "Obviously..."

Prefer:

• "The analysis suggests..."
• "Patterns in the data indicate..."
• "One interpretation is..."
• "This is consistent with..."
• "The evidence supports..."

Topic model language:

• Topics are "characterized by" words, not "about" concepts
• Topics "tend to appear in" documents, not "represent" ideas
• Prevalence is "estimated" with uncertainty

Classification language:

• "The classifier achieved X accuracy on held-out data"
• "Misclassifications tended to occur when..."
• Performance is on "this corpus," not "in general"

Dictionary language:

• "Documents mentioning X words..."
• Coverage and limitations should be noted
• "According to this measure..."

4. Write Limitations Section

Every text analysis has limitations. Document:

Method limitations:

• Topic models: K is a choice, not a truth
• Classification: Performance depends on training data
• Dictionary: Coverage and domain validity

Data limitations:

• Corpus scope: Findings apply to this corpus
• Selection: How texts were selected/sampled
• Quality: OCR errors, missing data

Interpretation limitations:

• Topics are statistical patterns, not concepts
• High probability ≠ topic is "about" that theme
• Classifiers learn correlations, not causation

5. Prepare Replication Materials

Create replication package:

replication/
├── README.md           # Instructions
├── requirements.R      # or requirements.txt
├── 01_preprocess.R     # or .py
├── 02_analysis.R       # or .py
├── 03_validation.R     # or .py
├── 04_figures.R        # or .py
└── session_info.txt    # Package versions

README.md for replication:

# Replication Materials

## Requirements
- R version X.X.X (or Python X.X)
- Packages: [list with versions]

## Data
Data files should be placed in `data/raw/`:
- [file1.csv]: [description]
- [file2.csv]: [description]

Note: Original data [is/is not] included due to [access/size/privacy].

## Replication Steps
1. Install requirements: `source("requirements.R")`
2. Preprocess data: `source("01_preprocess.R")`
3. Run analysis: `source("02_analysis.R")`
4. Validate: `source("03_validation.R")`
5. Generate figures: `source("04_figures.R")`

## Random Seed
All analyses use seed: [SEED]

## Expected Output
- [output1]: [description]
- [output2]: [description]

## Contact
[Author contact for questions]

6. Create Methods Section Draft

Write methods section following journal conventions:

## Text Analysis Methods

### Corpus
The corpus consists of N documents from [source],
spanning [time period]. Documents were [sampling description].

### Preprocessing
Text was preprocessed using [package/tool].
[Specific steps: tokenization, stopword removal, etc.]
Final vocabulary: N terms across N documents.

### Analysis
We used [method] implemented in [package] (version X.X).
[Key parameters: K topics, hyperparameters, etc.]
[Validation approach].

### Validation
[Human validation: sample size, procedure, results]
[Computational diagnostics: metrics, results]
[Robustness checks: what varied, results]

Output: Final Package

Create the following in output/:

output/
├── tables/
│   ├── table1_topic_summary.csv
│   ├── table2_prevalence.csv
│   └── ...
├── figures/
│   ├── fig1_topic_prevalence.pdf
│   ├── fig2_topic_words.pdf
│   └── ...
├── narrative/
│   ├── results_section.md
│   ├── methods_section.md
│   └── limitations.md
└── replication/
    └── [replication package]

Final Memo

Create memos/phase5-output-memo.md:

# Output Summary

## Deliverables

### Tables
1. [Table 1]: [description, location]
2. [Table 2]: [description, location]

### Figures
1. [Figure 1]: [description, location]
2. [Figure 2]: [description, location]

### Narrative Sections
- Results: [location]
- Methods: [location]
- Limitations: [location]

### Replication Materials
[Location, completeness status]

## Key Messages

### Main Findings (1-3 sentences)
[Summary of what the analysis shows]

### Required Caveats
1. [Caveat 1]
2. [Caveat 2]

### Claims Supported by Evidence
- [Claim 1]: [evidence]
- [Claim 2]: [evidence]

### Claims NOT Supported
- [Cannot claim 1]: [why not]
- [Cannot claim 2]: [why not]

## Quality Checklist

- [ ] All tables formatted consistently
- [ ] All figures publication-ready (300 dpi, clear labels)
- [ ] Narrative uses appropriate hedging
- [ ] Limitations section complete
- [ ] Replication package tested
- [ ] Methods section matches actual analysis
- [ ] Random seeds documented throughout

When You're Done

Return a summary to the orchestrator that includes: 1. List of deliverables produced 2. Key findings summary (2-3 sentences) 3. Main limitations to acknowledge 4. Replication package status 5. Any remaining concerns

Analysis is complete when user accepts the outputs.

Text Preprocessing in Python

Package Versions

# Tested with:
# Python 3.10
# nltk 3.8.1
# spacy 3.6.0
# scikit-learn 1.3.0

Installation

pip install nltk spacy scikit-learn pandas

# Download spaCy model
python -m spacy download en_core_web_sm

# Download NLTK data
python -c "import nltk; nltk.download('punkt'); nltk.download('stopwords'); nltk.download('wordnet')"

Three Approaches: NLTK vs spaCy vs sklearn

NLTK Workflow

Basic Tokenization

import nltk
from nltk.tokenize import word_tokenize, sent_tokenize
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer, WordNetLemmatizer

# Sample texts
texts = [
    "The quick brown fox jumps over the lazy dog.",
    "Machine learning is transforming social science research.",
    "Text analysis requires careful preprocessing decisions."
]

# Word tokenization
tokens = [word_tokenize(text.lower()) for text in texts]
print(tokens[0])
# ['the', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog', '.']

# Sentence tokenization
sentences = [sent_tokenize(text) for text in texts]

Stopword Removal

stop_words = set(stopwords.words('english'))

# Remove stopwords
tokens_clean = [[w for w in doc if w not in stop_words and w.isalpha()]
                for doc in tokens]

print(tokens_clean[0])
# ['quick', 'brown', 'fox', 'jumps', 'lazy', 'dog']

# Custom stopwords
custom_stops = {'data', 'analysis', 'research'}
stop_words = stop_words.union(custom_stops)

Stemming and Lemmatization

# Stemming (Porter)
stemmer = PorterStemmer()
tokens_stemmed = [[stemmer.stem(w) for w in doc] for doc in tokens_clean]

# Lemmatization
lemmatizer = WordNetLemmatizer()
tokens_lemma = [[lemmatizer.lemmatize(w) for w in doc] for doc in tokens_clean]

# Compare
print(f"Original: jumps")
print(f"Stemmed: {stemmer.stem('jumps')}")     # jump
print(f"Lemmatized: {lemmatizer.lemmatize('jumps', pos='v')}")  # jump

N-grams

from nltk import ngrams

# Bigrams
bigrams = [list(ngrams(doc, 2)) for doc in tokens_clean]
print(bigrams[0])
# [('quick', 'brown'), ('brown', 'fox'), ...]

# Trigrams
trigrams = [list(ngrams(doc, 3)) for doc in tokens_clean]

spaCy Workflow

Basic Processing

import spacy

# Load model
nlp = spacy.load('en_core_web_sm')

# Process text
doc = nlp("Machine learning is transforming social science research.")

# Tokens with attributes
for token in doc:
    print(f"{token.text:15} {token.pos_:8} {token.lemma_:15} {token.is_stop}")

Complete Preprocessing Pipeline

def preprocess_spacy(texts, nlp, remove_stops=True, lemmatize=True):
    """
    Preprocess texts using spaCy.

    Parameters:
    -----------
    texts : list of str
    nlp : spacy model
    remove_stops : bool
    lemmatize : bool

    Returns:
    --------
    list of list of str : processed tokens
    """
    processed = []
    for doc in nlp.pipe(texts, batch_size=50):
        tokens = []
        for token in doc:
            # Skip punctuation, spaces, and optionally stopwords
            if token.is_punct or token.is_space:
                continue
            if remove_stops and token.is_stop:
                continue

            # Lemmatize or use original
            if lemmatize:
                tokens.append(token.lemma_.lower())
            else:
                tokens.append(token.text.lower())

        processed.append(tokens)

    return processed

# Usage
tokens = preprocess_spacy(texts, nlp)
print(tokens[1])
# ['machine', 'learn', 'transform', 'social', 'science', 'research']

Named Entity Recognition

doc = nlp("Apple is looking at buying U.K. startup for $1 billion")

for ent in doc.ents:
    print(f"{ent.text:20} {ent.label_:10} {spacy.explain(ent.label_)}")

# Apple                ORG        Companies, agencies, institutions
# U.K.                 GPE        Countries, cities, states
# $1 billion           MONEY      Monetary values

Custom Pipeline Component

from spacy.language import Language

@Language.component("custom_cleaner")
def custom_cleaner(doc):
    """Remove URLs and email addresses."""
    cleaned_tokens = []
    for token in doc:
        if not token.like_url and not token.like_email:
            cleaned_tokens.append(token)
    return doc

# Add to pipeline
nlp.add_pipe("custom_cleaner", after="parser")

sklearn Workflow

CountVectorizer

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

# Basic count vectorizer
count_vec = CountVectorizer(
    lowercase=True,
    stop_words='english',
    max_features=1000,
    min_df=2,           # Minimum document frequency
    max_df=0.95,        # Maximum document frequency
    ngram_range=(1, 2)  # Unigrams and bigrams
)

# Fit and transform
dtm = count_vec.fit_transform(texts)

# Inspect
print(f"Shape: {dtm.shape}")
print(f"Vocabulary size: {len(count_vec.vocabulary_)}")
print(f"Feature names: {count_vec.get_feature_names_out()[:10]}")

TF-IDF Vectorizer

tfidf_vec = TfidfVectorizer(
    lowercase=True,
    stop_words='english',
    max_features=1000,
    min_df=2,
    max_df=0.95,
    ngram_range=(1, 1),
    sublinear_tf=True   # Apply log to TF
)

tfidf_matrix = tfidf_vec.fit_transform(texts)

Custom Tokenizer with sklearn

def custom_tokenizer(text):
    """Custom tokenizer with spaCy."""
    doc = nlp(text)
    return [token.lemma_.lower() for token in doc
            if not token.is_stop and not token.is_punct and token.is_alpha]

# Use in vectorizer
custom_vec = TfidfVectorizer(tokenizer=custom_tokenizer)
custom_matrix = custom_vec.fit_transform(texts)

Handling Special Cases

URLs and HTML

import re

def clean_text(text):
    """Basic text cleaning."""
    # Remove URLs
    text = re.sub(r'https?://\S+|www\.\S+', '', text)

    # Remove HTML tags
    text = re.sub(r'<[^>]+>', '', text)

    # Remove special characters but keep basic punctuation
    text = re.sub(r'[^\w\s.,!?]', '', text)

    # Remove extra whitespace
    text = ' '.join(text.split())

    return text

# Apply
texts_clean = [clean_text(t) for t in texts]

Handling Emojis and Special Characters

import emoji

def handle_emojis(text, mode='remove'):
    """Handle emojis in text."""
    if mode == 'remove':
        return emoji.replace_emoji(text, '')
    elif mode == 'demojize':
        return emoji.demojize(text)
    return text

# Example
text_with_emoji = "Great product! 😊👍"
print(handle_emojis(text_with_emoji, 'demojize'))
# Great product! :smiling_face_with_smiling_eyes::thumbs_up:

Encoding Issues

def fix_encoding(text):
    """Fix common encoding issues."""
    # Handle None
    if text is None:
        return ""

    # Ensure string
    if not isinstance(text, str):
        text = str(text)

    # Fix encoding
    try:
        text = text.encode('utf-8', errors='ignore').decode('utf-8')
    except:
        text = ""

    return text

Document-Term Matrix Operations

Converting to Dense

import pandas as pd
import numpy as np

# Sparse to dense (only for small matrices!)
dense_matrix = dtm.toarray()

# As DataFrame
dtm_df = pd.DataFrame(
    dense_matrix,
    columns=count_vec.get_feature_names_out()
)

Vocabulary Statistics

def get_vocab_stats(vectorizer, dtm):
    """Get vocabulary statistics."""
    feature_names = vectorizer.get_feature_names_out()
    freqs = np.asarray(dtm.sum(axis=0)).ravel()
    doc_freqs = np.asarray((dtm > 0).sum(axis=0)).ravel()

    stats = pd.DataFrame({
        'term': feature_names,
        'total_freq': freqs,
        'doc_freq': doc_freqs,
        'doc_freq_pct': doc_freqs / dtm.shape[0]
    }).sort_values('total_freq', ascending=False)

    return stats

stats = get_vocab_stats(count_vec, dtm)
print(stats.head(20))

Complete Preprocessing Pipeline

import pandas as pd
from typing import List, Optional

class TextPreprocessor:
    """Complete text preprocessing pipeline."""

    def __init__(
        self,
        language: str = 'english',
        remove_stops: bool = True,
        lemmatize: bool = True,
        min_token_len: int = 2,
        custom_stops: Optional[List[str]] = None
    ):
        self.language = language
        self.remove_stops = remove_stops
        self.lemmatize = lemmatize
        self.min_token_len = min_token_len

        # Load spaCy
        self.nlp = spacy.load('en_core_web_sm', disable=['parser', 'ner'])

        # Set stopwords
        self.stop_words = set(stopwords.words(language))
        if custom_stops:
            self.stop_words.update(custom_stops)

    def clean_text(self, text: str) -> str:
        """Basic text cleaning."""
        if not isinstance(text, str):
            return ""

        # Remove URLs
        text = re.sub(r'https?://\S+', '', text)

        # Remove HTML
        text = re.sub(r'<[^>]+>', '', text)

        # Normalize whitespace
        text = ' '.join(text.split())

        return text

    def tokenize(self, text: str) -> List[str]:
        """Tokenize and optionally lemmatize."""
        doc = self.nlp(text)
        tokens = []

        for token in doc:
            # Skip punctuation and spaces
            if token.is_punct or token.is_space:
                continue

            # Skip stopwords
            if self.remove_stops and token.text.lower() in self.stop_words:
                continue

            # Get token text
            if self.lemmatize:
                tok = token.lemma_.lower()
            else:
                tok = token.text.lower()

            # Skip short tokens
            if len(tok) < self.min_token_len:
                continue

            # Only alphabetic
            if not tok.isalpha():
                continue

            tokens.append(tok)

        return tokens

    def preprocess(self, texts: List[str]) -> List[List[str]]:
        """Full preprocessing pipeline."""
        processed = []
        for text in texts:
            clean = self.clean_text(text)
            tokens = self.tokenize(clean)
            processed.append(tokens)
        return processed

    def get_stats(self, processed: List[List[str]]) -> dict:
        """Get corpus statistics."""
        all_tokens = [t for doc in processed for t in doc]
        return {
            'n_documents': len(processed),
            'n_tokens': len(all_tokens),
            'n_types': len(set(all_tokens)),
            'mean_doc_length': len(all_tokens) / len(processed),
            'empty_docs': sum(1 for doc in processed if len(doc) == 0)
        }

# Usage
preprocessor = TextPreprocessor(
    remove_stops=True,
    lemmatize=True,
    custom_stops=['data', 'study']
)

processed = preprocessor.preprocess(texts)
stats = preprocessor.get_stats(processed)
print(stats)

Best Practices

1. Document all decisions - Keep preprocessing log 2. Start minimal - Add processing steps only if needed 3. Check vocabulary - Examine most/least frequent terms 4. Use batching - Process in batches for large corpora 5. Save intermediate results - For reproducibility 6. Set random seeds - For any stochastic elements

# Preprocessing log
preprocessing_log = {
    'date': pd.Timestamp.now().isoformat(),
    'n_docs_raw': len(texts),
    'n_docs_processed': len(processed),
    'vocab_size': stats['n_types'],
    'stopwords': 'english + custom',
    'lemmatization': 'spacy',
    'min_token_length': 2
}

# Save
pd.Series(preprocessing_log).to_json('data/processed/preprocessing_log.json')