
Microarray Analysis
- 1 installs
- 1 repo stars
- Updated February 8, 2026
- gexijin/vitiligo
Analyzes DNA microarray gene-expression data from GEO datasets, handling Affymetrix and Illumina platforms with QC, normalization, differential expression, and gene ID mapping.
About
Guides differential-expression analysis of DNA microarray data from GEO accessions across Affymetrix and Illumina platforms using numbered R scripts. A developer or bioinformatician uses it when analyzing microarray gene-expression data.
- Prescribes a numbered R script structure with QC, normalization, DE, annotation, and GO enrichment
- Uses the here package for portable paths and a standard analyses directory layout
Microarray Analysis by the numbers
- 1 all-time installs (skills.sh)
- Ranked #1,803 of 2,064 Data Science & ML skills by installs in the Skillselion catalog
- Data as of Jul 24, 2026 (Skillselion catalog sync)
npx skills add https://github.com/gexijin/vitiligo --skill microarray-analysisAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 1 |
|---|---|
| repo stars | ★ 1 |
| Last updated | February 8, 2026 |
| Repository | gexijin/vitiligo ↗ |
What it does
Analyzes DNA microarray gene-expression data from GEO datasets, handling Affymetrix and Illumina platforms with QC, normalization, differential expression, and gene ID mapping.
Files
DNA Microarray Analysis Skill
This skill guides analysis of DNA microarray gene expression data following established patterns from this project.
Quick Start
When asked to analyze microarray data:
1. Check if a CLAUDE.md file exists in the study directory - it contains study-specific metadata 2. Determine the platform (Affymetrix vs Illumina) to select appropriate methods 3. Create numbered scripts in scripts/ directory (01_, 02_, etc.) 4. Update analysis.md after each analysis step 5. Generate all plots in results/plots/ with plots/plots.md documentation
Directory Structure
Create this structure for each analysis:
analyses/[FirstAuthor]_[GSE#]/
├── CLAUDE.md # Study metadata (samples, platform, design)
├── analysis.md # Progressive documentation (update after each step)
├── scripts/
│ ├── 01_load_and_qc.R
│ ├── 02_normalize.R (or 02_normalize_filtered.R)
│ ├── 03_differential_expression.R
│ ├── 04_annotate_genes.R
│ └── 05_go_enrichment.R
├── data/ # Raw and processed data (gitignored)
└── results/
├── tables/ # CSV output files
└── plots/ # PNG/PDF figures
└── plots.md # Plot documentationScript Portability
Use the here package for all file paths to ensure scripts work regardless of working directory:
library(here)
# Set paths relative to project root
base_dir <- here("analyses", "Author_GSE#####")
data_dir <- file.path(base_dir, "data")
results_dir <- file.path(base_dir, "results")
plots_dir <- file.path(results_dir, "plots")
tables_dir <- file.path(results_dir, "tables")
# Source utility functions from project root
source(here("scripts", "utils", "collapse_to_gene.R"))
source(here("scripts", "utils", "enrichment_dotplot.R"))Never use hardcoded absolute paths like /Users/name/path/to/file. Always use here() for portability.
Probe Tracking
IMPORTANT: Track probe counts at each processing step and report a summary table at the end of QC and normalization scripts. This provides transparency about data filtering.
Initialize Tracking
# Initialize probe tracking at the start of script
probe_tracking <- data.frame(
step = character(),
n_probes = integer(),
n_removed = integer(),
description = character(),
stringsAsFactors = FALSE
)Track at Each Step
Add tracking after each filtering/processing step:
# After loading raw data
probe_tracking <- rbind(probe_tracking, data.frame(
step = "1. Raw data loaded",
n_probes = nrow(raw_data),
n_removed = 0,
description = "Total probes in raw data file"
))
# After creating expression matrix
probe_tracking <- rbind(probe_tracking, data.frame(
step = "2. Expression matrix created",
n_probes = nrow(expr_mat),
n_removed = nrow(raw_data) - nrow(expr_mat),
description = "Probes with valid signal columns"
))
# After detection filtering (Illumina)
probe_tracking <- rbind(probe_tracking, data.frame(
step = "3. Detection filter",
n_probes = nrow(expr_filtered),
n_removed = nrow(expr_mat) - nrow(expr_filtered),
description = sprintf("Probes detected (p < 0.05) in >= %d samples", min_samples_detected)
))
# After low-expression filtering
probe_tracking <- rbind(probe_tracking, data.frame(
step = "4. Expression filter",
n_probes = nrow(expr_filtered),
n_removed = n_before - nrow(expr_filtered),
description = "Probes with expression above threshold"
))Report Summary
At the end of QC script:
# Calculate percentage remaining
probe_tracking$pct_remaining <- round(100 * probe_tracking$n_probes / probe_tracking$n_probes[1], 1)
cat("\n=== Probe Tracking Summary ===\n\n")
print(probe_tracking)
# Save to file
write.csv(probe_tracking, file.path(tables_dir, "probe_tracking.csv"), row.names = FALSE)Expected Output
| Step | Probes | Removed | % Remaining | Description |
|---|---|---|---|---|
| 1. Raw data loaded | 29,377 | 0 | 100.0% | Total probes in raw data file |
| 2. Expression matrix | 29,377 | 0 | 100.0% | Probes with valid signal columns |
| 3. Detection filter | 20,233 | 9,144 | 68.9% | Probes detected in >= 3 samples |
NA Value Checking
IMPORTANT: Check for NA values in both signal and detection p-value matrices early in the QC script. This identifies data quality issues before they cause downstream problems.
# Check for NAs in signal data
na_signal <- is.na(expr_mat)
n_na_signal_total <- sum(na_signal)
n_na_signal_by_sample <- colSums(na_signal)
n_na_signal_by_probe <- rowSums(na_signal)
cat(sprintf("Signal data - Total NA values: %d (%.2f%%)\n",
n_na_signal_total,
100 * n_na_signal_total / length(expr_mat)))
if (n_na_signal_total > 0) {
cat("NA counts by sample (signal):\n")
for (i in seq_along(sample_names)) {
if (n_na_signal_by_sample[i] > 0) {
cat(sprintf(" %s: %d NAs (%.2f%%)\n",
sample_names[i],
n_na_signal_by_sample[i],
100 * n_na_signal_by_sample[i] / nrow(expr_mat)))
}
}
cat(sprintf("Probes with at least one NA: %d (%.2f%%)\n",
sum(n_na_signal_by_probe > 0),
100 * sum(n_na_signal_by_probe > 0) / nrow(expr_mat)))
}
# Check for NAs in detection p-values (Illumina)
na_detect <- is.na(detect_mat)
n_na_detect_total <- sum(na_detect)
# ... similar reporting
# Create NA summary table
na_summary <- data.frame(
sample = sample_names,
na_signal_count = n_na_signal_by_sample,
na_signal_pct = 100 * n_na_signal_by_sample / nrow(expr_mat),
na_detect_count = n_na_detect_by_sample,
na_detect_pct = 100 * n_na_detect_by_sample / nrow(detect_mat)
)
write.csv(na_summary, file.path(tables_dir, "01_na_value_summary.csv"), row.names = FALSE)NA Visualization
If NA values are present, generate diagnostic plots:
if (n_na_signal_total > 0 || n_na_detect_total > 0) {
# Bar plot comparing signal vs detection NAs by sample
na_summary_long <- na_summary %>%
pivot_longer(cols = c(na_signal_count, na_detect_count),
names_to = "data_type", values_to = "na_count")
p_na <- ggplot(na_summary_long, aes(x = sample, y = na_count, fill = data_type)) +
geom_bar(stat = "identity", position = "dodge") +
theme_minimal() +
theme(axis.text.x = element_text(angle = 45, hjust = 1))
ggsave(file.path(plots_dir, "01_na_comparison.png"), p_na, width = 12, height = 6)
# Optional: Heatmap of NA patterns if not too many probes affected
if (sum(n_na_signal_by_probe > 0) < 1000) {
# Create binary NA heatmap with pheatmap
}
}Handling NAs in Downstream Analysis
- Use
na.rm = TRUEfor summary statistics:colMeans(expr_mat, na.rm = TRUE) - Clean data before correlation/PCA:
expr_clean <- expr_mat[complete.cases(expr_mat), ] - Document the number of probes removed due to NA values in the probe tracking table
Platform Detection and Data Loading
Affymetrix Arrays (CEL files)
Use affy and affyPLM packages:
library(affy)
library(affyPLM)
# Load CEL files
raw_data <- ReadAffy(filenames = cel_files)
# Basic QC
pm_data <- pm(raw_data)
mm_data <- mm(raw_data)
avg_pm <- colMeans(pm_data)
# MAS5 present/absent calls
calls <- mas5calls(raw_data)
percent_present <- colSums(exprs(calls) == "P") / nrow(exprs(calls)) * 100
# PLM QC (NUSE and RLE)
plm_fit <- fitPLM(raw_data)
nuse_vals <- NUSE(plm_fit, type = "values")
rle_vals <- RLE(plm_fit, type = "values")
# RMA normalization
eset <- rma(raw_data)Annotation packages by platform:
- GPL570 (HG-U133_Plus_2):
hgu133plus2.db - GPL96 (HG-U133A):
hgu133a.db - GPL97 (HG-U133B):
hgu133b.db
Illumina BeadChip Arrays
Use limma with detection p-values:
library(limma)
# Data typically has AVG_Signal and Detection Pval columns
expr_mat <- as.matrix(data[, signal_cols])
detect_mat <- as.matrix(data[, pval_cols])
# Filter by detection (p < 0.05 in >= 3 samples)
probes_detected <- rowSums(detect_mat < 0.05) >= 3
expr_filtered <- expr_mat[probes_detected, ]
# Log2 transform and quantile normalize
expr_log2 <- log2(expr_filtered + 1)
expr_norm <- normalizeBetweenArrays(expr_log2, method = "quantile")Annotation packages by platform:
- GPL14951 (HumanHT-12 v4):
illuminaHumanv4.db - GPL10558 (HumanHT-12 v4.0):
illuminaHumanv4.db - GPL6884 (HumanHT-12 v3):
illuminaHumanv3.db
Quality Control Plots
Generate these QC plots (numbered sequentially):
1. Raw intensity boxplot - Distribution per sample 2. Raw intensity density - Overlapping density curves 3. Percent present/detected - Bar plot per sample 4. Average intensity - Bar plot per sample 5. Sample correlation heatmap - Hierarchical clustering 6. PCA plot - PC1 vs PC2 colored by condition 7. QC metrics by condition - Boxplots comparing groups
Post-normalization visualization:
- Overall expression heatmap (e.g., 19b) - Top 2000 most variable genes
- Median-centered, ±3 SD clipping for color utilization
- 1 - Pearson correlation distance, average linkage for gene clustering
- Samples ordered by condition (not clustered)
- Green-black-red color scheme with condition color bars
- Sample dendrogram (e.g., 19c) - Hierarchical clustering of all samples
- Uses all genes with 1 - Pearson correlation distance
- Average linkage, horizontal layout
- Branches and labels colored by sample condition
Affymetrix-specific:
- RNA degradation plot - 5' to 3' bias
- NUSE boxplot - Outliers if median > 1.05
- RLE boxplot - Outliers if |median| > 0.1
Outlier Detection Criteria
| Metric | Threshold | Action |
|---|---|---|
| NUSE median | > 1.05 | Flag as outlier |
| NUSE IQR | > 0.15 | Flag as outlier |
| RLE median | value | |
| RLE IQR | > 0.3 | Flag as outlier |
| Intensity z-score | z | |
| PCA extreme | Visual outlier | Investigate |
After excluding outliers: Re-run normalization (especially RMA which uses quantile normalization).
Statistical Modeling with limma
Paired Design (same subjects, multiple conditions)
For matched samples (e.g., lesional vs non-lesional from same patient):
# Create factors
tissue <- factor(sample_info$condition, levels = c("NonLesional", "Lesional"))
patient <- factor(sample_info$patient_id)
# Design with patient as fixed effect (n >= 10 patients)
design <- model.matrix(~ patient + tissue)
# Fit model
fit <- lmFit(expr_filtered, design)
# Contrasts (tissue effects are relative to baseline)
contrast_matrix <- makeContrasts(
Lesional_vs_NonLesional = tissueLesional,
levels = design
)
fit2 <- contrasts.fit(fit, contrast_matrix)
fit2 <- eBayes(fit2)Unpaired Design (independent groups)
For independent samples:
# Create factor
group <- factor(sample_info$condition, levels = c("Control", "Treatment"))
# Cell means model (no intercept)
design <- model.matrix(~ 0 + group)
colnames(design) <- levels(group)
# Fit model
fit <- lmFit(expr_filtered, design)
# Define contrast
contrast_matrix <- makeContrasts(
Treatment_vs_Control = Treatment - Control,
levels = design
)
fit2 <- contrasts.fit(fit, contrast_matrix)
fit2 <- eBayes(fit2)Interaction Terms (genotype x treatment)
For factorial designs:
# Create combined factor
group <- factor(paste(sample_info$genotype, sample_info$treatment, sep = "_"))
design <- model.matrix(~ 0 + group)
colnames(design) <- levels(group)
# Define biologically meaningful contrasts
contrast_matrix <- makeContrasts(
Treatment_in_WT = WT_Treated - WT_Control,
Treatment_in_Mut = Mut_Treated - Mut_Control,
Interaction = (Mut_Treated - Mut_Control) - (WT_Treated - WT_Control),
levels = design
)Model Diagnostics
Always generate these diagnostic plots:
1. P-value histogram - Should show uniform + spike near 0 2. MA plot - Check for intensity-dependent bias 3. Mean-variance trend (plotSA) - Validates eBayes assumptions
# P-value histogram
png(file.path(plots_dir, "03_pvalue_histogram.png"), width = 800, height = 600)
hist(results$P.Value, breaks = 50,
main = "P-value Distribution", xlab = "P-value",
col = "steelblue", border = "white")
abline(h = nrow(results) / 50, col = "red", lty = 2)
legend("topright", "Expected under null", col = "red", lty = 2)
dev.off()
# MA plot
png(file.path(plots_dir, "03_MA_plot.png"), width = 800, height = 600)
plotMA(fit2, coef = "contrast_name", main = "MA Plot")
dev.off()
# Mean-variance trend (validates eBayes assumptions)
png(file.path(plots_dir, "03_mean_variance_trend.png"), width = 800, height = 600)
plotSA(fit2, main = "Mean-Variance Trend (eBayes)")
dev.off()Volcano Plot Best Practices
Important: The y-axis should show FDR (adjusted p-value), not raw p-value:
p_volcano <- ggplot(results, aes(x = logFC, y = -log10(adj.P.Val), color = significance)) +
geom_point(alpha = 0.5, size = 1.5) +
geom_vline(xintercept = c(-1, 1), linetype = "dashed", color = "gray40") +
geom_hline(yintercept = -log10(0.05), linetype = "dashed", color = "gray40") +
labs(title = "Volcano Plot",
x = "log2 Fold Change",
y = "-log10(FDR)") # Use FDR, not P-valueGene ID Mapping
Using the collapse_to_gene Utility Function
Use the shared utility function scripts/utils/collapse_to_gene.R for probe-to-gene mapping. This function: 1. Queries the annotation database for the specified gene ID type 2. Removes probes that map to multiple gene IDs (ambiguous mapping) 3. Keeps the most significant probe per gene (by P.Value, |logFC| tiebreaker)
source(here("scripts", "utils", "collapse_to_gene.R"))
# Basic usage - collapse to gene symbols
de_annotated <- collapse_to_gene(
de_results,
annot_pkg = "illuminaHumanv4.db", # Platform-specific annotation package
id_column = "SYMBOL",
probe_column = "ProbeID",
keytype = "PROBEID",
collapse = TRUE, # Set FALSE to keep all probes with annotations
verbose = TRUE
) %>%
rename(gene_symbol = SYMBOL) %>%
relocate(ProbeID, gene_symbol)
# Get probe-level annotations without collapsing
de_probe_level <- collapse_to_gene(
de_results,
annot_pkg = "illuminaHumanv4.db",
id_column = "SYMBOL",
probe_column = "ProbeID",
collapse = FALSE # Keep all probes
)Annotation Packages by Platform
| Platform | GPL ID | Annotation Package |
|---|---|---|
| Illumina HumanHT-12 v4 | GPL14951, GPL10558 | illuminaHumanv4.db |
| Illumina HumanHT-12 v3 | GPL6884 | illuminaHumanv3.db |
| Affymetrix HG-U133 Plus 2 | GPL570 | hgu133plus2.db |
| Affymetrix HG-U133A | GPL96 | hgu133a.db |
Manual Probe-to-Gene Mapping (Alternative)
If not using the utility function:
library(AnnotationDbi)
# Get mapping (platform-specific)
mapping <- AnnotationDbi::select(
platform.db,
keys = probe_ids,
columns = c("SYMBOL", "ENTREZID", "ENSEMBL", "GENENAME"),
keytype = "PROBEID"
)
# Join with DE results
de_annotated <- de_results |>
left_join(mapping, by = c("ProbeID" = "PROBEID"))
# Collapse to unique gene symbols (keep most significant probe)
de_clean_symbol <- de_annotated %>%
filter(!is.na(gene_symbol)) %>%
arrange(P.Value, -abs(logFC)) %>%
filter(!duplicated(gene_symbol)) %>%
arrange(adj.P.Val)Output Format
Differential Expression Results
Required columns in output CSV:
| Column | Description |
|---|---|
| ensembl_id | Ensembl gene ID (primary identifier) |
| gene_symbol | HGNC gene symbol |
| logFC | Log2 fold change |
| P.Value | Raw p-value |
| adj.P.Val | FDR-adjusted p-value |
| ProbeID | Original probe/probeset ID |
| AveExpr | Average expression |
| t | Moderated t-statistic |
| B | Log-odds of differential expression |
File Naming Convention
Use numbered prefixes (01_, 02_, etc.) to indicate the script that generated each file:
results/tables/
├── 01_sample_metadata.csv # From 01_load_and_qc.R
├── 01_qc_metrics.csv
├── 01_expression_matrix_raw.csv
├── 01_probe_tracking.csv
├── 01_na_value_summary.csv # NA diagnostics
├── 02_expression_matrix_normalized.csv # From 02_normalize.R
├── 02_normalization_summary.csv
├── 03_DE_[Comparison].csv # From 03_differential_expression.R
├── 04_DE_[Comparison]_annotated.csv # From 04_annotate_genes.R
├── 04_DE_[Comparison]_clean_symbol.csv # Unique gene symbols
├── 04_genes_upregulated.csv
├── 04_genes_downregulated.csv
├── 05_GO_BP_upregulated.csv # From 05_go_enrichment.R
├── 05_GO_BP_downregulated.csv
├── 05_KEGG_upregulated.csv
├── 05_KEGG_downregulated.csv
└── 05_GSEA_GO_BP.csvresults/plots/
├── 01_raw_intensity_boxplot.png # From 01_load_and_qc.R
├── 01_raw_intensity_density.png
├── 01_PCA_raw.png
├── 01_sample_correlation_heatmap.png
├── 01_na_comparison.png # If NAs present
├── 02_normalized_boxplot.png # From 02_normalize.R
├── 02_PCA_normalized.png
├── 03_pvalue_histogram.png # From 03_differential_expression.R
├── 03_MA_plot.png
├── 03_mean_variance_trend.png
├── 03_volcano_plot.png
├── 03_heatmap_top_DE.png
├── 05_GO_BP_combined.png # From 05_go_enrichment.R
├── 05_KEGG_combined.png
├── 05_GSEA_GO_BP_combined.png
├── 05_GSEA_GO_BP_dotplot.png
└── 05_GSEA_GO_BP_ridgeplot.pngNote: The _clean_symbol.csv file contains one row per gene (collapsed from multiple probesets), selected by lowest P.Value. Use this for downstream analysis.
Progressive Documentation (analysis.md)
Update analysis.md after EACH analysis step with:
# [GSE#] Microarray Analysis Report
## Study Information
| Field | Value |
|-------|-------|
| **GEO Accession** | GSE##### |
| **Platform** | GPL### [Platform Name] |
| **Analysis Date** | YYYY-MM-DD |
## Methods
### Software Environment
| Package | Version | Purpose |
|---------|---------|---------|
| limma | X.X.X | Differential expression |
| affy | X.X.X | CEL file processing |
### Key Parameters
- [List all thresholds and parameters used]
## Results
### Quality Control
- [Summarize QC findings]
- [Document any outliers removed]
### Differential Expression
| Comparison | Total DEGs | Up | Down |
|------------|------------|-----|------|
| ... | ... | ... | ... |
### Key Findings
- [Biological interpretation]
## Output Files
[List all generated files]Plot Documentation (plots/plots.md)
Create results/plots/plots.md with embedded image references so plots render in markdown viewers:
# Plot Documentation - [GSE#]
## Quality Control
### 01_raw_intensity_boxplot.png
Raw probe intensity distribution by sample before normalization.

### 02_raw_intensity_density.png
Density curves of raw intensities colored by condition.

[Continue for all plots with image references...]
## Differential Expression
### 15_volcano_plot.png
Volcano plot showing -log10(p-value) vs log2 fold change.
- Red: Upregulated (FDR < 0.05, |logFC| > 1)
- Blue: Downregulated (FDR < 0.05, |logFC| > 1)
Important: Use ./ prefix for relative paths (e.g., ) to ensure images render correctly in VS Code and other markdown viewers.
GO and KEGG Enrichment Analysis
Use clusterProfiler for pathway analysis. Run both GO Biological Process and KEGG pathway enrichment for both upregulated AND downregulated genes in each comparison.
Background Gene List
Important: Use all genes that passed the filtering step as background, loaded from the annotation file:
library(clusterProfiler)
library(org.Hs.eg.db)
library(ggplot2)
library(dplyr)
# Load background from annotation file (all filtered genes)
annotation_df <- read.csv(file.path(tables_dir, "probeset_annotation.csv"))
background_genes <- annotation_df %>%
filter(!is.na(entrez_id)) %>%
pull(entrez_id) %>%
unique() %>%
as.character()GO Biological Process Enrichment (Both Directions)
# Get Entrez IDs for UPREGULATED genes
sig_up <- de_results %>%
filter(adj.P.Val < 0.05 & logFC > 0 & !is.na(entrez_id)) %>%
pull(entrez_id) %>%
unique() %>%
as.character()
# Get Entrez IDs for DOWNREGULATED genes
sig_down <- de_results %>%
filter(adj.P.Val < 0.05 & logFC < 0 & !is.na(entrez_id)) %>%
pull(entrez_id) %>%
unique() %>%
as.character()
# GO enrichment - Upregulated
ego_up <- enrichGO(
gene = sig_up,
universe = background_genes,
OrgDb = org.Hs.eg.db,
keyType = "ENTREZID",
ont = "BP",
pAdjustMethod = "BH",
pvalueCutoff = 0.05,
readable = TRUE
)
ego_up_simp <- simplify(ego_up, cutoff = 0.7, by = "p.adjust")
# GO enrichment - Downregulated
ego_down <- enrichGO(
gene = sig_down,
universe = background_genes,
OrgDb = org.Hs.eg.db,
keyType = "ENTREZID",
ont = "BP",
pAdjustMethod = "BH",
pvalueCutoff = 0.05,
readable = TRUE
)
ego_down_simp <- simplify(ego_down, cutoff = 0.7, by = "p.adjust")KEGG Pathway Enrichment (Both Directions)
# KEGG - Upregulated
kegg_up <- enrichKEGG(
gene = sig_up,
universe = background_genes,
organism = "hsa",
keyType = "ncbi-geneid",
pAdjustMethod = "BH",
pvalueCutoff = 0.05
)
# KEGG - Downregulated
kegg_down <- enrichKEGG(
gene = sig_down,
universe = background_genes,
organism = "hsa",
keyType = "ncbi-geneid",
pAdjustMethod = "BH",
pvalueCutoff = 0.05
)Combined Plots with Fold Enrichment
Use the utility functions from scripts/utils/enrichment_dotplot.R for creating combined enrichment plots:
source(here("scripts/utils/enrichment_dotplot.R"))
# Helper function to prepare ORA data with fold enrichment
prepare_ora_data <- function(enrichResult, direction_label) {
if (is.null(enrichResult) || nrow(as.data.frame(enrichResult)) == 0) return(NULL)
df <- as.data.frame(enrichResult)
df <- df %>%
mutate(
gene_ratio_num = as.numeric(sub("/.*", "", GeneRatio)),
gene_ratio_denom = as.numeric(sub(".*/", "", GeneRatio)),
bg_ratio_num = as.numeric(sub("/.*", "", BgRatio)),
bg_ratio_denom = as.numeric(sub(".*/", "", BgRatio)),
GeneRatioValue = gene_ratio_num / gene_ratio_denom,
BgRatioValue = bg_ratio_num / bg_ratio_denom,
FoldEnrichment = GeneRatioValue / BgRatioValue,
direction = direction_label
)
return(df)
}
# Prepare data for plotting
go_up_ora <- prepare_ora_data(ego_up_simp, "UP")
go_down_ora <- prepare_ora_data(ego_down_simp, "DOWN")
go_combined_ora <- bind_rows(go_up_ora, go_down_ora)
# Create combined GO BP plot using utility function
p_go_combined <- plot_ora_dotplot_combined(
go_combined_ora,
n_show = 10,
fdr_cutoff = 0.05,
title = "GO Biological Process: Comparison",
width = 10,
height = 8
)
ggsave(file.path(plots_dir, "05_GO_BP_combined.png"), p_go_combined,
width = attr(p_go_combined, "width"),
height = attr(p_go_combined, "height"), dpi = 300)
# Create combined KEGG plot
kegg_up_ora <- prepare_ora_data(kegg_up, "UP")
kegg_down_ora <- prepare_ora_data(kegg_down, "DOWN")
kegg_combined_ora <- bind_rows(kegg_up_ora, kegg_down_ora)
p_kegg_combined <- plot_ora_dotplot_combined(
kegg_combined_ora,
n_show = 10,
fdr_cutoff = 0.05,
title = "KEGG Pathways: Comparison"
)
ggsave(file.path(plots_dir, "05_KEGG_combined.png"), p_kegg_combined,
width = attr(p_kegg_combined, "width"),
height = attr(p_kegg_combined, "height"), dpi = 300)GSEA Combined Plots
For GSEA results, use plot_gsea_dotplot_combined():
# Run GSEA
gsea_go <- gseGO(geneList = gene_list, OrgDb = org.Hs.eg.db, ...)
gsea_df <- as.data.frame(gsea_go)
# Create combined GSEA plot
p_gsea_combined <- plot_gsea_dotplot_combined(
gsea_df,
n_show = 10,
fdr_cutoff = 0.05,
title = "GSEA GO BP: Comparison",
size_label = "Set Size"
)
ggsave(file.path(plots_dir, "05_GSEA_GO_BP_combined.png"), p_gsea_combined,
width = attr(p_gsea_combined, "width"),
height = attr(p_gsea_combined, "height"), dpi = 300)Output Files
For each comparison, produce:
- ORA combined plots:
05_GO_BP_combined.png- Up + down on same figure05_KEGG_combined.png- Up + down on same figure- GSEA plots:
05_GSEA_GO_BP_combined.png- Activated + suppressed05_GSEA_GO_BP_dotplot.png- Standard clusterProfiler dotplot05_GSEA_GO_BP_ridgeplot.png- Distribution ridge plot05_GSEA_GO_BP_running.png- Running enrichment for top terms- CSV tables:
05_GO_BP_upregulated.csv/05_GO_BP_downregulated.csv05_KEGG_upregulated.csv/05_KEGG_downregulated.csv05_GSEA_GO_BP.csv
Contrast Selection Guidelines
Only extract biologically meaningful contrasts:
Good contrasts:
- Treatment vs Control
- Disease vs Healthy
- Lesional vs Non-Lesional (paired)
- Time point comparisons in time series
Avoid:
- Redundant comparisons (if A-B and B-C, you may not need A-C)
- Comparisons without biological interpretation
- All pairwise when only specific comparisons matter
Common Issues and Solutions
| Issue | Solution |
|---|---|
| Few DEGs with FDR < 0.05 | Use nominal P < 0.05 for exploratory GO analysis |
| Batch effects visible in PCA | Consider ComBat or limma's removeBatchEffect |
| Low annotation rate | Try alternative ID mapping sources (biomaRt) |
| High proportion of outlier samples | Review sample preparation notes; may need to exclude |
Utility Functions
The project includes reusable utility functions in scripts/utils/:
| Function | File | Description |
|---|---|---|
collapse_to_gene() | collapse_to_gene.R | Maps probes to genes, removes ambiguous mappings, collapses to unique genes |
plot_ora_dotplot_combined() | enrichment_dotplot.R | Combined ORA dotplot for up/downregulated genes |
plot_gsea_dotplot_combined() | enrichment_dotplot.R | Combined GSEA dotplot for activated/suppressed terms |
Always source these utilities using here():
source(here("scripts", "utils", "collapse_to_gene.R"))
source(here("scripts", "utils", "enrichment_dotplot.R"))Reference Files
See these files in the project for complete examples:
analyses/Rashighi_GSE53146/- Illumina unpaired design (most current patterns)analyses/Regazzetti_GSE65127/- Affymetrix paired designanalyses/Natarajan_GSE75819/- Illumina paired design
Code Reference for Microarray Analysis
Standard Script Header
# ==============================================================================
# [GSE#] - [Author] et al. [Platform] Microarray Analysis
# Script ##: [Description]
# ==============================================================================
# Platform: GPL### [Platform name]
# Samples: [N] total ([description])
# ==============================================================================
# Load required libraries
.libPaths(c("~/R/library", .libPaths()))
library(limma)
library(ggplot2)
library(dplyr)
library(tidyr)
# Set paths
base_dir <- "/workspaces/vitiligo/analyses/[Author]_[GSE#]"
data_dir <- file.path(base_dir, "data")
results_dir <- file.path(base_dir, "results")
plots_dir <- file.path(results_dir, "plots")
tables_dir <- file.path(results_dir, "tables")
# Create directories if they don't exist
dir.create(plots_dir, recursive = TRUE, showWarnings = FALSE)
dir.create(tables_dir, recursive = TRUE, showWarnings = FALSE)
cat("=== [GSE#] [Analysis Type] ===\n\n")Color Palettes
# Two-group comparison
condition_colors <- c(
"Control" = "#2ecc71",
"Treatment" = "#e74c3c"
)
# Multi-group (vitiligo example)
condition_colors <- c(
"Healthy" = "#2ecc71",
"Lesional" = "#e74c3c",
"Non_Lesional" = "#3498db",
"Peri_Lesional" = "#f39c12"
)
# Generic multi-group
n_groups <- length(unique(sample_info$condition))
condition_colors <- setNames(
RColorBrewer::brewer.pal(max(3, n_groups), "Set1")[1:n_groups],
unique(sample_info$condition)
)Probe Tracking
Initialize probe tracking at the start of every QC script:
# ==============================================================================
# Initialize probe tracking
# ==============================================================================
probe_tracking <- data.frame(
step = character(),
n_probes = integer(),
n_removed = integer(),
description = character(),
stringsAsFactors = FALSE
)
# Helper function to add tracking entry
add_probe_tracking <- function(step, n_probes, n_removed, description) {
probe_tracking <<- rbind(probe_tracking, data.frame(
step = step,
n_probes = n_probes,
n_removed = n_removed,
description = description,
stringsAsFactors = FALSE
))
}Report probe tracking summary at the end of QC script:
# ==============================================================================
# Probe Tracking Summary
# ==============================================================================
cat("\n=== Probe Tracking Summary ===\n\n")
probe_tracking$pct_remaining <- round(100 * probe_tracking$n_probes / probe_tracking$n_probes[1], 1)
print(probe_tracking)
cat("\n")
# Save probe tracking table
write.csv(probe_tracking, file.path(tables_dir, "probe_tracking.csv"), row.names = FALSE)
cat(" Saved probe tracking to results/tables/probe_tracking.csv\n")Affymetrix Data Loading and QC
library(affy)
library(affyPLM)
# ==============================================================================
# Load CEL files
# ==============================================================================
cel_files <- list.files(data_dir, pattern = "\\.CEL(\\.gz)?$", full.names = TRUE)
cat(sprintf("Found %d CEL files\n", length(cel_files)))
setwd(data_dir)
raw_data <- ReadAffy(filenames = basename(cel_files))
sampleNames(raw_data) <- sample_info$sample_name
# Track: Raw data loaded
n_probes_raw <- nrow(pm(raw_data))
add_probe_tracking("1. Raw data loaded", n_probes_raw, 0, "Total probesets on array")
# ==============================================================================
# Basic QC metrics
# ==============================================================================
# PM/MM intensities
pm_data <- pm(raw_data)
mm_data <- mm(raw_data)
avg_pm <- colMeans(pm_data, na.rm = TRUE)
avg_mm <- colMeans(mm_data, na.rm = TRUE)
# Background estimate
avg_background <- apply(mm_data, 2, function(x) mean(x[x < quantile(x, 0.05)], na.rm = TRUE))
# MAS5 present/absent calls
calls <- mas5calls(raw_data)
percent_present <- colSums(exprs(calls) == "P") / nrow(exprs(calls)) * 100
# RNA degradation
rna_deg <- AffyRNAdeg(raw_data)
# ==============================================================================
# PLM QC (NUSE and RLE)
# ==============================================================================
plm_fit <- fitPLM(raw_data)
nuse_vals <- NUSE(plm_fit, type = "values")
rle_vals <- RLE(plm_fit, type = "values")
plm_qc <- data.frame(
sample_name = colnames(nuse_vals),
nuse_median = apply(nuse_vals, 2, median),
nuse_iqr = apply(nuse_vals, 2, IQR),
rle_median = apply(rle_vals, 2, median),
rle_iqr = apply(rle_vals, 2, IQR)
)
# Flag outliers
plm_qc$nuse_outlier <- plm_qc$nuse_median > 1.05 | plm_qc$nuse_iqr > 0.15
plm_qc$rle_outlier <- abs(plm_qc$rle_median) > 0.1 | plm_qc$rle_iqr > 0.3
# ==============================================================================
# RMA Normalization
# ==============================================================================
eset <- rma(raw_data)
expr_mat <- exprs(eset)
# Track: After RMA normalization
add_probe_tracking("2. RMA normalization", nrow(expr_mat), 0, "Probesets after RMA")
# ==============================================================================
# Filter low-expression probesets (optional)
# ==============================================================================
# Keep probesets with median expression > log2(100) in at least one group
expr_median_by_group <- apply(expr_mat, 1, function(x) {
tapply(x, sample_info$condition, median)
})
keep_probes <- apply(expr_median_by_group, 2, max) > log2(100)
n_before <- nrow(expr_mat)
expr_filtered <- expr_mat[keep_probes, ]
# Track: Expression filtering
add_probe_tracking("3. Expression filter", nrow(expr_filtered),
n_before - nrow(expr_filtered),
"Probesets with median > log2(100) in >= 1 group")Illumina Data Loading and QC
library(limma)
# ==============================================================================
# Load Illumina data (non-normalized)
# ==============================================================================
data_file <- file.path(data_dir, "GSE#####_non_normalized.txt")
raw_data <- read.delim(data_file, header = TRUE, stringsAsFactors = FALSE)
# Track: Raw data loaded
add_probe_tracking("1. Raw data loaded", nrow(raw_data), 0, "Total probes in raw data file")
# Parse signal and detection p-value columns
signal_cols <- grep("AVG_Signal", colnames(raw_data), value = TRUE)
pval_cols <- grep("Detection", colnames(raw_data), value = TRUE)
# Create expression matrix
expr_mat <- as.matrix(raw_data[, signal_cols])
rownames(expr_mat) <- raw_data$ID_REF
# Create detection p-value matrix
detect_mat <- as.matrix(raw_data[, pval_cols])
rownames(detect_mat) <- raw_data$ID_REF
# Standardize sample names
sample_names <- gsub("\\.AVG_Signal", "", signal_cols)
sample_names <- gsub("\\.", "", sample_names) # Clean up dots
colnames(expr_mat) <- sample_names
colnames(detect_mat) <- sample_names
# Track: Expression matrix created
add_probe_tracking("2. Expression matrix created", nrow(expr_mat), 0,
"Probes with valid signal columns")
# ==============================================================================
# QC metrics
# ==============================================================================
detect_threshold <- 0.05
percent_detected <- colSums(detect_mat < detect_threshold) / nrow(detect_mat) * 100
avg_intensity <- colMeans(expr_mat, na.rm = TRUE)
# ==============================================================================
# Filter probes by detection
# ==============================================================================
min_samples_detected <- 3 # Or 10% of samples
probes_detected <- rowSums(detect_mat < detect_threshold) >= min_samples_detected
expr_filtered <- expr_mat[probes_detected, ]
detect_filtered <- detect_mat[probes_detected, ]
# Track: Detection filtering
add_probe_tracking("3. Detection filter", nrow(expr_filtered),
nrow(expr_mat) - nrow(expr_filtered),
sprintf("Probes detected (p < 0.05) in >= %d samples", min_samples_detected))
# ==============================================================================
# Log2 transform and quantile normalize
# ==============================================================================
expr_log2 <- log2(expr_filtered + 1)
expr_norm <- normalizeBetweenArrays(expr_log2, method = "quantile")
# Track: After normalization (probe count unchanged)
add_probe_tracking("4. Normalized", nrow(expr_norm), 0,
"Log2 + quantile normalization applied")Differential Expression Analysis
Paired Design (Fixed Effects)
# Subset to samples with paired structure
vitiligo_samples <- sample_info$condition != "Healthy"
expr_subset <- expr_filtered[, vitiligo_samples]
info_subset <- sample_info[vitiligo_samples, ]
# Create factors
tissue <- factor(info_subset$condition, levels = c("NonLesional", "Lesional", "PeriLesional"))
patient <- factor(info_subset$subject_id)
# Design with patient as fixed effect
design <- model.matrix(~ patient + tissue)
# Fit model
fit <- lmFit(expr_subset, design)
# Contrasts for tissue effects
contrast_matrix <- makeContrasts(
Lesional_vs_NonLesional = tissueLesional,
PeriLesional_vs_NonLesional = tissuePeriLesional,
Lesional_vs_PeriLesional = tissueLesional - tissuePeriLesional,
levels = design
)
fit2 <- contrasts.fit(fit, contrast_matrix)
fit2 <- eBayes(fit2)
# Extract results
results <- topTable(fit2, coef = "Lesional_vs_NonLesional", number = Inf, sort.by = "P")Unpaired Design
# Create factor
group <- factor(sample_info$condition, levels = c("Control", "Treatment"))
# Cell means model
design <- model.matrix(~ 0 + group)
colnames(design) <- levels(group)
# Fit model
fit <- lmFit(expr_filtered, design)
# Define contrast
contrast_matrix <- makeContrasts(
Treatment_vs_Control = Treatment - Control,
levels = design
)
fit2 <- contrasts.fit(fit, contrast_matrix)
fit2 <- eBayes(fit2)
# Extract results
results <- topTable(fit2, coef = "Treatment_vs_Control", number = Inf, sort.by = "P")Gene Annotation
Affymetrix (using platform.db)
library(hgu133plus2.db) # Change based on platform
library(AnnotationDbi)
library(org.Hs.eg.db)
# Get probe-to-gene mapping
probe_ids <- rownames(results)
mapping <- AnnotationDbi::select(
hgu133plus2.db,
keys = probe_ids,
columns = c("SYMBOL", "ENTREZID", "ENSEMBL", "GENENAME"),
keytype = "PROBEID"
)
colnames(mapping) <- c("ProbeID", "gene_symbol", "entrez_id", "ensembl_id", "gene_name")
# Join with results
results$ProbeID <- rownames(results)
de_annotated <- results |>
left_join(mapping, by = "ProbeID")Illumina (using illuminaHumanv4.db)
library(illuminaHumanv4.db) # Change based on platform
library(AnnotationDbi)
mapping <- AnnotationDbi::select(
illuminaHumanv4.db,
keys = probe_ids,
columns = c("SYMBOL", "ENTREZID", "ENSEMBL", "GENENAME", "CHR"),
keytype = "PROBEID"
)Collapse to Unique Gene IDs
Export DE results in multiple ID formats. Keep probe with lowest P.Value when multiple map to same gene:
# Collapse to unique Ensembl IDs
collapse_to_ensembl <- function(annotated_results) {
annotated_results |>
filter(!is.na(.data$ensembl_id)) |>
group_by(.data$ensembl_id) |>
arrange(.data$P.Value, -abs(.data$logFC)) |>
slice(1) |>
ungroup() |>
arrange(-abs(.data$logFC)) |>
as.data.frame()
}
# Collapse to unique gene symbols
collapse_to_symbol <- function(annotated_results) {
annotated_results |>
filter(!is.na(.data$gene_symbol)) |>
group_by(.data$gene_symbol) |>
arrange(.data$P.Value, -abs(.data$logFC)) |>
slice(1) |>
ungroup() |>
arrange(-abs(.data$logFC)) |>
as.data.frame()
}
# Collapse to unique Entrez IDs
collapse_to_entrez <- function(annotated_results) {
annotated_results |>
filter(!is.na(.data$entrez_id)) |>
group_by(.data$entrez_id) |>
arrange(.data$P.Value, -abs(.data$logFC)) |>
slice(1) |>
ungroup() |>
arrange(-abs(.data$logFC)) |>
as.data.frame()
}
# Create all three versions and save
de_ensembl <- collapse_to_ensembl(de_annotated)
de_symbol <- collapse_to_symbol(de_annotated)
de_entrez <- collapse_to_entrez(de_annotated)
write.csv(de_ensembl, file.path(tables_dir, "DE_comparison_clean_ensembl.csv"),
row.names = FALSE)
write.csv(de_symbol, file.path(tables_dir, "DE_comparison_clean_symbol.csv"),
row.names = FALSE)
write.csv(de_entrez, file.path(tables_dir, "DE_comparison_clean_entrez.csv"),
row.names = FALSE)GO Enrichment Analysis
library(clusterProfiler)
library(org.Hs.eg.db)
library(enrichplot)
# Prepare gene lists
sig_up <- de_unique %>%
filter(adj.P.Val < 0.05 & logFC > 0) %>%
pull(entrez_id) %>%
unique() %>%
as.character()
sig_down <- de_unique %>%
filter(adj.P.Val < 0.05 & logFC < 0) %>%
pull(entrez_id) %>%
unique() %>%
as.character()
background <- de_unique %>%
filter(!is.na(entrez_id)) %>%
pull(entrez_id) %>%
unique() %>%
as.character()
# GO enrichment (require at least 5 genes)
if (length(sig_up) >= 5) {
ego_up <- enrichGO(
gene = sig_up,
universe = background,
OrgDb = org.Hs.eg.db,
keyType = "ENTREZID",
ont = "BP",
pAdjustMethod = "BH",
pvalueCutoff = 0.05,
qvalueCutoff = 0.2,
readable = TRUE
)
# Simplify redundant terms
ego_up <- simplify(ego_up, cutoff = 0.7, by = "p.adjust")
# Filter by FDR
ego_up_sig <- ego_up[ego_up@result$p.adjust < 0.05, ]
# Save results
write.csv(as.data.frame(ego_up_sig),
file.path(tables_dir, "GO_BP_upregulated.csv"),
row.names = FALSE)
}
# KEGG enrichment
if (length(sig_up) >= 5) {
kegg_up <- enrichKEGG(
gene = sig_up,
universe = background,
organism = "hsa",
pAdjustMethod = "BH",
pvalueCutoff = 0.05
)
}Visualization Functions
Volcano Plot
create_volcano <- function(results, title, fdr_thresh = 0.05, lfc_thresh = 1) {
results$significance <- case_when(
results$adj.P.Val < fdr_thresh & results$logFC > lfc_thresh ~ "Up",
results$adj.P.Val < fdr_thresh & results$logFC < -lfc_thresh ~ "Down",
results$adj.P.Val < fdr_thresh ~ "Significant",
TRUE ~ "Not significant"
)
ggplot(results, aes(x = logFC, y = -log10(P.Value), color = significance)) +
geom_point(alpha = 0.5, size = 1.5) +
scale_color_manual(values = c(
"Up" = "#e74c3c",
"Down" = "#3498db",
"Significant" = "#f39c12",
"Not significant" = "gray70"
)) +
geom_vline(xintercept = c(-lfc_thresh, lfc_thresh), linetype = "dashed", color = "gray40") +
geom_hline(yintercept = -log10(0.05), linetype = "dashed", color = "gray40") +
theme_minimal() +
labs(title = title,
x = "log2 Fold Change",
y = "-log10(P-value)",
color = "Significance") +
theme(legend.position = "bottom")
}Comparison-Specific DEG Heatmap (top 60 by |logFC|)
library(pheatmap)
create_comparison_heatmap <- function(de_results, expr_data, sample_info,
annotation_df, conditions_to_include,
filename, title, n_top = 60) {
# Get significant DEGs
sig_results <- de_results[de_results$adj.P.Val < 0.05, ]
if (nrow(sig_results) == 0) {
cat(sprintf(" Skipping %s (no DEGs)\n", title))
return(NULL)
}
# Sort by absolute logFC and take top n
sig_results <- sig_results[order(-abs(sig_results$logFC)), ]
top_probesets <- head(sig_results$probeset, n_top)
cat(sprintf(" %s: %d significant DEGs, showing top %d by |logFC|\n",
title, nrow(sig_results), length(top_probesets)))
# Filter samples to relevant conditions
samples_to_include <- rownames(sample_info)[
sample_info$condition %in% conditions_to_include
]
# Filter and order expression data
expr_subset <- expr_data[top_probesets, samples_to_include, drop = FALSE]
# Scale expression (z-score per gene)
expr_scaled <- t(scale(t(expr_subset)))
# Order samples by condition (not clustering)
sample_order <- samples_to_include[order(
factor(sample_info[samples_to_include, "condition"],
levels = conditions_to_include)
)]
expr_scaled <- expr_scaled[, sample_order, drop = FALSE]
# Get gene labels
row_labels <- annotation_df$gene_symbol[
match(rownames(expr_scaled), annotation_df$probeset)
]
row_labels[is.na(row_labels)] <- rownames(expr_scaled)[is.na(row_labels)]
# Column annotation
col_annotation <- data.frame(
Condition = sample_info[sample_order, "condition"],
row.names = sample_order
)
# Create heatmap - cluster genes but NOT samples
png(filename, width = 1200, height = 1200)
pheatmap(expr_scaled,
annotation_col = col_annotation,
annotation_colors = list(Condition = condition_colors),
labels_row = row_labels,
show_colnames = TRUE,
cluster_cols = FALSE, # Do not cluster samples
cluster_rows = TRUE, # Cluster genes
clustering_method = "ward.D2",
clustering_distance_rows = "correlation",
color = colorRampPalette(c("#3498db", "white", "#e74c3c"))(100),
main = sprintf("%s\nTop %d DEGs by |logFC| (FDR < 0.05)",
title, length(top_probesets)),
fontsize_col = 8,
fontsize_row = 7,
border_color = NA)
dev.off()
cat(sprintf(" Saved: %s\n", basename(filename)))
}
# Usage example:
# create_comparison_heatmap(
# res_lst_nlst, expr_mat, sample_info, annotation_df,
# conditions_to_include = c("Lesional", "Non_Lesional"),
# file.path(plots_dir, "29_heatmap_LST_vs_NLST.png"),
# "Lesional vs Non-Lesional"
# )Overall Expression Heatmap (top variable genes)
library(pheatmap)
# Order samples by condition (no clustering)
sample_order <- order(sample_info$condition)
expr_mat_ordered <- expr_mat[, sample_order]
sample_info_ordered <- sample_info[sample_order, ]
# Select top 2000 most variable genes by variance
gene_vars <- apply(expr_mat_ordered, 1, var)
top_genes <- names(sort(gene_vars, decreasing = TRUE))[1:2000]
expr_top <- expr_mat_ordered[top_genes, ]
# Median center expression for visualization
expr_centered <- expr_top - apply(expr_top, 1, median)
# Clip values to ±3 SD for better color utilization
mat_mean <- mean(expr_centered)
mat_sd <- sd(expr_centered)
lower_bound <- mat_mean - 3 * mat_sd
upper_bound <- mat_mean + 3 * mat_sd
expr_centered[expr_centered < lower_bound] <- lower_bound
expr_centered[expr_centered > upper_bound] <- upper_bound
# Create custom distance function using 1 - Pearson correlation
cor_dist <- function(x) {
as.dist(1 - cor(t(x), method = "pearson"))
}
# Create annotation for samples
annotation_col <- data.frame(
Condition = sample_info_ordered$condition,
row.names = sample_info_ordered$sample_name
)
# Traditional green-red color palette
heatmap_colors <- colorRampPalette(c("green", "black", "red"))(100)
# Generate heatmap - samples ordered by group, not clustered
png(file.path(plots_dir, "19b_expression_heatmap_top2000.png"),
width = 6, height = 8, units = "in", res = 150)
pheatmap(expr_centered,
color = heatmap_colors,
clustering_distance_rows = cor_dist(expr_centered),
clustering_method = "average",
cluster_cols = FALSE,
annotation_col = annotation_col,
annotation_colors = list(Condition = condition_colors),
show_rownames = FALSE,
show_colnames = TRUE,
fontsize_col = 6,
main = "Expression Heatmap (Top 2000 Variable Genes)",
treeheight_row = 50)
dev.off()Sample Hierarchical Clustering Dendrogram
library(dendextend)
# Use all genes, 1 - Pearson correlation distance, average linkage
sample_dist <- as.dist(1 - cor(expr_mat, method = "pearson"))
sample_hclust <- hclust(sample_dist, method = "average")
# Create dendrogram
dend <- as.dendrogram(sample_hclust)
# Get labels in dendrogram order and map to conditions
dend_labels <- labels(dend)
label_conditions <- sample_info[dend_labels, "condition"]
label_colors <- condition_colors[label_conditions]
# Color labels and branches by condition
labels_colors(dend) <- label_colors
dend <- branches_attr_by_labels(dend, dend_labels, label_colors, attr = "col")
png(file.path(plots_dir, "19c_sample_dendrogram.png"),
width = 8, height = 6, units = "in", res = 150)
par(mar = c(4, 2, 2, 8), cex = 0.6)
plot(dend, horiz = TRUE,
main = "Sample Hierarchical Clustering\n(1 - Pearson correlation, average linkage)")
legend("topleft", legend = names(condition_colors),
col = condition_colors, lwd = 2, cex = 0.8, bty = "n")
dev.off()Utility Functions
Extract and Summarize Results
extract_results <- function(fit, coef, name, fdr_threshold = 0.05) {
results <- topTable(fit, coef = coef, number = Inf, sort.by = "P")
results$ProbeID <- rownames(results)
results$contrast <- name
sig_fdr <- sum(results$adj.P.Val < fdr_threshold)
up_reg <- sum(results$adj.P.Val < fdr_threshold & results$logFC > 0)
down_reg <- sum(results$adj.P.Val < fdr_threshold & results$logFC < 0)
cat(sprintf("\n%s:\n", name))
cat(sprintf(" Total probes: %d\n", nrow(results)))
cat(sprintf(" FDR < 0.05: %d\n", sig_fdr))
cat(sprintf(" Up-regulated: %d\n", up_reg))
cat(sprintf(" Down-regulated: %d\n", down_reg))
return(results)
}Get Package Versions
get_package_versions <- function(packages) {
versions <- sapply(packages, function(pkg) {
tryCatch(
as.character(packageVersion(pkg)),
error = function(e) "Not installed"
)
})
data.frame(Package = packages, Version = versions)
}
# Usage
packages_used <- c("limma", "affy", "affyPLM", "ggplot2", "dplyr",
"pheatmap", "clusterProfiler", "org.Hs.eg.db")
print(get_package_versions(packages_used))Template Files for Microarray Analysis
analysis.md Template
Use this template structure for the analysis.md file in each study directory:
# [GSE#] Microarray Analysis Report
## Study Information
| Field | Value |
|-------|-------|
| **GEO Accession** | GSE##### |
| **Publication** | [Author et al. Journal Year] |
| **Title** | [Study title] |
| **Platform** | GPL### [Platform name] |
| **Organism** | Homo sapiens |
| **Analysis Date** | YYYY-MM-DD |
## Study Design
[Describe the experimental design, sample groups, and biological question]
---
## Methods
### Software Environment
| Component | Version |
|-----------|---------|
| R | X.X.X |
| Bioconductor | X.XX |
### R Packages Used
| Package | Version | Purpose |
|---------|---------|---------|
| limma | X.XX.X | Differential expression analysis |
| affy | X.XX.X | CEL file reading, RMA normalization |
| [package] | X.XX.X | [purpose] |
### Data Processing Pipeline
#### 1. Data Loading
[Describe how data was loaded]
#### 2. Quality Control
[Describe QC metrics computed]
#### 3. Normalization
[Describe normalization method]
#### 4. Differential Expression
[Describe design matrix and contrasts]
---
## Results
### Probe Tracking
| Step | Probes | Removed | % Remaining | Description |
|------|--------|---------|-------------|-------------|
| 1. Raw data loaded | X | 0 | 100.0% | Total probes in raw data file |
| 2. Expression matrix | X | 0 | 100.0% | Probes with valid signal columns |
| 3. Detection/Expression filter | X | X | XX.X% | [Filtering criteria] |
| 4. Normalized | X | 0 | XX.X% | After normalization |
### Sample Summary
| Condition | N | Description |
|-----------|---|-------------|
| [Group1] | X | [description] |
| [Group2] | X | [description] |
### Quality Control Results
#### Outlier Identification
| Sample | Issues | Recommendation |
|--------|--------|----------------|
| [sample] | [issues] | [action] |
### Differential Expression Results
| Comparison | Design | Total DEGs | Up | Down | |logFC| > 1 |
|------------|--------|------------|-----|------|-------------|
| [comparison] | [paired/unpaired] | X | X | X | X |
### Key Biological Findings
[Summarize main findings and biological interpretation]
---
## Output Files
### Tables (results/tables/)
| File | Description |
|------|-------------|
| sample_metadata.csv | Sample information |
| probe_tracking.csv | Probe counts at each processing step |
| qc_metrics.csv | Per-sample QC metrics |
| [file] | [description] |
### Plots (results/plots/)
| File | Description |
|------|-------------|
| 01_raw_intensity_boxplot.png | [description] |
| [file] | [description] |
---
## Next Steps
- [ ] [Planned follow-up analysis]
- [ ] [Additional comparisons]
---
## References
1. [Citation for original paper]
2. [Citation for methods papers]plots.md Template
Use this template structure for results/plots/plots.md:
# Plot Documentation
All plots are saved as PNG files at 300 DPI unless otherwise noted.
## Quality Control Plots
### 01_raw_intensity_boxplot.png
**Description:** Boxplot of raw probe intensities by sample before normalization.
**Interpretation:** All samples should have similar distributions. Outliers appear as samples with shifted medians or different spread.
### 02_raw_intensity_density.png
**Description:** Overlapping density curves of raw log2 intensities, colored by sample or condition.
**Interpretation:** Curves should overlap substantially; separated curves may indicate batch effects.
### 03_avg_intensity.png
**Description:** Bar plot of average signal intensity per sample.
**Interpretation:** Similar values expected; extreme outliers (z-score > 2) flagged.
### 04_percent_detected.png
**Description:** Percentage of probes with detection p-value < 0.05 (Illumina) or MAS5 "Present" calls (Affymetrix).
**Interpretation:** Low detection rates may indicate poor RNA quality or technical issues.
### 05_sample_correlation_heatmap.png
**Description:** Heatmap of pairwise Pearson correlations between samples with hierarchical clustering.
**Interpretation:** Samples should cluster by biological condition; outliers show lower correlation.
### 06_PCA_raw.png
**Description:** Principal Component Analysis of expression data, PC1 vs PC2.
**Interpretation:** Samples should separate by condition; extreme outliers visible as isolated points.
### 07_qc_by_condition.png
**Description:** QC metrics compared across experimental conditions.
**Interpretation:** Technical metrics should not differ systematically between conditions.
## Affymetrix-Specific QC
### 08_NUSE_boxplot.png
**Description:** Normalized Unscaled Standard Errors from probe-level model.
**Interpretation:** Median should be ~1.0; samples with median > 1.05 are potential outliers.
### 09_RLE_boxplot.png
**Description:** Relative Log Expression from probe-level model.
**Interpretation:** Median should be ~0; samples with |median| > 0.1 are potential outliers.
### 10_rna_degradation.png
**Description:** RNA degradation plot showing 5' to 3' bias.
**Interpretation:** Steeper slopes indicate greater degradation; all samples should have similar slopes.
## Post-Normalization Plots
### 11_normalized_boxplot.png
**Description:** Expression distribution after normalization.
**Interpretation:** After quantile normalization, all samples should have identical distributions.
### 12_normalized_density.png
**Description:** Density curves after normalization.
**Interpretation:** Curves should be nearly identical after quantile normalization.
### 13_PCA_normalized.png
**Description:** PCA after normalization.
**Interpretation:** Technical variation reduced; biological separation should be clearer.
### 19b_expression_heatmap_top2000.png
**Description:** Overall expression heatmap of top 2000 most variable genes.
**Methods:**
- Genes selected by highest variance across samples
- Median-centered expression values, clipped to ±3 SD
- Gene clustering: 1 - Pearson correlation distance, average linkage
- Samples ordered by condition (not clustered)
- Green-black-red color scheme
**Interpretation:** Reveals global expression patterns; samples should show condition-specific clustering patterns among genes.
### 19c_sample_dendrogram.png
**Description:** Hierarchical clustering dendrogram of all samples.
**Methods:**
- Uses all genes for clustering
- Distance: 1 - Pearson correlation coefficient
- Linkage: Average (UPGMA)
- Horizontal layout with branches/labels colored by condition
**Interpretation:** Shows sample relationships; samples from same condition should cluster together. Mixed clustering suggests biological heterogeneity or batch effects.
## Differential Expression Plots
### 20_volcano_[comparison].png
**Description:** Volcano plot showing -log10(p-value) vs log2 fold change.
**Color coding:**
- Red: Significantly upregulated (FDR < 0.05, logFC > 1)
- Blue: Significantly downregulated (FDR < 0.05, logFC < -1)
- Orange: FDR < 0.05 but |logFC| < 1
- Gray: Not significant
### 21_pvalue_histogram.png
**Description:** Distribution of raw p-values from differential expression test.
**Interpretation:** Should show uniform distribution (null) plus spike near 0 (true positives). Anti-conservative pattern (spike near 1) suggests model misspecification.
### 22_MA_plot.png
**Description:** M (log fold change) vs A (average expression) plot.
**Interpretation:** Points should be centered at M=0; funnel shape is expected (more variance at low expression).
### 29_heatmap_[comparison].png (e.g., 29_heatmap_LST_vs_NLST.png)
**Description:** Heatmap of top 60 DEGs by |logFC| for a specific comparison.
**Methods:**
- Genes selected: Top 60 significant DEGs (FDR < 0.05) ranked by absolute log fold change
- Expression values: Z-score normalized per gene
- Samples: Only the two conditions being compared, sorted by group (not clustered)
- Genes: Hierarchically clustered (Ward.D2, correlation distance)
- Color scheme: Blue-white-red
**Interpretation:** Shows the strongest expression differences between compared conditions. Samples sorted by group allow visual assessment of within-group consistency.
### 24_DE_summary_barplot.png
**Description:** Bar chart summarizing number of DEGs per comparison.
**Interpretation:** Quick overview of effect sizes across different contrasts.
## GO and KEGG Enrichment Plots
### [num]_GO_BP_[comparison].png (e.g., 34_GO_BP_LST_vs_NLST.png)
**Description:** Gene Ontology Biological Process enrichment for a specific comparison.
**Layout:** Two panels if both directions have significant terms (upregulated top, downregulated bottom), single panel otherwise.
**Visualization:** Dot plot with -log10(adj. p-value) on x-axis, terms on y-axis, color and size by gene count.
**Methods:**
- Background: All genes that passed filtering (from probeset_annotation.csv)
- GO terms simplified to reduce redundancy (cutoff = 0.7)
- FDR < 0.05 threshold

### [num]b_KEGG_[comparison].png (e.g., 34b_KEGG_LST_vs_NLST.png)
**Description:** KEGG pathway enrichment for a specific comparison.
**Layout:** Two panels if both directions have significant pathways, single panel otherwise.
**Visualization:** Same style as GO plot - dot plot with -log10(adj. p-value) on x-axis.
**Methods:**
- Background: All genes that passed filtering
- FDR < 0.05 threshold
CLAUDE.md Template for Study Metadata
Create this file in each study directory:
# [GSE#]
## Title
[Study title from GEO]
## Organism
Homo sapiens
## Experiment type
Expression profiling by array
## Summary
[Abstract/summary from GEO]
## Overall design
[Experimental design description]
## Platform
GPL### [Platform name]
## Citation
[Full citation with PMID]
## Samples
| Sample | Description | Condition | Subject |
|--------|-------------|-----------|---------|
| GSM### | [description] | [condition] | [subject_id] |
## Notes
[Any special considerations for analysis]