Folder Organization

Folder Organization Best Practices

Expert guidance for organizing project directories, establishing file naming conventions, and maintaining clean, navigable project structures for research and development work.

When to Use This Skill

• Setting up new projects
• Reorganizing existing projects
• Establishing team conventions
• Creating reproducible research structures
• Managing data-intensive projects

Core Principles

1. Predictability - Standard locations for common file types 2. Scalability - Structure grows gracefully with project 3. Separation of Concerns - Code, data, documentation, outputs separated 4. Discoverability - Easy for others (and future you) to navigate 5. Version Control Friendly - Large/generated files excluded appropriately

Quick Reference: Standard Project Structure

Use the structure matching your project type. Full templates in project-structures.md.

Research/Analysis: data/{raw,processed,external}, notebooks/, src/, scripts/, tests/, docs/, results/, config/

Development: src/package_name/, tests/, docs/, examples/, .github/workflows/

Bioinformatics: data/{raw,reference,processed}, workflows/, config/, scripts/, results/, logs/

Data Analysis with Notebooks: notebooks/, figures/, data/, tests/, scripts/, docs/, archives/

Add MANIFEST.md files for token-efficient navigation (see manifest-system.md).

File Naming Rules

Detailed conventions in naming-conventions.md. Key rules:

1. Use lowercase with hyphens or underscores (not spaces or CamelCase) 2. Be descriptive but concise - process-telomere-data.py not script.py 3. Use consistent separators - hyphens for files, underscores for Python modules 4. Include version/date for important outputs - report-2026-01-23.pdf 5. Zero-pad sequences - 01-exploration.ipynb, 02-analysis.ipynb 6. Standardize session notes in session-saves/ directory

Version Control

DO commit: Source code, documentation, config files, small test data (<1MB), requirements files, READMEs

DON'T commit: Large data files, generated outputs, environment dirs, logs, temp files, API keys/secrets

.gitignore Template

# Claude Code
.claude/

# Python
__pycache__/
*.py[cod]
.venv/
venv/

# Jupyter
.ipynb_checkpoints/

# Data
data/raw/
data/processed/
*.fastq.gz
*.bam

# Outputs
results/
outputs/
*.png
*.pdf

# Logs & Environment
logs/
*.log
.env
.DS_Store

Data Organization

• Never modify raw data - keep originals in data/raw/ (read-only if possible)
• Document data provenance (source, download date)
• Use hierarchy: raw/ -> interim/ -> processed/ -> external/

Common Anti-Patterns

Avoid these patterns:

• Flat structure: Everything in root directory with no organization
• Ambiguous naming: notebook1.ipynb, test.ipynb, analysis_new.ipynb
• Mixed concerns: Data files and output images inside src/ directory

Deprecation Strategy

Deprecate rather than delete to maintain history and recovery options. Full guide in deprecation-guide.md.

Key pattern: 1. Create deprecated/ subdirectory with descriptive name 2. Move old files there 3. Write README.md explaining what, when, why, and how to recover 4. For scattered deprecated dirs, consolidate to single root deprecated/

Project Reorganization

Full systematic guide in reorganization-guide.md. Critical steps:

1. Update all file paths in scripts and notebooks after moving files 2. Search for references: grep -r "\.json\|\.csv\|\.png" --include="*.ipynb" --include="*.py" 3. Verify file counts match expectations (no files lost) 4. Remove temp files (.bak, .ipynb_checkpoints) 5. Test that notebooks/scripts still run correctly

Documentation Standards

Full guide in documentation-organization.md. Key points:

• Every project needs a README with: description, installation, usage, structure, data, results
• After major changes, create dated summary documents
• Consolidate scattered docs into documentation/ directory
• Keep only README.md, LICENSE, .gitignore in project root

MANIFEST System

Token-efficient project navigation system. Full documentation in manifest-system.md.

• Provides 85-90% token reduction for session startup
• Root MANIFEST.md gives complete project overview in ~1,500 tokens
• Subdirectory MANIFESTs for data/, figures/, scripts/, documentation/
• Update at end of every session with /update-manifest

Project Cleanup: Identifying Essential Files

Detailed process in deprecation-guide.md (section "Project Cleanup"). Summary:

1. Analyze notebooks to find referenced figures 2. Map figures to generating scripts 3. Move unused files to deprecated/ with descriptive subdirectory names 4. Document what was kept in MINIMAL_ESSENTIAL_FILES.md 5. Verify notebooks still work with cleaned structure

Integration with Other Skills

• python-environment - Environment setup and management
• claude-collaboration - Team workflow best practices
• jupyter-notebook-analysis - Notebook organization standards
• data-backup - Backup system should include MANIFESTs
• project-sharing - Include MANIFESTs in shared packages

Quick Setup

# Create standard research project structure
mkdir -p data/{raw,processed,external} notebooks scripts src tests docs results config
touch README.md .gitignore environment.yml

Supporting Files

References and Resources

• Cookiecutter Data Science
• A Quick Guide to Organizing Computational Biology Projects
• Good Enough Practices in Scientific Computing

Deprecation and Cleanup Guide

Strategies for deprecating files, identifying essential files, and cleaning up projects.

Systematic Deprecation Strategy

When cleaning up projects, deprecate rather than delete to maintain project history and recovery options.

Pattern: Create Deprecated Subdirectories with Documentation

Instead of deleting old files:

# Create deprecated subdirectory with descriptive name
mkdir -p data-processing/phylo/deprecated_pre_feb20/

# Move old files
mv old_file1.txt old_file2.csv deprecated_pre_feb20/

# Create comprehensive README
cat > deprecated_pre_feb20/README.md << 'EOF'
# Deprecated [Category] - [Date/Reason]

**Date Deprecated**: YYYY-MM-DD
**Reason**: [Why these files are no longer active]

---

## Files ([N] files, ~[SIZE] total)

[List each file with explanation]

### 1. [Category of Files]
- **file1.txt** - What it was, why superseded
- **file2.csv** - Original purpose, replacement file

**Superseded by**: [Current active file(s)]

---

## Current Active Files (Use These Instead)

- `current_file.csv` - [Purpose and location]
- `updated_script.py` - [What it does now]

---

## Recovery Instructions

If you need to reference these old files:

cd deprecated_pre_feb20/ ls -lh

**Note**: These files should NOT be used for new analyses.
EOF

Benefits

1. Maintains history - Can review what was changed and why 2. Recovery option - Files aren't permanently lost 3. Clear documentation - Future users understand what happened 4. Clean working directory - Old files don't clutter main workspace 5. Audit trail - Shows project evolution over time

When to Use Deprecation

• Old analysis versions that have been superseded
• Intermediate processing files no longer needed
• Configuration files from previous approaches
• Batch processing logs from completed workflows

When to Delete Instead

• Truly temporary files (e.g., Python __pycache__)
• Large data files that are easily regenerated
• LaTeX build artifacts
• Duplicate copies with no historical value

Example: Multi-Directory Deprecation

When cleaning multiple subdirectories (phylo, telomeres, genomescope), create parallel deprecation structure:

data-processing/
├── phylo/
│   ├── deprecated_pre_feb20/
│   │   ├── README.md (explains 25 old files)
│   │   └── [old files]
│   └── [current files]
├── telomeres/
│   ├── deprecated_versions/
│   │   ├── README.md (explains 3 old CSV versions)
│   │   └── [old versions]
│   └── [current table]
└── genomescope_batch/
    ├── deprecated_batch_logs/
    │   ├── README.md (explains 11 log files)
    │   └── [processing logs]
    └── [current results]

Each README should document:

• What was deprecated and when
• Why these files are no longer active
• What replaced them (with file paths)
• How to recover if needed
• Warning not to use for new work

Consolidating Scattered Deprecated Directories

When projects have deprecated files scattered in multiple subdirectories, consolidate them to a single root deprecated/ location for easier management.

Problem: Scattered Deprecation

project/
├── deprecated/
│   └── notebooks_20260226/
├── documentation/
│   └── deprecated/          # Scattered
├── data-processing/
│   ├── telomeres/
│   │   └── deprecated_versions/  # Scattered
│   ├── phylo/
│   │   └── deprecated_pre_feb20/ # Scattered
│   └── genomescope_batch/
│       └── deprecated_batch_logs/ # Scattered

Issues:

• Hard to find all deprecated files
• No central inventory
• Inconsistent naming
• Clutters active subdirectories

Solution: Centralized Deprecation

Step 1: Find all deprecated directories

find . -type d -name "*deprecated*" -o -name "*OLD*" 2>/dev/null | grep -v ".git"

Step 2: Create organized structure in root

mkdir -p deprecated/data-processing

Step 3: Move with descriptive names

# Use clear, descriptive names that indicate origin
mv data-processing/telomeres/deprecated_versions \
   deprecated/data-processing/telomeres_versions

mv data-processing/phylo/deprecated_pre_feb20 \
   deprecated/data-processing/phylo_pre_feb20

mv documentation/deprecated \
   deprecated/documentation

Step 4: Create master README

cat > deprecated/README.md << 'EOF'
# Deprecated Files - [Project Name]

**Last Updated**: YYYY-MM-DD
**Purpose**: Central archive of all deprecated project files

## Organization

deprecated/
├── notebooks_YYYYMMDD/       # Deprecated notebooks
├── data-processing/          # Data processing deprecated
│   ├── telomeres_versions/
│   ├── phylo_pre_feb20/
│   └── genomescope_batch_logs/
├── documentation/            # Deprecated docs
└── [other categories]

## Contents

### 1. Category Name
- What was deprecated
- When and why
- What replaced it
- Where to find active files

[... sections for each category ...]

## Recovery

Browse deprecated files

cd deprecated/ ls -lh

**Status**: Archived - Complete (YYYY-MM-DD)
EOF

Step 5: Verify cleanup

# Check that source directories are gone
for dir in path/to/old/deprecated1 path/to/old/deprecated2; do
    if [ -d "$dir" ]; then
        echo "Still exists: $dir"
    else
        echo "Removed: $dir"
    fi
done

Result: Consolidated Structure

project/
└── deprecated/
    ├── README.md                # Master index
    ├── notebooks_YYYYMMDD/
    ├── data-processing/
    │   ├── category1/
    │   ├── category2/
    │   └── category3/
    ├── documentation/
    └── [other categories]

Benefits

1. Single location: All deprecated files in one place 2. Easy navigation: Master README explains everything 3. Clean subdirectories: No scattered deprecated_*/ folders 4. Preserved history: All files and their READMEs intact 5. Clear inventory: Know exactly what's archived

When to Consolidate

• During major cleanup - Part of project organization
• Before sharing - Single deprecated/ easier to exclude
• When scattered dirs > 3 - Multiple deprecation locations confusing
• After project maturity - Once deprecation patterns emerge

Project Cleanup: Identifying Essential Files

When projects accumulate many files over time, use this systematic approach to identify and keep only essential files:

1. Analyze Notebooks to Find Used Figures

# Extract figure references from Jupyter notebooks
grep -o "figures/[^'\"]*\.png" YourNotebook.ipynb | sort -u

# For multiple notebooks, check each one
for nb in *.ipynb; do
    echo "=== $nb ==="
    grep -o "figures/[^'\"]*\.png" "$nb" | sort -u
done

2. Map Figures to Generating Scripts

# Find which script generates a specific figure
grep -l "figure_name" scripts/*.py

# Search for output directory patterns
grep -l "figures/curation_impact" scripts/*.py

3. Organize Deprecated Files

Create clear structure:

mkdir -p deprecated/{figures,scripts,notebooks}
mkdir -p deprecated/figures/{unused_category1,unused_category2}
mkdir -p deprecated/scripts/unused_utilities

Use descriptive subdirectory names:

Good structure:

deprecated/
├── figures/
│   ├── unused_regression_plots/       # Category-based names
│   ├── unused_curation_impact/
│   └── exploratory_analysis/
├── scripts/
│   ├── unused_utilities/              # Purpose-based organization
│   ├── old_data_fetch/
│   └── notebook_fixes/
└── data/
    ├── intermediate_tables/
    └── old_versions/

Poor structure:

deprecated/
├── old_stuff/        # Too vague
├── misc/             # Unclear purpose
└── temp/             # Ambiguous

Benefits of good naming:

• Future-you understands what's in each folder
• Easy to restore specific categories
• Clear what can be safely deleted vs archived
• Documents project evolution

4. Document What Was Kept

Create MINIMAL_ESSENTIAL_FILES.md:

• List all active figures and their source scripts
• List essential scripts with their purposes
• Provide regeneration instructions
• Include restoration instructions for deprecated files

Example structure:

## Active Figures
1. figure_01.png - Used in Notebook A (Figure 1)
   - Generated by: script_14.py

## Essential Scripts
1. script_14.py - Generates Figures 1-4, 7
2. build_data.py - Required infrastructure

5. Verification Checklist

Before finalizing cleanup:

• [ ] All notebook-referenced figures identified
• [ ] Scripts generating those figures identified
• [ ] Unused files moved (not deleted) to deprecated/
• [ ] Documentation created (MINIMAL_ESSENTIAL_FILES.md)
• [ ] Regeneration commands tested
• [ ] Notebooks still work with cleaned structure

Benefits of This Approach

• Reduced confusion: Clear which files are active vs historical
• Easier maintenance: Only essential files to update
• Better documentation: Explicit mapping of figures to scripts
• Recoverable: Deprecated files preserved, not deleted
• Onboarding: New collaborators see minimal essential set

Identifying Files for Deprecation

When cleaning up analysis directories with multiple config file versions:

Patterns indicating old/superseded files:

1. Naming patterns:

• Files without _UPDATED suffix when _UPDATED versions exist
• Files with intermediate version numbers or dates
• Files named *_old.txt, *_backup.csv

2. Content indicators:

• Old parameter values (e.g., old color schemes)
• Old species names (e.g., Time Tree replacements)
• Incomplete coverage (fewer species than current)

3. Multiple similar files:

• itol_branch_colors.txt, itol_branch_colors_v2.txt, itol_branch_colors_UPDATED.txt
• Keep: _UPDATED.txt (current version)
• Deprecate: others (superseded)

Cleanup Strategy:

# Create deprecation directory with descriptive name
mkdir -p deprecated/phylo_old_configs

# Move superseded files (preserve for reference)
mv old_file1.txt old_file2.csv deprecated/phylo_old_configs/

# Verify move (should be empty or only current files)
ls *.txt *.csv

Files to keep in active directory:

• Current versions (e.g., *_UPDATED.*)
• Source scripts that generate configs
• Documentation (README, MANIFEST)
• Data files actively used

Files to deprecate:

• Superseded configs with old parameters
• Intermediate test versions
• Files from previous analysis versions
• Configs no longer referenced by notebooks

Example from phylo cleanup (18 files deprecated):

• Old: itol_3category_colorstrip.txt (old colors)
• Current: itol_3category_colorstrip_UPDATED.txt (new colors)
• Old: species_curation_methods.csv (2-category system)
• Current: species_3category_methods_UPDATED.csv (3-category system)

Benefits:

• Clear which files are current vs historical
• Reduced confusion when updating configs
• Preserved old versions for comparison
• Easier to identify files needing updates

Documentation Standards and Organization

Best practices for project documentation, change summaries, and documentation directory organization.

README.md Essentials

Every project should have a README with:

# Project Name

Brief description

## Installation

How to set up the environment

## Usage

How to run the analysis/code

## Project Structure

Brief overview of directories

## Data

Where data lives and how to access it

## Results

Where to find outputs

Code Documentation

• Docstrings for all functions/classes
• Comments for complex logic
• CHANGELOG.md for tracking changes
• TODO.md for tracking work (gitignored or removed before merge)

Change Documentation Best Practices

After major changes (cleanup, deprecation, restoration), create summary documents:

1. Create a dated summary document:

   OPERATION_SUMMARY_YYYY-MM-DD.md

2. Essential sections:

• Overview: What was done and why
• Problem: What issue was being addressed
• Solution: Actions taken
• Result: Current state after changes
• Files affected: What was moved/changed/restored
• Restoration: How to undo if needed

3. Examples of good summary docs:

• FIGURE_RESTORATION_SUMMARY.md - Documents restored files
• DEPRECATION_SUMMARY.md - Documents deprecated notebooks
• RECENT_CHANGES_SUMMARY.md - High-level overview

Template for Change Summaries

# [Operation] Summary - [Date]

## Problem
[Brief description of the issue]

## Solution
[What was done to address it]

### Files Changed
- **Moved**: [list]
- **Restored**: [list]
- **Updated**: [list]

## Current State
- **Active files**: [count and list]
- **Deprecated files**: [count and list]
- **Status**: [Ready/In Progress/etc.]

## Restoration Instructions

Commands to undo changes if needed

## Documentation Updated
- [List of docs that were updated]

---
**Date**: YYYY-MM-DD
**Status**: [Complete/Partial/etc.]

Why This Matters:

• Future users (including yourself) understand what changed
• Provides restoration instructions if needed
• Creates audit trail for project history
• Helps collaborators understand project evolution

Documentation Organization Strategy

Projects accumulate documentation files (.md, .log, .txt) in the root directory. Consolidate them effectively:

Structure

documentation/
├── README.md                    # Index to all documentation
├── logs/                        # Log files from processes
├── working_files/               # Temporary/working files
└── [organized .md files]

Implementation

# 1. Create structure
mkdir -p documentation/{logs,working_files}

# 2. Move documentation
mv *.md documentation/
mv *.log documentation/logs/
mv *.txt documentation/working_files/  # or keep essential ones in root

# 3. Create index (documentation/README.md)
cat > documentation/README.md << 'EOF'
# Project Documentation

## Quick Start
- ESSENTIAL_FILE.md - Start here
- RECENT_CHANGES.md - Latest updates

## By Category
### Analysis
- analysis_summary.md
- results.md

### Methods
- methods.md
- protocols.md

[etc...]
EOF

Documentation README Template

Include in documentation/README.md:

• Quick start section - Most important docs
• Categorical organization - Group by purpose
• File descriptions - One-line summaries
• File counts - Show organization scale
• Archive policy - Which docs are historical
• Access instructions - How to find specific info

What to Keep in Root

Keep in project root:

• README.md - Project overview
• LICENSE, CONTRIBUTING.md - Standard files
• .gitignore, config files

Move to documentation/:

• Analysis summaries
• Session notes
• Method descriptions
• Update logs
• All other markdown files

Benefits

• Clean root directory: Only essential project files visible
• Organized docs: Easy to find specific documentation
• Categorized: Logs separate from summaries separate from methods
• Indexed: README provides roadmap
• Scalable: Clear place for new documentation

Common Mistake

Do not delete old documentation - move it to documentation/archive/. Preserve history but organize it clearly.

MANIFEST System for Token-Efficient Navigation

For large data analysis projects, implement a MANIFEST system to enable efficient project navigation and minimize token usage in Claude Code sessions.

Problem Statement

Challenge: Large projects with many files consume excessive tokens during session startup:

• Reading large notebooks (4-6 MB) = 5,000-15,000 tokens
• Exploring data files and structure = 3,000-5,000 tokens
• Understanding scripts and workflows = 2,000-3,000 tokens
• Total: 15,000-23,000 tokens just for orientation!

Solution: MANIFEST files provide lightweight project indexes (~500-2,000 tokens each) that give complete context without reading actual files.

Token Efficiency Impact

Before MANIFESTs (reading actual files):

Root MANIFEST: N/A
Notebooks: ~10,000-15,000 tokens (reading 3 large notebooks)
Data exploration: ~3,000-5,000 tokens
Scripts analysis: ~2,000-3,000 tokens
---
Total: ~15,000-23,000 tokens for project orientation

After MANIFESTs (reading indexes):

Root MANIFEST: ~1,500 tokens (full project overview)
Subdirectory MANIFEST: ~500-1,000 tokens (specific area)
---
Total: ~2,000-2,500 tokens for complete context
Savings: 85-90% token reduction!

What is a MANIFEST?

A MANIFEST.md file is a comprehensive index for a directory that includes:

• Quick Reference: Entry points, key outputs, dependencies
• File Inventory: All files with descriptions, sizes, purposes
• Workflow Dependencies: How files relate and depend on each other
• Notes for Resuming Work: Current status, next steps, known issues
• Metadata: Tags, environment info, Obsidian notes links

Key Principle: MANIFEST provides 80% of context needed to resume work in 500-2,000 tokens instead of reading 5,000-15,000 tokens of actual files.

MANIFEST Structure

Root Directory MANIFEST Template

# [Project Name] - ROOT MANIFEST

**Last Updated**: YYYY-MM-DD
**Purpose**: [1-2 sentence description]
**Status**: Active/Deprecated/Archive

---

## Quick Reference

**Entry Points**: [Which files to read first]
**Key Outputs**: [Main deliverables]
**Dependencies**: [External requirements]

---

## Files

### Notebooks

#### `notebook_name.ipynb` (Size)
- **Purpose**: [What analysis/questions does this answer?]
- **Depends on**: [Input files, data, scripts]
- **Generates**: [Output files, figures]
- **Key findings**: [1-2 sentence summary]
- **Last modified**: YYYY-MM-DD
- **Execution time**: [~X minutes if relevant]
- **Priority**: [Main document or complementary analysis]

### Key Directories

#### `data/` (Size)
See `data/MANIFEST.md` for details.
- **Contents**: [Brief description]
- **Key files**: [Most important files]

[Repeat for figures/, scripts/, documentation/]

---

## Directory Structure

project/ ├── MANIFEST.md (this file) ├── data/ │ └── MANIFEST.md ├── figures/ │ └── MANIFEST.md └── [etc.]

---

## Workflow Dependencies

[Visual or text description of data -> processing -> outputs flow]

Example: 1. Data Acquisition: fetch_data.py -> data/raw/ 2. Processing: process.py -> data/processed/ 3. Analysis: analysis.ipynb -> figures/

---

## Notes for Resuming Work

**Current Status**: [What was last completed?]
**Next Steps**: [What needs to be done next?]
**Known Issues**: [Problems, TODOs, blockers]
**Reference**: [Links to related docs, other MANIFESTs]

---

## Metadata

**Created by**: Claude Code
**Project**: [Project name]
**Tags**: [#keywords #for #searching]
**Environment**: [conda env name or venv path]
**Obsidian notes path**: [Link to project notes]

---

## For Claude Code Sessions

**Quick Start for New Sessions**:
1. Read this MANIFEST.md (~500 tokens)
2. Read relevant subdirectory MANIFEST.md (~500 tokens)
3. Only read actual files when editing them

**Token Efficiency**:
- This MANIFEST provides 80% of context needed
- Subdirectory MANIFESTs provide detailed file info
- Read actual code/notebooks only when making changes

Subdirectory MANIFEST Template

Use the same structure but focused on the specific directory. For subdirectories:

data/MANIFEST.md - Focus on:

• Data provenance (where data came from)
• File formats and structure (rows, columns, size)
• Data dependencies (which files depend on which)
• Processing history (original -> processed versions)

figures/MANIFEST.md - Focus on:

• Which code generates which figure
• Figure purpose and key message
• Manuscript figure numbering
• Figure dependencies (data sources)

scripts/MANIFEST.md - Focus on:

• Script purpose and I/O
• Execution order and dependencies
• Usage examples and parameters
• Required dependencies

documentation/MANIFEST.md - Focus on:

• Document organization by category
• Critical entry points (RESUME_HERE.md)
• Active vs archived status
• Session summary locations

Linking MANIFESTs Across Directories

When implementing analysis_files/ (Iteration 2), create bidirectional links:

In figures/MANIFEST.md - Link to analysis files:

**01_figure_name.png** (318 KB)
- **Description**: Brief description
- **Analysis file**: `../analysis_files/figures/01_figure_name.md` - Detailed analysis

In analysis_files/MANIFEST.md - Link back to figures:

#### 01_figure_name.md
- **Figure file**: `figures/curation_impact_3cat/01_figure_name.png`
- **Purpose**: Detailed analysis and interpretation

In root MANIFEST.md - Reference both:

#### `analysis_files/` (~90 KB) **[NEW - ITERATION 2]**
- **Purpose**: Separate markdown files for figure analyses
- **Token efficiency**: ~98% reduction vs notebooks
- **Links to**: figures/ directory

This creates a navigable web of documentation.

MANIFEST Template File

Create MANIFEST_TEMPLATE.md in project root as a starting point. See the template in the Curation_Paper_figures project for a complete example with all sections and guidance.

Commands for MANIFEST Management

/generate-manifest Command

Create this command in .claude/commands/generate-manifest.md (or symlink from global commands):

Purpose: Automatically generates MANIFEST files by analyzing directory contents

Key Features:

• Analyzes directory type (root, data, figures, scripts, documentation)
• Extracts file information efficiently (sizes, dates, row counts)
• Identifies dependencies by searching code for file references
• Maps workflow relationships
• Uses AskUserQuestion for ambiguous information
• Marks fields requiring user input

Usage:

/generate-manifest              # Interactive mode
/generate-manifest data         # Generate for data/
/generate-manifest figures      # Generate for figures/

Implementation Tips:

• Don't read entire large files - use targeted searches
• Extract docstrings and header comments from scripts
• Use grep to find file references in code
• Check first/last cells of notebooks for descriptions
• Get row counts with wc -l for CSV files
• Target 1000-2000 tokens for root, 500-1000 for subdirectories

/update-manifest Command

Create this command in .claude/commands/update-manifest.md:

Purpose: Quickly updates existing MANIFESTs while preserving user content

Key Features:

• Preserves user-entered descriptions and notes
• Updates dates, sizes, and file existence
• Captures session context (asks "What did you accomplish?")
• Three modes: Minimal, Quick (default), Full
• Provides update summary

Usage:

/update-manifest              # Update current directory
/update-manifest data         # Update data/MANIFEST.md
/update-manifest --quick      # Force quick mode
/update-manifest --full       # Full re-analysis

Session End Pattern:

/update-manifest              # Capture session progress
/update-skills               # Save new knowledge
/safe-exit                   # Clean exit with notes

MANIFEST Workflow

Initial Setup

1. Create template:

   # Copy MANIFEST_TEMPLATE.md to project root
   # Or use /generate-manifest to create from scratch

2. Generate root MANIFEST:

   /generate-manifest
   # Choose "root directory"
   # Fill in user-specific fields

3. Generate subdirectory MANIFESTs:

   /generate-manifest data
   /generate-manifest figures
   /generate-manifest scripts
   /generate-manifest documentation

4. Customize MANIFESTs:

• Fill in [USER TO FILL] placeholders
• Add key findings summaries
• Document environment setup
• Add Obsidian notes paths

During Active Development

1. Start session - Read MANIFESTs for context:

   cat MANIFEST.md              # Project overview
   cat figures/MANIFEST.md      # If working on figures

2. Work on project - Normal development

3. End session - Update MANIFESTs:

   /update-manifest              # Captures session progress

After Major Changes

When you:

• Add new files or directories
• Reorganize structure
• Complete major analysis
• Make significant changes

Run full regeneration:

/generate-manifest --update     # Full re-analysis

MANIFEST Best Practices

Content Guidelines

1. Be concise but informative: Target 500-2,000 tokens 2. Front-load important info: Put critical details first 3. Use bullet points: Not paragraphs 4. Include dates: Everything should have timestamps 5. Think "6 months from now": What would you need to know?

What to Document

ALWAYS include:

• File purpose and key message
• Dependencies (inputs)
• Outputs (what it generates)
• Last modified date
• Size for large files

USER FILL fields for:

• Key findings (requires understanding)
• Priority classification (main vs complementary)
• Known issues and TODOs
• Environment names
• Obsidian note paths

Tip for filling user-specific fields:

• Obsidian notes path: Check .claude/project-config file - it often contains the vault path in the obsidian_vault or similar field
• Environment name: Check conda env list or look for environment.yml/requirements.txt
• Key findings: Analyze generation scripts or read notebook markdown cells for summaries

Auto-generate from code:

• File sizes and dates
• Row/column counts for data
• Dependencies (by searching code)
• Script usage (from docstrings)

Documenting New Analysis Notebooks in MANIFEST

Template for Analysis Notebook Entries:

When adding a new analysis notebook to MANIFEST.md, include:

#### `Notebook_Name.ipynb` (file size) **[NEW]**
- **Purpose**: One-sentence objective of the analysis
- **Type**: Category (e.g., "Confounding analysis", "Data enrichment", "Primary analysis")
- **Rationale**: Why this analysis is needed (2-3 sentences explaining motivation)
- **Approach**:
  - Bullet points of analytical steps
  - Key methodological decisions
- **Key Questions**:
  - Question 1 the analysis addresses
  - Question 2 the analysis addresses
- **Depends on**:
  - data/input_file.csv (description)
  - scripts/processing_script.py
- **Generates**:
  - figures/output_dir/figure1.png (what it shows)
  - results/statistics.csv
- **Dataset**: N assemblies/samples, key statistics
- **Last modified**: YYYY-MM-DD
- **Status**: Current state (e.g., "Code optimized", "In progress", "Complete")
- **Execution time**: ~XX minutes
- **Priority**: Role in project (e.g., "Confounding analysis - validates main findings")
- **Note**: Important caveats or special considerations

Example: Technology/Temporal Confounding Analysis

#### `Technology_Temporal_Analysis.ipynb` (32 KB) **[NEW]**
- **Purpose**: Investigate whether sequencing technology (CLR vs HiFi) and temporal trends confound the curation method comparisons
- **Type**: Confounding analysis - technology and temporal effects
- **Rationale**: Sequencing technology evolved rapidly (CLR -> HiFi), and assembly methods may correlate with technology era. Need to determine if observed quality differences are due to curation methods or underlying technology/temporal confounders.
- **Approach**:
  - Technology-separated analysis: Compare categories split by sequencing technology
  - Temporal trend analysis: Plot quality metrics over time (2019-2025)
  - HiFi-only temporal analysis: Eliminate technology confounding
- **Key Questions**:
  - Are quality differences consistent across technologies (HiFi vs CLR)?
  - Do quality metrics improve over time, and is this technology-driven?
  - Do temporal trends persist when technology is held constant?
- **Depends on**:
  - `data/vgp_assemblies_3categories_tech.csv` (3-category data with technology inference)
  - scipy for statistical tests (Mann-Whitney U, Spearman correlation)
- **Generates**:
  - `figures/technology_temporal/01_prialt_tech_comparison.png` (HiFi vs CLR)
  - `figures/technology_temporal/04_hifi_only_temporal_trends.png` (HiFi-only, 2021-2025)
  - `figures/technology_temporal/technology_effects_statistics.csv`
  - `figures/technology_temporal/temporal_trends_hifi_only_statistics.csv`
- **Dataset**: 541 VGP assemblies, 464/541 (86%) with technology assignment (355 HiFi, 107 CLR)
- **Last modified**: 2026-02-25
- **Status**: Code optimized (DPI reduced 300->150 to prevent image loading errors)
- **Execution time**: ~10-15 minutes
- **Priority**: Confounding analysis - validates that curation effects are not driven by technology or temporal biases
- **Note**: Figure sizes reduced (DPI 150) to prevent notebook image loading errors

Benefits of Comprehensive Documentation:

1. Resume work easily: Understand analysis purpose months later 2. Collaboration: Others can understand without reading code 3. Dependency tracking: Know what data/scripts are required 4. Output tracking: Know what files this notebook generates 5. Execution planning: Estimate time needed to re-run 6. Prioritization: Understand role in overall project

Update Frequency

• End of every session: Quick update with /update-manifest
• After adding files: Note new files, mark as [TO BE DOCUMENTED]
• After major changes: Full regeneration with /generate-manifest
• Before sharing: Ensure MANIFESTs are current

MANIFEST Session Context Updates

Structure for "Recent Session Work" Section:

When running /update-manifest, document the session in this format:

**Recent Session Work** (YYYY-MM-DD):
- **[Action taken]**:
  - Specific change 1 with details
  - Specific change 2 with quantitative results
  - Why this change was made
- Brief description of problem solved or feature added
- Any updates to directory structure or workflow

Example Session Documentation:

**Recent Session Work** (2026-02-25):
- **Updated Technology_Temporal_Analysis.ipynb code**:
  - Reduced DPI from 300->150 in global settings and all savefig calls
  - Reduced figure sizes: 01_prialt (15x10->12x8), 02_all_tech (18x12->14x9)
  - Prevents image loading errors while maintaining publication quality
  - File sizes reduced by ~75% (combination of DPI and size reduction)
- Added Technology_Temporal_Analysis.ipynb entry to root MANIFEST
- Updated directory structure to include figures/technology_temporal/
- Added HiFi-only temporal analysis section to eliminate technology confounding

Next Steps Format:

Prioritize and number action items:

**Next Steps**:
1. **[High priority action]** - [Why it's important]
2. **[Medium priority]** - [Context]
3. **[Future work]** - [When to tackle]

Example:

**Next Steps**:
1. **Re-run Technology_Temporal_Analysis.ipynb** - Execute cells to regenerate figures with optimized 150 DPI settings
2. **Generate notebook with temporal effect and only HiFi data** - Already added to notebook, need to execute new cells
3. **Write integrated manuscript Results section** - Combine findings from all 5 clades into cohesive narrative

This structure makes it easy to resume work by quickly understanding what was done and what's next.

Integration with Project Structure

Add MANIFESTs to standard project organization:

project/
├── MANIFEST.md                 # Root project index
├── MANIFEST_TEMPLATE.md        # Template for new MANIFESTs
├── data/
│   ├── MANIFEST.md            # Data inventory
│   └── [data files]
├── figures/
│   ├── MANIFEST.md            # Figure catalog
│   └── [figure files]
├── scripts/
│   ├── MANIFEST.md            # Script documentation
│   └── [script files]
├── documentation/
│   ├── MANIFEST.md            # Doc organization
│   └── [doc files]
└── [other directories with MANIFESTs as needed]

Common MANIFEST Patterns

For Data Directories

Emphasize:

• Data provenance and source
• File formats and structure
• Original vs processed versions
• Data dependencies and lineage
• Size and scale information

For Figure Directories

Emphasize:

• Generating code (which notebook/script)
• Data sources
• Manuscript figure numbers
• Key messages and findings
• Figure dependencies

For Script Directories

Emphasize:

• Input/output relationships
• Execution order
• Usage examples
• Dependencies (packages)
• Script purpose and logic

For Documentation Directories

Emphasize:

• Entry points (where to start)
• Document categories
• Active vs archived
• Session summaries location
• Critical documents for resuming work

Real-World Example

See the Curation_Paper_figures project for a complete implementation:

• 5 MANIFEST files (root + 4 subdirectories)
• ~10,000 lines of documentation
• Covers 3 notebooks, 12 scripts, 18 figures, 48 doc files
• Enables session startup in 2,000 tokens vs 15,000+ tokens
• Includes working examples of all MANIFEST types

Benefits Summary

For Claude Code:

• 85-90% reduction in session startup tokens
• Fast project orientation (2-3 MANIFESTs vs 20+ files)
• Clear entry points and workflow understanding
• Efficient file navigation without exploring

For Users:

• Quick work resumption (read 1 MANIFEST vs 10+ files)
• Clear project documentation
• Session continuity (Notes for Resuming Work)
• Workflow transparency (dependency maps)

For Teams:

• Faster onboarding for new members
• Shared understanding of project structure
• Clear documentation of decisions
• Easier code review (understand context quickly)

Troubleshooting

MANIFEST too long (>2,500 tokens):

• Break into subdirectory MANIFESTs
• Use "See subdirectory MANIFEST" links
• Summarize instead of listing all files

MANIFEST outdated:

• Set up session-end habit: /update-manifest before /safe-exit
• Use /generate-manifest --update for full refresh
• Add "Last Updated" reminders

Too many [USER TO FILL] fields:

• Fill in during active work, not after
• Use /update-manifest to capture context immediately
• Ask user questions during generation for key info

Unclear what to include:

• Think: "What would I need to resume work in 6 months?"
• Include anything that saves reading a file
• Front-load critical information

File Naming Conventions

Detailed naming rules and patterns for project files.

General Rules

1. Use lowercase with hyphens or underscores

• Good: data-analysis.py or data_analysis.py
• Bad: DataAnalysis.py or data analysis.py

2. Be descriptive but concise

• Good: process-telomere-data.py
• Bad: script.py or process_all_the_telomere_sequencing_data_from_experiments.py

3. Use consistent separators

• Choose either hyphens or underscores and stick with it
• Convention: hyphens for file names, underscores for Python modules

4. Include version/date for important outputs

• Good: report-2026-01-23.pdf or model-v2.pkl
• Bad: report-final-final-v3.pdf

Numbered Sequences

For sequential files (notebooks, scripts), use zero-padded numbers:

notebooks/
├── 01-data-exploration.ipynb
├── 02-quality-control.ipynb
├── 03-statistical-analysis.ipynb
└── 04-visualization.ipynb

Data Files

Include metadata in filename when possible:

data/raw/
├── sample-A_hifi_reads_2026-01-15.fastq.gz
├── sample-B_hifi_reads_2026-01-15.fastq.gz
└── reference_genome_v3.fasta

Deprecated File Naming

Use descriptive names that indicate origin:

Good:

• data-processing/telomeres_versions - Shows it's from telomeres
• data-processing/phylo_pre_feb20 - Shows origin and date
• notebooks_20260226 - Shows category and date

Avoid:

• old_stuff - Too vague
• backup1, backup2 - Not descriptive
• deprecated - Too generic (what is deprecated?)

Session Notes Storage

Standardize on `session-saves/` (not sessions-history/, Archive/, or other variants)

Benefits:

• Consistent with standard naming pattern (noun + plural)
• Matches archived/ pattern
• Works across all projects
• Prevents confusion when writing automation scripts

When reorganizing:

mv sessions-history/ session-saves/
mv Archive/ session-saves/  # if used for session notes

Project folder template (with session notes):

project-name/
├── TO-DOS.md                # Project-specific tasks
├── session-saves/           # Working session notes (tagged with #dump)
├── archived/                # Processed/consolidated notes
│   ├── daily/              # Daily consolidations
│   └── monthly/            # Monthly summaries
├── Planning/               # Planning documents
├── Development/            # Development notes
└── [other content folders]

Integration with Obsidian:

• Session notes should have dump tag in frontmatter for easy filtering
• See the obsidian skill for complete details on the dump tag requirement and frontmatter schema
• Archiving workflow: move from session-saves/ to archived/daily/ or archived/monthly/

Standard Project Structures

Detailed directory structure templates for different project types.

Research/Analysis Projects

project-name/
├── README.md                 # Project overview and getting started
├── .gitignore               # Exclude data, outputs, env files
├── environment.yml          # Conda environment (or requirements.txt)
├── data/                    # Input data (often gitignored)
│   ├── raw/                # Original, immutable data
│   ├── processed/          # Cleaned, transformed data
│   └── external/           # Third-party data
├── notebooks/               # Jupyter notebooks for exploration
│   ├── 01-exploration.ipynb
│   ├── 02-analysis.ipynb
│   └── figures/            # Notebook-generated figures
├── src/                     # Source code (reusable modules)
│   ├── __init__.py
│   ├── data_processing.py
│   ├── analysis.py
│   └── visualization.py
├── scripts/                 # Standalone scripts and workflows
│   ├── download_data.sh
│   └── run_pipeline.py
├── tests/                   # Unit tests
│   └── test_analysis.py
├── docs/                    # Documentation
│   ├── methods.md
│   └── references.md
├── results/                 # Analysis outputs (gitignored)
│   ├── figures/
│   ├── tables/
│   └── models/
└── config/                  # Configuration files
    └── analysis_config.yaml

Development Projects

project-name/
├── README.md
├── .gitignore
├── setup.py                 # Package configuration
├── requirements.txt         # or pyproject.toml
├── src/
│   └── package_name/
│       ├── __init__.py
│       ├── core.py
│       └── utils.py
├── tests/
│   ├── test_core.py
│   └── test_utils.py
├── docs/
│   ├── api.md
│   └── usage.md
├── examples/                # Example usage
│   └── example_workflow.py
└── .github/                 # CI/CD workflows
    └── workflows/
        └── tests.yml

Bioinformatics/Workflow Projects

project-name/
├── README.md
├── data/
│   ├── raw/                # Raw sequencing data
│   ├── reference/          # Reference genomes, annotations
│   └── processed/          # Workflow outputs
├── workflows/               # Galaxy .ga or Snakemake files
│   ├── preprocessing.ga
│   └── assembly.ga
├── config/
│   ├── workflow_params.yaml
│   └── sample_sheet.tsv
├── scripts/                # Helper scripts
│   ├── submit_workflow.py
│   └── quality_check.py
├── results/                # Final outputs
│   ├── figures/
│   ├── tables/
│   └── reports/
└── logs/                   # Workflow execution logs

Data Analysis Projects with Notebooks

For projects involving Jupyter notebooks, data analysis, and visualization with many generated figures:

project-name/
├── README.md                # Project overview
├── .gitignore
├── notebooks/               # Jupyter notebooks (analysis, exploration)
│   ├── 01-data-loading.ipynb
│   ├── 02-exploratory-analysis.ipynb
│   └── 03-final-analysis.ipynb
├── figures/                 # ALL generated visualizations (PNG, PDF, SVG)
│   ├── fig1_distribution.png
│   ├── fig2_correlation.png
│   └── supplementary_*.png
├── data/                    # Data files (JSON, CSV, TSV, Excel)
│   ├── raw/                # (optional) Original, unprocessed data
│   ├── processed/          # (optional) Cleaned, processed data
│   ├── input_data.json
│   └── metadata.tsv
├── tests/                   # Test scripts (test_*.py, pytest)
│   ├── test_processing.py
│   └── test_analysis.py
├── scripts/                 # Standalone Python/R scripts
│   ├── data_fetch.py
│   └── preprocessing.py
├── docs/                    # Documentation (MD, RST files)
│   ├── methods.md
│   └── analysis_notes.md
└── archives/                # Compressed archives, old versions
    └── backup_YYYYMMDD.tar.gz

Benefits of This Structure:

• Clear separation of concerns (code vs. data vs. outputs)
• Easy navigation: Find all figures in one place
• Scalability: Handles 50+ figures without cluttering root
• Git-friendly: Easy to .gitignore large data/figures
• Collaboration: Standard structure reduces onboarding time

When to Use This Structure:

• Projects with multiple notebooks
• Analysis generating many visualizations (10+ figures)
• Multiple data sources/formats
• Team collaboration
• Long-term research projects

MANIFEST Integration

For enhanced navigation and token efficiency, add MANIFEST.md files:

project-name/
├── MANIFEST.md                  # Root project index (~1,500 tokens)
├── MANIFEST_TEMPLATE.md         # Template for creating new MANIFESTs
├── notebooks/
│   ├── MANIFEST.md             # (optional) Notebook catalog
│   └── [notebook files]
├── figures/
│   ├── MANIFEST.md             # Figure catalog (~500-1,000 tokens)
│   └── [figure files]
├── data/
│   ├── MANIFEST.md             # Data inventory (~500-1,000 tokens)
│   └── [data files]
├── scripts/
│   ├── MANIFEST.md             # Script documentation (~500-1,000 tokens)
│   └── [script files]
└── documentation/
    ├── MANIFEST.md             # Doc organization (~500-1,000 tokens)
    └── [doc files]

Benefits:

• 85-90% token reduction for session startup
• Complete project context in ~2,000 tokens (vs 15,000+)
• Quick work resumption without reading files
• Clear workflow documentation

See manifest-system.md for complete MANIFEST documentation.

Project Reorganization Guide

Systematic approach to reorganizing existing project folder structures without breaking dependencies.

Post-Reorganization Path Updates

After reorganizing folders, ALWAYS verify and update file paths in all scripts and notebooks. Files often contain hardcoded paths that break after reorganization.

Systematic Path Update Process

1. Identify all files that reference other files:

   # Find notebooks
   find . -name "*.ipynb"

   # Find Python scripts
   find . -name "*.py"

   # Search for file references
   grep -r "\.json\|\.csv\|\.tsv\|\.png" --include="*.ipynb" --include="*.py"

2. Common file reference patterns to search for:

• Data files: .json, .csv, .tsv, .xlsx
• Figures: .png, .jpg, .pdf, .svg
• Python: open(), read_csv(), read_json(), load()
• Jupyter: savefig(), file I/O operations

3. Update paths using sed for batch replacements:

   # Update data file paths
   sed -i.bak "s|'filename.json'|'../data/filename.json'|g" notebook.ipynb

   # Update figure output paths
   sed -i.bak "s|savefig('fig|savefig('../figures/fig|g" notebook.ipynb

   # Update with double quotes (common in Python code)
   sed -i.bak 's|"filename.csv"|"../data/filename.csv"|g' script.py

4. Verify updates:

   # Check that paths were updated correctly
   grep -o "'../data/[^']*'" notebook.ipynb | head -5
   grep -o "'../figures/[^']*'" notebook.ipynb | head -5

5. Clean up backup files:

   rm *.bak *.bak2 *.bak3

Files to Check

Always check these file types after reorganization:

• Jupyter notebooks (.ipynb): Data loading, figure saving
• Python scripts (.py): File I/O operations
• Test files (test_*.py): Often reference data fixtures
• Data processing scripts: Input/output paths
• Documentation: Code examples with file paths
• Configuration files: Paths to resources

Common Path Patterns

Tips

• Use relative paths (../data/) not absolute paths for portability
• Batch process similar changes using sed or find/replace
• Test one file type at a time (notebooks first, then scripts)
• Keep backup files (.bak) until verification complete
• Grep to verify changes were applied correctly
• Consider case sensitivity on different operating systems

Verification and Cleanup After Reorganization

After completing a reorganization, always verify the results and clean up:

1. Verify File Counts

# Count files moved to each directory
for dir in figures data tests notebooks docs archives; do
  echo "$dir: $(ls $dir 2>/dev/null | wc -l | tr -d ' ') files"
done

2. Check Root Directory

# Ensure root is clean
ls -la

# Should only see:
# - Organized directories (figures/, data/, etc.)
# - Project-specific folders (Fetch_data/, sharing/)
# - Config directories (.claude/, .git/)
# - Essential files (README.md, .gitignore, etc.)

3. Remove Temporary Files

# Remove Jupyter checkpoints (auto-generated, not needed in version control)
rm -rf .ipynb_checkpoints

# Remove sed backup files
rm *.bak *.bak2 *.bak3

# Remove duplicate/backup data files
rm *.backup *.backup2 *.old

4. Display Final Structure

# Show clean directory tree
tree -L 2 -d

# Or list directories only
ls -d */

Verification Checklist

• [ ] All target directories created successfully
• [ ] File counts match expectations (no files lost)
• [ ] Root directory is clean (no scattered files)
• [ ] Temporary/backup files removed
• [ ] Paths in notebooks/scripts updated (see "Post-Reorganization Path Updates")
• [ ] Structure documented (README or similar)
• [ ] Test that notebooks/scripts still run correctly

After Reorganization: Session Folder Cleanup

Clean up old session folder variations:

If you renamed folders during reorganization (e.g., sessions-history/ -> session-saves/):

1. Move archived sessions:

mv old-folder/*.md session-saves/archived/daily/

2. Remove empty old folders safely:

rmdir old-folder/  # Fails if not empty - safety check

3. Update any links to old folder structure

Don't leave multiple session folders:

• Bad: sessions-history/ and session-saves/ both present
• Good: Single session-saves/ with archived content

Verification:

# Check for session folder variations
find . -type d -name "*session*" -not -path "*/archived/*"
# Should show only: ./session-saves/

Cleanup and Maintenance

Regular Maintenance Tasks

1. Archive old branches - Delete merged feature branches 2. Clean temp files - Remove TODO.md, NOTES.md from completed work 3. Update documentation - Keep README current with changes 4. Review .gitignore - Ensure large files aren't tracked 5. Organize notebooks - Rename/renumber as project evolves

End-of-Project Checklist

• [ ] README complete and accurate
• [ ] Code documented
• [ ] Tests passing
• [ ] Large files gitignored
• [ ] Working files removed (TODO.md, scratch notebooks)
• [ ] Final outputs in results/
• [ ] Environment files current
• [ ] License added (if applicable)