Documentation Organization

Documentation Organization for Research Projects

Supporting files in this directory:

- migration-guide.md - Migration from flat to directory-based organization, pattern reference, format debugging

- manifest-updates.md - MANIFEST update best practices, patterns, and checklists

- version-control-and-examples.md - Version control for document iterations, VGP project example

Overview

Organize project documentation in a structured way that separates internal working files from shareable content. This makes it easy to:

• Find documents during development
• Prepare sharing packages
• Maintain clean professional documentation
• Collaborate effectively

Recommended Structure

documentation/
├── README.md                          # Documentation index
│
├── data_descriptions/                 # SHARE: Data understanding
│   ├── dataset_name_README.md
│   ├── column_definitions.md
│   └── data_sources.md
│
├── methods/                           # SHARE: Methodology
│   ├── workflow.md
│   ├── analysis_plan.md
│   └── protocol.md
│
├── results/                           # SHARE: Key findings
│   ├── analysis_summary.md
│   └── key_findings.md
│
├── reference/                         # SHARE: External references (optional)
│   ├── citations.md
│   └── useful_resources.md
│
├── progress/                          # INTERNAL: Development tracking
│   ├── PROGRESS.md
│   ├── session_YYYY-MM-DD.md
│   └── RESUME_HERE.md
│
├── action_reports/                    # INTERNAL: What was done
│   ├── corrections_YYYY-MM-DD.md
│   ├── figure_regeneration.md
│   ├── data_verification.md
│   └── updates_summary.md
│
├── todos/                             # INTERNAL: Planning
│   ├── priorities.md
│   ├── search_lists.md
│   └── task_tracking.md
│
├── internal/                          # INTERNAL: Project management
│   ├── minimal_essential_files.md
│   ├── documentation_organization.md
│   └── notebook_issues.md
│
└── deprecated/                        # INTERNAL: Old versions
    ├── old_analysis_v1.md
    └── deprecated_workflow.md

Categorization Guide

data_descriptions/ (SHARE)

Purpose: Help recipients understand the data

Include:

• Dataset descriptions and README files
• Column/variable definitions
• Data sources and provenance
• Data quality notes
• Expected formats

Examples:

• vgp_assemblies_README.md
• column_definitions.md
• data_sources.md
• karyotype_data_README.md

methods/ (SHARE)

Purpose: Explain how analysis was done

Include:

• Methodology documentation
• Analysis workflows
• Protocols and procedures
• Step-by-step guides

Examples:

• karyotype_workflow.md
• analysis_plan.md
• data_fetching_protocol.md
• quality_control_methods.md

results/ (SHARE)

Purpose: Present findings and conclusions

Include:

• Analysis summaries
• Key findings
• Interpretation notes
• Publication-ready summaries

Examples:

• analysis_summary.md
• complete_results.md
• haplotype_analysis_summary.md

reference/ (SHARE - optional)

Purpose: Provide additional context

Include:

• Citations and references
• Useful external resources
• Related work
• Background reading

progress/ (INTERNAL)

Purpose: Track development and resume work

Include:

• Progress tracking files
• Session notes
• Resume-here files
• Status updates

Examples:

• PROGRESS.md
• session_2026-02-05.md
• RESUME_HERE.md
• tier1_search_progress.md

Pattern matching: *progress*, *session*, *resume*

action_reports/ (INTERNAL)

Purpose: Document what actions were taken

Include:

• Corrections and fixes
• Update summaries
• Figure regeneration notes
• Data verification reports
• Migration documentation

Examples:

• corrections_complete.md
• figure_regeneration_summary.md
• data_verification.md
• updates_summary.md
• migration_changes.md

Pattern matching: *correction*, *update*, *regeneration*, *restoration*, *verification*, *migration*

todos/ (INTERNAL)

Purpose: Planning and task management

Include:

• Priority lists
• Search task lists
• Task tracking
• To-do items

Examples:

• karyotype_search_priorities.md
• analysis_todos.md

Pattern matching: *todo*, *priority*, *task*

internal/ (INTERNAL)

Purpose: Project meta-documentation

Include:

• Essential files documentation
• Organization notes
• Issue tracking
• Project management notes

Examples:

• minimal_essential_files.md
• documentation_organization.md
• notebook_coherence_issues.md
• text_fixes_needed.md

Pattern matching: *essential*, *organization*, *issue*, *fixes*, *coherence*

deprecated/ (INTERNAL)

Purpose: Old versions kept for reference

Include:

• Deprecated files
• Old versions
• Superseded documentation

Pattern matching: *deprecated*, *old*, *backup*

Sharing Package Selection

When creating sharing packages, include only these directories:

SHARE_INCLUDE = [
    'data_descriptions',  # Essential for understanding data
    'methods',            # Essential for understanding methodology
    'results',            # Essential for understanding findings
    'reference'           # Optional: external references
]

INTERNAL_EXCLUDE = [
    'progress',           # Internal tracking
    'action_reports',     # Internal updates
    'todos',              # Internal planning
    'internal',           # Project management
    'deprecated',         # Old versions
    'logs',               # Runtime logs
    'working_files'       # Temporary files
]

Implementation

For New Projects

mkdir -p documentation/{data_descriptions,methods,results,reference,progress,action_reports,todos,internal,deprecated}

# Create documentation README
# For general project README templates, see the folder-organization skill
cat > documentation/README.md << 'EOF'
# Project Documentation

## For Recipients (Shareable)
- **data_descriptions/** - Understanding the data
- **methods/** - How the analysis was done
- **results/** - Key findings and summaries

## Internal (Development)
- **progress/** - Progress tracking and session notes
- **action_reports/** - Updates, corrections, verifications
- **todos/** - Task lists and priorities
- **internal/** - Project management
EOF

For Existing Projects

Reorganize documentation gradually:

1. Create new structure 2. Move files to appropriate folders 3. Update any references 4. Test that notebooks still work

See [migration-guide.md](migration-guide.md) for detailed migration commands, pattern reference table, and format debugging templates.

In share-project Command

Update filtering to use directory-based exclusion:

# Exclude entire directories
EXCLUDE_DIRS = ['progress', 'action_reports', 'todos', 'internal',
                'deprecated', 'logs', 'working_files']

# Or include only specific directories
INCLUDE_DIRS = ['data_descriptions', 'methods', 'results', 'reference']

Sharing Package Integration

Directory-Based Filtering (Recommended)

When creating sharing packages from projects with organized documentation:

Advantages:

• Simple and maintainable (no complex pattern matching)
• Clear intent (directory name = purpose)
• Easy to audit (just look at directory list)
• Scalable (add new categories without updating filters)

Implementation in share-project command:

def ignore_internal_dirs(dir, files):
    """Exclude internal documentation directories."""
    ignore_list = []
    for item in files:
        # Exclude internal directories
        if item in ['progress', 'action_reports', 'todos', 'internal',
                   'deprecated', 'logs', 'working_files', 'temp', 'tmp']:
            ignore_list.append(item)
        # Exclude hidden files except .gitkeep
        elif item.startswith('.') and item != '.gitkeep':
            ignore_list.append(item)
    return ignore_list

shutil.copytree("documentation", f"{SHARE_DIR}/documentation",
                ignore=ignore_internal_dirs,
                dirs_exist_ok=True)

Quick Reference:

• Include: data_descriptions/, methods/, results/, reference/
• Exclude: progress/, action_reports/, todos/, internal/, deprecated/

Why Directory-Based Over File-Pattern Matching?

Before (file-pattern matching):

# Complex, hard to maintain, easy to miss files
exclude_patterns = [
    '*CORRECTION*', '*UPDATE*', '*VERIFICATION*', '*REGENERATION*',
    '*RESTORATION*', '*PROGRESS*', '*SESSION*', '*RESUME*',
    '*PRIORITY*', '*TODO*', '*TASK*', '*ESSENTIAL*', '*ISSUE*',
    '*FIXES*', '*COHERENCE*', '*DEPRECATED*', '*CLEANUP*',
    '*MIGRATION*', # ... and many more
]

After (directory-based):

# Simple, clear, maintainable
exclude_dirs = ['progress', 'action_reports', 'todos', 'internal', 'deprecated']

Impact: In practice, this approach eliminated 15+ incorrectly shared files and reduced maintenance complexity from 50+ patterns to 5 directories.

Benefits

1. Clear Organization: Easy to find documents during development 2. Easy Sharing: Simply include/exclude directories when sharing 3. Professional: Recipients see clean, relevant documentation 4. Maintainable: Clear categories make it obvious where to put new docs 5. Collaborative: Team members understand the structure 6. Archival: Easy to separate essential from temporary documentation

Best Practices

1. Name files descriptively: Use clear prefixes/suffixes 2. Date internal docs: Add dates to action reports and progress notes 3. Update README: Keep documentation/README.md current 4. Regular cleanup: Move old files to deprecated/ 5. Consistent naming: Use established patterns for easy categorization 6. Test sharing: Verify shared packages have needed documentation

See also:

- manifest-updates.md for MANIFEST update patterns and checklists

- version-control-and-examples.md for document iteration tracking and a full VGP project example

MANIFEST Update Best Practices

When updating MANIFEST files (data/MANIFEST.md, scripts/MANIFEST.md, etc.) after significant project changes, follow these patterns.

Comprehensive Context Update Pattern

Purpose: Provide future sessions with complete context to resume work efficiently

Key Sections to Update:

1. Last Updated Date: Always update to current date

   **Last Updated**: 2026-02-25 (data consolidation complete, AWS QC enrichment added)

2. Notes for Resuming Work: Most critical section

   ## Notes for Resuming Work

   **Current Status** (YYYY-MM-DD):
   - Completed item 1: Brief description
   - Completed item 2: Brief description
   - Completed item 3: Brief description

   **Next Steps**:
   1. Priority task 1
   2. Priority task 2

   **Known Issues**:
   - Issue 1: Description and context
   - Issue 2: Description and context

3. File Sizes and Counts: Update actual metrics

   - **Size**: 482 KB, 716 rows, 77 columns

4. Quick Reference: Update entry points if data structure changed

   **Entry Points**:
   1. **Start here**: `file.csv` - Description (N rows, M columns)

Information to Capture

Completed Work:

• What was accomplished (data consolidation, enrichment, verification)
• How many files affected (e.g., "34 files reduced to 2")
• Where deprecated files moved (e.g., "archived in deprecated/data_backups_20260225/")
• New columns/features added (e.g., "added 3 QC columns: busco_completeness, busco_lineage, merqury_qv")

Next Steps:

• Immediate priorities (e.g., "Verify analyses with enriched data")
• Follow-up tasks (e.g., "Document enrichment methodology")
• Future improvements (e.g., "Run full AWS enrichment to increase coverage")

Known Issues:

• Incomplete work (e.g., "BUSCO/Merqury coverage only ~21% (TEST_MODE)")
• Missing dependencies (e.g., "VGP haplotype comparison files missing")
• Warnings about data quality or coverage

Ask User for Context

When updating MANIFEST, gather context from user via AskUserQuestion:

AskUserQuestion({
    "questions": [{
        "question": "What did we accomplish in this session?",
        "header": "Accomplished",
        "options": [
            {"label": "Data consolidation", "description": "..."},
            {"label": "New analysis", "description": "..."},
            # ... more options
        ],
        "multiSelect": True
    }]
})

Example: Data Consolidation Update

Before:

**Last Updated**: 2026-02-19
**Purpose**: VGP assembly metadata
**Status**: Active

After:

**Last Updated**: 2026-02-25 (data consolidation complete, AWS QC enrichment added)
**Purpose**: VGP assembly metadata and quality metrics in unified, optimized format
**Status**: Active - Consolidated to 2 core files only

## Notes for Resuming Work

**Current Status** (2026-02-25):
- Data consolidation complete: Reduced from 34 CSV files to 2 core files
- AWS QC enrichment added: BUSCO/Merqury data fetched from GenomeArk
- All 34 deprecated files safely archived in `deprecated/data_backups_20260225/`
- All notebooks verified to use correct consolidated data
- 3categories file rebuilt with correct filtering (541 assemblies)

**Next Steps**:
1. Verify analyses with enriched data
2. Document enrichment methodology

**Known Issues**:
- BUSCO/Merqury coverage only ~21% (TEST_MODE - need full enrichment)
- VGP haplotype comparison files missing

When to Update MANIFEST

Update MANIFEST whenever:

• Data structure changes (consolidation, new columns, file reorganization)
• Major analysis milestones (completed figure generation, finished enrichment)
• File deprecation (moved files to deprecated/, changed canonical files)
• Known issues discovered (missing data, coverage gaps, broken references)
• Before ending session (via /update-manifest command)

Benefits

1. Efficient resumption: Future sessions have complete context 2. Avoid redundant work: Known issues documented to prevent re-investigation 3. Clear next steps: Priorities explicitly listed 4. Accurate documentation: File sizes, counts, and status match reality 5. Session continuity: Each session builds on documented progress

---

MANIFEST Update Patterns

Common MANIFEST update patterns for different types of changes.

Pattern 1: Data Consolidation

Trigger: Reduced multiple data files to single source of truth

Update:

**Last Updated**: YYYY-MM-DD (data consolidation complete)
**Status**: Active - Consolidated to N core files only

**Current Status**:
- Data consolidation complete: Reduced from X files to N core files
- All deprecated files archived in `deprecated/data_backups_YYYYMMDD/`
- All notebooks verified to use consolidated data

**Known Issues**:
- [Any files still referencing deprecated data]

Also Update:

• File sizes (actual size after consolidation)
• Row/column counts
• Quick Reference (which file to use for what)
• Dependencies (update script references)

Pattern 2: Data Enrichment

Trigger: Added new columns from external sources (AWS, APIs, manual curation)

Update:

**Last Updated**: YYYY-MM-DD (enrichment added: [source])

**Current Status**:
- [Source] enrichment added: [What data] (coverage: X%)
- New columns: col1, col2, col3

**Next Steps**:
1. Re-run analyses with enriched data
2. Document enrichment methodology

**Known Issues**:
- Coverage only X% ([reason, e.g., TEST_MODE, missing source data])

Also Update:

• Column count (add new columns to total)
• Column descriptions in metadata files
• Dependencies (enrichment notebook/script)

Pattern 3: File Deprecation/Reorganization

Trigger: Moved files to deprecated/, changed canonical file locations

Update:

**Current Status**:
- Files reorganized: [describe change]
- Deprecated files archived in `deprecated/[subfolder]/`
- All references verified and updated

**Entry Points**:
1. **OLD (deprecated)**: `old_file.csv` -> moved to deprecated/
2. **NEW (use this)**: `new_file.csv` - Description

Also Update:

• Remove deprecated files from main file list
• Add note in Quick Reference about location change
• Update dependencies section

Pattern 4: Analysis Milestone

Trigger: Completed major analysis, generated figures, finished enrichment

Update:

**Current Status**:
- [Analysis name] complete: [brief description]
- Figures generated: [which figures, where saved]
- Results documented in: [notebook/report path]

**Next Steps**:
1. [Follow-up analysis]
2. [Documentation tasks]

Pattern 5: Discovered Issues

Trigger: Found missing data, broken references, coverage gaps

Update:

**Known Issues**:
- [Issue name]: [Description]
  - Impact: [What this affects]
  - Workaround: [If available]
  - Resolution: [What needs to be done]

Pattern 6: Notebook Splitting

Trigger: Split large analysis notebook into focused, modular notebooks

Mark the Original as Deprecated:

#### `Original_Combined_Analysis.ipynb` (1.5 MB) **[DEPRECATED - 2026-02-25]**
- **Status**: DEPRECATED - Split into Analysis_Part1.ipynb and Analysis_Part2.ipynb
- **Note**: This notebook combined both X and Y analyses. Now split for clarity.
- **Replacement notebooks**:
  - `Analysis_Part1.ipynb` - X analysis (Figures 01-02)
  - `Analysis_Part2.ipynb` - Y analysis (Figures 04-05)
- **Last modified**: 2026-02-25
- **Action**: Keep for reference, but use the split notebooks for future work

Document Each Split Notebook:

#### `Analysis_Part1.ipynb` (NEW - 2026-02-25)
- **Purpose**: X analysis only
- **Type**: Focused analysis - X effects
- **Rationale**: [Why this analysis is separate]
- **Approach**: [Brief methodology]
- **Generates**:
  - `figures/subfolder/01_figure_name.png`
  - `figures/subfolder/02_figure_name.png`
  - `statistics_file.csv`
- **Dataset**: N samples, X condition
- **Execution time**: ~3-5 minutes (vs ~10-15 for combined)
- **Priority**: [Where it fits in analysis workflow]

Update figures/MANIFEST.md:

### `figures/subfolder/`
**Generated by**:
  - `Analysis_Part1.ipynb` (Figures 01-02) - X analysis
  - `Analysis_Part2.ipynb` (Figures 04-05) - Y analysis
  - ~~Original_Combined_Analysis.ipynb~~ (DEPRECATED)

Also Update:

• "Current Status" in Notes for Resuming Work
• Any workflow diagrams or dependencies
• Quick Reference entry points

Pattern 7: MANIFEST Update After Major Changes

Trigger: Made significant project changes (notebook reorganization, deprecations, new files/scripts) that affect multiple MANIFESTs

Systematic Update Workflow:

1. Identify affected MANIFESTs:

# List all MANIFESTs in project
find . -name "MANIFEST.md" -not -path "*/deprecated/*"

2. Update in order of dependencies:

• data/MANIFEST.md (if data changes)
• scripts/MANIFEST.md (if scripts added/changed)
• figures/MANIFEST.md (if figures added/changed)
• Root MANIFEST.md (last - references all subdirectories)

3. Key sections to update in each MANIFEST:

• "Last Updated" date and summary
• New file entries with full metadata (purpose, size, dependencies)
• Deprecation entries with reason and replacement
• "Recent Session Work" with session accomplishments
• "Next Steps" with updated priorities
• "Known Issues" if new issues discovered

4. For deprecated notebooks:

• Keep entry in MANIFEST with DEPRECATED status
• Add deprecation date and reason
• Link to replacement notebook
• Document what's different in new version
• Note migration guide if created

Example deprecation entry:

#### `deprecated/notebooks_20260226/Temporal_Analysis_HiFi_OLD.ipynb` (916 KB) **[DEPRECATED - 2026-02-26]**
- **Status**: DEPRECATED - Reorganized for manuscript preparation
- **Original name**: `Temporal_Analysis_HiFi.ipynb`
- **Reason**: Unorganized structure (no TOC, methods scattered, phylo tree at end)
- **Replaced by**: `Temporal_Impact_Analysis.ipynb` (same analysis, better organization)
- **Content preserved**: All code, figures, statistics, interpretations, CSV outputs
- **What's different in new notebook**: TOC with anchor links, logical section flow (dataset -> analysis -> conclusions -> methods), clearer narrative for manuscript
- **Migration**: See `REORGANIZATION_GUIDE.md` for cell-by-cell copying instructions
- **Last modified**: 2026-02-26
- **Note**: Kept for reference, all scientific content is identical

5. Track todo items for systematic updates:

Use TodoWrite to ensure all MANIFESTs get updated:

todos = [
    "Update root MANIFEST.md with new files and deprecations",
    "Update data/MANIFEST.md with assembly year extraction status",
    "Update scripts/MANIFEST.md with extract_assembly_year.py",
    "Update figures/MANIFEST.md with new figure descriptions",
    "Verify all MANIFEST updates are complete"
]

Mark each complete as you finish it to track progress.

6. Verification checklist:

• [ ] All "Last Updated" dates match current session date
• [ ] New files documented with complete metadata
• [ ] Deprecated files marked with explanation
• [ ] "Recent Session Work" captures what was accomplished
• [ ] "Next Steps" reflects current priorities
• [ ] All file references are accurate (no broken paths)
• [ ] File sizes updated if files changed

Benefits:

• Prevents missing MANIFEST updates across multiple directories
• Maintains documentation consistency
• Clear tracking of what changed when
• Future sessions can quickly understand project state

Token efficiency: Update MANIFESTs immediately after changes to avoid context loss and reduce future re-reading.

Real example: Session updated 4 MANIFESTs (root, data, scripts, figures) after temporal notebook reorganization and deprecation, documenting 2 deprecated notebooks, new template notebook, new script, enhanced figures, and all related documentation files.

Quick Update Checklist

When updating MANIFEST after any change:

• [ ] Update "Last Updated" date (with brief description in parentheses)
• [ ] Add completed work to "Current Status"
• [ ] Update file sizes/counts if data changed
• [ ] Add any new issues to "Known Issues"
• [ ] Update "Next Steps" with new priorities
• [ ] Update Quick Reference if file paths changed
• [ ] Update Dependencies if scripts/notebooks changed
• [ ] Verify all file references are still accurate

Migration Guide

Migrating to Directory-Based Organization

To reorganize an existing project:

# Create new structure
mkdir -p documentation/{data_descriptions,methods,results,action_reports,progress,todos,internal}

# Move data descriptions
mv documentation/*README*.md documentation/data_descriptions/
mv documentation/*DATA*.md documentation/data_descriptions/

# Move methods
mv documentation/*workflow*.md documentation/methods/
mv documentation/*plan*.md documentation/methods/
mv documentation/*protocol*.md documentation/methods/

# Move results
mv documentation/*ANALYSIS_SUMMARY*.md documentation/results/
mv documentation/*COMPLETE*.md documentation/results/

# Move progress tracking
mv documentation/*PROGRESS*.md documentation/progress/
mv documentation/*SESSION*.md documentation/progress/
mv documentation/*RESUME*.md documentation/progress/

# Move action reports
mv documentation/*CORRECTION*.md documentation/action_reports/
mv documentation/*UPDATE*.md documentation/action_reports/
mv documentation/*REGENERATION*.md documentation/action_reports/
mv documentation/*RESTORATION*.md documentation/action_reports/
mv documentation/*VERIFICATION*.md documentation/action_reports/

# Move todos
mv documentation/*PRIORITY*.md documentation/todos/
mv documentation/*TODO*.md documentation/todos/

# Move internal/meta docs
mv documentation/*ESSENTIAL*.md documentation/internal/
mv documentation/*ORGANIZATION*.md documentation/internal/
mv documentation/*ISSUE*.md documentation/internal/
mv documentation/*FIXES*.md documentation/internal/

Pattern Reference

Quick reference for categorizing files:

Best Practices

1. Name files descriptively: Use clear prefixes/suffixes 2. Date internal docs: Add dates to action reports and progress notes 3. Update README: Keep documentation/README.md current 4. Regular cleanup: Move old files to deprecated/ 5. Consistent naming: Use established patterns for easy categorization 6. Test sharing: Verify shared packages have needed documentation

Documenting Format Debugging

When solving format/compatibility issues, create a brief summary doc for future reference:

Template (FORMAT_FIX.md or SPECIES_NAME_FIX.md):

# [Issue] Fix

**Date**: YYYY-MM-DD
**Issue**: Brief description of problem
**Root Cause**: Technical explanation
**Solution**: What was changed
**Files Updated**: List of affected files

## Verification
- [ ] Test case 1
- [ ] Test case 2

## Prevention
How to avoid this issue in future

Example: Species name case sensitivity fix

# Species Name Case Sensitivity Fix

**Date**: 2026-02-25
**Issue**: iTOL config missing one species (Alca_torda)
**Root Cause**: Dataset had "Alca Torda" (uppercase T) instead of "Alca torda"
**Solution**: Added case normalization regex to generation script
**Files Updated**:
- generate_tech_year_itol_configs.py
- itol_clr_technology_binary.txt (regenerated)
- itol_release_year_gradient.txt (regenerated)

## Verification
- [x] All 40 CLR species appear in config
- [x] All 446 year species appear in config
- [x] No species name mismatches between tree and configs

## Prevention
Always normalize case in data pipeline:

df['species_tree'] = df['species_tree'].str.replace( r'_([A-Z])', lambda m: '_' + m.group(1).lower(), regex=True )

Why document format fixes:

• Prevents re-debugging same issue months later
• Provides search keywords for similar issues
• Documents validation steps for future formats
• Shows prevention strategy for data pipeline

Location: Place in project root or documentation/working_files/

Version Control for Document Iterations

When creating documents through multiple iterations (especially for publication), maintain clear version history.

Version Numbering Scheme

• v1.0: Initial complete version
• v1.1: Minor fixes (typos, formatting)
• v2.0: Major changes (new sections, figures, analyses)
• vX.Y FINAL: Ready for submission/sharing

Version Documentation Template

Create FINAL_PACKAGE_*.md or VERSION_HISTORY.md:

## Version X.Y - What Changed

### From vX.0 to vX.Y:
**Changed**: [What specifically changed]
- **Was**: [Previous state]
- **Now**: [Current state]

**Why**: [Rationale for change]

**Important**: [What stayed the same]

**Added**: [New content]

## Version Timeline

- **v1.0** (timestamp): [Description]
- **v2.0** (timestamp): [Description]
- **v2.1** (timestamp): [Description] <- **CURRENT**

Benefits:

• Collaborators understand document evolution
• Clear indication of which version to use
• Rationale for changes documented
• Easy to track what stayed consistent vs. what changed

Example use case: LaTeX supplementary document went through v1.0 (4 figures, release year only) -> v2.0 (5 figures, both dating methods) -> v2.1 (clean temporal figures, methodological clarifications). Version history showed what changed at each step and why.

---

Notes for Resuming Work

Current Status (2026-02-25):

• Data consolidation complete: Reduced from 34 CSV files to 2 core files
• AWS QC enrichment added: BUSCO completeness/lineage and Merqury QV
• All 34 deprecated files safely archived in deprecated/data_backups_20260225/
• All notebooks and scripts verified to use correct data
• 3categories file properly rebuilt with correct filtering (541 assemblies)

Next Steps: 1. Verify analyses with enriched data: Re-run notebooks with new QC columns 2. Document enrichment methodology: GenomeArk sources, coverage stats

Known Issues:

• BUSCO/Merqury coverage only ~21%: Current enrichment in TEST_MODE
• Impact: Missing QC data for ~79% of assemblies
• Resolution: Set ENABLE_AWS_FETCH=True, TEST_MODE=False in enrich_unified_csv.ipynb
• Estimated time: 2-3 hours for full enrichment
• VGP haplotype comparison files missing: Haplotype_Comparison_Analysis.ipynb

references VGPPhase1-haplotype-comparison*.csv files not in project

• Impact: Cannot run haplotype comparison analysis
• Resolution: Locate or regenerate these files

Files Updated:

• Sizes: unified (482 KB), 3categories (422 KB)
• Columns: 74->77 (added 3 QC columns)
• Quick Reference: Noted new BUSCO/Merqury columns
• Dependencies: Added enrich_unified_csv.ipynb

This comprehensive update provides complete context for resuming work efficiently.

---

Example: VGP Curation Project

Current state reorganization:

SHARE:
data_descriptions/
  - HAPLOTYPE_COMPARISON_TABLE_README.md
  - vgp_assemblies_data_dictionary.md

methods/
  - KARYOTYPE_WORKFLOW.md
  - analysis_plan.md
  - data_fetching_plan.md
  - updated_methods_section.md

results/
  - ANALYSIS_SUMMARY.md
  - COMPLETE_ANALYSIS_SUMMARY.md
  - DUAL_HAPLOTYPE_ANALYSIS_SUMMARY.md
  - HAPLOTYPE_ANALYSIS_COMPLETE.md
  - DETAILED_METRICS_SUMMARY.md

INTERNAL:
action_reports/
  - CORRECTIONS_COMPLETE.md
  - DATA_TABLE_VERIFICATION.md
  - FIGURE_REGENERATION_SUMMARY.md
  - FIGURE_RESTORATION_SUMMARY.md
  - FINAL_UPDATE_SUMMARY.md
  - UPDATE_SUMMARY.md
  - GENOMESCOPE_DATA_RETRIEVAL_STATUS.md
  - IMPROVED_PLOTS_SUMMARY.md
  - KARYOTYPE_UPDATES_SUMMARY.md

progress/
  - PROGRESS.md
  - RESUME_HERE.md
  - KARYOTYPE_SESSION_SUMMARY.md
  - TIER1_SEARCH_PROGRESS.md
  - TIER2_SESSION_SUMMARY.md

todos/
  - KARYOTYPE_SEARCH_PRIORITY_LIST.md

internal/
  - MINIMAL_ESSENTIAL_FILES.md
  - DOCUMENTATION_ORGANIZATION.md
  - NOTEBOOK_COHERENCE_ISSUES.md
  - NOTEBOOK_UPDATE_COMMAND.md
  - TEXT_FIXES_NEEDED.md
  - OUTLIER_ANALYSIS_REPORT.md

deprecated/
  - CLEANUP_SUMMARY.md
  - MIGRATION_CHANGES.md
  - BOTH_HAPLOTYPES_DEPRECATION.md

This organization makes it immediately clear what should be shared and what is internal.