Data Backup

Smart Backup System with Skill Integration

Supporting files in this directory:

- MANIFEST_BACKUPS.md -- MANIFEST-aware intelligent backups

- FULL_PROJECT_BACKUPS.md -- Full project backups, selective inclusion/exclusion, path verification

- ADVANCED_USAGE.md -- Custom scripts, multiple file backups, real-world examples

When to Use This Skill

Use this skill when:

• Working on any project with files that change over time
• Jupyter notebooks, data files (CSV/TSV), HackMD presentations, or mixed projects
• Need intelligent cleanup before backup (clear outputs, remove debug code)
• Want to track what changed when (data provenance)
• Need professional backup workflow for collaboration or publication
• Want context-aware backups that use other skills intelligently

The Problem

Long-running data enrichment projects risk:

• Losing days of work from accidental overwrites
• Unable to revert to previous data states
• No documentation of what changed when
• Running out of disk space from manual backups
• Confusion about which version is current

Solution: Smart Two-Tier Backup System with Skill Integration

Core Features

1. Intelligent Detection - Automatically detects project type and files to backup 2. Skill Integration - Uses jupyter-notebook, hackmd, and other skills for pre-backup cleanup 3. Daily backups - Rolling 7-day window (auto-cleanup) 4. Milestone backups - Permanent, compressed (gzip ~80% reduction) 5. CHANGELOG - Automatic documentation of all changes 6. Session Integration - Prompts for backup when exiting Claude Code session

Smart Detection & Integration

The backup system automatically detects your project type and applies appropriate cleanup:

Jupyter Notebooks (uses jupyter-notebook skill):

• Detects: *.ipynb files
• Pre-backup cleanup: Clear all cell outputs, remove cells tagged 'debug' or 'remove', validate notebooks

HackMD/Presentations (uses hackmd skill):

• Detects: *.md files with slideOptions: frontmatter
• Pre-backup cleanup: Validate SVG elements, check slide separators, verify YAML frontmatter

Data Files (native handling):

• Detects: *.csv, *.tsv, *.xlsx files
• Pre-backup cleanup: Validate file integrity, check for corruption

Python Projects (uses managing-environments skill):

• Detects: requirements.txt, environment.yml, venv/, .venv/
• Pre-backup cleanup: Remove .pyc, __pycache__, .pytest_cache, clean build artifacts

Mixed Projects: Detects all of the above and applies appropriate cleanup for each file type.

Directory Structure

For data-only projects:

project/
├── your_data_file.csv          # Main working file
├── backup_project.sh           # Smart backup script
└── backups/
    ├── daily/                  # Rolling 7-day backups
    ├── milestones/             # Permanent compressed backups
    ├── CHANGELOG.md            # Auto-generated change log
    └── README.md               # User documentation

For mixed projects (notebooks + data):

project/
├── analysis.ipynb              # Jupyter notebooks
├── data.csv                    # Data files
├── backup_project.sh           # Smart backup script
└── backups/
    ├── daily/                  # Rolling 7-day backups
    │   └── backup_2026-01-17/
    │       ├── notebooks/      # Cleaned (no outputs)
    │       └── data/
    ├── milestones/             # Permanent compressed backups
    ├── CHANGELOG.md
    └── README.md

Storage Efficiency

• Daily backups: ~5.4 MB (7 days x 770KB)
• Milestone backups: ~200KB each compressed (80% size reduction with gzip)
• Total: <10 MB for complete project history
• Auto-cleanup: Old daily backups delete after 7 days

Implementation

Quick Start with /backup Command

First time - Setup the backup system:

/backup

This will:

• Detect your project type (notebooks, data files, presentations, etc.)
• Set up appropriate backup scripts with smart cleanup
• Create backup directory structure
• Optionally configure automated backups

Daily usage - Create backups:

/backup                    # Daily backup with smart cleanup
/backup milestone "desc"   # Milestone backup
/backup list              # View all backups
/backup restore DATE      # Restore from backup

What Happens During Backup

Smart cleanup before backup: 1. Detects file types in your project 2. Applies skill-specific cleanup:

• Notebooks: Clear outputs, remove debug cells
• HackMD: Validate SVG, check formatting
• Python: Remove .pyc, __pycache__
• Data: Validate integrity

3. Creates organized backup with cleaned files 4. Updates CHANGELOG with what was backed up

Manual Script Usage (Alternative)

./backup_project.sh                           # Daily backup
./backup_project.sh milestone "description"   # Milestone
./backup_project.sh list                      # List backups
./backup_project.sh restore 2026-01-23        # Restore

When to Create Milestones

• After adding new data sources (GenomeScope, karyotypes, external APIs)
• Before major data transformations or filtering
• When completing analysis sections
• Before submitting/publishing
• Before sharing with collaborators
• After recovering missing data

Key Features

Safety Features

1. Never overwrites without asking - Prompts before overwriting existing backups 2. Safety backup before restore - Creates backup of current state before any restore 3. Automatic cleanup - Old daily backups auto-delete (configurable) 4. Complete audit trail - CHANGELOG tracks everything 5. Milestone protection - Important versions preserved forever (compressed)

CHANGELOG Tracking

The CHANGELOG.md automatically documents:

• Date of each backup
• Type (daily vs milestone)
• Description of changes (for milestones)
• Major modifications made to data

Example CHANGELOG:

## 2026-01-23
- **MILESTONE**: Recovered VGP accessions (backup created)
  - Added columns: `accession_recovered`, `accession_recovered_all`
  - Recovered 5 VGP accessions from NCBI
- Daily backup created at 2026-01-23 15:00:00

## 2026-01-22
- Enriched GenomeScope data for 21 species from AWS repository
- Added column: `genomescope_path` with direct links to summary files

Using /backup Command

Setup mode (first run): /backup -- Detects project type, sets up scripts, creates directory structure.

Daily backup mode: /backup -- Quick daily backup.

Milestone mode: /backup milestone "description of changes" -- e.g., /backup milestone "added heterozygosity data"

List and restore:

/backup list              # Show all available backups
/backup restore 2026-01-23 # Restore from specific date

Configuration: Edit backup_project.sh to change retention days (default: 7), backup directory location, or custom cleanup rules.

Benefits for Data Analysis

• Data Provenance: CHANGELOG documents every modification; clear audit trail for methods sections in papers
• Confidence to Experiment: Easy rollback encourages trying different approaches safely
• Professional Workflow: Matches publication standards; reviewers can verify data processing steps
• Collaboration-Ready: Team members can understand data history and enrichment process

Session Integration with /safe-exit

When you end a Claude Code session with /safe-exit, the system automatically:

1. Detects if backup system exists in the current project 2. Prompts for backup if system is configured (daily, milestone, skip, or cancel) 3. Performs cleanup and backup if requested 4. Prompts for Obsidian session summary (if obsidian skill is available) 5. Exits session cleanly

This ensures you never forget to backup AND document your work at the end of your session!

Example Workflow

Monday Morning

/backup                          # Daily backup with smart cleanup
# Work on notebooks and data enrichment all day
/backup milestone "added karyotype data for 50 new species"

End of session

/safe-exit
# Prompted: daily backup -> backup complete -> session summary -> exit

Friday (oops, made a mistake!)

/backup list                     # Check available backups
/backup restore 2026-01-23       # Restore from Wednesday

MANIFEST-Aware Backups

For projects with MANIFEST files, use intelligent backups that include only essential files. See [MANIFEST_BACKUPS.md](./MANIFEST_BACKUPS.md) for the full pattern, script templates, inclusion/exclusion rules, and integration with the /backup command.

Full Project Backups

For projects where both code and data change, selective full-project backups capture the complete state without bloat. See [FULL_PROJECT_BACKUPS.md](./FULL_PROJECT_BACKUPS.md) for implementation patterns, backup strategy comparison, size benchmarks, and path verification guidance.

Advanced Usage

For custom backup script templates, handling multiple files, viewing compressed milestones, and real-world examples, see [ADVANCED_USAGE.md](./ADVANCED_USAGE.md).

Best Practices

1. Create daily backups at session start - Make it a habit 2. Milestone after every major change - Don't rely on memory 3. Use descriptive milestone names - "added genomescope" not "updates" 4. Check CHANGELOG before sharing - Verify data provenance is clear 5. List backups periodically - Ensure auto-cleanup is working 6. Test restore once - Verify you know how to recover

Troubleshooting

Backup script not found

ls -l backup_project.sh   # Check if backup system is set up
/backup                    # Set up if needed

Disk space running low

du -sh backups/            # Check backup sizes
# Reduce retention: edit DAYS_TO_KEEP=3 in backup_table.sh
# Manually clean old milestones if needed

CHANGELOG getting too large

tail -100 backups/CHANGELOG.md > backups/CHANGELOG_recent.md
mv backups/CHANGELOG.md backups/CHANGELOG_archive.md
mv backups/CHANGELOG_recent.md backups/CHANGELOG.md

Summary

• Two-tier system: Daily rolling + permanent milestones
• Storage efficient: Gzip compression (~80% reduction)
• Auto-cleanup: 7-day rolling window for dailies
• Complete audit trail: CHANGELOG tracks all changes
• Safety first: Never overwrites without confirmation
• Global installer: Use across all projects
• Professional workflow: Publication-ready data provenance

Advanced Usage

Custom Backup Script Template

The backup script can be customized for different file types or naming conventions:

#!/bin/bash
# Backup script for PROJECT_NAME

MAIN_TABLE="your_data_file.csv"
DAILY_DIR="backups/daily"
MILESTONE_DIR="backups/milestones"
CHANGELOG="backups/CHANGELOG.md"
DAYS_TO_KEEP=7

Viewing Compressed Milestones

# View without decompressing
gunzip -c milestone_file.csv.gz | less

# Decompress permanently
gunzip milestone_file.csv.gz

Multiple File Backups

For projects with multiple related data files, create separate backup scripts or modify the script to handle multiple files:

# Create separate backups
./backup_main_table.sh
./backup_metadata.sh

# Or modify script to backup multiple files
for file in *.csv; do
    cp "$file" "backups/daily/backup_${DATE}_$(basename $file)"
done

Token Efficiency

This backup system is token-efficient because:

• No need to read large files just to create backups (uses cp)
• Automated logging reduces manual documentation
• Quick restore prevents wasted time re-implementing lost work
• CHANGELOG serves as lightweight documentation

Real-World Example

VGP Phase 1 Enrichment Project:

• Main file: 716 assemblies, 127 columns, ~770KB
• Daily backups: 7 files = ~5.4 MB
• Milestones: 3 compressed files = ~600KB
• Total: ~6 MB for complete project history
• Tracked: 2 weeks of data enrichment, 5 major milestones
• Prevented: Multiple accidental overwrites during NCBI searches

Full Project Backups (vs Data-Only)

Problem

Data-only backups (single CSV file) don't capture the complete project state. But backing up EVERYTHING creates bloated backups with old/irrelevant files.

Solution: Selective Full Project Backup

What to Include:

• Main analysis notebook (e.g., Analysis.ipynb)
• Primary data file (e.g., data.csv)
• Current figure generation scripts only (e.g., python_scripts/)
• Current figures only (e.g., figures/*.png - root level)
• Active documentation (.md files, excluding backups)
• Utility scripts (.sh files)

What to Exclude:

• Backup notebooks (*backup*.ipynb, *Copy*.ipynb)
• Exploratory scripts in scripts/ (only keep figure generators)
• Old figure versions (only current in figures/)
• Jupyter checkpoints (.ipynb_checkpoints/)
• Python cache (__pycache__/, *.pyc)

Implementation Pattern

Bash script with rsync + selective copy:

# Copy specific directory with exclusions
if [ -d "python_scripts" ]; then
    rsync -a --exclude='__pycache__' --exclude='*.pyc' \
        "python_scripts/" "${BACKUP_DIR}/python_scripts/"
fi

# Copy only current figures (root level PNG files)
if [ -d "figures" ]; then
    if ls figures/*.png 1> /dev/null 2>&1; then
        cp figures/*.png "${BACKUP_DIR}/figures/"
    fi
fi

# Copy docs, excluding backups
shopt -s nullglob
for file in *.md *.sh; do
    if [[ ! "$file" =~ (backup|BACKUP|Copy) ]]; then
        cp "$file" "${BACKUP_DIR}/"
    fi
done
shopt -u nullglob

Archive with tar:

# Daily: uncompressed (fast restore)
tar -cf "backup_${DATE}.tar" "${PROJECT_NAME}/"

# Milestone: compressed (space efficient)
tar -czf "milestone_${DATE}_${NAME}.tar.gz" "${PROJECT_NAME}/"

Backup Strategy

Size Comparison

Real project example:

• Data-only backup: 211 KB (compressed CSV)
• Full project backup: 17 MB (notebook + data + scripts + 43 figures + docs)
• 7-day daily backups: ~120 MB total

When This Matters

• Projects with evolving analyses where both code and data change
• Jupyter notebook workflows with generated figures
• Research projects needing reproducibility (code + data + outputs)

Path Verification for Backups

Before creating milestone backups, verify that files use relative paths.

Why this matters:

• Backups may be restored to different locations
• Notebooks shared from backups must work for others
• Absolute paths break when directory structure changes

For complete path verification procedures and automated checking scripts, see the `folder-organization` skill.

Quick check:

# Check for absolute paths in notebooks
grep -l "/Users/" *.ipynb
grep -l "C:\\\\" *.ipynb

# Check in Python scripts
grep -l "/Users/" python_scripts/*.py

What to look for:

• /Users/yourname/project/data.csv (absolute) -- avoid
• data/data.csv (relative) -- preferred
• Image('/Users/you/figures/fig.png') (absolute) -- avoid
• Image('figures/fig.png') (relative) -- preferred

Best practice: 1. Run path check before milestone backups (see folder-organization skill) 2. Fix any absolute paths found 3. Test notebook runs from backup directory 4. Then create milestone backup

MANIFEST-Aware Backups

For projects with MANIFEST files, create intelligent backups that include only essential files based on project documentation.

Pattern: Read MANIFEST to Identify Key Files

Instead of backing up everything, read the project MANIFEST to determine what's important:

# Extract key file patterns from MANIFEST
grep -E "^(#### |### |##)" MANIFEST.md | head -40  # Get structure
grep -E "(\.csv|\.ipynb|\.md)" MANIFEST.md | grep "^\s*-" | head -20  # Get key files

Backup Script Template

Modify your milestone_backup() function to read MANIFEST:

milestone_backup() {
    DESCRIPTION="$1"
    TEMP_DIR=$(mktemp -d)
    BACKUP_ROOT="$TEMP_DIR/Project_${DATE}"
    mkdir -p "$BACKUP_ROOT"

    echo "📦 Backing up key files from MANIFEST..."

    # 1. Active notebooks (exclude deprecated based on MANIFEST)
    # Look for files marked "PRIMARY" or "ACTIVE" in MANIFEST
    for nb in Primary_Analysis.ipynb Supporting_Analysis.ipynb; do
        if [ -f "$nb" ]; then
            cp "$nb" "$BACKUP_ROOT/" 2>/dev/null || true
        fi
    done

    # 2. Key data files (from MANIFEST "Key files" section)
    mkdir -p "$BACKUP_ROOT/data"
    cp data/main_dataset_corrected.csv "$BACKUP_ROOT/data/" 2>/dev/null || true
    cp data/MIGRATION_GUIDE.md "$BACKUP_ROOT/data/" 2>/dev/null || true

    # 3. Analysis files (from MANIFEST entry points)
    if [ -d "analysis_files" ]; then
        cp -r analysis_files "$BACKUP_ROOT/" 2>/dev/null || true
    fi

    # 4. Scripts (selective - main scripts only)
    mkdir -p "$BACKUP_ROOT/scripts"
    cp scripts/*.py "$BACKUP_ROOT/scripts/" 2>/dev/null || true

    # 5. Figures (manuscript only, not exploratory)
    if [ -d "figures/manuscript" ]; then
        mkdir -p "$BACKUP_ROOT/figures"
        cp -r figures/manuscript "$BACKUP_ROOT/figures/" 2>/dev/null || true
    fi

    # 6. MANIFESTs and documentation
    for doc in MANIFEST.md README.md *.md; do
        if [ -f "$doc" ]; then
            cp "$doc" "$BACKUP_ROOT/" 2>/dev/null || true
        fi
    done

    # Compress
    echo "🗜️  Compressing..."
    tar -czf "$MILESTONE_FILE" -C "$TEMP_DIR" "$(basename "$BACKUP_ROOT")"
    rm -rf "$TEMP_DIR"
}

What to Include (from MANIFEST)

Include:

• Files marked as "PRIMARY" or "ACTIVE"
• Entry point notebooks/files
• Key data files explicitly mentioned
• Manuscript-ready figures
• Documentation (MANIFEST, README, guides)
• Analysis summaries and statistics
• Scripts that generate key outputs

Exclude:

• Files marked "DEPRECATED"
• Large data caches (unless marked critical)
• Temporary processing files
• Build artifacts
• Files in deprecated/ directories
• Exploratory/draft figures
• Python cache (__pycache__, .pyc)

Benefits

1. Smaller backups: 18 MB -> 11 MB compressed (vs. 50+ MB if backing up everything) 2. Focused recovery: Only important files included 3. Manuscript-ready: Backup contains publication-relevant files 4. Documented selection: MANIFEST shows what was chosen and why 5. Faster backup/restore: Less to compress and transfer

Example Output

📦 Backing up key files from MANIFEST...
  → Notebooks (5 active)...
  → Key data files...
  → Analysis files...
  → Scripts...
  → Figures (manuscript)...
  → Documentation...

📊 Backup size (uncompressed):  18M
🗜️  Compressing...
  ✓ Compressed to:  11M

✅ Milestone backup created

Cache Preservation Documentation

If MANIFEST identifies caches that should NOT be backed up but also NOT deleted:

## GenomeScope QC Data Cache

**Purpose**: Cached QC data from AWS S3
**Status**: Active cache - DO NOT DELETE
**Backup**: Not included in backups (easily regenerated from S3)

Document these clearly in backup README to prevent accidental deletion during cleanup.

Integration with /backup Command

When implementing /backup command, add MANIFEST detection:

1. Check for MANIFEST file: if [ -f "MANIFEST.md" ]; then ... 2. Use MANIFEST-aware backup function for milestone backups 3. Show what was included based on MANIFEST categories 4. Update backup README with MANIFEST reference