Genomeark Aws

GenomeArk AWS S3 Data Repository

Comprehensive guide for accessing and navigating the GenomeArk AWS S3 public bucket containing Vertebrate Genomes Project (VGP) assemblies and quality control data.

Supporting files (read as needed for detailed code and strategies):

• assembly-date-extraction.md - Extract assembly dates from FASTA filenames, validation rules
• qc-data-fetching.md - GenomeScope, BUSCO, Merqury, Meryl fetching code and parsing
• best-practices.md - AWS CLI patterns, batch processing, common pitfalls, testing examples, version history

When to Use This Skill

Use this skill when:

• Accessing VGP genome assemblies from GenomeArk AWS S3
• Fetching QC metrics (GenomeScope, BUSCO, Merqury) for genomic analyses
• Downloading genome evaluation data for comparative studies
• Accessing meryl k-mer histograms for GenomeScope analysis
• Building automated pipelines that fetch VGP data
• Troubleshooting S3 path issues or missing data
• Working with species-specific genome data from VGP

Repository Overview

GenomeArk is a public AWS S3 bucket (s3://genomeark/) hosting:

• VGP genome assemblies (primary, alternate, trio)
• Quality control metrics (GenomeScope, BUSCO, Merqury)
• Intermediate files (meryl databases, k-mer histograms)
• Assembly evaluation reports
• Haplotype-resolved assemblies

Access Method: Public bucket requiring no AWS credentials when using --no-sign-request

Critical Discovery: GenomeArk structure has evolved over time (2022 -> 2024+). Always implement fallback path patterns for reliability.

Directory Structure

Base Structure

s3://genomeark/
└── species/
    └── {Genus_species}/          # e.g., Rhinolophus_ferrumequinum
        └── {ToLID}/               # e.g., mRhiFer1 (VGP specimen ID)
            ├── assembly_vgp_{type}_{version}/
            │   ├── evaluation/     # QC metrics (MAIN ACCESS POINT)
            │   │   ├── genomescope/
            │   │   ├── busco/
            │   │   ├── merqury/
            │   │   └── ...
            │   └── intermediates/  # K-mer databases, temp files
            │       └── meryl/
            └── genomic_data/      # Raw sequencing data folders

Assembly Directory Variations

assembly_vgp_{type}_{version} - Standard VGP Patterns:

• assembly_vgp_HiC_2.0 - Hi-C phased assembly (case-sensitive!)
• assembly_vgp_standard_2.0 - Standard assembly without Hi-C
• assembly_vgp_hic_2.0 - Alternative Hi-C naming
• assembly_vgp_trio_2.0 - Trio-binned assembly

Legacy Versions (2019-2021 assemblies):

• assembly_vgp_standard_1.6 - Version 1.6 (common in fish, birds)
• assembly_vgp_standard_1.0 - Version 1.0 (early assemblies)
• assembly_vgp_HiC_1.6 - Hi-C version 1.6
• assembly_vgp_HiC_1.0 - Hi-C version 1.0
• assembly_vgp_HiC_1.4 - Hi-C version 1.4

Verkko Assemblies (diploid assemblies):

• assembly_verkko_1.4/ - Verkko version 1.4
• assembly_verkko_1.1-0.1/ - Verkko version 1.1-0.1
• assembly_verkko_1.1-0.1-freeze/ - Frozen version
• assembly_verkko_1.1-0.2/ - Version 1.1-0.2
• assembly_verkko_1.4.1r/ - Revised version 1.4.1

Clade-Specific Directories (2023+ specialized assemblies):

• assembly_primate_v1.4.2/ - Primate-specific pipeline
• assembly_fish_* - Fish-specific (potential)
• assembly_bird_* - Bird-specific (potential)

Institution-Specific Directories:

• assembly_rockefeller/ - Rockefeller University assemblies
• assembly_cambridge/ - Cambridge assemblies
• assembly_MT_rockefeller/ - Case variation
• assembly_mt_rockefeller/ - Lowercase variation
• assembly_mt_milan/ - Milan institute

Directories Without "assembly_" Prefix (rare):

• vgp_standard_1.6/ - Standard v1.6 without prefix
• vgp_standard_1.0/ - Standard v1.0 without prefix
• vgp_HiC_1.6/ - Hi-C v1.6 without prefix

Curated Assemblies (post-manual curation):

• assembly_curated/ - Exclude for date extraction (post-curation dates)

CRITICAL CASE SENSITIVITY:

• Metadata may store: assembly_vgp_hic_2.0 (lowercase)
• S3 requires: assembly_vgp_HiC_2.0 (mixed case!)
• Always normalize before fetching

COMPREHENSIVE PATTERN MATCHING:

• Don't stop at first match: Try ALL valid paths
• Pri/alt assemblies often use legacy versions (1.6, 1.0)
• Phased assemblies typically use version 2.0
• Verkko assemblies are diploid, use different naming
• Coverage improvement: Using all patterns -> 47-62% vs 27% with basic patterns

Data Access Summary

For detailed fetching code and parsing logic, see qc-data-fetching.md.

Path Normalization (used by all fetching functions)

def normalize_s3_path(s3_path):
    """Normalize path for GenomeArk (case sensitivity!)"""
    if not s3_path:
        return None
    s3_path = s3_path.replace('/assembly_vgp_hic_2.0/', '/assembly_vgp_HiC_2.0/')
    if not s3_path.endswith('/'):
        s3_path += '/'
    return s3_path

GenomeScope Filename Patterns (TRY ALL THREE!)

• Pattern A: {ToLID}_genomescope__Summary.txt (double underscore, most common)
• Pattern C: {ToLID}_genomescope_Summary.txt (single underscore, easily missed)
• Pattern B: {ToLID}_Summary.txt (no prefix, older assemblies)

Checking only A and B causes ~30-40% of data to be missed.

GenomeScope Validation

Reject failed runs where heterozygosity range > 50% or max > 95%. A range of 0%-100% indicates complete model failure.

Meryl Histograms - Direct HTTPS URLs (for Galaxy import)

https://genomeark.s3.amazonaws.com/species/{species}/{tolid}/assembly_vgp_standard_1.0/intermediates/meryl/{tolid}.cut.meryl.hist

Quick Reference

AWS CLI pattern (prefer over boto3 for public buckets):

cmd = ['aws', 's3', 'cp', s3_path, '-', '--no-sign-request']
result = subprocess.run(cmd, capture_output=True, text=True, timeout=30)

Rate limiting: 0.2s delay between requests.

Common pitfalls: Case sensitivity (hic vs HiC), directory evolution (2022 vs 2024 layouts), downloading full meryl databases instead of .hist files. See best-practices.md for full list.

Assembly Date Extraction

Location: Assembly FASTA files with YYYYMMDD timestamps Purpose: Extract accurate assembly completion dates (vs NCBI release dates)

Why Important

• NCBI release dates may be 1-2 years after actual assembly completion
• Average delay: 1.1 years (64% of assemblies have delayed release)
• Critical for temporal analyses and technology transition studies

File Naming Pattern

{ToLID}.{type}.{haplotype}.YYYYMMDD.fasta.gz

Examples:

mLoxAfr1.HiC.hap1.20221209.fasta.gz         → 2022 (Released 2023)
mPanPan1.dip.20230906.fasta                 → 2023 (Verkko)
fAngAng1.standard.hap1.20190815.fasta.gz    → 2019 (Legacy 1.6)

Assembly Types:

• HiC - Hi-C phased assemblies
• standard - Standard assemblies
• dip - Diploid (Verkko assemblies)
• hap1, hap2 - Haplotype-specific
• pri, alt - Primary/alternate haplotypes

Extraction Strategy

Algorithm: 1. Find ALL valid assembly directories for a ToLID 2. Try each directory (prioritize recent versions first) 3. List files recursively, excluding _curated directories 4. Extract all 8-digit dates (YYYYMMDD) from filenames 5. Validate dates (2000-2030, valid month/day) 6. Return most recent year found

Critical: Try ALL Paths:

def extract_assembly_year_comprehensive(tolid, scientific_name):
    """
    Extract assembly year using comprehensive pattern matching.

    Returns most recent assembly year from date-stamped files.
    Excludes _curated directories (post-curation dates).
    """
    species_parts = scientific_name.strip().split()
    species_name = f"{species_parts[0]}_{species_parts[1]}"  # Handle subspecies

    # Build comprehensive path list (30+ patterns)
    all_paths = []

    # Standard patterns (version 2.0)
    patterns_v2 = [
        f"s3://genomeark/species/{species_name}/{tolid}/assembly_vgp_HiC_2.0/",
        f"s3://genomeark/species/{species_name}/{tolid}/assembly_vgp_standard_2.0/",
        f"s3://genomeark/species/{species_name}/{tolid}/assembly_vgp_hic_2.0/",
    ]

    # Legacy patterns (version 1.6, 1.0, 1.4)
    patterns_legacy = [
        f"s3://genomeark/species/{species_name}/{tolid}/assembly_vgp_standard_1.6/",
        f"s3://genomeark/species/{species_name}/{tolid}/assembly_vgp_standard_1.0/",
        f"s3://genomeark/species/{species_name}/{tolid}/assembly_vgp_HiC_1.6/",
        f"s3://genomeark/species/{species_name}/{tolid}/assembly_vgp_HiC_1.0/",
        f"s3://genomeark/species/{species_name}/{tolid}/assembly_vgp_HiC_1.4/",
    ]

    # Without "assembly_" prefix
    patterns_no_prefix = [
        f"s3://genomeark/species/{species_name}/{tolid}/vgp_standard_1.6/",
        f"s3://genomeark/species/{species_name}/{tolid}/vgp_standard_1.0/",
        f"s3://genomeark/species/{species_name}/{tolid}/vgp_HiC_1.6/",
    ]

    # Institution-specific
    patterns_institution = [
        f"s3://genomeark/species/{species_name}/{tolid}/assembly_rockefeller/",
        f"s3://genomeark/species/{species_name}/{tolid}/assembly_cambridge/",
        f"s3://genomeark/species/{species_name}/{tolid}/assembly_MT_rockefeller/",
        f"s3://genomeark/species/{species_name}/{tolid}/assembly_mt_milan/",
    ]

    # Try standard paths first
    all_paths.extend(patterns_v2 + patterns_legacy + patterns_no_prefix + patterns_institution)

    # Check which paths exist
    valid_paths = []
    for path in all_paths:
        result = subprocess.run(
            ['aws', 's3', 'ls', path, '--no-sign-request'],
            capture_output=True, timeout=10
        )
        if result.returncode == 0:
            valid_paths.append(path)

    # Dynamic discovery for verkko and clade-specific
    base_path = f"s3://genomeark/species/{species_name}/{tolid}/"
    list_result = subprocess.run(
        ['aws', 's3', 'ls', base_path, '--no-sign-request'],
        capture_output=True, text=True, timeout=10
    )

    if list_result.returncode == 0:
        for line in list_result.stdout.split('\n'):
            if 'PRE' in line and 'assembly' in line:
                dir_name = line.split('PRE')[1].strip().rstrip('/')
                dir_path = base_path + dir_name + '/'

                # Check for verkko or clade-specific
                if any(keyword in dir_name.lower() for keyword in ['verkko', 'primate', 'fish', 'bird']):
                    if dir_path not in valid_paths:
                        valid_paths.append(dir_path)

    # Add curated LAST (often lacks dates, but try if nothing else works)
    curated_path = f"s3://genomeark/species/{species_name}/{tolid}/assembly_curated/"
    if curated_path not in valid_paths:
        valid_paths.append(curated_path)

    # Try each path until we find dates
    for s3_path in valid_paths:
        # List files recursively
        result = subprocess.run(
            ['aws', 's3', 'ls', s3_path, '--recursive', '--no-sign-request'],
            capture_output=True, text=True, timeout=60
        )

        if result.returncode != 0:
            continue

        # Extract dates from filenames (exclude _curated in path)
        dates = []
        for line in result.stdout.split('\n'):
            if line.strip() and '_curated' not in line:
                # Find 8-digit dates
                matches = re.findall(r'(\d{8})', line)
                for date_str in matches:
                    year = int(date_str[:4])
                    month = int(date_str[4:6])
                    day = int(date_str[6:8])

                    # Validate
                    if 2000 <= year <= 2030 and 1 <= month <= 12 and 1 <= day <= 31:
                        dates.append(date_str)

        # If dates found, return most recent year
        if dates:
            most_recent = max(dates)
            return int(most_recent[:4])

    return None  # No dates found in any path

Expected Coverage

• Comprehensive approach (30+ patterns): 47-62% of assemblies
• Basic approach (3 patterns): 24% of assemblies
• Intermediate approach (8 patterns): 27% of assemblies

Assembly Type Differences:

• Phased assemblies: 55% found (typically use version 2.0)
• Pri/alt assemblies: 43% found (often use legacy versions 1.6, 1.0)
• Verkko assemblies: Found with dynamic discovery

Date Validation Results:

• Average delay: 1.1 years between assembly and NCBI release
• 64% of assemblies released 1-2 years after completion
• 35% released same year as assembled
• Maximum observed delay: 2 years

Critical Validation: assembly_year vs release_year

The extraction script can produce false positives by matching S3 file sizes as YYYYMMDD patterns (e.g., a 2028-byte file -> year 2028). Always validate:

# Invalidate assembly_year where it exceeds release_year (impossible)
bad = df[df['assembly_year'] > df['release_year']]
print(f"False positives: {len(bad)} assemblies")
df.loc[df['assembly_year'] > df['release_year'], 'assembly_year'] = float('nan')

Known false positives found (2026-03-11):

• bFalBia1 (Falco biarmicus): asm=2028 from 2028-byte BUSCO .gff file
• aRhiBiv1 (Rhinatrema bivittatum): asm=2026 from 202611172-byte .las file
• 6 additional off-by-one errors (asm=2023, rel=2022)

Rule: assembly_year must be <= release_year. Remove any that violate this.

GenomeArk Best Practices and Reference

Best Practices

Path Construction

1. Always normalize case: hic -> HiC before fetching 2. Try multiple patterns: Implement all filename variations 3. Use timeouts: 10-30s per fetch to avoid hanging 4. Add rate limiting: 0.2s delay between requests (respectful to AWS) 5. Handle failures gracefully: Continue on errors, report successes

Data Validation

1. Validate GenomeScope ranges: Reject values with >50% range or max >95% 2. Check for empty results: Verify data before parsing 3. Verify file existence: S3 returns 0 bytes for non-existent files 4. Skip failed QC runs: Filter out unrealistic values

AWS CLI vs boto3

For public buckets like GenomeArk:

• Prefer: subprocess + aws s3 CLI with --no-sign-request
• Avoid: boto3 (requires credential config even for public access)

# Simple and works
cmd = ['aws', 's3', 'cp', s3_path, '-', '--no-sign-request']
result = subprocess.run(cmd, capture_output=True, text=True, timeout=30)

# More complex, requires config (avoid)
import boto3
from botocore import UNSIGNED
from botocore.config import Config
s3 = boto3.client('s3', config=Config(signature_version=UNSIGNED))

Batch Processing

import time
from concurrent.futures import ThreadPoolExecutor, as_completed

def batch_fetch_qc_data(species_list, max_workers=5):
    """Fetch QC data for multiple species with rate limiting"""
    results = []

    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        futures = {
            executor.submit(fetch_all_qc, species, tolid): (species, tolid)
            for species, tolid in species_list
        }

        for future in as_completed(futures):
            species, tolid = futures[future]
            try:
                data = future.result()
                results.append({
                    'species': species,
                    'tolid': tolid,
                    **data
                })
            except Exception as e:
                print(f"Error fetching {species} ({tolid}): {e}")

            time.sleep(0.2)  # Rate limiting

    return results

def fetch_all_qc(species, tolid):
    """Fetch all QC metrics for a species"""
    data = {}

    # GenomeScope
    genomescope = fetch_genomescope_data(species, tolid)
    if genomescope:
        data.update(genomescope)

    # BUSCO
    busco = fetch_busco_data(species, tolid)
    if busco:
        data['busco_completeness'] = busco['completeness']

    # Merqury
    merqury = fetch_merqury_data(species, tolid)
    if merqury:
        data['qv'] = merqury['qv']

    return data

Common Pitfalls

1. Case sensitivity: assembly_vgp_hic_2.0 (table) -> assembly_vgp_HiC_2.0 (S3) 2. Missing pattern C: Single underscore GenomeScope files often missed 3. Directory evolution: Merqury structure changed 2022 -> 2024 4. Failed QC runs: Always validate GenomeScope ranges before use 5. Subdirectory variations: BUSCO/Merqury use different subdir names 6. File format variations: Merqury may/may not have header line 7. Downloading full meryl databases: Use .hist files only! 8. Timeout issues: Set reasonable timeouts for S3 operations 9. Rate limiting: Add delays between requests to avoid throttling

Testing Examples

Confirmed working paths (as of 2026-02-26):

# GenomeScope - Pattern A (double underscore)
aws s3 cp s3://genomeark/species/Gastrophryne_carolinensis/aGasCar1/assembly_vgp_HiC_2.0/evaluation/genomescope/aGasCar1_genomescope__Summary.txt - --no-sign-request

# GenomeScope - Pattern C (single underscore)
aws s3 cp s3://genomeark/species/Platysternon_megacephalum/rPlaMeg1/assembly_vgp_HiC_2.0/evaluation/genomescope/rPlaMeg1_genomescope_Summary.txt - --no-sign-request

# GenomeScope - Pattern B (no prefix)
aws s3 cp s3://genomeark/species/Spea_bombifrons/aSpeBom1/assembly_vgp_standard_2.0/evaluation/genomescope/aSpeBom1_Summary.txt - --no-sign-request

# BUSCO
aws s3 cp s3://genomeark/species/Gastrophryne_carolinensis/aGasCar1/assembly_vgp_HiC_2.0/evaluation/busco/c/aGasCar1_HiC__busco_hap1_busco_short_summary.txt - --no-sign-request

# Merqury - Direct path (2024+)
aws s3 cp s3://genomeark/species/Ia_io/mIaxIox2/assembly_vgp_HiC_2.0/evaluation/merqury/mIaxIox2_qv/output_merqury.tabular - --no-sign-request

# Merqury - Nested path (2022)
aws s3 cp s3://genomeark/species/Gastrophryne_carolinensis/aGasCar1/assembly_vgp_HiC_2.0/evaluation/merqury/c/aGasCar1_qv/output_merqury.tabular - --no-sign-request

# Meryl histogram
aws s3 cp s3://genomeark/species/Rhinolophus_ferrumequinum/mRhiFer1/assembly_vgp_standard_1.0/intermediates/meryl/mRhiFer1.cut.meryl.hist - --no-sign-request

Integration with VGP Workflows

This skill integrates with:

• vgp-pipeline: Fetching GenomeScope summaries for workflow analysis
• bioinformatics-fundamentals: General S3 access patterns
• galaxy-automation: Meryl histogram URLs for Galaxy import

See vgp-pipeline skill for species-specific workflow integration patterns.

References

• GenomeArk Homepage: https://www.genomeark.org/
• VGP Project: https://vertebrategenomesproject.org/
• AWS S3 CLI Documentation: https://docs.aws.amazon.com/cli/latest/reference/s3/

Version History

• 1.1 (2026-02-26): Assembly date extraction patterns added
• NEW: Comprehensive assembly directory patterns (30+ patterns)
• NEW: Legacy versions (1.6, 1.0, 1.4) for Pri/alt assemblies
• NEW: Verkko assembly patterns (diploid assemblies)
• NEW: Clade-specific directories (primate_v*, etc.)
• NEW: Institution-specific patterns (rockefeller, cambridge, milan)
• NEW: Patterns without "assembly_" prefix
• NEW: Assembly date extraction strategy
• NEW: Date validation results (1.1 year average delay)
• Updated: Coverage improvement: 47-62% vs 27% with basic patterns

• 1.0 (2026-02-26): Initial skill creation, consolidated from fundamentals and vgp-pipeline skills
• Comprehensive S3 structure documentation
• All three GenomeScope filename patterns
• GenomeScope validation logic
• Meryl histogram access patterns
• Best practices and common pitfalls

QC Data Fetching from GenomeArk

Detailed fetching strategies for GenomeScope, BUSCO, Merqury, and Meryl data.

1. GenomeScope Data (Genome Characteristics)

Location: {assembly}/evaluation/genomescope/

Metrics Available:

• Genome haploid length (genome size estimate)
• Heterozygosity percentage
• Repeat content percentage
• Unique sequence length

Filename Patterns (TRY ALL THREE!)

Pattern A - Double Underscore (most common):

{ToLID}_genomescope__Summary.txt

Example: aGasCar1_genomescope__Summary.txt

Pattern C - Single Underscore (easily missed, discovered 2026):

{ToLID}_genomescope_Summary.txt

Example: rPlaMeg1_genomescope_Summary.txt

Pattern B - No Prefix (older assemblies):

{ToLID}_Summary.txt

Example: aSpeBom1_Summary.txt

CRITICAL: Checking only patterns A and B causes ~30-40% of data to be missed! Pattern C (single underscore) is common but was only discovered during Feb 2026 debugging.

File Format

GenomeScope version 2.0
...
property                      min               max
Genome Haploid Length         4,077,481,159 bp  4,095,803,536 bp
Heterozygous (ab)             1.43264%          1.47696%
Genome Repeat Length          2,528,408,288 bp  2,539,769,824 bp
Genome Unique Length          1,567,234,248 bp  1,556,033,712 bp

Parsing Rules:

• Always use max value (second column) for measurements
• Genome size: Take max haploid length, remove commas
• Heterozygosity: Take max percentage (validate first!)
• Repeat content: Calculate (repeat_length / genome_size) * 100

Data Validation - CRITICAL!

Failed GenomeScope runs produce unrealistic ranges that MUST be filtered:

Failed Run (DO NOT USE):

Heterozygous (ab)    0%    100%

Valid Run (ACCEPT):

Heterozygous (ab)    0.49%    0.54%

Validation Logic:

def validate_genomescope(min_het, max_het):
    """Validate GenomeScope heterozygosity estimates"""
    range_width = max_het - min_het

    # Reject if:
    if range_width > 50.0:        # Range too wide = model failure
        return False
    if max_het > 95.0:            # Unrealistic for diploid genomes
        return False
    if min_het == 0 and max_het == 100:  # Complete failure
        return False

    return True  # ACCEPT

Skip values if:

• Range width > 50% (indicates model failure)
• Max value > 95% (unrealistic for most genomes)
• Range is exactly 0%-100% (complete model failure)

Fetching Strategy

import subprocess
import re
import time

def normalize_s3_path(s3_path):
    """Normalize path for GenomeArk (case sensitivity!)"""
    if not s3_path:
        return None
    # Critical: HiC capitalization
    s3_path = s3_path.replace('/assembly_vgp_hic_2.0/', '/assembly_vgp_HiC_2.0/')
    if not s3_path.endswith('/'):
        s3_path += '/'
    return s3_path

def fetch_genomescope_data(species_name, tolid, assembly_type='HiC_2.0'):
    """Fetch GenomeScope summary with all pattern attempts"""
    species_s3 = species_name.replace(' ', '_')
    base_path = f"s3://genomeark/species/{species_s3}/{tolid}/assembly_vgp_{assembly_type}/"
    base_path = normalize_s3_path(base_path)

    # Try ALL THREE filename patterns in order
    patterns = [
        f'{tolid}_genomescope__Summary.txt',   # Pattern A: double underscore
        f'{tolid}_genomescope_Summary.txt',    # Pattern C: single underscore
        f'{tolid}_Summary.txt'                  # Pattern B: no prefix
    ]

    for pattern in patterns:
        file_path = f"{base_path}evaluation/genomescope/{pattern}"

        try:
            result = subprocess.run(
                ['aws', 's3', 'cp', file_path, '-', '--no-sign-request'],
                capture_output=True,
                text=True,
                timeout=30
            )

            if result.returncode == 0 and result.stdout:
                data = parse_genomescope_summary(result.stdout)
                if data:
                    return data
        except subprocess.TimeoutExpired:
            continue

        time.sleep(0.2)  # Rate limiting

    return None

def parse_genomescope_summary(content):
    """Extract and validate genome characteristics"""
    data = {}

    # Genome Haploid Length (max value - second column)
    match = re.search(r'Genome Haploid Length\s+[\d,]+\s*bp\s+([\d,]+)\s*bp', content)
    if match:
        data['genome_size'] = int(match.group(1).replace(',', ''))

    # Heterozygosity percentage (extract both min and max)
    match = re.search(r'Heterozygous \(ab\)\s+([\d.]+)%\s+([\d.]+)%', content)
    if match:
        min_het = float(match.group(1))
        max_het = float(match.group(2))

        # VALIDATE before accepting
        if validate_genomescope(min_het, max_het):
            data['heterozygosity'] = max_het
        # Otherwise skip - don't include invalid value

    # Repeat content calculation
    repeat_match = re.search(r'Genome Repeat Length\s+[\d,]+\s*bp\s+([\d,]+)\s*bp', content)
    unique_match = re.search(r'Genome Unique Length\s+[\d,]+\s*bp\s+([\d,]+)\s*bp', content)

    if repeat_match and unique_match:
        repeat_length = float(repeat_match.group(1).replace(',', ''))
        unique_length = float(unique_match.group(1).replace(',', ''))
        total_length = repeat_length + unique_length
        if total_length > 0:
            repeat_percent = (repeat_length / total_length) * 100
            data['repeat_content'] = round(repeat_percent, 2)

    return data if data else None

Expected Success Rate: ~80-90% for VGP species

2. BUSCO Data (Assembly Completeness)

Location: {assembly}/evaluation/busco/{subdir}/

Subdirectory Variations:

• c/, c1/ - primary chromosome results
• p/, p1/ - primary scaffold results
• Search dynamically, don't hardcode

Files: *short_summary*.txt (case-insensitive)

Filename Patterns

HiC assemblies:

{ToLID}_HiC__busco_hap1_busco_short_summary.txt
{ToLID}_HiC__busco_hap2_busco_short_summary.txt

Standard assemblies:

{ToLID}_busco_short_summary.txt

File Format

# BUSCO version is: 5.2.2
# The lineage dataset is: vertebrata_odb10
# Summarized benchmarking in BUSCO notation for file primary.fa
...
	C:94.0%[S:92.4%,D:1.6%],F:2.7%,M:3.3%,n:3354
	3152	Complete BUSCOs (C)
	3099	Complete and single-copy BUSCOs (S)
	53	Complete and duplicated BUSCOs (D)
	91	Fragmented BUSCOs (F)
	111	Missing BUSCOs (M)
	3354	Total BUSCO groups searched

Parsing: Extract completeness from line starting with C: -> 94.0 from C:94.0%

Fetching Strategy

def fetch_busco_data(base_path, tolid):
    """Fetch BUSCO completeness with dynamic subdir search"""
    base_path = normalize_s3_path(base_path)

    # List busco subdirectories
    list_cmd = ['aws', 's3', 'ls', f"{base_path}evaluation/busco/", '--no-sign-request']
    result = subprocess.run(list_cmd, capture_output=True, text=True, timeout=10)

    if result.returncode != 0:
        return None

    # Find subdirectories (lines with 'PRE')
    subdirs = [line.split('PRE')[1].strip().rstrip('/')
               for line in result.stdout.split('\n') if 'PRE' in line]

    # Try each subdirectory
    for subdir in subdirs:
        # List files in subdirectory
        files_cmd = ['aws', 's3', 'ls', f"{base_path}evaluation/busco/{subdir}/", '--no-sign-request']
        files_result = subprocess.run(files_cmd, capture_output=True, text=True, timeout=10)

        # Find short_summary files
        for line in files_result.stdout.split('\n'):
            if 'short_summary' in line.lower():
                filename = line.split()[-1]
                file_path = f"{base_path}evaluation/busco/{subdir}/{filename}"

                # Fetch and parse
                content_result = subprocess.run(
                    ['aws', 's3', 'cp', file_path, '-', '--no-sign-request'],
                    capture_output=True, text=True, timeout=30
                )

                if content_result.returncode == 0:
                    completeness = parse_busco_completeness(content_result.stdout)
                    if completeness:
                        return {'completeness': completeness, 'subdir': subdir}

        time.sleep(0.2)  # Rate limiting

    return None

def parse_busco_completeness(content):
    """Extract completeness percentage"""
    for line in content.split('\n'):
        if line.strip().startswith('C:'):
            match = re.search(r'C:([\d.]+)%', line)
            if match:
                return float(match.group(1))
    return None

3. Merqury Data (Assembly QV Scores)

Location: Two path patterns (structure changed 2022 -> 2024)

Pattern A - Direct (2024+, newer):

{assembly}/evaluation/merqury/{ToLID}_qv/output_merqury.tabular

Pattern B - Nested (2022, older):

{assembly}/evaluation/merqury/{c,p}/{ToLID}_qv/output_merqury.tabular

Strategy: Try direct path first, then search nested subdirectories

File Format

Tab-separated, may have header:

assembly	unique k-mers	common k-mers	QV	error rate
assembly_01	20197	2133011206	63.4592	4.50896e-07
assembly_02	19654	2304717679	63.9138	4.06084e-07
Both	39851	4437728885	63.6894	4.27623e-07

Parsing:

• Skip header line if starts with assembly\t
• QV is always column 4 (index 3)
• Take first data line (usually assembly_01 or Both)

Fetching Strategy

def fetch_merqury_data(base_path, tolid):
    """Fetch Merqury QV with fallback patterns"""
    base_path = normalize_s3_path(base_path)

    # Try direct path first (newer structure)
    direct_path = f"{base_path}evaluation/merqury/{tolid}_qv/output_merqury.tabular"
    result = subprocess.run(
        ['aws', 's3', 'cp', direct_path, '-', '--no-sign-request'],
        capture_output=True, text=True, timeout=30
    )

    if result.returncode == 0 and result.stdout:
        qv = parse_merqury_qv(result.stdout)
        if qv:
            return {'qv': qv, 'path_type': 'direct'}

    # Fallback: search nested subdirectories (older structure)
    list_result = subprocess.run(
        ['aws', 's3', 'ls', f"{base_path}evaluation/merqury/", '--no-sign-request'],
        capture_output=True, text=True, timeout=10
    )

    if list_result.returncode == 0:
        subdirs = [line.split('PRE')[1].strip().rstrip('/')
                   for line in list_result.stdout.split('\n') if 'PRE' in line]

        for subdir in subdirs:
            nested_path = f"{base_path}evaluation/merqury/{subdir}/{tolid}_qv/output_merqury.tabular"
            result = subprocess.run(
                ['aws', 's3', 'cp', nested_path, '-', '--no-sign-request'],
                capture_output=True, text=True, timeout=30
            )

            if result.returncode == 0 and result.stdout:
                qv = parse_merqury_qv(result.stdout)
                if qv:
                    return {'qv': qv, 'path_type': 'nested', 'subdir': subdir}

            time.sleep(0.2)

    return None

def parse_merqury_qv(content):
    """Extract QV score from merqury output"""
    for line in content.split('\n'):
        line = line.strip()
        if not line or line.startswith('assembly\t'):
            continue  # Skip header

        parts = line.split('\t')
        if len(parts) >= 4:
            try:
                return float(parts[3])  # QV is column 4
            except ValueError:
                continue
    return None

4. Meryl K-mer Histograms

Location: {assembly}/intermediates/meryl/

Key Insight: GenomeScope only needs .hist files, not full meryl databases!

File Structure:

s3://genomeark/species/{species}/{tolid}/assembly_*/intermediates/meryl/
├── {tolid}.cut.meryl.hist          # ~700KB - DOWNLOAD THIS
└── {tolid}.cut.meryl/              # Many GB - full database (DON'T DOWNLOAD)
    ├── 0x000000.merylData
    ├── 0x000000.merylIndex
    └── ...

Direct HTTPS URLs (for Galaxy import):

https://genomeark.s3.amazonaws.com/species/{species}/{tolid}/assembly_vgp_standard_1.0/intermediates/meryl/{tolid}.cut.meryl.hist

Example:

https://genomeark.s3.amazonaws.com/species/Rhinolophus_ferrumequinum/mRhiFer1/assembly_vgp_standard_1.0/intermediates/meryl/mRhiFer1.cut.meryl.hist

Benefits:

• 1000x smaller download (~700KB vs ~10GB)
• Can import directly into Galaxy via URL
• No need to download full meryl database
• Much faster for batch GenomeScope analysis

Common Mistake: Downloading entire meryl directory when only .hist file is needed

Fetching Strategy

def fetch_meryl_histogram(species_name, tolid, assembly_type='standard_1.0'):
    """Fetch meryl histogram file (not full database)"""
    species_s3 = species_name.replace(' ', '_')
    file_path = (
        f"s3://genomeark/species/{species_s3}/{tolid}/"
        f"assembly_vgp_{assembly_type}/intermediates/meryl/{tolid}.cut.meryl.hist"
    )

    result = subprocess.run(
        ['aws', 's3', 'cp', file_path, f'./{tolid}.cut.meryl.hist', '--no-sign-request'],
        capture_output=True, text=True, timeout=60
    )

    return result.returncode == 0

def get_histogram_url(species_name, tolid, assembly_type='standard_1.0'):
    """Get direct HTTPS URL for Galaxy import"""
    species_s3 = species_name.replace(' ', '_')
    return (
        f"https://genomeark.s3.amazonaws.com/species/{species_s3}/{tolid}/"
        f"assembly_vgp_{assembly_type}/intermediates/meryl/{tolid}.cut.meryl.hist"
    )

5. Technology Verification from genomic_data/

When Assembly tech is missing, check GenomeArk genomic_data/ subfolders:

aws s3 ls --no-sign-request \
  s3://genomeark/species/Genus_species/tolid/genomic_data/

Folder to Technology mapping:

Fallback: If not in GenomeArk, check sequencing_tech column:

• "PacBio Sequel" or "PacBio Sequel I/II CLR" -> CLR
• "PacBio Sequel II HiFi" or "PacBio Revio HiFi" -> HiFi
• "PacBio" + "Oxford Nanopore" -> check for pacbio_hifi/ to distinguish CLR+ONT vs HiFi+ONT