Devtu Optimize Skills

Optimizing ToolUniverse Skills

Best practices for high-quality research skills with evidence grading and source attribution.

Tool Quality Standards

1. Error messages must be actionable — tell the user what went wrong AND what to do 2. Schema must match API reality — run python3 -m tooluniverse.cli run <Tool> '<json>' to verify 3. Coverage transparency — state what data is NOT included 4. Input validation before API calls — don't silently send invalid values 5. Cross-tool routing — name the correct tool when query is out-of-scope 6. No silent parameter dropping — if a parameter is ignored, say so

Core Principles (13 Patterns)

Full details: references/optimization-patterns.md

Optimized Skill Workflow

Phase -1: Tool Verification (check params)
Phase  0: Foundation Data (aggregator query)
Phase  1: Disambiguation (IDs, collisions, baseline)
Phase  2: Specialized Queries (fill gaps)
Phase  3: Report Synthesis (evidence-graded narrative)

Testing Standards

Full details: references/testing-standards.md

Critical rule: NEVER write skill docs without testing all tool calls first.

• 30+ tests per skill, 100% pass rate
• All tests use real data (no placeholders)
• Phase + integration + edge case tests
• SOAP tools (IMGT, SAbDab, TheraSAbDab) need operation parameter
• Distinguish transient errors (retry) from real bugs (fix)
• API docs are often wrong — always verify with actual calls

Pattern 14: Reasoning Frameworks Over Tool Catalogs (CRITICAL)

Skills that just list tools ("call A, then B, then C") score 3-5/10 in usefulness tests. Skills that explain HOW to interpret and combine data score 7-9/10. Every skill MUST include:

14a. Interpretation Tables

Map raw API data to biological/clinical meaning. Don't just retrieve — explain.

14b. Synthesis Phases

Every multi-phase skill needs a final phase that answers "so what?" — not just collecting data:

• "What changed and why does it matter?"
• "Is this cause or consequence?"
• "What's the actionable recommendation?"

14c. Honest Limitations

If a tool API can't deliver what the skill promises, say so explicitly. Don't describe aspirational capabilities. Example: "DepMap_get_gene_dependencies returns gene metadata only, NOT per-cell-line CRISPR scores."

Pattern 15: Computational Procedures When Tools Can't Help

Some scientific analyses require computation, not just API queries. When no tool exists for a capability, embed a Python code procedure directly in the skill using packages available in ToolUniverse (pandas, scipy, numpy, statsmodels, biopython, networkx).

When to use computational procedures:

Template for computational procedures in skills:

**Computational procedure: [Name]**
[When to use this: explain the gap it fills]

\`\`\`python
# [What this computes]
# Requires: [packages] (included in ToolUniverse dependencies)
import pandas as pd
from scipy.stats import mannwhitneyu

# Input: [describe expected input format]
# Output: [describe output]
# [Full working code with example data]
\`\`\`

[Interpretation guidance for the output]

Key rules for computational procedures:

1. Only use packages in ToolUniverse dependencies (pyproject.toml): pandas, scipy, numpy, networkx, requests, biopython (optional extra) 2. Include example data so the procedure is immediately testable 3. Explain the output — a code block without interpretation is useless 4. Note when external data download is needed (e.g., DepMap CSV from depmap.org)

Pattern 15b: Download-and-Process for Datasets Without REST APIs

Many critical scientific datasets have NO REST API but provide bulk download files. Skills should include concrete download-and-process instructions when this is the only path to essential data.

Template for download-and-process procedures:

**Step 1: Download data files**
- URL: [exact download page URL]
- Files needed: [filename] (~[size]) — [what it contains]
- Registration: [required/not required]
- Update frequency: [quarterly/annually/etc.]

**Step 2: Process with Python**
[Working code with pandas/scipy that loads the CSV and produces the analysis]

**Step 3: Interpret results**
[Table mapping output values to biological/clinical meaning]

**When files are not available**: [Fallback strategy using API tools]

Known download-only datasets that skills reference:

Critical rule: Always include a fallback for when the download is unavailable (user may not have registration, file may be too large, etc.). The fallback should use available API tools even if they provide less complete data.

Common Anti-Patterns

Quick Fixes for User Complaints

Skill Template

---
name: [domain]-research
description: [What + when triggers]
---

# [Domain] Research

## Workflow
Phase -1: Tool Verification → Phase 0: Foundation → Phase 1: Disambiguate
→ Phase 2: Search → Phase 3: Report

## Phase -1: Tool Verification
[Parameter corrections table]

## Phase 0: Foundation Data
[Aggregator query]

## Phase 1: Disambiguation
[IDs, collisions, baseline]

## Phase 2: Specialized Queries
[Query strategy, fallbacks]

## Phase 3: Report Synthesis
[Evidence grading, mandatory sections]

## Output Files
- [topic]_report.md, [topic]_bibliography.json

## Quantified Minimums
[Numbers per section]

## Completeness Checklist
[Required sections with checkboxes]

Additional References

• Detailed patterns: references/optimization-patterns.md
• Testing standards: references/testing-standards.md
• Case studies (4 real fixes): references/case-studies.md
• Checklists (review + release): references/checklists.md

Case Studies: Fixing Non-Functional Skills

Real-world examples from fixing 4 broken skills in February 2026.

Case 1: Drug-Drug Interaction Skill

Original: 0% functional. Docs showed drugbank_get_drug_basic_info_by_drug_name_or_id(drug_name_or_drugbank_id="...") but tool requires query.

Fixes:

Lesson: Function names are misleading. get_drug_basic_info_by_drug_name_or_id takes query, not drug_name_or_id.

Case 2: Antibody Engineering Skill

Original: 0% functional. All SOAP tool calls missing operation parameter.

Fix: Added operation parameter to all SOAP calls (IMGT, SAbDab, TheraSAbDab).

Lesson: SOAP tools have special requirements not obvious from function signatures.

Case 3: CRISPR Screen Analysis Skill

Original: 20% functional. Primary API (DepMap) completely down (404).

Fix: Implemented Pharos TDL fallback. TDL classification (Tclin/Tchem/Tbio/Tdark) as essentiality proxy.

Lesson: External APIs fail. Always implement fallback chains.

Case 4: Clinical Trial Design Skill

Original: 0% functional. All DrugBank tool parameters wrong throughout.

Fix: ALL DrugBank tools use query, not the parameter names in their function names.

Lesson: Even when multiple tools have similar naming patterns, always verify each one.

Common Thread

All 4 skills had excellent documentation (300+ lines) but were never tested. The fix was always the same: test with real ToolUniverse instance first, then write docs from working code.

Skill Quality Checklists

Skill Review Checklist

Tool Contract

• [ ] Tool parameters verified via get_tool_info() or documented corrections
• [ ] Versioned vs unversioned ID handling specified
• [ ] Foundation data source identified (if available for domain)

Report Quality

• [ ] Report focuses on content, not search process
• [ ] Methodology in separate appendix (optional)
• [ ] Evidence grades applied to claims (T1-T4)
• [ ] Source attribution on every fact
• [ ] Sections exist even if "limited evidence"

Query Strategy

• [ ] Disambiguation phase before search
• [ ] Collision detection for ambiguous names
• [ ] High-precision seeds before broad search
• [ ] Citation expansion for sparse topics
• [ ] Negative filters documented

Tool Usage

• [ ] Annotation tools used (not just literature)
• [ ] Fallback chains defined
• [ ] Failure handling with retry
• [ ] OA handling (full or best-effort)

Completeness

• [ ] Quantified minimums defined per section
• [ ] Completeness checklist with checkboxes
• [ ] Data Gaps section aggregates all missing data
• [ ] "Negative results" explicitly documented

Output Structure

• [ ] Main report is narrative-focused
• [ ] Bibliography in separate JSON/CSV
• [ ] Synthesis sections required

Implementation & Testing (2026-02 Standards)

• [ ] All tool calls tested in real ToolUniverse instance (MANDATORY)
• [ ] Test script with >=30 tests
• [ ] 100% test pass rate
• [ ] All tests use real data (no placeholders)
• [ ] Edge cases: empty, large, invalid, boundary
• [ ] Phase-level + integration + cross-example tests
• [ ] SOAP tools have operation parameter (if applicable)
• [ ] Fallback strategies implemented and tested
• [ ] API quirks documented

Documentation (2026-02 Standards)

• [ ] SKILL.md is implementation-agnostic (no Python/MCP code)
• [ ] Working python_implementation.py
• [ ] QUICK_START.md with both SDK and MCP examples
• [ ] TOOLS_REFERENCE.md with verified parameters
• [ ] All code examples actually work (copy-paste ready)

Pre-Release Final Check

# 1. Run test suite
python test_*.py  # Expect 100% pass

# 2. Check for placeholders
grep -r "TEST\|DUMMY\|PLACEHOLDER" *.md *.py  # Should find none

# 3. Performance benchmark
time python test_*.py  # Document time

# 4. Edge case coverage
grep "def test_edge" test_*.py  # Should have 5+

Optimization Patterns

Detailed patterns for improving ToolUniverse skill quality.

Table of Contents

• Tool Interface Verification
• Foundation Data Layer
• Versioned Identifier Handling
• Disambiguation Before Research
• Report-Only Output
• Evidence Grading
• Quantified Completeness
• Mandatory Completeness Checklist
• Aggregated Data Gaps
• Query Strategy Optimization
• Tool Failure Handling
• Scalable Output Structure
• Synthesis Sections

---

1. Tool Interface Verification

Verify tool parameters before calling unfamiliar tools:

tool_info = tu.tools.get_tool_info(tool_name="Reactome_map_uniprot_to_pathways")
# Reveals: takes `id` not `uniprot_id`

Known corrections table:

Rule: Before calling any tool for the first time, verify params via get_tool_info().

2. Foundation Data Layer

Query a comprehensive aggregator FIRST before specialized tools:

Pattern: Phase 0 (aggregator) → Phase 1 (disambiguate) → Phase 2 (specialized) → Phase 3 (report)

3. Versioned Identifier Handling

Capture BOTH versioned and unversioned forms during ID resolution:

ids = {
    'ensembl': 'ENSG00000123456',              # Most APIs
    'ensembl_versioned': 'ENSG00000123456.12'   # GTEx, GENCODE
}

Fallback: try unversioned first → versioned if empty → document which worked.

4. Disambiguation Before Research

Add disambiguation phase before literature search: 1. Resolve official identifiers (UniProt, Ensembl, NCBI Gene, ChEMBL) 2. Gather synonyms and aliases 3. Detect naming collisions (search "[SYMBOL]"[Title], check if >20% off-topic) 4. Build negative filters for collisions 5. Get baseline profile from annotation DBs (not literature)

5. Report-Only Output

DO: "The literature reveals three main therapeutic approaches..." DON'T: "I searched PubMed, OpenAlex, and EuropePMC, finding 342 papers..."

6. Evidence Grading

Required in: Executive Summary, Disease Associations, Key Papers table, Recommendations.

7. Quantified Completeness

8. Mandatory Completeness Checklist

All sections must exist, even if "Limited evidence":

• Identity: IDs resolved, synonyms, collisions
• Biology: architecture, localization, expression (>=10 tissues), pathways (>=10)
• Mechanism: core function with evidence, model organisms, key assays
• Disease/Clinical: variants, constraint scores (all 4), disease links (>=10)
• Druggability: tractability, known drugs, probes, clinical pipeline
• Synthesis: themes (>=3 papers each), open questions, biological model, hypotheses (>=3)

9. Aggregated Data Gaps

Consolidate all gaps into one section:

## Data Gaps & Limitations
| Section | Expected | Actual | Reason | Alternative |
|---------|----------|--------|--------|-------------|
| PPIs | >=20 | 8 | Novel target | Literature review |
| Expression | GTEx TPM | None | ID not recognized | HPA data |

10. Query Strategy Optimization

Three-step collision-aware strategy: 1. High-precision seeds (15-30 papers): "[SYMBOL]"[Title] AND mechanism 2. Citation expansion: forward (cited_by), related, backward (references) 3. Collision-filtered broad: apply negative filters for known collisions

11. Tool Failure Handling

NEVER silently skip failed tools. Document in Data Gaps section.

12. Scalable Output Structure

Narrative report (~20-50 pages): executive summary, key findings by theme, top 20-50 papers, conclusions. Bibliography files (unlimited): JSON + CSV with evidence tiers, themes, OA status.

13. Synthesis Sections

Required:

• Biological Model (3-5 paragraphs): integrate all evidence
• Testable Hypotheses (>=3): hypothesis, perturbation, readout, expected result
• Suggested Experiments: how to test each hypothesis

Test-Driven Skill Development Standards

The Golden Rule

NEVER write skill documentation without first testing all tool calls with real ToolUniverse instance.

All 4 broken skills discovered in Feb 2026 had excellent docs but 0% functionality because tools were never tested.

Test-First Workflow

1. Write skill implementation (phases, tool calls) 2. Write comprehensive test suite 3. Run tests, achieve 100% pass rate 4. Fix all failures 5. ONLY THEN mark skill as complete

Test Suite Structure

#!/usr/bin/env python3
"""Comprehensive Test Suite for [Skill Name]"""

# Naming: test_phase[N]_[description]
def test_phase1_gene_resolution():
    result = resolve_gene("BRCA1")  # NOT "TEST_GENE"
    assert result['ensembl_id'] == "ENSG00000012048"

def test_phase1_edge_cases():
    result = resolve_gene("FAKE_GENE_XYZ")
    assert result is None or 'error' in result

def test_integration_full_workflow():
    result = analyze("EGFR", "L858R", "lung adenocarcinoma")
    assert result['clinical_evidence']
    assert result['completeness_score'] >= 80

What to Test

1. All use cases from SKILL.md (4-6) 2. Every documented parameter 3. All response fields 4. Edge cases:

• Empty/minimal data
• Large data (500+ genes)
• Invalid data (unknown gene, typos)
• Boundary values (TMB=0, TMB=999)
• Conflicting data (high TMB + low PD-L1)

Test Output Format

PASS Phase1: Gene resolution - BRCA1 -> ENSG00000012048
PASS Phase2: CIViC evidence - Found 12 entries
WARN Phase4: Clinical trials - API timeout (transient)
FAIL Phase5: Pathway enrichment - Missing 'gene_list'

Total: 80 | PASS: 78 | FAIL: 1 | WARN: 1 | Rate: 97.5%

Transient vs Real Errors

Transient (retry): timeouts, rate limiting (429), service overload (503) Real (fix): wrong parameters, missing fields, logic errors

Handle transient errors with exponential backoff. In tests, mark as PASS with note.

Minimum Requirements

• 30+ tests per skill
• 100% pass rate (transient errors = PASS with warning)
• All tests use real data (no "TEST", "DUMMY", "PLACEHOLDER")
• Phase-level + integration + edge case + cross-example tests
• Performance benchmarks documented

SOAP Tools Special Handling

IMGT, SAbDab, TheraSAbDab require operation parameter as first param:

tu.tools.IMGT_search_genes(operation="search_genes", gene_type="IGHV", species="Homo sapiens")

API Documentation Is Often Wrong

Always verify with actual calls: 1. Check tool parameters via get_tool_info() 2. Test with real data 3. Inspect actual response structure 4. Document findings in TOOLS_REFERENCE.md