Devtu Fix Tool

Fix ToolUniverse Tools

Diagnose and fix failing ToolUniverse tools through systematic error identification, targeted fixes, and validation.

First Principles for Bug Fixes

Before writing any fix, ask: why does the user reach this failure state?

1. Prevent, don't recover — fix the root cause so the failure can't happen, rather than adding hint text after it does 2. Validate at input, not at output — wrong parameters, unknown disease names, unsupported drugs should be caught and rejected early with clear guidance, not discovered after a silent API call 3. Don't mask silent mutations — if input is auto-normalized (fusion notation, Title Case), either accept both forms natively OR reject with explicit guidance; never silently transform and hide it 4. Distinguish "no data" from "bad query" — zero results because the filter is wrong is different from zero results because the data doesn't exist; the response must distinguish these clearly 5. Fix the abstraction, not the instance — if a parameter name is inconsistent, fix the interface; don't add an alias list that grows forever

Anti-patterns to avoid:

• Adding hint text to zero-result messages instead of validating upfront
• Adding parameter aliases instead of fixing naming consistency
• Post-hoc probing to rescue a failed query instead of pre-validating

Bug Verification (CRITICAL)

Before implementing any bug report, verify it via CLI first:

python3 -m tooluniverse.cli run <ToolName> '<json_args>'

Many agent-reported bugs are false positives caused by MCP interface confusion. Always confirm the bug is reproducible before implementing a fix.

---

Instructions

When fixing a failing tool:

1. Run targeted test to identify error:

python scripts/test_new_tools.py <tool-pattern> -v

2. Verify API is correct - search online for official API documentation to confirm endpoints, parameters, and patterns are correct

3. Identify error type (see Error Types section)

4. Apply appropriate fix based on error pattern

4. Regenerate tools if you modified JSON configs or tool classes:

python -m tooluniverse.generate_tools

5. Check and update tool tests if they exist in tests/tools/:

ls tests/tools/test_<tool-name>_tool.py

6. Verify fix by re-running both integration and unit tests

7. Provide fix summary with problem, root cause, solution, and test results

Where to Fix

Error Types

1. JSON Parsing Errors

Symptom: Expecting value: line 1 column 1 (char 0)

Cause: Tool expects JSON but receives binary data (images, PDFs, files)

Fix: Check Content-Type header. For binary responses, return a description string instead of parsing JSON. Update return_schema to {"type": "string"}.

2. Schema Validation Errors

Symptom: Schema Mismatch: At root: ... is not of type 'object' or Data: None

Cause: Missing data field wrapper OR wrong schema type

Fix depends on the error:

• If Data: None → Add data wrapper to ALL operation methods (see Multi-Operation Pattern below)
• If type mismatch → Update return_schema in JSON config:
• Data is string: {"type": "string"}
• Data is array: {"type": "array", "items": {...}}
• Data is object: {"type": "object", "properties": {...}}

Key concept: Schema validates the data field content, NOT the full response.

3. Nullable Field Errors

Symptom: Schema Mismatch: At N->fieldName: None is not of type 'integer'

Cause: API returns None/null for optional fields

Fix: Allow nullable types in JSON config using {"type": ["<base_type>", "null"]}. Use for optional fields, not required identifiers.

4. Mutually Exclusive Parameter Errors

Symptom: Parameter validation failed for 'param_name': None is not of type 'integer' when passing a different parameter

Cause: Tool accepts EITHER paramA OR paramB (mutually exclusive), but both are defined with fixed types. When only one is provided, validation fails because the other is None.

Example:

{
  "neuron_id": {"type": "integer"},      // ❌ Fails when neuron_name is used
  "neuron_name": {"type": "string"}      // ❌ Fails when neuron_id is used
}

Fix: Make mutually exclusive parameters nullable:

{
  "neuron_id": {"type": ["integer", "null"]},      // ✅ Allows None
  "neuron_name": {"type": ["string", "null"]}      // ✅ Allows None
}

Common patterns:

• id OR name parameters (get by ID or by name)
• acronym OR name parameters (search by symbol or full name)
• Optional filter parameters that may not be provided

Important: Also make truly optional parameters (like filter_field, filter_value) nullable even if not mutually exclusive.

5. Mixed Type Field Errors

Symptom: Schema Mismatch: At N->field: {object} is not of type 'string', 'null'

Cause: Field returns different structures depending on context

Fix: Use oneOf in JSON config for fields with multiple distinct schemas. Different from nullable ({"type": ["string", "null"]}) which is same base type + null.

6. Invalid Test Examples

Symptom: 404 ERROR - Not found or 400 Bad Request

Cause: Test example uses invalid/outdated IDs

Fix: Discover valid examples using the List → Get or Search → Details patterns below.

7. API Parameter Errors

Symptom: 400 Bad Request or parameter validation errors

Fix: Update parameter schema in JSON config with correct types, required fields, and enums.

8. API Key Configuration Errors

Symptom: Tool not loading when API key is optional, or api_key parameter causing confusion

Cause: Using required_api_keys for keys that should be optional, or exposing API key as tool parameter

Key differences:

• required_api_keys: Tool is skipped if keys are missing
• optional_api_keys: Tool loads and works without keys (with reduced performance)

Fix: Use optional_api_keys in JSON config for APIs that work anonymously but have better rate limits with keys. Read API key from environment only (os.environ.get()), never as a tool parameter.

9. API Endpoint Pattern Errors

Symptom: 404 for valid resources, or unexpected results

Fix: Verify official API docs - check if values belong in URL path vs query parameters.

10. Transient API Failures

Symptom: Tests fail intermittently with timeout/connection/5xx errors

Fix: Use pytest.skip() for transient errors in unit tests - don't fail on external API outages.

Common Fix Patterns

Schema Validation Pattern

Schema validates the data field content, not the full response. Match return_schema type to what's inside data (array, object, or string).

Multi-Operation Tool Pattern

Every internal method must return {"status": "...", "data": {...}}. Don't use alternative field names at top level.

Finding Valid Test Examples

When test examples fail with 400/404, discover valid IDs by:

• List → Get: Call a list endpoint first, extract ID from results
• Search → Details: Search for a known entity, use returned ID
• Iterate Versions: Try different dataset versions if supported

Unit Test Management

Check for Unit Tests

After fixing a tool, check if unit tests exist:

ls tests/tools/test_<tool-name>_tool.py

When to Update Unit Tests

Update unit tests when you:

1. Change return structure: Update assertions checking result["data"] structure 2. Add/modify operations: Add test cases for new operations 3. Change error handling: Update error assertions 4. Modify required parameters: Update parameter validation tests 5. Fix schema issues: Ensure tests validate correct data structure 6. Add binary handling: Add tests for binary responses

Running Unit Tests

# Run specific tool tests
pytest tests/tools/test_<tool-name>_tool.py -v

# Run all unit tests
pytest tests/tools/ -v

Unit Test Checklist

• [ ] Check if tests/tools/test_<tool-name>_tool.py exists
• [ ] Run unit tests before and after fix
• [ ] Update assertions if data structure changed
• [ ] Ensure both direct and interface tests pass

For detailed unit test patterns and examples, see unit-tests-reference.md.

Verification

Run Integration Tests

python scripts/test_new_tools.py <pattern> -v

Run Unit Tests (if exist)

pytest tests/tools/test_<tool-name>_tool.py -v

Regenerate Tools

After modifying JSON configs or tool classes:

python -m tooluniverse.generate_tools

Regenerate after:

• Changing src/tooluniverse/data/*_tools.json files
• Modifying tool class implementations

Not needed for test script changes.

Output Format

After fixing, provide this summary:

Problem: [Brief description]

Root Cause: [Why it failed]

Solution: [What was changed]

Changes Made:

• File 1: [Description]
• File 2: [Description]
• File 3 (if applicable): [Unit test updates]

Integration Test Results:

• Before: X tests, Y passed (Z%), N failed, M schema invalid
• After: X tests, X passed (100.0%), 0 failed, 0 schema invalid

Unit Test Results (if applicable):

• Before: X tests, Y passed, Z failed
• After: X tests, X passed, 0 failed

Testing Best Practices

Verify Parameter Names Before Testing

CRITICAL: Always read the tool's JSON config or generated wrapper to get the correct parameter names. Don't assume parameter names.

Example of incorrect testing:

# ❌ WRONG - assumed parameter name
AllenBrain_search_genes(query='Gad1')  # Fails: unexpected keyword 'query'

Correct approach:

# ✅ RIGHT - checked config first
# Config shows parameters: gene_acronym, gene_name
AllenBrain_search_genes(gene_acronym='Gad1')  # Works!

How to find correct parameter names: 1. Read the JSON config: src/tooluniverse/data/*_tools.json 2. Check the generated wrapper: src/tooluniverse/tools/<ToolName>.py 3. Look at test_examples in the JSON config

Systematic Testing Approach

When testing multiple tools:

1. Sample first: Test 1-2 tools per API to identify patterns 2. Categorize errors: Group by error type (param validation, API errors, data structure) 3. Fix systematically: Fix all tools with same issue type together 4. Regenerate once: Run python -m tooluniverse.generate_tools after all JSON changes 5. Verify all: Test all fixed tools comprehensively

Understanding Data Structure

Tools can return different data structures:

• Object: {"data": {"id": 1, "name": "..."}} - single result
• Array: {"data": [{"id": 1}, {"id": 2}]} - multiple results
• String: {"data": "description text"} - text response

Test accordingly:

# For object data
result = tool()
data = result.get('data', {})
value = data.get('field_name')  # ✅

# For array data
result = tool()
items = result.get('data', [])
count = len(items)  # ✅
first = items[0] if items else {}  # ✅

Common Pitfalls

1. Schema validates `data` field, not full response 2. All methods need `{"status": "...", "data": {...}}` wrapper 3. JSON config changes require regeneration 4. Use `optional_api_keys` for APIs that work without keys 5. Check official API docs for correct endpoint patterns 6. Unit tests should skip on transient API failures, not fail 7. Mutually exclusive parameters MUST be nullable - most common new tool issue 8. Verify parameter names from configs - don't assume or guess 9. Test with correct data structure expectations - list vs dict vs string

Debugging

• Inspect API response: Check status code, Content-Type header, and body preview
• Check tool config: Load ToolUniverse and inspect the tool's configuration
• Add debug prints: Log URL, params, status, and Content-Type in the run method

Quick Reference

ToolUniverse Tool Fix Examples

Real-world examples of fixing ToolUniverse tools, including the complete ChEMBL image endpoint fix.

Example 1: Binary Response Handling (ChEMBL Image)

Initial Error

$ python scripts/test_new_tools.py chembl -v

Testing ChEMBL_get_molecule_image (1 examples)...
  ❌ ChEMBL_get_molecule_image Ex 1: Failed - 
     ChEMBL API request failed: Expecting value: line 1 column 1 (char 0)

Tests Run:        64
Passed:           63 (98.4%)
Failed:           1

Diagnosis

Step 1: Run direct test

from tooluniverse import ToolUniverse
import json

tu = ToolUniverse()
tu.load_tools()

result = tu.run_one_function({
    'name': 'ChEMBL_get_molecule_image',
    'arguments': {
        'chembl_id': 'CHEMBL25',
        'format': 'svg'
    }
})

print(json.dumps(result, indent=2))

Output: Same JSON parsing error

Step 2: Check API directly

import requests

url = "https://www.ebi.ac.uk/chembl/api/data/image/CHEMBL25?format=svg"
response = requests.get(url)

print("Status:", response.status_code)              # 200
print("Content-Type:", response.headers.get("Content-Type"))  # image/svg+xml
print("Is JSON?:", response.text[:10])              # <?xml... (SVG content)

Root cause: API returns SVG image (binary), but tool tries to parse as JSON.

Solution

File 1: src/tooluniverse/chem_tool.py

Located the issue in ChEMBLRESTTool.run() method:

# BEFORE (line 146)
def run(self, arguments: Dict[str, Any]) -> Dict[str, Any]:
    url = self._build_url(arguments)
    params = self._build_params(arguments)
    response = request_with_retry(...)
    response.raise_for_status()
    
    data = response.json()  # ❌ Fails for binary data
    
    return {
        "status": "success",
        "data": data,
        "url": response.url,
    }

Added binary detection:

# AFTER
def run(self, arguments: Dict[str, Any]) -> Dict[str, Any]:
    url = self._build_url(arguments)
    params = self._build_params(arguments)
    tool_name = self.tool_config.get("name", "")
    
    # Check if this is an image endpoint
    is_image_endpoint = "get_molecule_image" in tool_name.lower() or "/image/" in url
    
    response = request_with_retry(...)
    response.raise_for_status()

    # Handle image endpoints differently
    if is_image_endpoint:
        content_type = response.headers.get("Content-Type", "")
        if "image" in content_type or "svg" in content_type:
            return {
                "status": "success",
                "data": f"Image data available at URL (Content-Type: {content_type})",
                "url": response.url,
                "content_type": content_type,
                "image_size_bytes": len(response.content)
            }

    data = response.json()  # ✅ Only called for JSON endpoints
    
    return {
        "status": "success",
        "data": data,
        "url": response.url,
    }

File 2: src/tooluniverse/data/chembl_tools.json

Initially tried this schema (wrong):

{
  "name": "ChEMBL_get_molecule_image",
  "return_schema": {
    "type": "object",
    "properties": {
      "status": {"type": "string"},
      "data": {"type": "string"},
      "url": {"type": "string"}
    }
  }
}

Result: Still failed schema validation because test validates data field (a string) against entire schema (which says root type is "object").

Correct schema:

{
  "name": "ChEMBL_get_molecule_image",
  "return_schema": {
    "type": "string",
    "description": "Description of image data availability and access information"
  }
}

File 3: Regenerate tools

python -m tooluniverse.generate_tools

Verification

$ python scripts/test_new_tools.py chembl -v

Testing ChEMBL_get_molecule_image (1 examples)...
  ✅ Ex 1: Passed

Tests Run:        64
Passed:           64 (100.0%)
Failed:           0
Schema Valid:     64
Schema Invalid:   0

Direct test:

result = tu.run_one_function({
    'name': 'ChEMBL_get_molecule_image',
    'arguments': {'chembl_id': 'CHEMBL25', 'format': 'svg'}
})

print(json.dumps(result, indent=2))

Output:

{
  "status": "success",
  "data": "Image data available at URL (Content-Type: image/svg+xml)",
  "url": "https://www.ebi.ac.uk/chembl/api/data/image/CHEMBL25?format=svg",
  "content_type": "image/svg+xml",
  "image_size_bytes": 8476
}

Key Learnings

1. Content-Type detection is critical - Check headers before parsing 2. Schema describes data field only - Not the full response structure 3. jsonschema is permissive - {"type": "object"} validates any dict 4. Test both ways - Use test script AND direct execution 5. Binary data needs special handling - Return metadata instead of content

---

Example 2: Schema Validation Fix Pattern

Problem

⚠️  Tool_search Ex 1: Schema Mismatch: 
   At root: 'result_string' is not of type 'object'

Diagnosis

Check what data field contains:

result = tu.run_one_function({'name': 'Tool_search', 'arguments': {...}})
print("Data type:", type(result.get('data')))  # <class 'str'>
print("Data value:", result.get('data'))       # "Found 5 results"

Check schema:

{
  "return_schema": {
    "type": "object",
    "properties": {
      "results": {"type": "array"}
    }
  }
}

Problem: Data is string, but schema expects object.

Solution

Option A: Change tool to return object

# In tool class
return {
    "status": "success",
    "data": {
        "message": "Found 5 results",
        "count": 5
    }
}

Option B: Change schema to match string (if tool design is correct)

{
  "return_schema": {
    "type": "string",
    "description": "Search result message"
  }
}

Choose based on:

• Tool's intended design
• Consistency with similar tools
• User expectations

---

Example 3: Endpoint URL Fix

Problem

❌ Tool_get_data Ex 1: Failed - ChEMBL API returned HTTP 404

Diagnosis

Check endpoint configuration:

{
  "name": "Tool_get_data",
  "fields": {
    "endpoint": "/data/{id}.json"
  },
  "test_examples": [
    {"id": "12345"}
  ]
}

Test endpoint manually:

curl https://api.example.com/data/12345.json
# Returns 404

Find correct endpoint:

curl https://api.example.com/records/12345.json
# Returns 200 ✓

Solution

{
  "name": "Tool_get_data",
  "fields": {
    "endpoint": "/records/{id}.json"  // ✅ Fixed
  }
}

---

Example 4: Parameter Schema Fix

Problem

❌ Tool_search Ex 1: Failed - 400 Bad Request: 
   Parameter 'limit' must be integer

Diagnosis

Check parameter schema:

{
  "parameter": {
    "type": "object",
    "properties": {
      "query": {"type": "string"},
      "limit": {"type": "string"}  // ❌ Wrong type
    }
  }
}

Check test example:

{
  "test_examples": [
    {"query": "test", "limit": "10"}  // String passed
  ]
}

Solution

{
  "parameter": {
    "type": "object",
    "properties": {
      "query": {"type": "string"},
      "limit": {
        "type": "integer",           // ✅ Fixed type
        "default": 20,
        "minimum": 1,
        "maximum": 1000
      }
    }
  },
  "test_examples": [
    {"query": "test", "limit": 10}   // ✅ Integer
  ]
}

---

Example 5: Complex Response Handling

Problem

Tool returns nested structure but schema is too simple.

Original Schema

{
  "return_schema": {
    "type": "object"
  }
}

Better Schema

{
  "return_schema": {
    "type": "object",
    "properties": {
      "results": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "id": {"type": "string"},
            "name": {"type": "string"},
            "score": {"type": "number"}
          }
        }
      },
      "total": {"type": "integer"},
      "page": {"type": "integer"}
    }
  }
}

Trade-off: More specific schema provides better validation but requires maintenance if API changes.

Recommendation: Use detailed schemas for critical tools, simple schemas for exploratory tools.

---

Debugging Workflow Example

Complete debugging session for a failing tool:

# 1. Identify failure
$ python scripts/test_new_tools.py category -v
# Note: Tool_name failed with error X

# 2. Test directly with Python
$ python -c "
from tooluniverse import ToolUniverse
import json

tu = ToolUniverse()
tu.load_tools()

result = tu.run_one_function({
    'name': 'Tool_name',
    'arguments': {'param': 'value'}
})

print(json.dumps(result, indent=2))
"

# 3. Check API directly
$ curl -v https://api.example.com/endpoint

# 4. Compare with working tool
$ grep -A 20 "Working_tool" src/tooluniverse/data/category_tools.json

# 5. Implement fix
$ vim src/tooluniverse/category_tool.py
$ vim src/tooluniverse/data/category_tools.json

# 6. Regenerate
$ python -m tooluniverse.generate_tools

# 7. Verify
$ python scripts/test_new_tools.py category -v

# 8. Document
$ vim TOOL_FIX_SUMMARY.md

---

Testing Patterns

Test Individual Tool

from tooluniverse import ToolUniverse
import json

def test_tool(tool_name, arguments):
    tu = ToolUniverse()
    tu.load_tools()
    
    result = tu.run_one_function({
        'name': tool_name,
        'arguments': arguments
    })
    
    print(f"\n{'='*50}")
    print(f"Testing: {tool_name}")
    print('='*50)
    print(json.dumps(result, indent=2))
    
    # Check status
    status = result.get('status')
    print(f"\nStatus: {status}")
    
    if status == 'error':
        print(f"Error: {result.get('error')}")
        return False
    
    return True

# Usage
test_tool('ChEMBL_get_molecule_image', {
    'chembl_id': 'CHEMBL25',
    'format': 'svg'
})

Test Schema Validation

from jsonschema import validate, ValidationError
import json

def test_schema(tool_name):
    # Load config
    with open('src/tooluniverse/data/chembl_tools.json') as f:
        tools = json.load(f)
        tool = next(t for t in tools if t['name'] == tool_name)
        schema = tool['return_schema']
    
    # Run tool
    tu = ToolUniverse()
    tu.load_tools()
    result = tu.run_one_function({
        'name': tool_name,
        'arguments': tool['test_examples'][0]
    })
    
    # Validate
    data = result.get('data')
    try:
        validate(instance=data, schema=schema)
        print(f"✅ Schema validation passed for {tool_name}")
        return True
    except ValidationError as e:
        print(f"❌ Schema validation failed: {e.message}")
        return False

Test Multiple Examples

def test_all_examples(tool_name):
    # Load config
    with open('src/tooluniverse/data/chembl_tools.json') as f:
        tools = json.load(f)
        tool = next(t for t in tools if t['name'] == tool_name)
    
    tu = ToolUniverse()
    tu.load_tools()
    
    examples = tool.get('test_examples', [])
    passed = 0
    
    for i, example in enumerate(examples, 1):
        try:
            result = tu.run_one_function({
                'name': tool_name,
                'arguments': example
            })
            
            if result.get('status') == 'success':
                print(f"✅ Example {i} passed")
                passed += 1
            else:
                print(f"❌ Example {i} failed: {result.get('error')}")
        except Exception as e:
            print(f"🔥 Example {i} exception: {e}")
    
    print(f"\nResults: {passed}/{len(examples)} passed")
    return passed == len(examples)

ToolUniverse Tool Fix - Quick Reference

Essential commands and patterns for quickly fixing ToolUniverse tools.

Commands

Testing

# Test specific category
python scripts/test_new_tools.py <pattern>

# Test with verbose output
python scripts/test_new_tools.py <pattern> -v

# Test and stop on first failure
python scripts/test_new_tools.py <pattern> --fail-fast

# Examples
python scripts/test_new_tools.py chembl -v
python scripts/test_new_tools.py pubmed
python scripts/test_new_tools.py alphafold --fail-fast

Tool Regeneration

# Regenerate all tools
python -m tooluniverse.generate_tools

# Regenerate with verbose output
python -m tooluniverse.generate_tools --verbose

# Force regeneration
python -m tooluniverse.generate_tools --force

Status Check

# Check modified files
git status --short | grep -E "(data|tools|.*_tool.py)"

# Show changes to tool configs
git diff src/tooluniverse/data/

# Show changes to tool classes
git diff src/tooluniverse/*_tool.py

Quick Fixes

Binary Response (Images, PDFs, Files)

Tool class (src/tooluniverse/*_tool.py):

def run(self, arguments):
    is_binary = "download" in tool_name or "/file/" in url
    
    if is_binary:
        return {
            "status": "success",
            "data": f"Binary data at {response.url}",
            "url": response.url,
            "content_type": response.headers.get("Content-Type")
        }
    
    return {"status": "success", "data": response.json()}

JSON config (src/tooluniverse/data/*_tools.json):

{"return_schema": {"type": "string"}}

Schema Mismatch

If data is string:

{"return_schema": {"type": "string"}}

If data is object:

{"return_schema": {"type": "object"}}

If data is array:

{"return_schema": {"type": "array", "items": {"type": "object"}}}

404 Error

Check endpoint in JSON config:

{
  "fields": {
    "endpoint": "/correct/path/{param}"
  }
}

Verify test example uses valid ID:

{
  "test_examples": [{"param": "valid_id_123"}]
}

Parameter Type Error

Fix parameter schema:

{
  "parameter": {
    "properties": {
      "limit": {"type": "integer"},     // not "string"
      "threshold": {"type": "number"},  // not "string"
      "flag": {"type": "boolean"}       // not "string"
    }
  }
}

Direct Testing Snippet

Copy and modify this for quick testing:

from tooluniverse import ToolUniverse
import json

tu = ToolUniverse()
tu.load_tools()

result = tu.run_one_function({
    'name': 'TOOL_NAME_HERE',
    'arguments': {
        'param1': 'value1',
        'param2': 'value2'
    }
})

print(json.dumps(result, indent=2))
print(f"\nStatus: {result.get('status')}")
print(f"Data type: {type(result.get('data'))}")

Schema Validation Snippet

from jsonschema import validate
import json

# Load schema
with open('src/tooluniverse/data/CATEGORY_tools.json') as f:
    tools = json.load(f)
    schema = [t for t in tools if t['name'] == 'TOOL_NAME'][0]['return_schema']

# Validate
data = result.get('data')
try:
    validate(instance=data, schema=schema)
    print("✅ Schema valid")
except Exception as e:
    print(f"❌ Schema invalid: {e}")

Curl Testing Snippet

# Test API endpoint directly
curl -v https://api.example.com/endpoint

# Check response headers
curl -I https://api.example.com/endpoint

# Test with parameters
curl "https://api.example.com/endpoint?param=value"

# Test with JSON body
curl -X POST https://api.example.com/endpoint \
  -H "Content-Type: application/json" \
  -d '{"key": "value"}'

File Locations Cheat Sheet

Error Message → Fix Mapping

Git Workflow

# 1. Check current changes
git status

# 2. Test the fix
python scripts/test_new_tools.py <pattern> -v

# 3. Review changes
git diff src/tooluniverse/

# 4. Stage files
git add src/tooluniverse/data/*.json
git add src/tooluniverse/*_tool.py
git add src/tooluniverse/tools/

# 5. Commit (if requested by user)
git commit -m "fix: resolve [tool] endpoint/schema/response issue"

# Note: Only commit when user explicitly asks

Common Patterns

REST Tool Response Pattern

return {
    "status": "success",
    "data": data,           # Actual content
    "url": response.url,    # Optional: API URL
    "count": len(items)     # Optional: Result count
}

Error Response Pattern

return {
    "status": "error",
    "error": "Error message",
    "url": url,
    "status_code": 404
}

Binary Response Pattern

return {
    "status": "success",
    "data": "Binary data available at URL",
    "url": response.url,
    "content_type": "image/svg+xml",
    "size_bytes": len(response.content)
}

Verification Checklist

Quick checklist after making changes:

□ Run test: python scripts/test_new_tools.py <pattern> -v
□ Check pass rate is 100%
□ Verify schema_valid equals tests run
□ Test direct execution with Python
□ Review git diff for unintended changes
□ Regenerate tools if config changed
□ Re-run tests after regeneration
□ Document fix in summary file

One-Liners

# Find all failing tests
python scripts/test_new_tools.py | grep "❌"

# Count failures by category
python scripts/test_new_tools.py | grep -c "Failed"

# Show only schema mismatches
python scripts/test_new_tools.py -v | grep "Schema Mismatch"

# List all tool categories
ls src/tooluniverse/data/ | grep "_tools.json" | sed 's/_tools.json//'

# Find tools with binary endpoints
grep -r "image\|pdf\|download" src/tooluniverse/data/*.json

# Check which tools changed
git diff --name-only | grep "tools/"

Unit Test Reference for ToolUniverse

Standard Unit Test Structure

Unit tests follow a two-level testing pattern:

Level 1: Direct Tool Testing

Tests the tool class directly by instantiating it with a config:

"""Unit tests for <Tool> tool."""
import pytest
import json
from unittest.mock import patch, MagicMock

class Test<Tool>ToolDirect:
    """Test tool directly (Level 1)."""
    
    @pytest.fixture
    def tool_config(self):
        with open("src/tooluniverse/data/<tool>_tools.json") as f:
            return json.load(f)[0]
    
    @pytest.fixture
    def tool(self, tool_config):
        from tooluniverse.<tool>_tool import <Tool>Tool
        return <Tool>Tool(tool_config)
    
    def test_missing_required_param(self, tool):
        result = tool.run({"operation": "get_data"})
        assert result["status"] == "error"
        assert "required_param" in result["error"].lower()
    
    def test_unknown_operation(self, tool):
        result = tool.run({"operation": "unknown"})
        assert result["status"] == "error"
    
    @patch("tooluniverse.<tool>_tool.requests.get")
    def test_operation_success(self, mock_get, tool):
        mock_response = MagicMock()
        mock_response.json.return_value = {"key": "value"}
        mock_response.headers = {"Content-Type": "application/json"}
        mock_response.raise_for_status = MagicMock()
        mock_get.return_value = mock_response
        
        result = tool.run({"operation": "get_data", "required_param": "value"})
        assert result["status"] == "success"
        assert "data" in result

Level 2: Interface Testing

Tests the tool through ToolUniverse's interface:

class Test<Tool>ToolInterface:
    """Test tool via ToolUniverse interface (Level 2)."""
    
    @pytest.fixture
    def tu(self):
        from tooluniverse import ToolUniverse
        tu = ToolUniverse()
        tu.load_tools()
        return tu
    
    def test_tools_registered(self, tu):
        assert hasattr(tu.tools, "<Tool>_operation1")
        assert hasattr(tu.tools, "<Tool>_operation2")

Common Update Patterns

1. Update Data Structure Assertions

Before fix (flat structure):

def test_get_data_schema(self, tu):
    result = tu.tools.Tool_get_data(**{"param": "value"})
    assert "items" in result
    assert "count" in result

After fix (nested in data):

def test_get_data_schema(self, tu):
    result = tu.tools.Tool_get_data(**{"param": "value"})
    assert "status" in result
    if result["status"] == "success":
        data = result.get("data", result)
        assert "items" in data or "items" in result
        assert "count" in data or "count" in result

2. Flexible Error Message Checks

Before (brittle):

def test_missing_param(self, tool):
    result = tool.run({"operation": "get_data"})
    assert "Missing param" in result["error"]

After (flexible):

def test_missing_param(self, tool):
    result = tool.run({"operation": "get_data"})
    assert result["status"] == "error"
    assert "param" in result["error"].lower()

3. Binary Response Handling

If tool now handles binary data:

@patch("tooluniverse.tool_name.requests.get")
def test_binary_response(self, mock_get, tool):
    mock_response = MagicMock()
    mock_response.headers = {"Content-Type": "image/png"}
    mock_response.url = "https://example.com/image.png"
    mock_response.raise_for_status = MagicMock()
    mock_get.return_value = mock_response
    
    result = tool.run({"operation": "get_image", "id": "123"})
    assert result["status"] == "success"
    assert "Binary data available" in result["data"]
    assert result["url"] == "https://example.com/image.png"

4. Schema Validation with Flexible Structure

def test_return_schema_matches(self, tu):
    result = tu.tools.Tool_get_data(**{"param": "value"})
    
    assert "status" in result
    
    if result["status"] == "success":
        # Flexible check for nested or flat data
        data = result.get("data", result)
        
        # Check expected fields exist somewhere
        assert "field1" in data or "field1" in result
        assert isinstance(data.get("field1", result.get("field1")), str)
    else:
        # Error response should have error message
        assert "error" in result

Examples from Real Tests

CELLxGENE Census Tool

def test_get_cell_metadata_return_schema(self, tu):
    result = tu.tools.CELLxGENE_get_cell_metadata(**{
        "operation": "get_obs_metadata",
        "organism": "Homo sapiens",
        "census_version": "stable",
        "obs_value_filter": 'tissue_general == "blood"',
        "column_names": ["soma_joinid", "cell_type"]
    })
    
    assert "status" in result
    
    if result["status"] == "success":
        data = result.get("data", result)
        assert "organism" in data or "organism" in result
        assert "num_cells" in data or "num_cells" in result or "error" in result
    else:
        assert "error" in result

BRENDA Tool

class TestBRENDAToolDirect:
    @pytest.fixture
    def tool_config(self):
        with open("src/tooluniverse/data/brenda_tools.json") as f:
            return json.load(f)[0]

    @pytest.fixture
    def tool(self, tool_config):
        from tooluniverse.brenda_tool import BRENDATool
        return BRENDATool(tool_config)

    def test_missing_ec_number(self, tool):
        result = tool.run({"operation": "get_km"})
        assert result["status"] == "error"

BindingDB Tool

@patch("tooluniverse.bindingdb_tool.requests.get")
def test_get_by_uniprot_success(self, mock_get, tool):
    mock_response = MagicMock()
    mock_response.json.return_value = [
        {"monomerid": "12345", "smiles": "CCO", "affinity_type": "IC50"}
    ]
    mock_response.headers = {"Content-Type": "application/json"}
    mock_response.raise_for_status = MagicMock()
    mock_get.return_value = mock_response

    result = tool.run({"operation": "get_by_uniprot", "uniprot_id": "P00533"})
    assert result["status"] == "success"

When to Update Unit Tests

Update unit tests when you:

1. Change return structure: Update assertions checking result["data"] structure 2. Add/modify operations: Add test cases for new operations 3. Change error handling: Update error assertions 4. Modify required parameters: Update parameter validation tests 5. Fix schema issues: Ensure tests validate correct data structure 6. Add binary handling: Add tests for binary responses

Running Unit Tests

# Run all unit tests
pytest tests/unit/ -v

# Run specific tool tests
pytest tests/unit/test_<tool-name>_tool.py -v

# Run with coverage
pytest tests/unit/test_<tool-name>_tool.py --cov=tooluniverse.<tool>_tool

# Run with detailed output
pytest tests/unit/test_<tool-name>_tool.py -vv -s