Skill Tester
- 585 installs
- 23.5k repo stars
- Updated July 17, 2026
- alirezarezvani/claude-skills
skill-tester is a Claude Code skill that smoke-tests another skill's text-processing behavior using bundled sample text and CSV fixtures with expected word-count stats for developers who need repeatable skill validation
About
skill-tester is a Claude Code skill from alirezarezvani/claude-skills that validates text-processing skills against bundled fixtures. The skill runs sample plain-text files and CSV inputs through a target skill, then compares output against expected word-count statistics for word counting, character analysis, line counting, and text transformations. Developers reach for skill-tester when authoring or refactoring agent skills that parse, count, or transform text and need a deterministic regression check without writing a full test harness. The bundled fixtures include multi-line prose, punctuation, numbers, special characters, and mixed-case tokens to exercise common edge cases.
- Sample plain-text file with mixed punctuation, cases, and lorem content for processor checks
- Companion CSV rows with headers for structured-data parsing tests
- Embedded expected stats snapshot (e.g., total_words 116, unique_words 87) for regression comparison
- Documents five test dimensions: word count, character analysis, lines, transforms, statistics
Skill Tester by the numbers
- 585 all-time installs (skills.sh)
- Ranked #92 of 782 Skill Development skills by installs in the Skillselion catalog
- Security screen: LOW risk (skills.sh audit)
- Data as of Jul 31, 2026 (Skillselion catalog sync)
npx skills add https://github.com/alirezarezvani/claude-skills --skill skill-testerAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 585 |
|---|---|
| repo stars | ★ 23.5k |
| Security audit | 3 / 3 scanners passed |
| Last updated | July 17, 2026 |
| Repository | alirezarezvani/claude-skills ↗ |
How do you smoke-test a Claude skill's text output?
Smoke-test another skill’s text-processing behavior using bundled sample text and CSV fixtures plus expected word-count stats.
Who is it for?
Developers authoring Claude Code text-processing skills who need quick fixture-based regression checks before publishing.
Skip if: Teams validating REST APIs, database queries, or UI components should use framework test runners instead of skill-tester.
When should I use this skill?
A developer asks to verify, regression-test, or smoke-test a skill's text-processing or word-count behavior.
What you get
Pass or fail validation against expected word-count statistics on bundled text and CSV fixtures.
- Pass/fail smoke-test result
- Word-count comparison against expected stats
By the numbers
- Bundled fixtures test five text-processing capabilities: word counting, character analysis, line counting, text transfor
Files
Sample Text Processor
---
Name: sample-text-processor Tier: BASIC Category: Text Processing Dependencies: None (Python Standard Library Only) Author: Claude Skills Engineering Team Version: 1.0.0 Last Updated: 2026-02-16
---
Description
The Sample Text Processor is a simple skill designed to demonstrate the basic structure and functionality expected in the claude-skills ecosystem. This skill provides fundamental text processing capabilities including word counting, character analysis, and basic text transformations.
This skill serves as a reference implementation for BASIC tier requirements and can be used as a template for creating new skills. It demonstrates proper file structure, documentation standards, and implementation patterns that align with ecosystem best practices.
The skill processes text files and provides statistics and transformations in both human-readable and JSON formats, showcasing the dual output requirement for skills in the claude-skills repository.
Features
Core Functionality
- Word Count Analysis: Count total words, unique words, and word frequency
- Character Statistics: Analyze character count, line count, and special characters
- Text Transformations: Convert text to uppercase, lowercase, or title case
- File Processing: Process single text files or batch process directories
- Dual Output Formats: Generate results in both JSON and human-readable formats
Technical Features
- Command-line interface with comprehensive argument parsing
- Error handling for common file and processing issues
- Progress reporting for batch operations
- Configurable output formatting and verbosity levels
- Cross-platform compatibility with standard library only dependencies
Usage
Basic Text Analysis
python text_processor.py analyze document.txt
python text_processor.py analyze document.txt --output results.jsonText Transformation
python text_processor.py transform document.txt --mode uppercase
python text_processor.py transform document.txt --mode title --output transformed.txtBatch Processing
python text_processor.py batch text_files/ --output results/
python text_processor.py batch text_files/ --format json --output batch_results.jsonExamples
Example 1: Basic Word Count
$ python text_processor.py analyze sample.txt
=== TEXT ANALYSIS RESULTS ===
File: sample.txt
Total words: 150
Unique words: 85
Total characters: 750
Lines: 12
Most frequent word: "the" (8 occurrences)Example 2: JSON Output
$ python text_processor.py analyze sample.txt --format json
{
"file": "sample.txt",
"statistics": {
"total_words": 150,
"unique_words": 85,
"total_characters": 750,
"lines": 12,
"most_frequent": {
"word": "the",
"count": 8
}
}
}Example 3: Text Transformation
$ python text_processor.py transform sample.txt --mode title
Original: "hello world from the text processor"
Transformed: "Hello World From The Text Processor"Installation
This skill requires only Python 3.7 or later with the standard library. No external dependencies are required.
1. Clone or download the skill directory 2. Navigate to the scripts directory 3. Run the text processor directly with Python
cd scripts/
python text_processor.py --helpConfiguration
The text processor supports various configuration options through command-line arguments:
--format: Output format (json, text)--verbose: Enable verbose output and progress reporting--output: Specify output file or directory--encoding: Specify text file encoding (default: utf-8)
Architecture
The skill follows a simple modular architecture:
- TextProcessor Class: Core processing logic and statistics calculation
- OutputFormatter Class: Handles dual output format generation
- FileManager Class: Manages file I/O operations and batch processing
- CLI Interface: Command-line argument parsing and user interaction
Error Handling
The skill includes comprehensive error handling for:
- File not found or permission errors
- Invalid encoding or corrupted text files
- Memory limitations for very large files
- Output directory creation and write permissions
- Invalid command-line arguments and parameters
Performance Considerations
- Efficient memory usage for large text files through streaming
- Optimized word counting using dictionary lookups
- Batch processing with progress reporting for large datasets
- Configurable encoding detection for international text
Contributing
This skill serves as a reference implementation and contributions are welcome to demonstrate best practices:
1. Follow PEP 8 coding standards 2. Include comprehensive docstrings 3. Add test cases with sample data 4. Update documentation for any new features 5. Ensure backward compatibility
Limitations
As a BASIC tier skill, some advanced features are intentionally omitted:
- Complex text analysis (sentiment, language detection)
- Advanced file format support (PDF, Word documents)
- Database integration or external API calls
- Parallel processing for very large datasets
This skill demonstrates the essential structure and quality standards required for BASIC tier skills in the claude-skills ecosystem while remaining simple and focused on core functionality.
This is a sample text file for testing the text processor skill.
It contains multiple lines of text with various words and punctuation.
The quick brown fox jumps over the lazy dog.
This sentence contains all 26 letters of the English alphabet.
Some additional content:
- Numbers: 123, 456, 789
- Special characters: !@#$%^&*()
- Mixed case: CamelCase, snake_case, PascalCase
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Ut enim ad minim veniam, quis nostrud exercitation ullamco.
This file serves as a basic test case for:
1. Word counting functionality
2. Character analysis
3. Line counting
4. Text transformations
5. Statistical analysis
The text processor should handle this content correctly and produce
meaningful statistics and transformations for testing purposes.name,age,city,country
John Doe,25,New York,USA
Jane Smith,30,London,UK
Bob Johnson,22,Toronto,Canada
Alice Brown,28,Sydney,Australia
Charlie Wilson,35,Berlin,Germany
This CSV file contains sample data with headers and multiple rows.
It can be used to test the text processor's ability to handle
structured data formats and count words across different content types.
The file includes:
- Header row with column names
- Data rows with mixed text and numbers
- Various city and country names
- Different age values for statistical analysis{
"file": "assets/sample_text.txt",
"file_size": 855,
"total_words": 116,
"unique_words": 87,
"total_characters": 855,
"lines": 19,
"average_word_length": 4.7,
"most_frequent": {
"word": "the",
"count": 5
}
}Sample Text Processor
A basic text processing skill that demonstrates BASIC tier requirements for the claude-skills ecosystem.
Quick Start
# Analyze a text file
python scripts/text_processor.py analyze sample.txt
# Get JSON output
python scripts/text_processor.py analyze sample.txt --format json
# Transform text to uppercase
python scripts/text_processor.py transform sample.txt --mode upper
# Process multiple files
python scripts/text_processor.py batch text_files/ --verboseFeatures
- Word count and text statistics
- Text transformations (upper, lower, title, reverse)
- Batch file processing
- JSON and human-readable output formats
- Comprehensive error handling
Requirements
- Python 3.7 or later
- No external dependencies (standard library only)
Usage
See SKILL.md for comprehensive documentation and examples.
Testing
Sample data files are provided in the assets/ directory for testing the functionality.
Text Processor API Reference
Classes
TextProcessor
Main class for text processing operations.
__init__(self, encoding: str = 'utf-8')
Initialize the text processor with specified encoding.
Parameters:
encoding(str): Character encoding for file operations. Default: 'utf-8'
analyze_text(self, text: str) -> Dict[str, Any]
Analyze text and return comprehensive statistics.
Parameters:
text(str): Text content to analyze
Returns:
dict: Statistics including word count, character count, lines, most frequent word
Example:
processor = TextProcessor()
stats = processor.analyze_text("Hello world")
# Returns: {'total_words': 2, 'unique_words': 2, ...}transform_text(self, text: str, mode: str) -> str
Transform text according to specified mode.
Parameters:
text(str): Text to transformmode(str): Transformation mode ('upper', 'lower', 'title', 'reverse')
Returns:
str: Transformed text
Raises:
ValueError: If mode is not supported
OutputFormatter
Static methods for output formatting.
format_json(data: Dict[str, Any]) -> str
Format data as JSON string.
format_human_readable(data: Dict[str, Any]) -> str
Format data as human-readable text.
FileManager
Handles file operations and batch processing.
find_text_files(self, directory: str) -> List[str]
Find all text files in a directory recursively.
Supported Extensions:
- .txt
- .md
- .rst
- .csv
- .log
Command Line Interface
Commands
analyze
Analyze text file statistics.
python text_processor.py analyze <file> [options]transform
Transform text file content.
python text_processor.py transform <file> --mode <mode> [options]batch
Process multiple files in a directory.
python text_processor.py batch <directory> [options]Global Options
--format {json,text}: Output format (default: text)--output FILE: Output file path (default: stdout)--encoding ENCODING: Text file encoding (default: utf-8)--verbose: Enable verbose output
Error Handling
The text processor handles several error conditions:
- FileNotFoundError: When input file doesn't exist
- UnicodeDecodeError: When file encoding doesn't match specified encoding
- PermissionError: When file access is denied
- ValueError: When invalid transformation mode is specified
All errors are reported to stderr with descriptive messages.
#!/usr/bin/env python3
"""
Sample Text Processor - Basic text analysis and transformation tool
This script demonstrates the basic structure and functionality expected in
BASIC tier skills. It provides text processing capabilities with proper
argument parsing, error handling, and dual output formats.
Usage:
python text_processor.py analyze <file> [options]
python text_processor.py transform <file> --mode <mode> [options]
python text_processor.py batch <directory> [options]
Author: Claude Skills Engineering Team
Version: 1.0.0
Dependencies: Python Standard Library Only
"""
import argparse
import json
import os
import sys
from collections import Counter
from pathlib import Path
from typing import Dict, List, Any, Optional
class TextProcessor:
"""Core text processing functionality"""
def __init__(self, encoding: str = 'utf-8'):
self.encoding = encoding
def analyze_text(self, text: str) -> Dict[str, Any]:
"""Analyze text and return statistics"""
lines = text.split('\n')
words = text.lower().split()
# Calculate basic statistics
stats = {
'total_words': len(words),
'unique_words': len(set(words)),
'total_characters': len(text),
'lines': len(lines),
'average_word_length': sum(len(word) for word in words) / len(words) if words else 0
}
# Find most frequent word
if words:
word_counts = Counter(words)
most_common = word_counts.most_common(1)[0]
stats['most_frequent'] = {
'word': most_common[0],
'count': most_common[1]
}
else:
stats['most_frequent'] = {'word': '', 'count': 0}
return stats
def transform_text(self, text: str, mode: str) -> str:
"""Transform text according to specified mode"""
if mode == 'upper':
return text.upper()
elif mode == 'lower':
return text.lower()
elif mode == 'title':
return text.title()
elif mode == 'reverse':
return text[::-1]
else:
raise ValueError(f"Unknown transformation mode: {mode}")
def process_file(self, file_path: str) -> Dict[str, Any]:
"""Process a single text file"""
try:
with open(file_path, 'r', encoding=self.encoding) as file:
content = file.read()
stats = self.analyze_text(content)
stats['file'] = file_path
stats['file_size'] = os.path.getsize(file_path)
return stats
except FileNotFoundError:
raise FileNotFoundError(f"File not found: {file_path}")
except UnicodeDecodeError:
raise UnicodeDecodeError(f"Cannot decode file with {self.encoding} encoding: {file_path}")
except PermissionError:
raise PermissionError(f"Permission denied accessing file: {file_path}")
class OutputFormatter:
"""Handles dual output format generation"""
@staticmethod
def format_json(data: Dict[str, Any]) -> str:
"""Format data as JSON"""
return json.dumps(data, indent=2, ensure_ascii=False)
@staticmethod
def format_human_readable(data: Dict[str, Any]) -> str:
"""Format data as human-readable text"""
lines = []
lines.append("=== TEXT ANALYSIS RESULTS ===")
lines.append(f"File: {data.get('file', 'Unknown')}")
lines.append(f"File size: {data.get('file_size', 0)} bytes")
lines.append(f"Total words: {data.get('total_words', 0)}")
lines.append(f"Unique words: {data.get('unique_words', 0)}")
lines.append(f"Total characters: {data.get('total_characters', 0)}")
lines.append(f"Lines: {data.get('lines', 0)}")
lines.append(f"Average word length: {data.get('average_word_length', 0):.1f}")
most_frequent = data.get('most_frequent', {})
lines.append(f"Most frequent word: \"{most_frequent.get('word', '')}\" ({most_frequent.get('count', 0)} occurrences)")
return "\n".join(lines)
class FileManager:
"""Manages file I/O operations and batch processing"""
def __init__(self, verbose: bool = False):
self.verbose = verbose
def log_verbose(self, message: str):
"""Log verbose message if verbose mode enabled"""
if self.verbose:
print(f"[INFO] {message}", file=sys.stderr)
def find_text_files(self, directory: str) -> List[str]:
"""Find all text files in directory"""
text_extensions = {'.txt', '.md', '.rst', '.csv', '.log'}
text_files = []
try:
for file_path in Path(directory).rglob('*'):
if file_path.is_file() and file_path.suffix.lower() in text_extensions:
text_files.append(str(file_path))
except PermissionError:
raise PermissionError(f"Permission denied accessing directory: {directory}")
return text_files
def write_output(self, content: str, output_path: Optional[str] = None):
"""Write content to file or stdout"""
if output_path:
try:
# Create directory if needed
output_dir = os.path.dirname(output_path)
if output_dir and not os.path.exists(output_dir):
os.makedirs(output_dir)
with open(output_path, 'w', encoding='utf-8') as file:
file.write(content)
self.log_verbose(f"Output written to: {output_path}")
except PermissionError:
raise PermissionError(f"Permission denied writing to: {output_path}")
else:
print(content)
def analyze_command(args: argparse.Namespace) -> int:
"""Handle analyze command"""
try:
processor = TextProcessor(args.encoding)
file_manager = FileManager(args.verbose)
file_manager.log_verbose(f"Analyzing file: {args.file}")
# Process the file
results = processor.process_file(args.file)
# Format output
if args.format == 'json':
output = OutputFormatter.format_json(results)
else:
output = OutputFormatter.format_human_readable(results)
# Write output
file_manager.write_output(output, args.output)
return 0
except FileNotFoundError as e:
print(f"Error: {e}", file=sys.stderr)
return 1
except UnicodeDecodeError as e:
print(f"Error: {e}", file=sys.stderr)
print(f"Try using --encoding option with different encoding", file=sys.stderr)
return 1
except Exception as e:
print(f"Error: {e}", file=sys.stderr)
return 1
def transform_command(args: argparse.Namespace) -> int:
"""Handle transform command"""
try:
processor = TextProcessor(args.encoding)
file_manager = FileManager(args.verbose)
file_manager.log_verbose(f"Transforming file: {args.file}")
# Read and transform the file
with open(args.file, 'r', encoding=args.encoding) as file:
content = file.read()
transformed = processor.transform_text(content, args.mode)
# Write transformed content
file_manager.write_output(transformed, args.output)
return 0
except FileNotFoundError as e:
print(f"Error: {e}", file=sys.stderr)
return 1
except ValueError as e:
print(f"Error: {e}", file=sys.stderr)
return 1
except Exception as e:
print(f"Error: {e}", file=sys.stderr)
return 1
def batch_command(args: argparse.Namespace) -> int:
"""Handle batch command"""
try:
processor = TextProcessor(args.encoding)
file_manager = FileManager(args.verbose)
file_manager.log_verbose(f"Finding text files in: {args.directory}")
# Find all text files
text_files = file_manager.find_text_files(args.directory)
if not text_files:
print(f"No text files found in directory: {args.directory}", file=sys.stderr)
return 1
file_manager.log_verbose(f"Found {len(text_files)} text files")
# Process all files
all_results = []
for i, file_path in enumerate(text_files, 1):
try:
file_manager.log_verbose(f"Processing {i}/{len(text_files)}: {file_path}")
results = processor.process_file(file_path)
all_results.append(results)
except Exception as e:
print(f"Warning: Failed to process {file_path}: {e}", file=sys.stderr)
continue
if not all_results:
print("Error: No files could be processed successfully", file=sys.stderr)
return 1
# Format batch results
batch_summary = {
'total_files': len(all_results),
'total_words': sum(r.get('total_words', 0) for r in all_results),
'total_characters': sum(r.get('total_characters', 0) for r in all_results),
'files': all_results
}
if args.format == 'json':
output = OutputFormatter.format_json(batch_summary)
else:
lines = []
lines.append("=== BATCH PROCESSING RESULTS ===")
lines.append(f"Total files processed: {batch_summary['total_files']}")
lines.append(f"Total words across all files: {batch_summary['total_words']}")
lines.append(f"Total characters across all files: {batch_summary['total_characters']}")
lines.append("")
lines.append("Individual file results:")
for result in all_results:
lines.append(f" {result['file']}: {result['total_words']} words")
output = "\n".join(lines)
# Write output
file_manager.write_output(output, args.output)
return 0
except PermissionError as e:
print(f"Error: {e}", file=sys.stderr)
return 1
except Exception as e:
print(f"Error: {e}", file=sys.stderr)
return 1
def main():
"""Main entry point with argument parsing"""
parser = argparse.ArgumentParser(
description="Sample Text Processor - Basic text analysis and transformation",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="""
Examples:
Analysis:
python text_processor.py analyze document.txt
python text_processor.py analyze document.txt --format json --output results.json
Transformation:
python text_processor.py transform document.txt --mode upper
python text_processor.py transform document.txt --mode title --output transformed.txt
Batch processing:
python text_processor.py batch text_files/ --verbose
python text_processor.py batch text_files/ --format json --output batch_results.json
Transformation modes:
upper - Convert to uppercase
lower - Convert to lowercase
title - Convert to title case
reverse - Reverse the text
"""
)
parser.add_argument('--format',
choices=['json', 'text'],
default='text',
help='Output format (default: text)')
parser.add_argument('--output',
help='Output file path (default: stdout)')
parser.add_argument('--encoding',
default='utf-8',
help='Text file encoding (default: utf-8)')
parser.add_argument('--verbose',
action='store_true',
help='Enable verbose output')
subparsers = parser.add_subparsers(dest='command', help='Available commands')
# Analyze subcommand
analyze_parser = subparsers.add_parser('analyze', help='Analyze text file statistics')
analyze_parser.add_argument('file', help='Text file to analyze')
# Transform subcommand
transform_parser = subparsers.add_parser('transform', help='Transform text file')
transform_parser.add_argument('file', help='Text file to transform')
transform_parser.add_argument('--mode',
required=True,
choices=['upper', 'lower', 'title', 'reverse'],
help='Transformation mode')
# Batch subcommand
batch_parser = subparsers.add_parser('batch', help='Process multiple files')
batch_parser.add_argument('directory', help='Directory containing text files')
args = parser.parse_args()
if not args.command:
parser.print_help()
return 1
try:
if args.command == 'analyze':
return analyze_command(args)
elif args.command == 'transform':
return transform_command(args)
elif args.command == 'batch':
return batch_command(args)
else:
print(f"Unknown command: {args.command}", file=sys.stderr)
return 1
except KeyboardInterrupt:
print("\nOperation interrupted by user", file=sys.stderr)
return 130
except Exception as e:
print(f"Unexpected error: {e}", file=sys.stderr)
return 1
if __name__ == "__main__":
sys.exit(main()){
"skill_path": "assets/sample-skill",
"timestamp": "2026-02-16T16:41:00Z",
"overall_score": 85.0,
"compliance_level": "GOOD",
"checks": {
"skill_md_exists": {
"passed": true,
"message": "SKILL.md found",
"score": 1.0
},
"readme_exists": {
"passed": true,
"message": "README.md found",
"score": 1.0
},
"skill_md_length": {
"passed": true,
"message": "SKILL.md has 145 lines (≥100)",
"score": 1.0
},
"frontmatter_complete": {
"passed": true,
"message": "All required frontmatter fields present",
"score": 1.0
},
"required_sections": {
"passed": true,
"message": "All required sections present",
"score": 1.0
},
"dir_scripts_exists": {
"passed": true,
"message": "scripts/ directory found",
"score": 1.0
},
"min_scripts_count": {
"passed": true,
"message": "Found 1 Python scripts (≥1)",
"score": 1.0
},
"script_syntax_text_processor.py": {
"passed": true,
"message": "text_processor.py has valid Python syntax",
"score": 1.0
},
"script_argparse_text_processor.py": {
"passed": true,
"message": "Uses argparse in text_processor.py",
"score": 1.0
},
"script_main_guard_text_processor.py": {
"passed": true,
"message": "Has main guard in text_processor.py",
"score": 1.0
},
"tier_compliance": {
"passed": true,
"message": "Meets BASIC tier requirements",
"score": 1.0
}
},
"warnings": [],
"errors": [],
"suggestions": [
"Consider adding optional directories: references, expected_outputs"
]
}Skill Tester - Quality Assurance Meta-Skill
A POWERFUL-tier skill that provides comprehensive validation, testing, and quality scoring for skills in the claude-skills ecosystem.
Overview
The Skill Tester is a meta-skill that ensures quality and consistency across all skills in the repository through:
- Structure Validation - Verifies directory structure, file presence, and documentation standards
- Script Testing - Tests Python scripts for syntax, functionality, and compliance
- Quality Scoring - Provides comprehensive quality assessment across multiple dimensions
Quick Start
Validate a Skill
# Basic validation
python scripts/skill_validator.py engineering/my-skill
# Validate against specific tier
python scripts/skill_validator.py engineering/my-skill --tier POWERFUL --jsonTest Scripts
# Test all scripts in a skill
python scripts/script_tester.py engineering/my-skill
# Test with custom timeout
python scripts/script_tester.py engineering/my-skill --timeout 60 --jsonScore Quality
# Get quality assessment
python scripts/quality_scorer.py engineering/my-skill
# Detailed scoring with improvement suggestions
python scripts/quality_scorer.py engineering/my-skill --detailed --jsonComponents
Scripts
- skill_validator.py (700+ LOC) - Validates skill structure and compliance
- script_tester.py (800+ LOC) - Tests script functionality and quality
- quality_scorer.py (1100+ LOC) - Multi-dimensional quality assessment
Reference Documentation
- skill-structure-specification.md - Complete structural requirements
- tier-requirements-matrix.md - Tier-specific quality standards
- quality-scoring-rubric.md - Detailed scoring methodology
Sample Assets
- sample-skill/ - Complete sample skill for testing the tester itself
Features
Validation Capabilities
- SKILL.md format and content validation
- Directory structure compliance checking
- Python script syntax and import validation
- Argparse implementation verification
- Tier-specific requirement enforcement
Testing Framework
- Syntax validation using AST parsing
- Import analysis for external dependencies
- Runtime execution testing with timeout protection
- Help functionality verification
- Sample data processing validation
- Output format compliance checking
Quality Assessment
- Documentation quality scoring (25%)
- Code quality evaluation (25%)
- Completeness assessment (25%)
- Usability analysis (25%)
- Letter grade assignment (A+ to F)
- Tier recommendation generation
- Improvement roadmap creation
CI/CD Integration
GitHub Actions Example
name: Skill Quality Gate
on:
pull_request:
paths: ['engineering/**']
jobs:
validate-skills:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: '3.11'
- name: Validate Skills
run: |
for skill in $(git diff --name-only ${{ github.event.before }} | grep -E '^engineering/[^/]+/' | cut -d'/' -f1-2 | sort -u); do
python engineering/skill-tester/scripts/skill_validator.py $skill --json
python engineering/skill-tester/scripts/script_tester.py $skill
python engineering/skill-tester/scripts/quality_scorer.py $skill --minimum-score 75
donePre-commit Hook
#!/bin/bash
# .git/hooks/pre-commit
python engineering/skill-tester/scripts/skill_validator.py engineering/my-skill --tier STANDARD
if [ $? -ne 0 ]; then
echo "Skill validation failed. Commit blocked."
exit 1
fiQuality Standards
All Scripts
- Zero External Dependencies - Python standard library only
- Comprehensive Error Handling - Meaningful error messages and recovery
- Dual Output Support - Both JSON and human-readable formats
- Proper Documentation - Comprehensive docstrings and comments
- CLI Best Practices - Full argparse implementation with help text
Validation Accuracy
- Structure Checks - 100% accurate directory and file validation
- Content Analysis - Deep parsing of SKILL.md and documentation
- Code Analysis - AST-based Python code validation
- Compliance Scoring - Objective, repeatable quality assessment
Self-Testing
The skill-tester can validate itself:
# Validate the skill-tester structure
python scripts/skill_validator.py . --tier POWERFUL
# Test the skill-tester scripts
python scripts/script_tester.py .
# Score the skill-tester quality
python scripts/quality_scorer.py . --detailedAdvanced Usage
Batch Validation
# Validate all skills in repository
find engineering/ -maxdepth 1 -type d | while read skill; do
echo "Validating $skill..."
python engineering/skill-tester/scripts/skill_validator.py "$skill"
doneQuality Monitoring
# Generate quality report for all skills
python engineering/skill-tester/scripts/quality_scorer.py engineering/ \
--batch --json > quality_report.jsonCustom Scoring Thresholds
# Enforce minimum quality scores
python scripts/quality_scorer.py engineering/my-skill --minimum-score 80
# Exit code 0 = passed, 1 = failed, 2 = needs improvementError Handling
All scripts provide comprehensive error handling:
- File System Errors - Missing files, permission issues, invalid paths
- Content Errors - Malformed YAML, invalid JSON, encoding issues
- Execution Errors - Script timeouts, runtime failures, import errors
- Validation Errors - Standards violations, compliance failures
Output Formats
Human-Readable
=== SKILL VALIDATION REPORT ===
Skill: engineering/my-skill
Overall Score: 85.2/100 (B+)
Tier Recommendation: STANDARD
STRUCTURE VALIDATION:
✓ PASS: SKILL.md found
✓ PASS: README.md found
✓ PASS: scripts/ directory found
SUGGESTIONS:
• Add references/ directory
• Improve error handling in main.pyJSON Format
{
"skill_path": "engineering/my-skill",
"overall_score": 85.2,
"letter_grade": "B+",
"tier_recommendation": "STANDARD",
"dimensions": {
"Documentation": {"score": 88.5, "weight": 0.25},
"Code Quality": {"score": 82.0, "weight": 0.25},
"Completeness": {"score": 85.5, "weight": 0.25},
"Usability": {"score": 84.8, "weight": 0.25}
}
}Requirements
- Python 3.7+ - No external dependencies required
- File System Access - Read access to skill directories
- Execution Permissions - Ability to run Python scripts for testing
Contributing
See SKILL.md for comprehensive documentation and contribution guidelines.
The skill-tester itself serves as a reference implementation of POWERFUL-tier quality standards.
Quality Scoring Rubric
Version: 2.0.0 Last Updated: 2026-03-27 Authority: Claude Skills Engineering Team
Overview
This document defines the comprehensive quality scoring methodology used to assess skills within the claude-skills ecosystem. The scoring system evaluates four key dimensions by default (each weighted at 25%), with an optional fifth Security dimension (enabled via --include-security flag).
Dimension Configuration
Default Mode (backward compatible):
- Documentation Quality: 25%
- Code Quality: 25%
- Completeness: 25%
- Usability: 25%
With `--include-security` flag:
- Documentation Quality: 20%
- Code Quality: 20%
- Completeness: 20%
- Security: 20%
- Usability: 20%
Scoring Framework
Overall Scoring Scale
- A+ (95-100): Exceptional quality, exceeds all standards
- A (90-94): Excellent quality, meets highest standards consistently
- A- (85-89): Very good quality, minor areas for improvement
- B+ (80-84): Good quality, meets most standards well
- B (75-79): Satisfactory quality, meets standards adequately
- B- (70-74): Below average, several areas need improvement
- C+ (65-69): Poor quality, significant improvements needed
- C (60-64): Minimal acceptable quality, major improvements required
- C- (55-59): Unacceptable quality, extensive rework needed
- D (50-54): Very poor quality, fundamental issues present
- F (0-49): Failing quality, does not meet basic standards
Dimension Weights (Default: 4 dimensions × 25%)
Each dimension contributes equally to the overall score:
- Documentation Quality: 25%
- Code Quality: 25%
- Completeness: 25%
- Usability: 25%
When --include-security is used, all five dimensions are weighted at 20% each.
Documentation Quality (20% Weight)
Scoring Components
SKILL.md Quality (40% of Documentation Score)
Component Breakdown:
- Length and Depth (25%): Line count and content substance
- Frontmatter Quality (25%): Completeness and accuracy of YAML metadata
- Section Coverage (25%): Required and recommended section presence
- Content Depth (25%): Technical detail and comprehensiveness
Scoring Criteria:
| Score Range | Length | Frontmatter | Sections | Depth |
|---|---|---|---|---|
| 90-100 | 400+ lines | All fields complete + extras | All required + 4+ recommended | Rich technical detail, examples |
| 80-89 | 300-399 lines | All required fields complete | All required + 2-3 recommended | Good technical coverage |
| 70-79 | 200-299 lines | Most required fields | All required + 1 recommended | Adequate technical content |
| 60-69 | 150-199 lines | Some required fields | Most required sections | Basic technical information |
| 50-59 | 100-149 lines | Minimal frontmatter | Some required sections | Limited technical detail |
| Below 50 | <100 lines | Missing/invalid frontmatter | Few/no required sections | Insufficient content |
README.md Quality (25% of Documentation Score)
Scoring Criteria:
- Excellent (90-100): 1000+ chars, comprehensive usage guide, examples, troubleshooting
- Good (75-89): 500-999 chars, clear usage instructions, basic examples
- Satisfactory (60-74): 200-499 chars, minimal usage information
- Poor (40-59): <200 chars or confusing content
- Failing (0-39): Missing or completely inadequate
Reference Documentation (20% of Documentation Score)
Scoring Criteria:
- Excellent (90-100): Multiple comprehensive reference docs (2000+ chars total)
- Good (75-89): 2-3 reference files with substantial content
- Satisfactory (60-74): 1-2 reference files with adequate content
- Poor (40-59): Minimal reference content or poor quality
- Failing (0-39): No reference documentation
Examples and Usage Clarity (15% of Documentation Score)
Scoring Criteria:
- Excellent (90-100): 5+ diverse examples, clear usage patterns
- Good (75-89): 3-4 examples covering different scenarios
- Satisfactory (60-74): 2-3 basic examples
- Poor (40-59): 1-2 minimal examples
- Failing (0-39): No examples or unclear usage
Code Quality (20% Weight)
Scoring Components
Script Complexity and Architecture (25% of Code Score)
Evaluation Criteria:
- Lines of code per script relative to tier requirements
- Function and class organization
- Code modularity and reusability
- Algorithm sophistication
Scoring Matrix:
| Tier | Excellent (90-100) | Good (75-89) | Satisfactory (60-74) | Poor (Below 60) |
|---|---|---|---|---|
| BASIC | 200-300 LOC, well-structured | 150-199 LOC, organized | 100-149 LOC, basic | <100 LOC, minimal |
| STANDARD | 400-500 LOC, modular | 350-399 LOC, structured | 300-349 LOC, adequate | <300 LOC, basic |
| POWERFUL | 600-800 LOC, sophisticated | 550-599 LOC, advanced | 500-549 LOC, solid | <500 LOC, simple |
Error Handling Quality (25% of Code Score)
Scoring Criteria:
- Excellent (90-100): Comprehensive exception handling, specific error types, recovery mechanisms
- Good (75-89): Good exception handling, meaningful error messages, logging
- Satisfactory (60-74): Basic try/except blocks, simple error messages
- Poor (40-59): Minimal error handling, generic exceptions
- Failing (0-39): No error handling or inappropriate handling
Error Handling Checklist:
- [ ] Try/except blocks for risky operations
- [ ] Specific exception types (not just Exception)
- [ ] Meaningful error messages for users
- [ ] Proper error logging or reporting
- [ ] Graceful degradation where possible
- [ ] Input validation and sanitization
Code Structure and Organization (25% of Code Score)
Evaluation Elements:
- Function decomposition and single responsibility
- Class design and inheritance patterns
- Import organization and dependency management
- Documentation and comments quality
- Consistent naming conventions
- PEP 8 compliance
Scoring Guidelines:
- Excellent (90-100): Exemplary structure, comprehensive docstrings, perfect style
- Good (75-89): Well-organized, good documentation, minor style issues
- Satisfactory (60-74): Adequate structure, basic documentation, some style issues
- Poor (40-59): Poor organization, minimal documentation, style problems
- Failing (0-39): No clear structure, no documentation, major style violations
Output Format Support (25% of Code Score)
Required Capabilities:
- JSON output format support
- Human-readable output format
- Proper data serialization
- Consistent output structure
- Error output handling
Scoring Criteria:
- Excellent (90-100): Dual format + custom formats, perfect serialization
- Good (75-89): Dual format support, good serialization
- Satisfactory (60-74): Single format well-implemented
- Poor (40-59): Basic output, formatting issues
- Failing (0-39): Poor or no structured output
Completeness (20% Weight)
Scoring Components
Directory Structure Compliance (25% of Completeness Score)
Required Directories by Tier:
- BASIC: scripts/ (required), assets/ + references/ (recommended)
- STANDARD: scripts/ + assets/ + references/ (required), expected_outputs/ (recommended)
- POWERFUL: scripts/ + assets/ + references/ + expected_outputs/ (all required)
Scoring Calculation:
Structure Score = (Required Present / Required Total) * 0.6 +
(Recommended Present / Recommended Total) * 0.4Asset Availability and Quality (25% of Completeness Score)
Scoring Criteria:
- Excellent (90-100): 5+ diverse assets, multiple file types, realistic data
- Good (75-89): 3-4 assets, some diversity, good quality
- Satisfactory (60-74): 2-3 assets, basic variety
- Poor (40-59): 1-2 minimal assets
- Failing (0-39): No assets or unusable assets
Asset Quality Factors:
- File diversity (JSON, CSV, YAML, etc.)
- Data realism and complexity
- Coverage of use cases
- File size appropriateness
- Documentation of asset purpose
Expected Output Coverage (25% of Completeness Score)
Evaluation Criteria:
- Correspondence with asset files
- Coverage of success and error scenarios
- Output format variety
- Reproducibility and accuracy
Scoring Matrix:
- Excellent (90-100): Complete output coverage, all scenarios, verified accuracy
- Good (75-89): Good coverage, most scenarios, mostly accurate
- Satisfactory (60-74): Basic coverage, main scenarios
- Poor (40-59): Minimal coverage, some inaccuracies
- Failing (0-39): No expected outputs or completely inaccurate
Test Coverage and Validation (25% of Completeness Score)
Assessment Areas:
- Sample data processing capability
- Output verification mechanisms
- Edge case handling
- Error condition testing
- Integration test scenarios
Scoring Guidelines:
- Excellent (90-100): Comprehensive test coverage, automated validation
- Good (75-89): Good test coverage, manual validation possible
- Satisfactory (60-74): Basic testing capability
- Poor (40-59): Minimal testing support
- Failing (0-39): No testing or validation capability
Usability (20% Weight)
Scoring Components
Installation and Setup Simplicity (25% of Usability Score)
Evaluation Factors:
- Dependency requirements (Python stdlib preferred)
- Setup complexity
- Environment requirements
- Installation documentation clarity
Scoring Criteria:
- Excellent (90-100): Zero external dependencies, single-file execution
- Good (75-89): Minimal dependencies, simple setup
- Satisfactory (60-74): Some dependencies, documented setup
- Poor (40-59): Complex dependencies, unclear setup
- Failing (0-39): Unable to install or excessive complexity
Usage Clarity and Help Quality (25% of Usability Score)
Assessment Elements:
- Command-line help comprehensiveness
- Usage example clarity
- Parameter documentation quality
- Error message helpfulness
Help Quality Checklist:
- [ ] Comprehensive --help output
- [ ] Clear parameter descriptions
- [ ] Usage examples included
- [ ] Error messages are actionable
- [ ] Progress indicators where appropriate
Scoring Matrix:
- Excellent (90-100): Exemplary help, multiple examples, perfect error messages
- Good (75-89): Good help quality, clear examples, helpful errors
- Satisfactory (60-74): Adequate help, basic examples
- Poor (40-59): Minimal help, confusing interface
- Failing (0-39): No help or completely unclear interface
Documentation Accessibility (25% of Usability Score)
Evaluation Criteria:
- README quick start effectiveness
- SKILL.md navigation and structure
- Reference material organization
- Learning curve considerations
Accessibility Factors:
- Information hierarchy clarity
- Cross-reference quality
- Beginner-friendly explanations
- Advanced user shortcuts
- Troubleshooting guidance
Practical Example Quality (25% of Usability Score)
Assessment Areas:
- Example realism and relevance
- Complexity progression (simple to advanced)
- Output demonstration
- Common use case coverage
- Integration scenarios
Scoring Guidelines:
- Excellent (90-100): 5+ examples, perfect progression, real-world scenarios
- Good (75-89): 3-4 examples, good variety, practical scenarios
- Satisfactory (60-74): 2-3 examples, adequate coverage
- Poor (40-59): 1-2 examples, limited practical value
- Failing (0-39): No examples or completely impractical
Scoring Calculations
Dimension Score Calculation
Each dimension score is calculated as a weighted average of its components:
def calculate_dimension_score(components):
total_weighted_score = 0
total_weight = 0
for component_name, component_data in components.items():
score = component_data['score']
weight = component_data['weight']
total_weighted_score += score * weight
total_weight += weight
return total_weighted_score / total_weight if total_weight > 0 else 0Overall Score Calculation
The overall score combines all dimensions with equal weighting:
def calculate_overall_score(dimensions):
return sum(dimension.score * 0.25 for dimension in dimensions.values())Letter Grade Assignment
def assign_letter_grade(overall_score):
if overall_score >= 95: return "A+"
elif overall_score >= 90: return "A"
elif overall_score >= 85: return "A-"
elif overall_score >= 80: return "B+"
elif overall_score >= 75: return "B"
elif overall_score >= 70: return "B-"
elif overall_score >= 65: return "C+"
elif overall_score >= 60: return "C"
elif overall_score >= 55: return "C-"
elif overall_score >= 50: return "D"
else: return "F"Quality Improvement Recommendations
Score-Based Recommendations
For Scores Below 60 (C- or Lower)
Priority Actions: 1. Address fundamental structural issues 2. Implement basic error handling 3. Add essential documentation sections 4. Create minimal viable examples 5. Fix critical functionality issues
For Scores 60-74 (C+ to B-)
Improvement Areas: 1. Expand documentation comprehensiveness 2. Enhance error handling sophistication 3. Add more diverse examples and use cases 4. Improve code organization and structure 5. Increase test coverage and validation
For Scores 75-84 (B to B+)
Enhancement Opportunities: 1. Refine documentation for expert-level quality 2. Implement advanced error recovery mechanisms 3. Add comprehensive reference materials 4. Optimize code architecture and performance 5. Develop extensive example library
For Scores 85+ (A- or Higher)
Excellence Maintenance: 1. Regular quality audits and updates 2. Community feedback integration 3. Best practice evolution tracking 4. Mentoring lower-quality skills 5. Innovation and cutting-edge feature adoption
Dimension-Specific Improvement Strategies
Low Documentation Scores
- Expand SKILL.md with technical details
- Add comprehensive API reference
- Include architecture diagrams and explanations
- Develop troubleshooting guides
- Create contributor documentation
Low Code Quality Scores
- Refactor for better modularity
- Implement comprehensive error handling
- Add extensive code documentation
- Apply advanced design patterns
- Optimize performance and efficiency
Low Completeness Scores
- Add missing directories and files
- Develop comprehensive sample datasets
- Create expected output libraries
- Implement automated testing
- Add integration examples
Low Usability Scores
- Simplify installation process
- Improve command-line interface design
- Enhance help text and documentation
- Create beginner-friendly tutorials
- Add interactive examples
Security (Optional, 20% Weight when enabled)
Overview
The Security dimension evaluates Python scripts for security vulnerabilities and best practices. This dimension is optional and only evaluated when the --include-security flag is passed to the quality scorer.
Important: By default, the quality scorer uses 4 dimensions × 25% weights for backward compatibility. To include Security assessment, use:
python quality_scorer.py <skill_path> --include-securityWhen Security is enabled, all dimensions are rebalanced to 20% each (5 dimensions × 20% = 100%).
This dimension is critical for ensuring that skills do not introduce security risks into the claude-skills ecosystem.
Scoring Components
Sensitive Data Exposure Prevention (25% of Security Score)
Component Breakdown:
- Hardcoded Credentials Detection: Passwords, API keys, tokens, secrets
- AWS Credential Detection: Access keys and secret keys
- Private Key Detection: RSA, SSH, and other private keys
- JWT Token Detection: JSON Web Tokens in code
Scoring Criteria:
| Score Range | Criteria |
|---|---|
| 90-100 | No hardcoded credentials, uses environment variables properly |
| 75-89 | Minor issues (e.g., placeholder values that aren't real secrets) |
| 60-74 | One or two low-severity issues |
| 40-59 | Multiple medium-severity issues |
| Below 40 | Critical hardcoded secrets detected |
Safe File Operations (25% of Security Score)
Component Breakdown:
- Path Traversal Detection:
../, URL-encoded variants, Unicode variants - String Concatenation Risks:
open(path + user_input) - Null Byte Injection:
%00,\x00 - Safe Pattern Usage:
pathlib.Path,os.path.basename
Scoring Criteria:
| Score Range | Criteria |
|---|---|
| 90-100 | Uses pathlib/os.path safely, no path traversal vulnerabilities |
| 75-89 | Minor issues, uses safe patterns mostly |
| 60-74 | Some path concatenation with user input |
| 40-59 | Path traversal patterns detected |
| Below 40 | Critical vulnerabilities present |
Command Injection Prevention (25% of Security Score)
Component Breakdown:
- Dangerous Functions:
os.system(),eval(),exec(),subprocesswithshell=True - Safe Alternatives:
subprocess.run(args, shell=False),shlex.quote(),shlex.split()
Scoring Criteria:
| Score Range | Criteria |
|---|---|
| 90-100 | No command injection risks, uses subprocess safely |
| 75-89 | Minor issues, mostly safe patterns |
| 60-74 | Some use of shell=True or eval with safe context |
| 40-59 | Command injection patterns detected |
| Below 40 | Critical vulnerabilities (unfiltered user input to shell) |
Input Validation Quality (25% of Security Score)
Component Breakdown:
- Argparse Usage: CLI argument validation
- Type Checking:
isinstance(), type hints - Error Handling:
try/exceptblocks - Input Sanitization: Regex validation, input cleaning
Scoring Criteria:
| Score Range | Criteria |
|---|---|
| 90-100 | Comprehensive input validation, proper error handling |
| 75-89 | Good validation coverage, most inputs checked |
| 60-74 | Basic validation present |
| 40-59 | Minimal input validation |
| Below 40 | No input validation |
Security Best Practices
Recommended Patterns:
# Use environment variables for secrets
import os
password = os.environ.get("PASSWORD")
# Use pathlib for safe path operations
from pathlib import Path
safe_path = Path(base_dir) / user_input
# Use subprocess safely
import subprocess
result = subprocess.run(["ls", user_input], capture_output=True)
# Use shlex for shell argument safety
import shlex
safe_arg = shlex.quote(user_input)Patterns to Avoid:
# Don't hardcode secrets
password = "my_secret_password" # BAD
# Don't use string concatenation for paths
open(base_path + "/" + user_input) # BAD
# Don't use shell=True with user input
os.system(f"ls {user_input}") # BAD
# Don't use eval on user input
eval(user_input) # VERY BADSecurity Score Impact on Tiers
Note: Security requirements only apply when --include-security is used.
When Security dimension is enabled:
- POWERFUL Tier: Requires Security score ≥ 70
- STANDARD Tier: Requires Security score ≥ 50
- BASIC Tier: No minimum Security requirement
When Security dimension is not enabled (default):
- Tier recommendations are based on the 4 core dimensions (Documentation, Code Quality, Completeness, Usability)
Low Security Scores
- Remove hardcoded credentials, use environment variables
- Fix path traversal vulnerabilities
- Replace dangerous functions with safe alternatives
- Add input validation and error handling
Quality Assurance Process
Automated Scoring
The quality scorer runs automated assessments based on this rubric: 1. File system analysis for structure compliance 2. Content analysis for documentation quality 3. Code analysis for quality metrics 4. Asset inventory and quality assessment
Manual Review Process
Human reviewers validate automated scores and provide qualitative insights: 1. Content quality assessment beyond automated metrics 2. Usability testing with real-world scenarios 3. Technical accuracy verification 4. Community value assessment
Continuous Improvement
The scoring rubric evolves based on:
- Community feedback and usage patterns
- Industry best practice changes
- Tool capability enhancements
- Quality trend analysis
This quality scoring rubric ensures consistent, objective, and comprehensive assessment of all skills within the claude-skills ecosystem while providing clear guidance for quality improvement.
Skill Structure Specification
Version: 1.0.0 Last Updated: 2026-02-16 Authority: Claude Skills Engineering Team
Overview
This document defines the mandatory and optional components that constitute a well-formed skill within the claude-skills ecosystem. All skills must adhere to these structural requirements to ensure consistency, maintainability, and quality across the repository.
Directory Structure
Mandatory Components
skill-name/
├── SKILL.md # Primary skill documentation (REQUIRED)
├── README.md # Usage instructions and quick start (REQUIRED)
└── scripts/ # Python implementation scripts (REQUIRED)
└── *.py # At least one Python scriptRecommended Components
skill-name/
├── SKILL.md
├── README.md
├── scripts/
│ └── *.py
├── assets/ # Sample data and input files (RECOMMENDED)
│ ├── samples/
│ ├── examples/
│ └── data/
├── references/ # Reference documentation (RECOMMENDED)
│ ├── api-reference.md
│ ├── specifications.md
│ └── external-links.md
└── expected_outputs/ # Expected results for testing (RECOMMENDED)
├── sample_output.json
├── example_results.txt
└── test_cases/Optional Components
skill-name/
├── [mandatory and recommended components]
├── tests/ # Unit tests and validation scripts
├── examples/ # Extended examples and tutorials
├── docs/ # Additional documentation
├── config/ # Configuration files
└── templates/ # Template files for code generationFile Requirements
SKILL.md Requirements
The SKILL.md file serves as the primary documentation for the skill and must contain:
Mandatory YAML Frontmatter
---
Name: skill-name
Tier: [BASIC|STANDARD|POWERFUL]
Category: [Category Name]
Dependencies: [None|List of dependencies]
Author: [Author Name]
Version: [Semantic Version]
Last Updated: [YYYY-MM-DD]
---Required Sections
- Description: Comprehensive overview of the skill's purpose and capabilities
- Features: Detailed list of key features and functionality
- Usage: Instructions for using the skill and its components
- Examples: Practical usage examples with expected outcomes
Recommended Sections
- Architecture: Technical architecture and design decisions
- Installation: Setup and installation instructions
- Configuration: Configuration options and parameters
- Troubleshooting: Common issues and solutions
- Contributing: Guidelines for contributors
- Changelog: Version history and changes
Content Requirements by Tier
- BASIC: Minimum 100 lines of substantial content
- STANDARD: Minimum 200 lines of substantial content
- POWERFUL: Minimum 300 lines of substantial content
README.md Requirements
The README.md file provides quick start instructions and must include:
Mandatory Content
- Brief description of the skill
- Quick start instructions
- Basic usage examples
- Link to full SKILL.md documentation
Recommended Content
- Installation instructions
- Prerequisites and dependencies
- Command-line usage examples
- Troubleshooting section
- Contributing guidelines
Length Requirements
- Minimum 200 characters of substantial content
- Recommended 500+ characters for comprehensive coverage
Scripts Directory Requirements
The scripts/ directory contains all Python implementation files:
Mandatory Requirements
- At least one Python (.py) file
- All scripts must be executable Python 3.7+
- No external dependencies outside Python standard library
- Proper file naming conventions (lowercase, hyphens for separation)
Script Content Requirements
- Shebang line:
#!/usr/bin/env python3 - Module docstring: Comprehensive description of script purpose
- Argparse implementation: Command-line argument parsing
- Main guard:
if __name__ == "__main__":protection - Error handling: Appropriate exception handling and user feedback
- Dual output: Support for both JSON and human-readable output formats
Script Size Requirements by Tier
- BASIC: 100-300 lines of code per script
- STANDARD: 300-500 lines of code per script
- POWERFUL: 500-800 lines of code per script
Assets Directory Structure
The assets/ directory contains sample data and supporting files:
assets/
├── samples/ # Sample input data
│ ├── simple_example.json
│ ├── complex_dataset.csv
│ └── test_configuration.yaml
├── examples/ # Example files demonstrating usage
│ ├── basic_workflow.py
│ ├── advanced_usage.sh
│ └── integration_example.md
└── data/ # Static data files
├── reference_data.json
├── lookup_tables.csv
└── configuration_templates/Content Requirements
- At least 2 sample files demonstrating different use cases
- Files should represent realistic usage scenarios
- Include both simple and complex examples where applicable
- Provide diverse file formats (JSON, CSV, YAML, etc.)
References Directory Structure
The references/ directory contains detailed reference documentation:
references/
├── api-reference.md # Complete API documentation
├── specifications.md # Technical specifications and requirements
├── external-links.md # Links to related resources
├── algorithms.md # Algorithm descriptions and implementations
└── best-practices.md # Usage best practices and patternsContent Requirements
- Each file should contain substantial technical content (500+ words)
- Include code examples and technical specifications
- Provide external references and links where appropriate
- Maintain consistent documentation format and style
Expected Outputs Directory Structure
The expected_outputs/ directory contains reference outputs for testing:
expected_outputs/
├── basic_example_output.json
├── complex_scenario_result.txt
├── error_cases/
│ ├── invalid_input_error.json
│ └── timeout_error.txt
└── test_cases/
├── unit_test_outputs/
└── integration_test_results/Content Requirements
- Outputs correspond to sample inputs in assets/ directory
- Include both successful and error case examples
- Provide outputs in multiple formats (JSON, text, CSV)
- Ensure outputs are reproducible and verifiable
Naming Conventions
Directory Names
- Use lowercase letters only
- Use hyphens (-) to separate words
- Keep names concise but descriptive
- Avoid special characters and spaces
Examples: data-processor, api-client, ml-trainer
File Names
- Use lowercase letters for Python scripts
- Use hyphens (-) to separate words in script names
- Use underscores (_) only when required by Python conventions
- Use descriptive names that indicate purpose
Examples: data-processor.py, api-client.py, quality_scorer.py
Script Internal Naming
- Use PascalCase for class names
- Use snake_case for function and variable names
- Use UPPER_CASE for constants
- Use descriptive names that indicate purpose
Quality Standards
Documentation Standards
- All documentation must be written in clear, professional English
- Use proper Markdown formatting and structure
- Include code examples with syntax highlighting
- Provide comprehensive coverage of all features
- Maintain consistent terminology throughout
Code Standards
- Follow PEP 8 Python style guidelines
- Include comprehensive docstrings for all functions and classes
- Implement proper error handling with meaningful error messages
- Use type hints where appropriate
- Maintain reasonable code complexity and readability
Testing Standards
- Provide sample data that exercises all major functionality
- Include expected outputs for verification
- Cover both successful and error scenarios
- Ensure reproducible results across different environments
Validation Criteria
Skills are validated against the following criteria:
Structural Validation
- All mandatory files and directories present
- Proper file naming conventions followed
- Directory structure matches specification
- File permissions and accessibility correct
Content Validation
- SKILL.md meets minimum length and section requirements
- README.md provides adequate quick start information
- Scripts contain required components (argparse, main guard, etc.)
- Sample data and expected outputs are complete and realistic
Quality Validation
- Documentation is comprehensive and accurate
- Code follows established style and quality guidelines
- Examples are practical and demonstrate real usage
- Error handling is appropriate and user-friendly
Compliance Levels
Full Compliance
- All mandatory components present and complete
- All recommended components present with substantial content
- Exceeds minimum quality thresholds for tier
- Demonstrates best practices throughout
Partial Compliance
- All mandatory components present
- Most recommended components present
- Meets minimum quality thresholds for tier
- Generally follows established patterns
Non-Compliance
- Missing mandatory components
- Inadequate content quality or length
- Does not meet minimum tier requirements
- Significant deviations from established standards
Migration and Updates
Existing Skills
Skills created before this specification should be updated to comply within:
- POWERFUL tier: 30 days
- STANDARD tier: 60 days
- BASIC tier: 90 days
Specification Updates
- Changes to this specification require team consensus
- Breaking changes must provide 90-day migration period
- All changes must be documented with rationale and examples
- Automated validation tools must be updated accordingly
Tools and Automation
Validation Tools
skill_validator.py- Validates structure and content compliancescript_tester.py- Tests script functionality and qualityquality_scorer.py- Provides comprehensive quality assessment
Integration Points
- Pre-commit hooks for basic validation
- CI/CD pipeline integration for pull request validation
- Automated quality reporting and tracking
- Integration with code review processes
Examples and Templates
Minimal BASIC Tier Example
basic-skill/
├── SKILL.md # 100+ lines
├── README.md # Basic usage instructions
└── scripts/
└── main.py # 100-300 lines with argparseComplete POWERFUL Tier Example
powerful-skill/
├── SKILL.md # 300+ lines with comprehensive sections
├── README.md # Detailed usage and setup
├── scripts/ # Multiple sophisticated scripts
│ ├── main_processor.py # 500-800 lines
│ ├── data_analyzer.py # 500-800 lines
│ └── report_generator.py # 500-800 lines
├── assets/ # Diverse sample data
│ ├── samples/
│ ├── examples/
│ └── data/
├── references/ # Comprehensive documentation
│ ├── api-reference.md
│ ├── specifications.md
│ └── best-practices.md
└── expected_outputs/ # Complete test outputs
├── json_outputs/
├── text_reports/
└── error_cases/This specification serves as the authoritative guide for skill structure within the claude-skills ecosystem. Adherence to these standards ensures consistency, quality, and maintainability across all skills in the repository.
Tier Requirements Matrix
Version: 2.0.0 Last Updated: 2026-03-27 Authority: Claude Skills Engineering Team
Overview
This document provides a comprehensive matrix of requirements for each skill tier within the claude-skills ecosystem. Skills are classified into three tiers based on complexity, functionality, and comprehensiveness: BASIC, STANDARD, and POWERFUL.
Note: Security dimension requirements are optional and only apply when --include-security flag is used. By default, tier recommendations are based on 4 core dimensions (Documentation, Code Quality, Completeness, Usability) at 25% weight each.
Tier Classification Philosophy
BASIC Tier
Entry-level skills that provide fundamental functionality with minimal complexity. Suitable for simple automation tasks, basic data processing, or straightforward utilities.
STANDARD Tier
Intermediate skills that offer enhanced functionality with moderate complexity. Suitable for business processes, advanced data manipulation, or multi-step workflows.
POWERFUL Tier
Advanced skills that provide comprehensive functionality with sophisticated implementation. Suitable for complex systems, enterprise-grade tools, or mission-critical applications.
Requirements Matrix
| Component | BASIC | STANDARD | POWERFUL |
|---|---|---|---|
| SKILL.md Lines | ≥100 | ≥200 | ≥300 |
| Scripts Count | ≥1 | ≥1 | ≥2 |
| Script Size (LOC) | 100-300 | 300-500 | 500-800 |
| Required Directories | scripts | scripts, assets, references | scripts, assets, references, expected_outputs |
| Argparse Implementation | Basic | Advanced | Complex with subcommands |
| Output Formats | Human-readable | JSON + Human-readable | JSON + Human-readable + Custom |
| Error Handling | Basic | Comprehensive | Advanced with recovery |
| Documentation Depth | Functional | Comprehensive | Expert-level |
| Examples Provided | ≥1 | ≥3 | ≥5 |
| Test Coverage | Basic validation | Sample data testing | Comprehensive test suite |
| Security Score (opt-in) | ≥40 | ≥50 | ≥70 |
| Hardcoded Secrets (opt-in) | None | None | None |
| Input Validation (opt-in) | Basic | Comprehensive | Advanced with sanitization |
Detailed Requirements by Tier
BASIC Tier Requirements
Documentation Requirements
- SKILL.md: Minimum 100 lines of substantial content
- Required Sections: Name, Description, Features, Usage, Examples
- README.md: Basic usage instructions (200+ characters)
- Content Quality: Clear and functional documentation
- Examples: At least 1 practical usage example
Code Requirements
- Scripts: Minimum 1 Python script (100-300 LOC)
- Argparse: Basic command-line argument parsing
- Main Guard:
if __name__ == "__main__":protection - Dependencies: Python standard library only
- Output: Human-readable format with clear messaging
- Error Handling: Basic exception handling with user-friendly messages
Structure Requirements
- Mandatory Directories:
scripts/ - Recommended Directories:
assets/,references/ - File Organization: Logical file naming and structure
- Assets: Optional sample data files
Quality Standards
- Code Style: Follows basic Python conventions
- Documentation: Adequate coverage of functionality
- Usability: Clear usage instructions and examples
- Completeness: All essential components present
Security Requirements (opt-in with --include-security)
Note: These requirements only apply when the Security dimension is enabled via --include-security flag.
- Security Score: Minimum 40/100
- Hardcoded Secrets: No hardcoded passwords, API keys, or tokens
- Input Validation: Basic validation for user inputs
- Error Handling: User-friendly error messages without exposing sensitive info
- Safe Patterns: Avoid obvious security anti-patterns
STANDARD Tier Requirements
Documentation Requirements
- SKILL.md: Minimum 200 lines with comprehensive coverage
- Required Sections: All BASIC sections plus Architecture, Installation
- README.md: Detailed usage instructions (500+ characters)
- References: Technical documentation in
references/directory - Content Quality: Professional-grade documentation with technical depth
- Examples: At least 3 diverse usage examples
Code Requirements
- Scripts: 1-2 Python scripts (300-500 LOC each)
- Argparse: Advanced argument parsing with subcommands and validation
- Output Formats: Both JSON and human-readable output support
- Error Handling: Comprehensive exception handling with specific error types
- Code Structure: Well-organized classes and functions
- Documentation: Comprehensive docstrings for all functions
Structure Requirements
- Mandatory Directories:
scripts/,assets/,references/ - Recommended Directories:
expected_outputs/ - Assets: Multiple sample files demonstrating different use cases
- References: Technical specifications and API documentation
- Expected Outputs: Sample results for validation
Quality Standards
- Code Quality: Advanced Python patterns and best practices
- Documentation: Expert-level technical documentation
- Testing: Sample data processing with validation
- Integration: Consideration for CI/CD and automation use
Security Requirements (opt-in with --include-security)
Note: These requirements only apply when the Security dimension is enabled via --include-security flag.
- Security Score: Minimum 50/100
- Hardcoded Secrets: No hardcoded credentials (zero tolerance)
- Input Validation: Comprehensive validation with error messages
- File Operations: Safe path handling, no path traversal vulnerabilities
- Command Execution: No shell injection risks, safe subprocess usage
- Security Patterns: Use of environment variables for secrets
POWERFUL Tier Requirements
Documentation Requirements
- SKILL.md: Minimum 300 lines with expert-level comprehensiveness
- Required Sections: All STANDARD sections plus Troubleshooting, Contributing, Advanced Usage
- README.md: Comprehensive guide with installation and setup (1000+ characters)
- References: Multiple technical documents with specifications
- Content Quality: Publication-ready documentation with architectural details
- Examples: At least 5 examples covering simple to complex scenarios
Code Requirements
- Scripts: 2-3 Python scripts (500-800 LOC each)
- Argparse: Complex argument parsing with multiple modes and configurations
- Output Formats: JSON, human-readable, and custom format support
- Error Handling: Advanced error handling with recovery mechanisms
- Code Architecture: Sophisticated design patterns and modular structure
- Performance: Optimized for efficiency and scalability
Structure Requirements
- Mandatory Directories:
scripts/,assets/,references/,expected_outputs/ - Optional Directories:
tests/,examples/,docs/ - Assets: Comprehensive sample data covering edge cases
- References: Complete technical specification suite
- Expected Outputs: Full test result coverage including error cases
- Testing: Comprehensive validation and test coverage
Quality Standards
- Enterprise Grade: Production-ready code with enterprise patterns
- Documentation: Comprehensive technical documentation suitable for technical teams
- Integration: Full CI/CD integration capabilities
- Maintainability: Designed for long-term maintenance and extension
Security Requirements (opt-in with --include-security)
Note: These requirements only apply when the Security dimension is enabled via --include-security flag.
- Security Score: Minimum 70/100
- Hardcoded Secrets: Zero tolerance for hardcoded credentials
- Input Validation: Advanced validation with sanitization and type checking
- File Operations: All file operations use safe patterns (pathlib, validation)
- Command Execution: All subprocess calls use safe patterns (no shell=True)
- Security Patterns: Comprehensive security practices including:
- Environment variables for all secrets
- Input sanitization for all user inputs
- Safe deserialization practices
- Secure error handling without info leakage
- Security Documentation: Security considerations documented in code and docs
Tier Assessment Criteria
Automatic Tier Classification
Skills are automatically classified based on quantitative metrics:
def classify_tier(skill_metrics):
if (skill_metrics['skill_md_lines'] >= 300 and
skill_metrics['script_count'] >= 2 and
skill_metrics['min_script_size'] >= 500 and
all_required_dirs_present(['scripts', 'assets', 'references', 'expected_outputs'])):
return 'POWERFUL'
elif (skill_metrics['skill_md_lines'] >= 200 and
skill_metrics['script_count'] >= 1 and
skill_metrics['min_script_size'] >= 300 and
all_required_dirs_present(['scripts', 'assets', 'references'])):
return 'STANDARD'
else:
return 'BASIC'Manual Tier Override
Manual tier assignment may be considered when:
- Skill provides exceptional value despite not meeting all quantitative requirements
- Skill addresses critical infrastructure or security needs
- Skill demonstrates innovative approaches or cutting-edge techniques
- Skill provides essential integration or compatibility functions
Tier Promotion Criteria
Skills may be promoted to higher tiers when:
- All quantitative requirements for higher tier are met
- Quality assessment scores exceed tier thresholds
- Community usage and feedback indicate higher value
- Continuous integration and maintenance demonstrate reliability
Tier Demotion Criteria
Skills may be demoted to lower tiers when:
- Quality degradation below tier standards
- Lack of maintenance or updates
- Compatibility issues or security vulnerabilities
- Community feedback indicates reduced value
Implementation Guidelines by Tier
BASIC Tier Implementation
# Example argparse implementation for BASIC tier
parser = argparse.ArgumentParser(description="Basic skill functionality")
parser.add_argument("input", help="Input file or parameter")
parser.add_argument("--output", help="Output destination")
parser.add_argument("--verbose", action="store_true", help="Verbose output")
# Basic error handling
try:
result = process_input(args.input)
print(f"Processing completed: {result}")
except FileNotFoundError:
print("Error: Input file not found")
sys.exit(1)
except Exception as e:
print(f"Error: {str(e)}")
sys.exit(1)STANDARD Tier Implementation
# Example argparse implementation for STANDARD tier
parser = argparse.ArgumentParser(
description="Standard skill with advanced functionality",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="Examples:\n python script.py input.json --format json\n python script.py data/ --batch --output results/"
)
parser.add_argument("input", help="Input file or directory")
parser.add_argument("--format", choices=["json", "text"], default="json", help="Output format")
parser.add_argument("--batch", action="store_true", help="Process multiple files")
parser.add_argument("--output", help="Output destination")
# Advanced error handling with specific exception types
try:
if args.batch:
results = batch_process(args.input)
else:
results = single_process(args.input)
if args.format == "json":
print(json.dumps(results, indent=2))
else:
print_human_readable(results)
except FileNotFoundError as e:
logging.error(f"File not found: {e}")
sys.exit(1)
except ValueError as e:
logging.error(f"Invalid input: {e}")
sys.exit(2)
except Exception as e:
logging.error(f"Unexpected error: {e}")
sys.exit(1)POWERFUL Tier Implementation
# Example argparse implementation for POWERFUL tier
parser = argparse.ArgumentParser(
description="Powerful skill with comprehensive functionality",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="""
Examples:
Basic usage:
python script.py process input.json --output results/
Advanced batch processing:
python script.py batch data/ --format json --parallel 4 --filter "*.csv"
Custom configuration:
python script.py process input.json --config custom.yaml --dry-run
"""
)
subparsers = parser.add_subparsers(dest="command", help="Available commands")
# Process subcommand
process_parser = subparsers.add_parser("process", help="Process single file")
process_parser.add_argument("input", help="Input file path")
process_parser.add_argument("--config", help="Configuration file")
process_parser.add_argument("--dry-run", action="store_true", help="Show what would be done")
# Batch subcommand
batch_parser = subparsers.add_parser("batch", help="Process multiple files")
batch_parser.add_argument("directory", help="Input directory")
batch_parser.add_argument("--parallel", type=int, default=1, help="Number of parallel processes")
batch_parser.add_argument("--filter", help="File filter pattern")
# Comprehensive error handling with recovery
try:
if args.command == "process":
result = process_with_recovery(args.input, args.config, args.dry_run)
elif args.command == "batch":
result = batch_process_with_monitoring(args.directory, args.parallel, args.filter)
else:
parser.print_help()
sys.exit(1)
# Multiple output format support
output_formatter = OutputFormatter(args.format)
output_formatter.write(result, args.output)
except KeyboardInterrupt:
logging.info("Processing interrupted by user")
sys.exit(130)
except ProcessingError as e:
logging.error(f"Processing failed: {e}")
if e.recoverable:
logging.info("Attempting recovery...")
# Recovery logic here
sys.exit(1)
except ValidationError as e:
logging.error(f"Validation failed: {e}")
logging.info("Check input format and try again")
sys.exit(2)
except Exception as e:
logging.critical(f"Critical error: {e}")
logging.info("Please report this issue")
sys.exit(1)Quality Scoring by Tier
Scoring Thresholds
- POWERFUL Tier: Overall score ≥80, all dimensions ≥75, Security ≥70
- STANDARD Tier: Overall score ≥70, 4+ dimensions ≥65, Security ≥50
- BASIC Tier: Overall score ≥60, meets minimum requirements, Security ≥40
Dimension Weights (All Tiers)
- Documentation: 20%
- Code Quality: 20%
- Completeness: 20%
- Security: 20%
- Usability: 20%
Tier-Specific Quality Expectations
BASIC Tier Quality Profile
- Documentation: Functional and clear (60+ points expected)
- Code Quality: Clean and maintainable (60+ points expected)
- Completeness: Essential components present (60+ points expected)
- Security: Basic security practices (40+ points expected)
- Usability: Easy to understand and use (60+ points expected)
STANDARD Tier Quality Profile
- Documentation: Professional and comprehensive (70+ points expected)
- Code Quality: Advanced patterns and best practices (70+ points expected)
- Completeness: All recommended components (70+ points expected)
- Security: Good security practices, no hardcoded secrets (50+ points expected)
- Usability: Well-designed user experience (70+ points expected)
POWERFUL Tier Quality Profile
- Documentation: Expert-level and publication-ready (80+ points expected)
- Code Quality: Enterprise-grade implementation (80+ points expected)
- Completeness: Comprehensive test and validation coverage (80+ points expected)
- Security: Advanced security practices, comprehensive validation (70+ points expected)
- Usability: Exceptional user experience with extensive help (80+ points expected)
Tier Migration Process
Promotion Process
1. Assessment: Quality scorer evaluates skill against higher tier requirements 2. Review: Engineering team reviews assessment and implementation 3. Testing: Comprehensive testing against higher tier standards 4. Approval: Team consensus on tier promotion 5. Update: Skill metadata and documentation updated to reflect new tier
Demotion Process
1. Issue Identification: Quality degradation or standards violation identified 2. Assessment: Current quality evaluated against tier requirements 3. Notice: Skill maintainer notified of potential demotion 4. Grace Period: 30-day period for remediation 5. Final Review: Re-assessment after grace period 6. Action: Tier adjustment or removal if standards not met
Tier Change Communication
- All tier changes logged in skill CHANGELOG.md
- Repository-level tier change notifications
- Integration with CI/CD systems for automated handling
- Community notifications for significant changes
Compliance Monitoring
Automated Monitoring
- Daily quality assessment scans
- Tier compliance validation in CI/CD
- Automated reporting of tier violations
- Integration with code review processes
Manual Review Process
- Quarterly tier review cycles
- Community feedback integration
- Expert panel reviews for complex cases
- Appeals process for tier disputes
Enforcement Actions
- Warning: First violation or minor issues
- Probation: Repeated violations or moderate issues
- Demotion: Serious violations or quality degradation
- Removal: Critical violations or abandonment
This tier requirements matrix serves as the definitive guide for skill classification and quality standards within the claude-skills ecosystem. Regular updates ensure alignment with evolving best practices and community needs.
#!/usr/bin/env python3
"""
Script Tester - Tests Python scripts in a skill directory
This script validates and tests Python scripts within a skill directory by checking
syntax, imports, runtime execution, argparse functionality, and output formats.
It ensures scripts meet quality standards and function correctly.
Usage:
python script_tester.py <skill_path> [--timeout SECONDS] [--json] [--verbose]
Author: Claude Skills Engineering Team
Version: 1.0.0
Dependencies: Python Standard Library Only
"""
import argparse
import ast
import json
import os
import subprocess
import sys
import tempfile
import time
from datetime import datetime
from pathlib import Path
from typing import Dict, List, Any, Optional, Tuple, Union
import threading
class TestError(Exception):
"""Custom exception for testing errors"""
pass
class ScriptTestResult:
"""Container for individual script test results"""
def __init__(self, script_path: str):
self.script_path = script_path
self.script_name = Path(script_path).name
self.timestamp = datetime.utcnow().isoformat() + "Z"
self.tests = {}
self.overall_status = "PENDING"
self.execution_time = 0.0
self.errors = []
self.warnings = []
def add_test(self, test_name: str, passed: bool, message: str = "", details: Dict = None):
"""Add a test result"""
self.tests[test_name] = {
"passed": passed,
"message": message,
"details": details or {}
}
def add_error(self, error: str):
"""Add an error message"""
self.errors.append(error)
def add_warning(self, warning: str):
"""Add a warning message"""
self.warnings.append(warning)
def calculate_status(self):
"""Calculate overall test status"""
if not self.tests:
self.overall_status = "NO_TESTS"
return
failed_tests = [name for name, result in self.tests.items() if not result["passed"]]
if not failed_tests:
self.overall_status = "PASS"
elif len(failed_tests) <= len(self.tests) // 2:
self.overall_status = "PARTIAL"
else:
self.overall_status = "FAIL"
class TestSuite:
"""Container for all test results"""
def __init__(self, skill_path: str):
self.skill_path = skill_path
self.timestamp = datetime.utcnow().isoformat() + "Z"
self.script_results = {}
self.summary = {}
self.global_errors = []
def add_script_result(self, result: ScriptTestResult):
"""Add a script test result"""
self.script_results[result.script_name] = result
def add_global_error(self, error: str):
"""Add a global error message"""
self.global_errors.append(error)
def calculate_summary(self):
"""Calculate summary statistics"""
if not self.script_results:
self.summary = {
"total_scripts": 0,
"passed": 0,
"partial": 0,
"failed": 0,
"overall_status": "NO_SCRIPTS"
}
return
statuses = [result.overall_status for result in self.script_results.values()]
self.summary = {
"total_scripts": len(self.script_results),
"passed": statuses.count("PASS"),
"partial": statuses.count("PARTIAL"),
"failed": statuses.count("FAIL"),
"no_tests": statuses.count("NO_TESTS")
}
# Determine overall status
if self.summary["failed"] == 0 and self.summary["no_tests"] == 0:
self.summary["overall_status"] = "PASS"
elif self.summary["passed"] > 0:
self.summary["overall_status"] = "PARTIAL"
else:
self.summary["overall_status"] = "FAIL"
class ScriptTester:
"""Main script testing engine"""
def __init__(self, skill_path: str, timeout: int = 30, verbose: bool = False):
self.skill_path = Path(skill_path).resolve()
self.timeout = timeout
self.verbose = verbose
self.test_suite = TestSuite(str(self.skill_path))
def log_verbose(self, message: str):
"""Log verbose message if verbose mode enabled"""
if self.verbose:
print(f"[VERBOSE] {message}", file=sys.stderr)
def test_all_scripts(self) -> TestSuite:
"""Main entry point - test all scripts in the skill"""
try:
self.log_verbose(f"Starting script testing for {self.skill_path}")
# Check if skill path exists
if not self.skill_path.exists():
self.test_suite.add_global_error(f"Skill path does not exist: {self.skill_path}")
return self.test_suite
scripts_dir = self.skill_path / "scripts"
if not scripts_dir.exists():
self.test_suite.add_global_error("No scripts directory found")
return self.test_suite
# Find all Python scripts
python_files = list(scripts_dir.glob("*.py"))
if not python_files:
self.test_suite.add_global_error("No Python scripts found in scripts directory")
return self.test_suite
self.log_verbose(f"Found {len(python_files)} Python scripts to test")
# Test each script
for script_path in python_files:
try:
result = self.test_single_script(script_path)
self.test_suite.add_script_result(result)
except Exception as e:
# Create a failed result for the script
result = ScriptTestResult(str(script_path))
result.add_error(f"Failed to test script: {str(e)}")
result.overall_status = "FAIL"
self.test_suite.add_script_result(result)
# Calculate summary
self.test_suite.calculate_summary()
except Exception as e:
self.test_suite.add_global_error(f"Testing failed with exception: {str(e)}")
return self.test_suite
def test_single_script(self, script_path: Path) -> ScriptTestResult:
"""Test a single Python script comprehensively"""
result = ScriptTestResult(str(script_path))
start_time = time.time()
try:
self.log_verbose(f"Testing script: {script_path.name}")
# Read script content
try:
content = script_path.read_text(encoding='utf-8')
except Exception as e:
result.add_test("file_readable", False, f"Cannot read file: {str(e)}")
result.add_error(f"Cannot read script file: {str(e)}")
result.overall_status = "FAIL"
return result
result.add_test("file_readable", True, "Script file is readable")
# Test 1: Syntax validation
self._test_syntax(content, result)
# Test 2: Import validation
self._test_imports(content, result)
# Test 3: Argparse validation
self._test_argparse_implementation(content, result)
# Test 4: Main guard validation
self._test_main_guard(content, result)
# Test 5: Runtime execution tests
if result.tests.get("syntax_valid", {}).get("passed", False):
self._test_script_execution(script_path, result)
# Test 6: Help functionality
if result.tests.get("syntax_valid", {}).get("passed", False):
self._test_help_functionality(script_path, result)
# Test 7: Sample data processing (if available)
self._test_sample_data_processing(script_path, result)
# Test 8: Output format validation
self._test_output_formats(script_path, result)
except Exception as e:
result.add_error(f"Unexpected error during testing: {str(e)}")
finally:
result.execution_time = time.time() - start_time
result.calculate_status()
return result
def _test_syntax(self, content: str, result: ScriptTestResult):
"""Test Python syntax validity"""
self.log_verbose("Testing syntax...")
try:
ast.parse(content)
result.add_test("syntax_valid", True, "Python syntax is valid")
except SyntaxError as e:
result.add_test("syntax_valid", False, f"Syntax error: {str(e)}",
{"error": str(e), "line": getattr(e, 'lineno', 'unknown')})
result.add_error(f"Syntax error: {str(e)}")
def _test_imports(self, content: str, result: ScriptTestResult):
"""Test import statements for external dependencies"""
self.log_verbose("Testing imports...")
try:
tree = ast.parse(content)
external_imports = self._find_external_imports(tree)
if not external_imports:
result.add_test("imports_valid", True, "Uses only standard library imports")
else:
result.add_test("imports_valid", False,
f"Uses external imports: {', '.join(external_imports)}",
{"external_imports": external_imports})
result.add_error(f"External imports detected: {', '.join(external_imports)}")
except Exception as e:
result.add_test("imports_valid", False, f"Error analyzing imports: {str(e)}")
def _find_external_imports(self, tree: ast.AST) -> List[str]:
"""Find external (non-stdlib) imports"""
# Comprehensive standard library module list
stdlib_modules = {
# Built-in modules
'argparse', 'ast', 'json', 'os', 'sys', 'pathlib', 'datetime', 'typing',
'collections', 're', 'math', 'random', 'itertools', 'functools', 'operator',
'csv', 'sqlite3', 'urllib', 'http', 'html', 'xml', 'email', 'base64',
'hashlib', 'hmac', 'secrets', 'tempfile', 'shutil', 'glob', 'fnmatch',
'subprocess', 'threading', 'multiprocessing', 'queue', 'time', 'calendar',
'locale', 'gettext', 'logging', 'warnings', 'unittest', 'doctest',
'pickle', 'copy', 'pprint', 'reprlib', 'enum', 'dataclasses',
'contextlib', 'abc', 'atexit', 'traceback', 'gc', 'weakref', 'types',
'decimal', 'fractions', 'statistics', 'cmath', 'platform', 'errno',
'io', 'codecs', 'unicodedata', 'stringprep', 'textwrap', 'string',
'struct', 'difflib', 'heapq', 'bisect', 'array', 'uuid', 'mmap',
'ctypes', 'winreg', 'msvcrt', 'winsound', 'posix', 'pwd', 'grp',
'crypt', 'termios', 'tty', 'pty', 'fcntl', 'resource', 'nis',
'syslog', 'signal', 'socket', 'ssl', 'select', 'selectors',
'asyncio', 'asynchat', 'asyncore', 'netrc', 'xdrlib', 'plistlib',
'mailbox', 'mimetypes', 'encodings', 'pkgutil', 'modulefinder',
'runpy', 'importlib', 'imp', 'zipimport', 'zipfile', 'tarfile',
'gzip', 'bz2', 'lzma', 'zlib', 'binascii', 'quopri', 'uu',
'configparser', 'netrc', 'xdrlib', 'plistlib', 'token', 'tokenize',
'keyword', 'heapq', 'bisect', 'array', 'weakref', 'types',
'copyreg', 'shelve', 'marshal', 'dbm', 'sqlite3', 'zoneinfo'
}
external_imports = []
for node in ast.walk(tree):
if isinstance(node, ast.Import):
for alias in node.names:
module_name = alias.name.split('.')[0]
if module_name not in stdlib_modules and not module_name.startswith('_'):
external_imports.append(alias.name)
elif isinstance(node, ast.ImportFrom) and node.module:
module_name = node.module.split('.')[0]
if module_name not in stdlib_modules and not module_name.startswith('_'):
external_imports.append(node.module)
return list(set(external_imports))
def _test_argparse_implementation(self, content: str, result: ScriptTestResult):
"""Test argparse implementation"""
self.log_verbose("Testing argparse implementation...")
try:
tree = ast.parse(content)
# Check for argparse import
has_argparse_import = False
has_parser_creation = False
has_parse_args = False
for node in ast.walk(tree):
if isinstance(node, (ast.Import, ast.ImportFrom)):
if (isinstance(node, ast.Import) and
any(alias.name == 'argparse' for alias in node.names)):
has_argparse_import = True
elif (isinstance(node, ast.ImportFrom) and
node.module == 'argparse'):
has_argparse_import = True
elif isinstance(node, ast.Call):
# Check for ArgumentParser creation
if (isinstance(node.func, ast.Attribute) and
isinstance(node.func.value, ast.Name) and
node.func.value.id == 'argparse' and
node.func.attr == 'ArgumentParser'):
has_parser_creation = True
# Check for parse_args call
if (isinstance(node.func, ast.Attribute) and
node.func.attr == 'parse_args'):
has_parse_args = True
argparse_score = sum([has_argparse_import, has_parser_creation, has_parse_args])
if argparse_score == 3:
result.add_test("argparse_implementation", True, "Complete argparse implementation found")
elif argparse_score > 0:
result.add_test("argparse_implementation", False,
"Partial argparse implementation",
{"missing_components": [
comp for comp, present in [
("import", has_argparse_import),
("parser_creation", has_parser_creation),
("parse_args", has_parse_args)
] if not present
]})
result.add_warning("Incomplete argparse implementation")
else:
result.add_test("argparse_implementation", False, "No argparse implementation found")
result.add_error("Script should use argparse for command-line arguments")
except Exception as e:
result.add_test("argparse_implementation", False, f"Error analyzing argparse: {str(e)}")
def _test_main_guard(self, content: str, result: ScriptTestResult):
"""Test for if __name__ == '__main__' guard"""
self.log_verbose("Testing main guard...")
has_main_guard = 'if __name__ == "__main__"' in content or "if __name__ == '__main__'" in content
if has_main_guard:
result.add_test("main_guard", True, "Has proper main guard")
else:
result.add_test("main_guard", False, "Missing main guard")
result.add_error("Script should have 'if __name__ == \"__main__\"' guard")
def _test_script_execution(self, script_path: Path, result: ScriptTestResult):
"""Test basic script execution"""
self.log_verbose("Testing script execution...")
try:
# Try to run the script with no arguments (should not crash immediately)
process = subprocess.run(
[sys.executable, str(script_path)],
capture_output=True,
text=True,
timeout=self.timeout,
cwd=script_path.parent
)
# Script might exit with error code if no args provided, but shouldn't crash
if process.returncode in (0, 1, 2): # 0=success, 1=general error, 2=misuse
result.add_test("basic_execution", True,
f"Script runs without crashing (exit code: {process.returncode})")
else:
result.add_test("basic_execution", False,
f"Script crashed with exit code {process.returncode}",
{"stdout": process.stdout, "stderr": process.stderr})
except subprocess.TimeoutExpired:
result.add_test("basic_execution", False,
f"Script execution timed out after {self.timeout} seconds")
result.add_error(f"Script execution timeout ({self.timeout}s)")
except Exception as e:
result.add_test("basic_execution", False, f"Execution error: {str(e)}")
result.add_error(f"Script execution failed: {str(e)}")
def _test_help_functionality(self, script_path: Path, result: ScriptTestResult):
"""Test --help functionality"""
self.log_verbose("Testing help functionality...")
try:
# Test --help flag
process = subprocess.run(
[sys.executable, str(script_path), '--help'],
capture_output=True,
text=True,
timeout=self.timeout,
cwd=script_path.parent
)
if process.returncode == 0:
help_output = process.stdout
# Check for reasonable help content
help_indicators = ['usage:', 'positional arguments:', 'optional arguments:',
'options:', 'description:', 'help']
has_help_content = any(indicator in help_output.lower() for indicator in help_indicators)
if has_help_content and len(help_output.strip()) > 50:
result.add_test("help_functionality", True, "Provides comprehensive help text")
else:
result.add_test("help_functionality", False,
"Help text is too brief or missing key sections",
{"help_output": help_output})
result.add_warning("Help text could be more comprehensive")
else:
result.add_test("help_functionality", False,
f"Help command failed with exit code {process.returncode}",
{"stderr": process.stderr})
result.add_error("--help flag does not work properly")
except subprocess.TimeoutExpired:
result.add_test("help_functionality", False, "Help command timed out")
except Exception as e:
result.add_test("help_functionality", False, f"Help test error: {str(e)}")
def _test_sample_data_processing(self, script_path: Path, result: ScriptTestResult):
"""Test script against sample data if available"""
self.log_verbose("Testing sample data processing...")
assets_dir = self.skill_path / "assets"
if not assets_dir.exists():
result.add_test("sample_data_processing", True, "No sample data to test (assets dir missing)")
return
# Look for sample input files
sample_files = list(assets_dir.rglob("*sample*")) + list(assets_dir.rglob("*test*"))
sample_files = [f for f in sample_files if f.is_file() and not f.name.startswith('.')]
if not sample_files:
result.add_test("sample_data_processing", True, "No sample data files found to test")
return
tested_files = 0
successful_tests = 0
for sample_file in sample_files[:3]: # Test up to 3 sample files
try:
self.log_verbose(f"Testing with sample file: {sample_file.name}")
# Try to run script with the sample file as input
process = subprocess.run(
[sys.executable, str(script_path), str(sample_file)],
capture_output=True,
text=True,
timeout=self.timeout,
cwd=script_path.parent
)
tested_files += 1
if process.returncode == 0:
successful_tests += 1
else:
self.log_verbose(f"Sample test failed for {sample_file.name}: {process.stderr}")
except subprocess.TimeoutExpired:
tested_files += 1
result.add_warning(f"Sample data test timed out for {sample_file.name}")
except Exception as e:
tested_files += 1
self.log_verbose(f"Sample test error for {sample_file.name}: {str(e)}")
if tested_files == 0:
result.add_test("sample_data_processing", True, "No testable sample data found")
elif successful_tests == tested_files:
result.add_test("sample_data_processing", True,
f"Successfully processed all {tested_files} sample files")
elif successful_tests > 0:
result.add_test("sample_data_processing", False,
f"Processed {successful_tests}/{tested_files} sample files",
{"success_rate": successful_tests / tested_files})
result.add_warning("Some sample data processing failed")
else:
result.add_test("sample_data_processing", False,
"Failed to process any sample data files")
result.add_error("Script cannot process sample data")
def _test_output_formats(self, script_path: Path, result: ScriptTestResult):
"""Test output format compliance"""
self.log_verbose("Testing output formats...")
# Test if script supports JSON output
json_support = False
human_readable_support = False
try:
# Read script content to check for output format indicators
content = script_path.read_text(encoding='utf-8')
# Look for JSON-related code
if any(indicator in content.lower() for indicator in ['json.dump', 'json.load', '"json"', '--json']):
json_support = True
# Look for human-readable output indicators
if any(indicator in content for indicator in ['print(', 'format(', 'f"', "f'"]):
human_readable_support = True
# Try running with --json flag if it looks like it supports it
if '--json' in content:
try:
process = subprocess.run(
[sys.executable, str(script_path), '--json', '--help'],
capture_output=True,
text=True,
timeout=10,
cwd=script_path.parent
)
if process.returncode == 0:
json_support = True
except:
pass
# Evaluate dual output support
if json_support and human_readable_support:
result.add_test("output_formats", True, "Supports both JSON and human-readable output")
elif json_support or human_readable_support:
format_type = "JSON" if json_support else "human-readable"
result.add_test("output_formats", False,
f"Supports only {format_type} output",
{"json_support": json_support, "human_readable_support": human_readable_support})
result.add_warning("Consider adding dual output format support")
else:
result.add_test("output_formats", False, "No clear output format support detected")
result.add_warning("Output format support is unclear")
except Exception as e:
result.add_test("output_formats", False, f"Error testing output formats: {str(e)}")
class TestReportFormatter:
"""Formats test reports for output"""
@staticmethod
def format_json(test_suite: TestSuite) -> str:
"""Format test suite as JSON"""
return json.dumps({
"skill_path": test_suite.skill_path,
"timestamp": test_suite.timestamp,
"summary": test_suite.summary,
"global_errors": test_suite.global_errors,
"script_results": {
name: {
"script_path": result.script_path,
"timestamp": result.timestamp,
"overall_status": result.overall_status,
"execution_time": round(result.execution_time, 2),
"tests": result.tests,
"errors": result.errors,
"warnings": result.warnings
}
for name, result in test_suite.script_results.items()
}
}, indent=2)
@staticmethod
def format_human_readable(test_suite: TestSuite) -> str:
"""Format test suite as human-readable text"""
lines = []
lines.append("=" * 60)
lines.append("SCRIPT TESTING REPORT")
lines.append("=" * 60)
lines.append(f"Skill: {test_suite.skill_path}")
lines.append(f"Timestamp: {test_suite.timestamp}")
lines.append("")
# Summary
if test_suite.summary:
lines.append("SUMMARY:")
lines.append(f" Total Scripts: {test_suite.summary['total_scripts']}")
lines.append(f" Passed: {test_suite.summary['passed']}")
lines.append(f" Partial: {test_suite.summary['partial']}")
lines.append(f" Failed: {test_suite.summary['failed']}")
lines.append(f" Overall Status: {test_suite.summary['overall_status']}")
lines.append("")
# Global errors
if test_suite.global_errors:
lines.append("GLOBAL ERRORS:")
for error in test_suite.global_errors:
lines.append(f" • {error}")
lines.append("")
# Individual script results
for script_name, result in test_suite.script_results.items():
lines.append(f"SCRIPT: {script_name}")
lines.append(f" Status: {result.overall_status}")
lines.append(f" Execution Time: {result.execution_time:.2f}s")
lines.append("")
# Tests
if result.tests:
lines.append(" TESTS:")
for test_name, test_result in result.tests.items():
status = "✓ PASS" if test_result["passed"] else "✗ FAIL"
lines.append(f" {status}: {test_result['message']}")
lines.append("")
# Errors
if result.errors:
lines.append(" ERRORS:")
for error in result.errors:
lines.append(f" • {error}")
lines.append("")
# Warnings
if result.warnings:
lines.append(" WARNINGS:")
for warning in result.warnings:
lines.append(f" • {warning}")
lines.append("")
lines.append("-" * 40)
lines.append("")
return "\n".join(lines)
def main():
"""Main entry point"""
parser = argparse.ArgumentParser(
description="Test Python scripts in a skill directory",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="""
Examples:
python script_tester.py engineering/my-skill
python script_tester.py engineering/my-skill --timeout 60 --json
python script_tester.py engineering/my-skill --verbose
Test Categories:
- Syntax validation (AST parsing)
- Import validation (stdlib only)
- Argparse implementation
- Main guard presence
- Basic execution testing
- Help functionality
- Sample data processing
- Output format compliance
"""
)
parser.add_argument("skill_path",
help="Path to the skill directory containing scripts to test")
parser.add_argument("--timeout",
type=int,
default=30,
help="Timeout for script execution tests in seconds (default: 30)")
parser.add_argument("--json",
action="store_true",
help="Output results in JSON format")
parser.add_argument("--verbose",
action="store_true",
help="Enable verbose logging")
args = parser.parse_args()
try:
# Create tester and run tests
tester = ScriptTester(args.skill_path, args.timeout, args.verbose)
test_suite = tester.test_all_scripts()
# Format and output results
if args.json:
print(TestReportFormatter.format_json(test_suite))
else:
print(TestReportFormatter.format_human_readable(test_suite))
# Exit with appropriate code
if test_suite.global_errors:
sys.exit(1)
elif test_suite.summary.get("overall_status") == "FAIL":
sys.exit(1)
elif test_suite.summary.get("overall_status") == "PARTIAL":
sys.exit(2) # Partial success
else:
sys.exit(0) # Success
except KeyboardInterrupt:
print("\nTesting interrupted by user", file=sys.stderr)
sys.exit(130)
except Exception as e:
print(f"Testing failed: {str(e)}", file=sys.stderr)
if args.verbose:
import traceback
traceback.print_exc()
sys.exit(1)
if __name__ == "__main__":
main()Related skills
Find SkillsSo their agent can intelligently recommend and install the right modular skills from the open ecosystem when they ask how to do something.3M27.6k
Lark Skill MakerRapidly create custom skills for Lark (Feishu) that can be invoked by their AI agents and automation workflows.493k
Skill CreatorGenerate, test, and iterate on new reusable agent skills directly inside their Claude or Cursor workflow.332k165k
BrainstormingTurn vague product ideas into clear, approved designs and specifications through guided collaborative dialogue before writing any code.301k263k
Writing Great SkillsCreate high-quality, reusable agent skills that other developers can discover and install from the Skillselion catalog.262k195k
Find SkillsInstantly surface the best relevant agent skills, MCP servers, and workflows across multiple registries before starting any new project.226k12How it compares
Pick skill-tester over generic unit-test skills when validating Claude skill text output with zero custom harness setup.
FAQ
What does skill-tester validate?
skill-tester validates another Claude Code skill's text-processing behavior by running bundled sample text and CSV fixtures and comparing results to expected word-count statistics for counting, character analysis, lines, and transformations.
What fixtures does skill-tester include?
skill-tester ships bundled sample plain-text files and CSV inputs containing multi-line prose, punctuation, numbers, special characters, and mixed-case tokens such as CamelCase and snake_case for edge-case coverage.
Is Skill Tester safe to install?
skills.sh reports 3 of 3 security scanners passed. Review the Security Audits panel on this page before installing in production.