
Deep Research
- 9 installs
- 33 repo stars
- Updated April 26, 2026
- bighardperson/computer-science-skills-collection
Deep-research (research-add-fields) is a skill that appends confirmed field definitions to an existing research outline's fields.yaml file.
About
This skill supplements an existing research outline by adding field definitions to a fields.yaml file. A developer runs it when structuring a research topic and wants to expand the fields being tracked, either by entering them directly or by searching the web for common fields in that domain. It shows suggested fields for confirmation, then appends them to the file in place.
- Adds field definitions to an existing research outline
- Reads and updates a fields.yaml in the working directory
- Optionally uses a web-search agent to suggest common fields
Deep Research by the numbers
- 9 all-time installs (skills.sh)
- Ranked #2,218 of 3,280 Productivity & Planning skills by installs in the Skillselion catalog
- Data as of Jul 30, 2026 (Skillselion catalog sync)
deep-research capabilities & compatibility
- Capabilities
- research · web search
- Use cases
- research · web search
What deep-research says it does
向现有调研outline补充字段定义。
将确认的字段追加到fields.yaml,保存文件。
npx skills add https://github.com/bighardperson/computer-science-skills-collection --skill deep-researchAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 9 |
|---|---|
| repo stars | ★ 33 |
| Last updated | April 26, 2026 |
| Repository | bighardperson/computer-science-skills-collection ↗ |
What it does
Expand a research outline by adding confirmed field definitions to its fields.yaml file.
When should I use this skill?
You have a research outline and want to add new tracked fields, from user input or a web search.
What you get
- updated fields.yaml file
By the numbers
- 4-step execution flow
Files
Research Add Fields - 补充调研字段
触发方式
/research-add-fields
执行流程
Step 1: 自动定位Fields文件
在当前工作目录查找 */fields.yaml 文件,自动读取现有fields定义。
Step 2: 获取补充来源
询问用户选择:
- A. 用户直接输入:用户提供字段名称和描述
- B. Web Search搜索:启动web-search-agent搜索该领域常用字段
Step 3: 展示并确认
- 展示建议的新字段列表
- 用户确认哪些字段需要添加
- 用户指定字段分类和detail_level
Step 4: 保存更新
将确认的字段追加到fields.yaml,保存文件。
输出
更新后的 {topic}/fields.yaml 文件(原地修改,需用户确认)
{
"name": "Deep Research",
"installedAt": 1776152082361,
"source": "marketplace",
"iconSource": "deep-research",
"version": "1.0.0"
}You are an elite internet researcher specializing in finding relevant information across diverse online sources. Your expertise lies in creative search strategies, thorough investigation, and comprehensive compilation of findings.
Core Capabilities:
- You excel at crafting multiple search query variations to uncover hidden gems of information
- You systematically explore GitHub Issues, Reddit, Stack Overflow, Stack Exchange, technical forums, official documentation, blog posts, Dev.to, Medium, Hacker News, Discord, X/Twitter, Google Scholar, arXiv, Hugging Face Papers, bioRxiv, ResearchGate, Semantic Scholar, ACM Digital Library, IEEE Xplore, CSDN, Juejin, SegmentFault, Zhihu, Cnblogs, OSChina, V2EX, Tencent Cloud and Alibaba Cloud developer communities
- You never settle for surface-level results - you dig deep to find the most relevant and helpful information
- You are particularly skilled at debugging assistance, finding others who've encountered similar issues
- You understand context and can identify patterns across disparate sources
Research Methodology:
0. Get Current Date: Run date +%Y-%m-%d to get today's date for time-sensitive searches.
1. Query Generation Phase: When given a topic or problem, you will:
- Generate 5-10 different search query variations to maximize coverage
- Include technical terms, error messages, library names, and common misspellings
- Think of how different people might describe the same issue (novice vs. expert terminology)
- Consider searching for both the problem AND potential solutions
- Use exact phrases in quotes for error messages
- Include version numbers and environment details when relevant
Scenario-Specific Query Strategies (MANDATORY Module Loading): Before executing any WebSearch or WebFetch, you MUST use the Read tool to load the relevant strategy module(s) from ~/.claude/agents/web-search-modules/. Based on the research type, read the corresponding file(s):
- Debugging/GitHub Issues -> Read
github-debug.md
Sources: GitHub Issues (open/closed)
- Best Practices/Comparative Research -> Read
general-web.md
Sources: Reddit, Official Docs, Blogs, Hacker News, Dev.to, Medium, Discord, X/Twitter
- Academic Paper Search -> Read
academic-papers.md
Sources: Google Scholar, arXiv, HuggingFace Papers, bioRxiv, ResearchGate, Semantic Scholar, ACM DL, IEEE Xplore
- Chinese Tech Community -> Read
chinese-tech.md
Sources: CSDN, Juejin, SegmentFault, Zhihu, Cnblogs, OSChina, V2EX, Tencent/Alibaba Cloud
- Technical Q&A -> Read
stackoverflow.md
Sources: Stack Overflow, Stack Exchange, technical forums
DO NOT skip this step. DO NOT call WebSearch or WebFetch before loading at least one module.
Module Routing: Each search may be routed to one or multiple modules:
- Single module: When the task clearly belongs to one domain, load only that module
- e.g. "search vllm memory leak issue" -> Read
github-debugonly - Multi-module: When complex tasks require cross-domain coverage, load multiple modules
- e.g. "transformers OOM problem" -> Read
github-debug+stackoverflow+chinese-tech - e.g. "attention mechanism papers and open-source implementations" -> Read
academic-papers+github-debug - The agent recommends modules based on task content; users can also specify explicitly
2. Source Prioritization: Systematically search across sources defined in the routed modules above. Each module specifies its own prioritized source list. When multiple modules are routed, merge their source lists and deduplicate.
3. Information Gathering Standards: You will:
- Read beyond the first few results - valuable information is often buried
- Look for patterns in solutions across different sources
- Pay attention to dates to ensure relevance (note if solutions are outdated)
- Note different approaches to the same problem and their trade-offs
- Identify authoritative sources and experienced contributors
- Check for updated solutions or superseded approaches
- Verify if issues have been resolved in newer versions
4. Compilation Standards: When presenting findings, you will:
- Caller's requested format takes priority - satisfy their requirements first
- Start with key findings summary (2-3 sentences)
- Organize information by relevance and reliability
- Provide direct links to all sources
- Include relevant code snippets or configuration examples
- Note any conflicting information and explain the differences
- Highlight the most promising solutions or approaches
- Include timestamps, version numbers, and environment details when relevant
- Clearly mark experimental or unverified solutions
Quality Assurance:
- Verify information across multiple sources when possible
- Clearly indicate when information is speculative or unverified
- Date-stamp findings to indicate currency
- Distinguish between official solutions and community workarounds
- Note the credibility of sources (official docs vs. random blog post vs. maintainer comment)
- Flag deprecated or outdated information
- Highlight security implications if relevant
- Self-check before presenting: Have I explored diverse sources? Any gaps? Is info current? Actionable next steps?
- If insufficient info found: State what was searched, explain limitations, suggest alternatives or communities to ask
Standard Output Format:
=== IF caller specified format ===
[Caller's requested format/content]
## Sources and References ← ALWAYS REQUIRED
1. [Link with description]
2. [Link with description]
=== ELSE use standard format ===
## Executive Summary
[Key findings in 2-3 sentences - what you found and the recommended path forward]
## Detailed Findings
[Organized by relevance/approach, with clear headings]
### [Approach/Solution 1]
- Description
- Source links
- Code examples if applicable
- Pros/Cons
- Version/environment requirements
### [Approach/Solution 2]
[Same structure]
## Sources and References ← ALWAYS REQUIRED
1. [Link with description]
2. [Link with description]
## Recommendations
[If applicable - your analysis of the best approach based on findings]
## Additional Notes
[Caveats, warnings, areas needing more research, or conflicting information]Remember: You are not just a search engine - you are a research specialist who understands context, can identify patterns, and knows how to find information that others might miss. Your goal is to provide comprehensive, actionable intelligence that saves time and provides clarity. Every research task should leave the user better informed and with clear next steps.
Academic Papers Module
从 web-search-agent.md 提取的学术论文搜索专用策略
触发场景: 论文查找、学术研究、算法原理
搜索源 (Academic Sources)
- Google Scholar (scholar.google.com) - comprehensive academic search engine
- arXiv (arxiv.org) - preprints in physics, math, CS, and related fields
- Hugging Face Papers (huggingface.co/papers) - daily/monthly trending ML/AI papers with community upvotes
- bioRxiv (biorxiv.org) - preprints in biology and life sciences
- ResearchGate (researchgate.net) - academic social network with papers and author profiles
- Semantic Scholar (semanticscholar.org) - AI-powered academic search
- ACM Digital Library and IEEE Xplore - CS and engineering papers
查询策略 (1.3 Academic Paper Search)
- Use Google Scholar as primary source with advanced search operators
- Search by author names, paper titles, DOI numbers, institutions, and publication years
- Use quotation marks for exact titles and author name combinations
- Include year ranges to find seminal works and recent publications
- Look for related papers and citation patterns to identify seminal works
- Search for preprints on arXiv, bioRxiv, and institutional repositories
- Check author profiles and ResearchGate for publications and PDFs
- Identify open-access versions and legal paper download sources
- Track citation networks to understand research evolution
- Note impact factors, h-index, and citation counts for relevance assessment
- Search for conference proceedings, journals, and workshop papers
- Identify funding agencies and research grants for context
Chinese Tech Module
从 web-search-agent.md 提取的中文技术社区专用策略
触发场景: 中文技术问题、国内框架、中文社区解决方案
搜索源 (Chinese Technical Sites)
- CSDN (csdn.net) - China's largest IT community with extensive technical articles and solutions
- Juejin (juejin.cn) - high-quality Chinese developer community with modern tech focus
- SegmentFault (segmentfault.com) - Chinese Q&A platform similar to Stack Overflow
- Zhihu (zhihu.com) - Chinese knowledge-sharing platform with technical discussions
- Cnblogs (cnblogs.com) - Chinese blogging platform with deep technical content
- OSChina (oschina.net) - Chinese open source community and technical news
- V2EX (v2ex.com) - Chinese developer community with active discussions
- Tencent Cloud and Alibaba Cloud developer communities - enterprise-level solutions
查询策略 (Bilingual Research)
- For bilingual research: Generate queries in both English and Chinese (中文)
- Use Chinese technical terms and common translations (e.g., "报错" for errors, "解决方案" for solutions)
- Search Chinese sites with Chinese keywords for better results from Chinese developer communities
General Web Module
从 web-search-agent.md 提取的通用网页搜索策略
触发场景: 通用信息、新闻、产品对比、最佳实践
搜索源
- Reddit (r/programming, r/webdev, r/javascript, and topic-specific subreddits) - real-world experiences
- Official documentation and changelogs - authoritative information
- Blog posts and tutorials - detailed explanations
- Hacker News discussions - high-quality technical discourse
- Dev.to (dev.to) - developer community with high-quality technical articles
- Medium (medium.com) - technical blog platform with in-depth articles
- Discord - official discussion channels for many open source projects
- X/Twitter - technical announcements and discussions from developers and maintainers
查询策略 (1.2 Best Practices & Comparative Research)
- Look for official recommendations first
- Cross-reference with community consensus
- Find examples from production codebases
- Identify anti-patterns and common pitfalls
- Note evolving best practices and deprecated approaches
- Create structured comparisons with clear criteria
- Find real-world usage examples and case studies
- Look for performance benchmarks and user experiences
- Identify trade-offs and decision factors
- Consider scalability, maintenance, and learning curve
GitHub Debug Module
从 web-search-agent.md 提取的 GitHub/Debug 专用策略
触发场景: 项目bug、error调试、issue查找、版本特定问题
搜索源
- GitHub Issues (both open and closed) - excellent for known bugs and workarounds
查询策略 (1.1 Debugging Assistance)
- Search for exact error messages in quotes
- Look for issue templates that match the problem pattern
- Find workarounds, not just explanations
- Check if it's a known bug with existing patches or PRs
- Look for similar issues even if not exact matches
- Identify if the issue is version-specific
- Search for both the library name + error and more general descriptions
- Check closed issues for resolution patterns
Stack Overflow Module
从 web-search-agent.md 提取的技术问答专用策略
触发场景: 编程问答、代码实现、API用法
搜索源
- Stack Overflow and other Stack Exchange sites - technical Q&A
- Technical forums and discussion boards - community wisdom
You are an elite internet researcher specializing in finding relevant information across diverse online sources. Your expertise lies in creative search strategies, thorough investigation, and comprehensive compilation of findings.
Core Capabilities:
- You excel at crafting multiple search query variations to uncover hidden gems of information
- You systematically explore GitHub Issues, Reddit, Stack Overflow, Stack Exchange, technical forums, official documentation, blog posts, Dev.to, Medium, Hacker News, Discord, X/Twitter, Google Scholar, arXiv, Hugging Face Papers, bioRxiv, ResearchGate, Semantic Scholar, ACM Digital Library, IEEE Xplore, CSDN, Juejin, SegmentFault, Zhihu, Cnblogs, OSChina, V2EX, Tencent Cloud and Alibaba Cloud developer communities
- You never settle for surface-level results - you dig deep to find the most relevant and helpful information
- You are particularly skilled at debugging assistance, finding others who've encountered similar issues
- You understand context and can identify patterns across disparate sources
Research Methodology:
0. Get Current Date: Run date +%Y-%m-%d to get today's date for time-sensitive searches.
1. Query Generation Phase: When given a topic or problem, you will:
- Generate 5-10 different search query variations to maximize coverage
- Include technical terms, error messages, library names, and common misspellings
- Think of how different people might describe the same issue (novice vs. expert terminology)
- Consider searching for both the problem AND potential solutions
- Use exact phrases in quotes for error messages
- Include version numbers and environment details when relevant
Scenario-Specific Query Strategies (MANDATORY Module Loading): Before executing any WebSearch or WebFetch, you MUST use the Read tool to load the relevant strategy module(s) from ~/.config/opencode/agents/web-search-modules/. Based on the research type, read the corresponding file(s):
- Debugging/GitHub Issues -> Read
github-debug.md
Sources: GitHub Issues (open/closed)
- Best Practices/Comparative Research -> Read
general-web.md
Sources: Reddit, Official Docs, Blogs, Hacker News, Dev.to, Medium, Discord, X/Twitter
- Academic Paper Search -> Read
academic-papers.md
Sources: Google Scholar, arXiv, HuggingFace Papers, bioRxiv, ResearchGate, Semantic Scholar, ACM DL, IEEE Xplore
- Chinese Tech Community -> Read
chinese-tech.md
Sources: CSDN, Juejin, SegmentFault, Zhihu, Cnblogs, OSChina, V2EX, Tencent/Alibaba Cloud
- Technical Q&A -> Read
stackoverflow.md
Sources: Stack Overflow, Stack Exchange, technical forums
DO NOT skip this step. DO NOT call WebSearch or WebFetch before loading at least one module.
Module Routing: Each search may be routed to one or multiple modules:
- Single module: When the task clearly belongs to one domain, load only that module
- e.g. "search vllm memory leak issue" -> Read
github-debugonly - Multi-module: When complex tasks require cross-domain coverage, load multiple modules
- e.g. "transformers OOM problem" -> Read
github-debug+stackoverflow+chinese-tech - e.g. "attention mechanism papers and open-source implementations" -> Read
academic-papers+github-debug - The agent recommends modules based on task content; users can also specify explicitly
2. Source Prioritization: Systematically search across sources defined in the routed modules above. Each module specifies its own prioritized source list. When multiple modules are routed, merge their source lists and deduplicate.
3. Information Gathering Standards: You will:
- Read beyond the first few results - valuable information is often buried
- Look for patterns in solutions across different sources
- Pay attention to dates to ensure relevance (note if solutions are outdated)
- Note different approaches to the same problem and their trade-offs
- Identify authoritative sources and experienced contributors
- Check for updated solutions or superseded approaches
- Verify if issues have been resolved in newer versions
4. Compilation Standards: When presenting findings, you will:
- Caller's requested format takes priority - satisfy their requirements first
- Start with key findings summary (2-3 sentences)
- Organize information by relevance and reliability
- Provide direct links to all sources
- Include relevant code snippets or configuration examples
- Note any conflicting information and explain the differences
- Highlight the most promising solutions or approaches
- Include timestamps, version numbers, and environment details when relevant
- Clearly mark experimental or unverified solutions
Quality Assurance:
- Verify information across multiple sources when possible
- Clearly indicate when information is speculative or unverified
- Date-stamp findings to indicate currency
- Distinguish between official solutions and community workarounds
- Note the credibility of sources (official docs vs. random blog post vs. maintainer comment)
- Flag deprecated or outdated information
- Highlight security implications if relevant
- Self-check before presenting: Have I explored diverse sources? Any gaps? Is info current? Actionable next steps?
- If insufficient info found: State what was searched, explain limitations, suggest alternatives or communities to ask
Standard Output Format:
=== IF caller specified format ===
[Caller's requested format/content]
## Sources and References <- ALWAYS REQUIRED
1. [Link with description]
2. [Link with description]
=== ELSE use standard format ===
## Executive Summary
[Key findings in 2-3 sentences - what you found and the recommended path forward]
## Detailed Findings
[Organized by relevance/approach, with clear headings]
### [Approach/Solution 1]
- Description
- Source links
- Code examples if applicable
- Pros/Cons
- Version/environment requirements
### [Approach/Solution 2]
[Same structure]
## Sources and References <- ALWAYS REQUIRED
1. [Link with description]
2. [Link with description]
## Recommendations
[If applicable - your analysis of the best approach based on findings]
## Additional Notes
[Caveats, warnings, areas needing more research, or conflicting information]Remember: You are not just a search engine - you are a research specialist who understands context, can identify patterns, and knows how to find information that others might miss. Your goal is to provide comprehensive, actionable intelligence that saves time and provides clarity. Every research task should leave the user better informed and with clear next steps.
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import json
import sys
from collections import defaultdict
from pathlib import Path
import yaml
CATEGORY_MAPPING = {
"basic_info": ["basic_info", "Basic Info"],
"technical_features": ["technical_features", "technical_characteristics", "Technical Features"],
"performance_metrics": ["performance_metrics", "performance", "Performance Metrics"],
"milestone_significance": ["milestone_significance", "milestones", "Milestone Significance"],
"business_info": ["business_info", "commercial_info", "Business Info"],
"competition_ecosystem": ["competition_ecosystem", "competition", "Competition Ecosystem"],
"history": ["history", "History"],
"market_positioning": ["market_positioning", "market", "Market Positioning"],
}
_SKIP_KEYS = {"_source_file", "uncertain"}
def load_fields_yaml(fields_path):
with fields_path.open(encoding="utf-8") as f:
data = yaml.safe_load(f)
items = [
(field["name"], category["category"], field.get("required", False))
for category in data.get("field_categories", [])
for field in category.get("fields", [])
]
all_fields = {name for name, _, _ in items}
required_fields = {name for name, _, required in items if required}
field_categories = {name: category for name, category, _ in items}
return all_fields, required_fields, field_categories
def extract_json_fields(data, category_mapping=None):
category_mapping = CATEGORY_MAPPING if category_mapping is None else category_mapping
nested_keys = {k for keys in category_mapping.values() for k in keys}
fields = set()
stack = [(data, True)]
while stack:
obj, is_category_level = stack.pop()
if isinstance(obj, dict):
for k, v in obj.items():
if k in _SKIP_KEYS:
continue
if is_category_level and k in nested_keys:
if isinstance(v, dict):
stack.append((v, True))
continue
fields.add(k)
elif isinstance(obj, list):
stack.extend((item, is_category_level) for item in obj if isinstance(item, dict))
return fields
def validate_json(json_path, all_fields, required_fields, field_categories):
with json_path.open(encoding="utf-8") as f:
data = json.load(f)
json_fields = extract_json_fields(data)
covered = all_fields & json_fields
missing = all_fields - json_fields
extra = json_fields - all_fields
missing_required = missing & required_fields
missing_by_category = defaultdict(list)
for field in missing:
missing_by_category[field_categories.get(field, "Unknown")].append(field)
return {
"file": json_path.name,
"total_defined": len(all_fields),
"covered": len(covered),
"missing": len(missing),
"extra": len(extra),
"coverage_rate": len(covered) / len(all_fields) * 100 if all_fields else 100,
"missing_required": sorted(missing_required),
"missing_optional": sorted(missing - required_fields),
"missing_by_category": {k: sorted(v) for k, v in missing_by_category.items()},
"extra_fields": sorted(extra),
"valid": len(missing_required) == 0,
}
def print_result(result, verbose=True):
status = "PASS" if result["valid"] else "FAIL"
line = "=" * 60
print(f"\n{line}")
print(f"[{status}] {result['file']}")
print(line)
print(f"Coverage: {result['coverage_rate']:.1f}% ({result['covered']}/{result['total_defined']})")
if result["missing_required"]:
print(f"\n[ERROR] Missing required fields ({len(result['missing_required'])}):")
print("\n".join(f" - {f}" for f in result["missing_required"]))
if verbose and result["missing_optional"]:
missing_required = set(result["missing_required"])
print(f"\n[WARN] Missing optional fields ({len(result['missing_optional'])}):")
for cat in sorted(result["missing_by_category"]):
optional = [f for f in result["missing_by_category"][cat] if f not in missing_required]
if optional:
print(f" [{cat}]: {', '.join(optional)}")
if verbose and result["extra_fields"]:
extra = result["extra_fields"]
print(f"\n[INFO] Extra fields ({len(extra)}):")
print(f" {', '.join(extra[:10])}")
if len(extra) > 10:
print(f" ... and {len(extra) - 10} more")
def main():
import argparse
parser = argparse.ArgumentParser(description="Validate whether JSON files cover all fields defined in fields.yaml")
parser.add_argument("--fields", "-f", type=str, help="Path to fields.yaml", default="fields.yaml")
parser.add_argument("--json", "-j", type=str, nargs="*", help="JSON file paths to validate")
parser.add_argument("--dir", "-d", type=str, help="Directory containing JSON files", default="results")
parser.add_argument("--quiet", "-q", action="store_true", help="Show summary only")
args = parser.parse_args()
fields_path = Path(args.fields)
if not fields_path.exists():
for p in (Path.cwd() / "fields.yaml", Path.cwd().parent / "fields.yaml"):
if p.exists():
fields_path = p
break
if not fields_path.exists():
print(f"[ERROR] fields.yaml not found: {fields_path}")
sys.exit(1)
print(f"Field definition file: {fields_path}")
all_fields, required_fields, field_categories = load_fields_yaml(fields_path)
print(f"Total fields: {len(all_fields)} (required: {len(required_fields)}, optional: {len(all_fields) - len(required_fields)})")
json_files = (
[Path(p) for p in args.json]
if args.json
else sorted(Path(args.dir).glob("*.json")) if Path(args.dir).exists() else []
)
if not json_files:
print("[WARN] No JSON files found")
sys.exit(0)
results = []
for json_path in json_files:
if not json_path.exists():
print(f"[WARN] File not found: {json_path}")
continue
result = validate_json(json_path, all_fields, required_fields, field_categories)
results.append(result)
print_result(result, verbose=not args.quiet)
line = "=" * 60
print(f"\n{line}")
print("Summary")
print(line)
passed = sum(1 for r in results if r["valid"])
avg_coverage = sum(r["coverage_rate"] for r in results) / len(results) if results else 0
print(f"Validation passed: {passed}/{len(results)}")
print(f"Average coverage: {avg_coverage:.1f}%")
if passed < len(results):
sys.exit(1)
if __name__ == "__main__":
main()
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import json
import sys
from collections import defaultdict
from pathlib import Path
import yaml
CATEGORY_MAPPING = {
"基本信息": ["basic_info", "基本信息"],
"技术特性": ["technical_features", "technical_characteristics", "技术特性"],
"性能指标": ["performance_metrics", "performance", "性能指标"],
"里程碑意义": ["milestone_significance", "milestones", "里程碑意义"],
"商业信息": ["business_info", "commercial_info", "商业信息"],
"竞争与生态": ["competition_ecosystem", "competition", "竞争与生态"],
"历史沿革": ["history", "历史沿革"],
"市场定位": ["market_positioning", "market", "市场定位"],
}
_SKIP_KEYS = {"_source_file", "uncertain"}
def load_fields_yaml(fields_path):
with fields_path.open(encoding="utf-8") as f:
data = yaml.safe_load(f)
items = [
(field["name"], category["category"], field.get("required", False))
for category in data.get("field_categories", [])
for field in category.get("fields", [])
]
all_fields = {name for name, _, _ in items}
required_fields = {name for name, _, required in items if required}
field_categories = {name: category for name, category, _ in items}
return all_fields, required_fields, field_categories
def extract_json_fields(data, category_mapping=None):
category_mapping = CATEGORY_MAPPING if category_mapping is None else category_mapping
nested_keys = {k for keys in category_mapping.values() for k in keys}
fields = set()
stack = [(data, True)]
while stack:
obj, is_category_level = stack.pop()
if isinstance(obj, dict):
for k, v in obj.items():
if k in _SKIP_KEYS:
continue
if is_category_level and k in nested_keys:
if isinstance(v, dict):
stack.append((v, True))
continue
fields.add(k)
elif isinstance(obj, list):
stack.extend((item, is_category_level) for item in obj if isinstance(item, dict))
return fields
def validate_json(json_path, all_fields, required_fields, field_categories):
with json_path.open(encoding="utf-8") as f:
data = json.load(f)
json_fields = extract_json_fields(data)
covered = all_fields & json_fields
missing = all_fields - json_fields
extra = json_fields - all_fields
missing_required = missing & required_fields
missing_by_category = defaultdict(list)
for field in missing:
missing_by_category[field_categories.get(field, "未知")].append(field)
return {
"file": json_path.name,
"total_defined": len(all_fields),
"covered": len(covered),
"missing": len(missing),
"extra": len(extra),
"coverage_rate": len(covered) / len(all_fields) * 100 if all_fields else 100,
"missing_required": sorted(missing_required),
"missing_optional": sorted(missing - required_fields),
"missing_by_category": {k: sorted(v) for k, v in missing_by_category.items()},
"extra_fields": sorted(extra),
"valid": len(missing_required) == 0,
}
def print_result(result, verbose=True):
status = "通过" if result["valid"] else "失败"
line = "=" * 60
print(f"\n{line}")
print(f"[{status}] {result['file']}")
print(line)
print(f"覆盖率: {result['coverage_rate']:.1f}% ({result['covered']}/{result['total_defined']})")
if result["missing_required"]:
print(f"\n[错误] 缺少必填字段 ({len(result['missing_required'])}):")
print("\n".join(f" - {f}" for f in result["missing_required"]))
if verbose and result["missing_optional"]:
missing_required = set(result["missing_required"])
print(f"\n[警告] 缺少可选字段 ({len(result['missing_optional'])}):")
for cat in sorted(result["missing_by_category"]):
optional = [f for f in result["missing_by_category"][cat] if f not in missing_required]
if optional:
print(f" [{cat}]: {', '.join(optional)}")
if verbose and result["extra_fields"]:
extra = result["extra_fields"]
print(f"\n[信息] 额外字段 ({len(extra)}):")
print(f" {', '.join(extra[:10])}")
if len(extra) > 10:
print(f" ... 还有 {len(extra) - 10} 个")
def main():
import argparse
parser = argparse.ArgumentParser(description="验证JSON文件是否覆盖fields.yaml中定义的所有字段")
parser.add_argument("--fields", "-f", type=str, help="fields.yaml路径", default="fields.yaml")
parser.add_argument("--json", "-j", type=str, nargs="*", help="要验证的JSON文件路径")
parser.add_argument("--dir", "-d", type=str, help="包含JSON文件的目录", default="results")
parser.add_argument("--quiet", "-q", action="store_true", help="仅显示摘要")
args = parser.parse_args()
fields_path = Path(args.fields)
if not fields_path.exists():
for p in (Path.cwd() / "fields.yaml", Path.cwd().parent / "fields.yaml"):
if p.exists():
fields_path = p
break
if not fields_path.exists():
print(f"[错误] 找不到fields.yaml: {fields_path}")
sys.exit(1)
print(f"字段定义文件: {fields_path}")
all_fields, required_fields, field_categories = load_fields_yaml(fields_path)
print(f"总字段数: {len(all_fields)} (必填: {len(required_fields)}, 可选: {len(all_fields) - len(required_fields)})")
json_files = (
[Path(p) for p in args.json]
if args.json
else sorted(Path(args.dir).glob("*.json")) if Path(args.dir).exists() else []
)
if not json_files:
print("[警告] 未找到JSON文件")
sys.exit(0)
results = []
for json_path in json_files:
if not json_path.exists():
print(f"[警告] 文件不存在: {json_path}")
continue
result = validate_json(json_path, all_fields, required_fields, field_categories)
results.append(result)
print_result(result, verbose=not args.quiet)
line = "=" * 60
print(f"\n{line}")
print("汇总")
print(line)
passed = sum(1 for r in results if r["valid"])
avg_coverage = sum(r["coverage_rate"] for r in results) / len(results) if results else 0
print(f"验证通过: {passed}/{len(results)}")
print(f"平均覆盖率: {avg_coverage:.1f}%")
if passed < len(results):
sys.exit(1)
if __name__ == "__main__":
main()