Novelty Assessment
Run multi-round literature search with a harsh critic to decide if a research idea is truly novel before you invest months building.
Overview
Novelty Assessment is an agent skill most often used in Idea (also Validate scope) that systematically searches literature and delivers a justified novel-or-not-novel decision before you commit.
Install
npx skills add https://github.com/lingzhi227/agent-research-skills --skill novelty-assessmentWhat is this skill?
- Multi-round literature search-evaluate loop (up to 10 rounds) with targeted queries each round
- Top-10 paper review per round with abstract-level overlap assessment
- Harsh critic persona and binary novel / not-novel decision with written justification
- Optional `novelty_check.py` automation and Semantic Scholar search script hooks
- Structured workflow: decompose idea → search → decide whether more rounds are needed
- Up to 10 literature search rounds
- Top-10 results reviewed per round
Adoption & trust: 734 installs on skills.sh; 114 GitHub stars; 2/3 security scanners passed (skills.sh audits).
What problem does it solve?
You are excited about a research idea but cannot tell if nearly identical work already exists in the literature.
Who is it for?
Indie researchers and agent builders evaluating paper ideas, hackathon ML projects, or R&D bets where duplicate work would waste months.
Skip if: Pure product-market validation with no research claim, or topics where you already have a completed related-work section from a trusted advisor.
When should I use this skill?
Use before committing to a research direction, when you have an idea title or description and need a justified novelty decision.
What do I get? / Deliverables
You receive a documented novelty verdict after up to ten search rounds, plus justification you can use to pivot, narrow scope, or proceed to implementation planning.
- Novel / not-novel decision with justification
- novelty_report.json when using automation
- Documented search rounds and overlap notes
Recommended Skills
Journey fit
Spans multiple journey phases - primary shelf plus alternate fits below.
Canonical shelf is Idea/research because the skill’s stated trigger is assessing novelty before committing to a research direction. Research subphase matches systematic Semantic Scholar/arXiv-style search, overlap analysis, and a binary novel/not-novel verdict.
Where it fits
Compare your proposed architecture against recent arXiv hits before writing a full spec.
Map subfield terminology into better search queries when exploring adjacent topics.
Decide whether to narrow the claim or abandon a line of work after overlap with a strong prior paper.
Gate an agent-research automation feature so it does not reproduce an existing benchmark contribution.
How it compares
Systematic literature gate with binary outcomes—not casual web search or generic brainstorming without citation discipline.
Common Questions / FAQ
Who is novelty-assessment for?
Solo builders and small research teams using coding agents who need rigorous prior-art checking before committing to a technical or academic direction.
When should I use novelty-assessment?
In Idea/research when framing a new contribution; in Validate/scope when deciding whether to prototype; anytime SKILL.md says to assess novelty before committing to a research direction.
Is novelty-assessment safe to install?
Review the Security Audits panel on this page; scripts may call external scholarly APIs and run Python on your machine—verify queries and outputs before sharing sensitive ideas.
SKILL.md
READMESKILL.md - Novelty Assessment
# Novelty Assessment Rigorously assess whether a research idea is novel through systematic literature search. ## Input - `$0` — Research idea description, title, or JSON file ## Scripts ### Automated novelty check ```bash python ~/.claude/skills/idea-generation/scripts/novelty_check.py \ --idea "Your research idea description" \ --max-rounds 10 --output novelty_report.json ``` ### Literature search ```bash python ~/.claude/skills/deep-research/scripts/search_semantic_scholar.py \ --query "relevant search query" --max-results 10 ``` ## References - Assessment prompts and criteria: `~/.claude/skills/novelty-assessment/references/assessment-prompts.md` ## Workflow ### Step 1: Understand the Idea - Identify the core contribution - List the key technical components - Determine the research area and subfield ### Step 2: Multi-Round Literature Search (up to 10 rounds) For each round: 1. Generate a targeted search query 2. Search Semantic Scholar / arXiv / OpenAlex 3. Review top-10 results with abstracts 4. Assess overlap with the idea 5. Decide: need more searching, or ready to decide ### Step 3: Make Decision - **Novel**: After sufficient searching, no paper significantly overlaps - **Not Novel**: Found a paper that significantly overlaps ### Step 4: Position the Idea If novel, identify: - Most similar existing papers (for Related Work) - How the idea differs from each - The specific gap this idea fills ## Harsh Critic Persona ``` Be a harsh critic for novelty. Ensure there is a sufficient contribution for a new conference or workshop paper. A trivial extension of existing work is NOT novel. The idea must offer a meaningfully different approach, formulation, or insight. ``` ## Output Format ```json { "decision": "novel" | "not_novel", "confidence": "high" | "medium" | "low", "justification": "After searching X rounds...", "most_similar_papers": [ {"title": "...", "year": 2024, "overlap": "..."} ], "differentiation": "Our idea differs because..." } ``` ## Rules - Minimum 3 search rounds before declaring novel - Try to recall exact paper names for targeted queries - A paper idea is NOT novel if it's a trivial extension - Consider both methodology novelty AND application novelty - Check for concurrent/recent arXiv submissions ## Related Skills - Upstream: [literature-search](../literature-search/), [deep-research](../deep-research/) - Downstream: [idea-generation](../idea-generation/), [research-planning](../research-planning/) - See also: [related-work-writing](../related-work-writing/) # Novelty Assessment Prompts Extracted from AI-Scientist (check_idea_novelty in generate_ideas.py), data-to-paper, and SciMON. ## Harsh Critic System Prompt (AI-Scientist) ``` You are a harsh but fair academic critic. Your task is to determine whether a research idea is truly novel or merely a trivial extension of existing work. Be a harsh critic for novelty. Ensure there is a sufficient contribution for a new conference or workshop paper. You will be given access to the Semantic Scholar API to survey the literature and check whether the proposed idea already exists. A trivial extension of existing work is NOT novel. The idea must offer a meaningfully different approach, formulation, or insight. ``` ## Multi-Round Search Protocol (AI-Scientist) ``` Round {N} of novelty assessment: Idea: {idea_description} Previous search queries and results: {previous_rounds} Instructions: 1. Generate a NEW search query that hasn't been tried before 2. Focus on the most specific aspect of the idea 3. Try to find the MOST SIMILAR existing paper 4. Consider different phrasings and related concepts Search query: [your query] A