Idea Generation

Idea Generation

Generate and refine novel research ideas with literature-backed novelty assessment.

Input

• $0 — Research area, task description, or existing codebase context
• $1 — Optional: additional context (e.g., "for NeurIPS", constraints)

Scripts

Novelty check against Semantic Scholar

python ~/.claude/skills/idea-generation/scripts/novelty_check.py \
  --idea "Adaptive attention head pruning via gradient-guided importance" \
  --max-rounds 5

Performs iterative literature search to assess if an idea is novel.

References

• Ideation prompts (generation, reflection, novelty): ~/.claude/skills/idea-generation/references/ideation-prompts.md

Workflow

Step 1: Generate Ideas

Given a research area and optional code/paper context: 1. Generate 3-5 diverse research ideas 2. For each idea, provide: Name, Title, Experiment plan, and ratings 3. Use the ideation prompt templates from references

Step 2: Iterative Refinement (up to 5 rounds per idea)

For each idea: 1. Critically evaluate quality, novelty, and feasibility 2. Refine the idea while preserving its core spirit 3. Stop when converged ("I am done") or max rounds reached

Step 3: Novelty Assessment

For each promising idea: 1. Run novelty_check.py or manually search Semantic Scholar / arXiv 2. Use the novelty checking prompts from references 3. Multi-round search: generate queries, review results, decide 4. Binary decision: Novel / Not Novel with justification

Step 4: Rank and Select

• Score each idea on three dimensions (1-10): Interestingness, Feasibility, Novelty
• Be cautious and realistic on ratings
• Select the top idea(s) for development

Output Format

{
  "Name": "adaptive_attention_pruning",
  "Title": "Adaptive Attention Head Pruning via Gradient-Guided Importance Scoring",
  "Experiment": "Detailed implementation plan...",
  "Interestingness": 8,
  "Feasibility": 7,
  "Novelty": 9,
  "novel": true,
  "most_similar_papers": ["paper1", "paper2"]
}

Rules

• Ideas must be feasible with available resources (no requiring new datasets or massive compute)
• Do not overfit ideas to a specific dataset or model — aim for wider significance
• Be a harsh critic for novelty — ensure sufficient contribution for a conference paper
• Each idea should stem from a simple, elegant question or hypothesis
• Always check novelty before committing to an idea

Related Skills

• Upstream: literature-search, deep-research
• Downstream: research-planning, experiment-design
• See also: novelty-assessment

Idea Generation Prompts Reference

Extracted from AI-Scientist, AI-Scientist-v2, and AI-Researcher.

1. Idea Generation Prompt (AI-Scientist)

System:

You are an ambitious AI researcher who is looking to publish a paper that will contribute significantly to the field.

User (first idea):

{task_description}
<experiment.py>
{code}
</experiment.py>

Here are the ideas that you have already generated:

'''
{prev_ideas_string}
'''

Come up with the next impactful and creative idea for research experiments and directions you can feasibly investigate with the code provided.
Note that you will not have access to any additional resources or datasets.
Make sure any idea is not overfit the specific training dataset or model, and has wider significance.

Respond in the following format:

THOUGHT:
<THOUGHT>

NEW IDEA JSON:

<JSON>

In <THOUGHT>, first briefly discuss your intuitions and motivations for the idea. Detail your high-level plan, necessary design choices and ideal outcomes of the experiments. Justify how the idea is different from the existing ones.

In <JSON>, provide the new idea in JSON format with the following fields:
- "Name": A shortened descriptor of the idea. Lowercase, no spaces, underscores allowed.
- "Title": A title for the idea, will be used for the report writing.
- "Experiment": An outline of the implementation. E.g. which functions need to be added or modified, how results will be obtained, ...
- "Interestingness": A rating from 1 to 10 (lowest to highest).
- "Feasibility": A rating from 1 to 10 (lowest to highest).
- "Novelty": A rating from 1 to 10 (lowest to highest).

Be cautious and realistic on your ratings.
This JSON will be automatically parsed, so ensure the format is precise.
You will have {num_reflections} rounds to iterate on the idea, but do not need to use them all.

2. Reflection Prompt (AI-Scientist)

Round {current_round}/{num_reflections}.
In your thoughts, first carefully consider the quality, novelty, and feasibility of the idea you just created.
Include any other factors that you think are important in evaluating the idea.
Ensure the idea is clear and concise, and the JSON is the correct format.
Do not make things overly complicated.
In the next attempt, try and refine and improve your idea.
Stick to the spirit of the original idea unless there are glaring issues.

Respond in the same format as before:
THOUGHT:
<THOUGHT>

NEW IDEA JSON:

<JSON>

If there is nothing to improve, simply repeat the previous JSON EXACTLY after the thought and include "I am done" at the end of the thoughts but before the JSON.
ONLY INCLUDE "I am done" IF YOU ARE MAKING NO MORE CHANGES.

3. Novelty Check System Message (AI-Scientist)

You are an ambitious AI PhD student who is looking to publish a paper that will contribute significantly to the field.
You have an idea and you want to check if it is novel or not. I.e., not overlapping significantly with existing literature or already well explored.
Be a harsh critic for novelty, ensure there is a sufficient contribution in the idea for a new conference or workshop paper.
You will be given access to the Semantic Scholar API, which you may use to survey the literature and find relevant papers to help you make your decision.
The top 10 results for any search query will be presented to you with the abstracts.

You will be given {num_rounds} to decide on the paper, but you do not need to use them all.
At any round, you may exit early and decide on the novelty of the idea.
Decide a paper idea is novel if after sufficient searching, you have not found a paper that significantly overlaps with your idea.
Decide a paper idea is not novel, if you have found a paper that significantly overlaps with your idea.

{task_description}
<experiment.py>
{code}
</experiment.py>

4. Novelty Check Per-Round Prompt (AI-Scientist)

Round {current_round}/{num_rounds}.
You have this idea:

"""
{idea}
"""

The results of the last query are (empty on first round):
"""
{last_query_results}
"""

Respond in the following format:

THOUGHT:
<THOUGHT>

RESPONSE:

<JSON>

In <THOUGHT>, first briefly reason over the idea and identify any query that could help you make your decision.
If you have made your decision, add "Decision made: novel." or "Decision made: not novel." to your thoughts.

In <JSON>, respond in JSON format with ONLY the following field:
- "Query": An optional search query to search the literature (e.g. attention is all you need). You must make a query if you have not decided this round.

A query will work best if you are able to recall the exact name of the paper you are looking for, or the authors.
This JSON will be automatically parsed, so ensure the format is precise.

5. AI-Scientist-v2: Enhanced Ideation

System:

You are an experienced AI researcher who aims to propose high-impact research ideas resembling exciting grant proposals. Feel free to propose any novel ideas or experiments; make sure they are novel. Be very creative and think out of the box. Each proposal should stem from a simple and elegant question, observation, or hypothesis about the topic.

Ensure that the proposal does not require resources beyond what an academic lab could afford. These proposals should lead to papers that are publishable at top ML conferences.

Idea Finalization (v2 fields):

{
  "Name": "...",
  "Title": "...",
  "Short Hypothesis": "A concise statement of the main hypothesis. Clarify the need for this direction, ensure this is the best setting to investigate.",
  "Related Work": "Brief discussion of most relevant related work and how the proposal clearly distinguishes from it.",
  "Abstract": "Conference-format abstract (~250 words).",
  "Experiments": "List of experiments to validate the proposal. Be specific in how you would test the hypothesis and detail precise algorithmic changes. Include evaluation metrics.",
  "Risk Factors and Limitations": "Potential risks and limitations of the proposal."
}

6. AI-Researcher: Idea Agent

You are an Idea Generation Agent specialized in analyzing academic papers and generating innovative ideas.

OBJECTIVE:
- Conduct thorough literature review of provided papers
- Identify research gaps and challenges
- Generate innovative and feasible ideas
- Provide detailed technical solutions

Generate comprehensive proposals including:
a) Challenges: Current technical limitations, unsolved problems
b) Existing Methods: Summary of approaches, advantages, limitations
c) Motivation: Why the problem is important, gaps to address
d) Proposed Method: Detailed technical solution, mathematical formulations, key innovations
e) Technical Details: Architecture design, algorithm specs, data flow
f) Expected Outcomes: Anticipated improvements, evaluation metrics, applications

Key Parameters

Scoring Rubric