Gpt Researcher

GPT Researcher Development Skill

GPT Researcher is an LLM-based autonomous agent using a planner-executor-publisher pattern with parallelized agent work for speed and reliability.

Quick Start

Basic Python Usage

from gpt_researcher import GPTResearcher
import asyncio

async def main():
    researcher = GPTResearcher(
        query="What are the latest AI developments?",
        report_type="research_report",  # or detailed_report, deep, outline_report
        report_source="web",            # or local, hybrid
    )
    await researcher.conduct_research()
    report = await researcher.write_report()
    print(report)

asyncio.run(main())

Run Servers

# Backend
python -m uvicorn backend.server.server:app --reload --port 8000

# Frontend
cd frontend/nextjs && npm install && npm run dev

---

Key File Locations

---

Architecture Overview

User Query → GPTResearcher.__init__()
                │
                ▼
         choose_agent() → (agent_type, role_prompt)
                │
                ▼
         ResearchConductor.conduct_research()
           ├── plan_research() → sub_queries
           ├── For each sub_query:
           │     └── _process_sub_query() → context
           └── Aggregate contexts
                │
                ▼
         [Optional] ImageGenerator.plan_and_generate_images()
                │
                ▼
         ReportGenerator.write_report() → Markdown report

For detailed architecture diagrams: See references/architecture.md

---

Core Patterns

Adding a New Feature (8-Step Pattern)

1. Config → Add to gpt_researcher/config/variables/default.py 2. Provider → Create in gpt_researcher/llm_provider/my_feature/ 3. Skill → Create in gpt_researcher/skills/my_feature.py 4. Agent → Integrate in gpt_researcher/agent.py 5. Prompts → Update gpt_researcher/prompts.py 6. WebSocket → Events via stream_output() 7. Frontend → Handle events in useWebSocket.ts 8. Docs → Create docs/docs/gpt-researcher/gptr/my_feature.md

For complete feature addition guide with Image Generation case study: See references/adding-features.md

Adding a New Retriever

# 1. Create: gpt_researcher/retrievers/my_retriever/my_retriever.py
class MyRetriever:
    def __init__(self, query: str, headers: dict = None):
        self.query = query
    
    async def search(self, max_results: int = 10) -> list[dict]:
        # Return: [{"title": str, "href": str, "body": str}]
        pass

# 2. Register in gpt_researcher/actions/retriever.py
case "my_retriever":
    from gpt_researcher.retrievers.my_retriever import MyRetriever
    return MyRetriever

# 3. Export in gpt_researcher/retrievers/__init__.py

For complete retriever documentation: See references/retrievers.md

---

Configuration

Config keys are lowercased when accessed:

# In default.py: "SMART_LLM": "gpt-4o"
# Access as: self.cfg.smart_llm  # lowercase!

Priority: Environment Variables → JSON Config File → Default Values

For complete configuration reference: See references/config-reference.md

---

Common Integration Points

WebSocket Streaming

class WebSocketHandler:
    async def send_json(self, data):
        print(f"[{data['type']}] {data.get('output', '')}")

researcher = GPTResearcher(query="...", websocket=WebSocketHandler())

MCP Data Sources

researcher = GPTResearcher(
    query="Open source AI projects",
    mcp_configs=[{
        "name": "github",
        "command": "npx",
        "args": ["-y", "@modelcontextprotocol/server-github"],
        "env": {"GITHUB_TOKEN": os.getenv("GITHUB_TOKEN")}
    }],
    mcp_strategy="deep",  # or "fast", "disabled"
)

For MCP integration details: See references/mcp.md

Deep Research Mode

researcher = GPTResearcher(
    query="Comprehensive analysis of quantum computing",
    report_type="deep",  # Triggers recursive tree-like exploration
)

For deep research configuration: See references/deep-research.md

---

Error Handling

Always use graceful degradation in skills:

async def execute(self, ...):
    if not self.is_enabled():
        return []  # Don't crash
    
    try:
        result = await self.provider.execute(...)
        return result
    except Exception as e:
        await stream_output("logs", "error", f"⚠️ {e}", self.websocket)
        return []  # Graceful degradation

---

Critical Gotchas

---

Reference Documentation

Adding Features Guide

Table of Contents

• The 8-Step Pattern
• Image Generation Case Study
• Testing New Features

---

The 8-Step Pattern

┌────────┐    ┌────────┐    ┌────────┐    ┌────────┐
│1.CONFIG│ →  │2.PROVIDER│ → │3.SKILL │ →  │4.AGENT │
└────────┘    └────────┘    └────────┘    └────────┘
     ↓             ↓             ↓             ↓
┌────────┐    ┌────────┐    ┌────────┐    ┌────────┐
│5.PROMPTS│ → │6.WEBSOCKET│→ │7.FRONTEND│→ │8.DOCS  │
└────────┘    └────────┘    └────────┘    └────────┘

Step 1: Add Configuration

File: gpt_researcher/config/variables/default.py

DEFAULT_CONFIG: BaseConfig = {
    "MY_FEATURE_ENABLED": False,
    "MY_FEATURE_MODEL": "model-name",
    "MY_FEATURE_MAX_ITEMS": 3,
}

File: gpt_researcher/config/variables/base.py

class BaseConfig(TypedDict):
    "MY_FEATURE_ENABLED": bool
    "MY_FEATURE_MODEL": Union[str, None]
    "MY_FEATURE_MAX_ITEMS": int

Step 2: Create Provider

File: gpt_researcher/llm_provider/my_feature/my_provider.py

class MyFeatureProvider:
    def __init__(self, api_key: str = None, model: str = None):
        self.api_key = api_key or os.getenv("MY_API_KEY")
        self.model = model
    
    def is_enabled(self) -> bool:
        return bool(self.api_key and self.model)
    
    async def execute(self, input_data: str) -> Dict[str, Any]:
        # API implementation
        pass

Export in gpt_researcher/llm_provider/__init__.py.

Step 3: Create Skill

File: gpt_researcher/skills/my_feature.py

class MyFeatureSkill:
    def __init__(self, researcher):
        self.researcher = researcher
        self.config = researcher.cfg
        self.provider = MyFeatureProvider(...)
    
    def is_enabled(self) -> bool:
        return getattr(self.config, 'my_feature_enabled', False) and self.provider.is_enabled()
    
    async def execute(self, context: str, query: str) -> List[Dict]:
        if not self.is_enabled():
            return []
        
        await stream_output("logs", "my_feature_start", "🚀 Starting...", self.researcher.websocket)
        results = await self.provider.execute(context)
        await stream_output("logs", "my_feature_complete", "✅ Done", self.researcher.websocket)
        
        return results

Export in gpt_researcher/skills/__init__.py.

Step 4: Integrate into Agent

File: gpt_researcher/agent.py

def __init__(self, ...):
    if self.cfg.my_feature_enabled:
        from gpt_researcher.skills import MyFeatureSkill
        self.my_feature = MyFeatureSkill(self)
    else:
        self.my_feature = None
    self.my_feature_results = []

async def conduct_research(self, ...):
    # ... existing ...
    if self.my_feature and self.my_feature.is_enabled():
        self.my_feature_results = await self.my_feature.execute(self.context, self.query)

Step 5: Update Prompts

File: gpt_researcher/prompts.py

@staticmethod
def generate_my_feature_prompt(context: str, query: str) -> str:
    return f"""..."""

Step 6: WebSocket Events

Already handled via stream_output() in skill.

Step 7: Frontend (if needed)

File: frontend/nextjs/hooks/useWebSocket.ts

if (data.content === 'my_feature_start') {
    setStatus('processing');
}

Step 8: Documentation

Create docs/docs/gpt-researcher/gptr/my_feature.md.

---

Image Generation Case Study

This section shows the actual implementation of the Image Generation feature as a reference.

1. Configuration Added

File: gpt_researcher/config/variables/default.py

DEFAULT_CONFIG: BaseConfig = {
    # ... existing ...
    "IMAGE_GENERATION_MODEL": "models/gemini-2.5-flash-image",
    "IMAGE_GENERATION_MAX_IMAGES": 3,
    "IMAGE_GENERATION_ENABLED": False,
    "IMAGE_GENERATION_STYLE": "dark",  # dark, light, auto
}

2. Provider Created

File: gpt_researcher/llm_provider/image/image_generator.py

class ImageGeneratorProvider:
    def __init__(self, api_key: str = None, model: str = None):
        self.api_key = api_key or os.getenv("GOOGLE_API_KEY")
        self.model = model or "models/gemini-2.5-flash-image"
        self._client = None
    
    def is_enabled(self) -> bool:
        return bool(self.api_key and self.model)
    
    def _build_enhanced_prompt(self, prompt: str, context: str = "", style: str = "dark") -> str:
        """Add styling instructions to prompt."""
        if style == "dark":
            style_instructions = """
            Style: Dark mode professional infographic
            - Background: Dark (#0d1117)
            - Accents: Teal/cyan (#14b8a6)
            - Clean, modern, minimalist
            """
        # ... handle light, auto
        return f"{style_instructions}\n\nCreate: {prompt}\n\nContext: {context}"
    
    async def generate_image(
        self,
        prompt: str,
        context: str = "",
        research_id: str = "",
        style: str = "dark",
    ) -> List[Dict[str, Any]]:
        """Generate image using Gemini."""
        full_prompt = self._build_enhanced_prompt(prompt, context, style)
        
        # Call Gemini API
        response = await self._generate_with_gemini(full_prompt, output_path, ...)
        
        return [{"url": f"/outputs/images/{research_id}/img_{hash}.png", ...}]

3. Skill Created

File: gpt_researcher/skills/image_generator.py

class ImageGenerator:
    def __init__(self, researcher):
        self.researcher = researcher
        self.config = researcher.cfg
        self.image_provider = ImageGeneratorProvider(
            api_key=os.getenv("GOOGLE_API_KEY"),
            model=getattr(self.config, 'image_generation_model', None),
        )
        self.max_images = getattr(self.config, 'image_generation_max_images', 3)
        self.style = getattr(self.config, 'image_generation_style', 'dark')
    
    def is_enabled(self) -> bool:
        enabled = getattr(self.config, 'image_generation_enabled', False)
        return enabled and self.image_provider.is_enabled()
    
    async def plan_and_generate_images(
        self,
        research_context: str,
        research_query: str,
        research_id: str,
        websocket: Any,
    ) -> List[Dict[str, Any]]:
        """
        1. Use LLM to identify visual concepts from context
        2. Generate images in parallel
        3. Return list of image metadata
        """
        # Stream progress
        await stream_output("logs", "image_planning", "🎨 Planning images...", websocket)
        
        # LLM identifies concepts
        concepts = await self._plan_image_concepts(research_context, research_query)
        
        # Generate images in parallel
        generated_images = []
        for i, concept in enumerate(concepts[:self.max_images]):
            await stream_output("logs", "image_generating", 
                f"🖼️ Generating image {i+1}/{len(concepts)}...", websocket)
            
            images = await self.image_provider.generate_image(
                prompt=concept["prompt"],
                context=concept.get("context", ""),
                research_id=research_id,
                style=self.style,
            )
            generated_images.extend(images)
        
        await stream_output("logs", "images_ready", 
            f"✅ Generated {len(generated_images)} images", websocket)
        
        return generated_images

4. Agent Integration

File: gpt_researcher/agent.py

class GPTResearcher:
    def __init__(self, ...):
        # ... existing init ...
        
        # Initialize image generator if enabled
        if self.cfg.image_generation_enabled:
            from gpt_researcher.skills import ImageGenerator
            self.image_generator = ImageGenerator(self)
        else:
            self.image_generator = None
        
        self.available_images: List[Dict[str, Any]] = []
        self.research_id = self._generate_research_id(query)
    
    async def conduct_research(self, on_progress=None):
        # ... existing research ...
        
        self.context = await self.research_conductor.conduct_research()
        
        # Pre-generate images after research, before report writing
        if self.cfg.image_generation_enabled and self.image_generator and self.image_generator.is_enabled():
            self.available_images = await self.image_generator.plan_and_generate_images(
                research_context=self.context,
                research_query=self.query,
                research_id=self.research_id,
                websocket=self.websocket,
            )
        
        return self.context
    
    async def write_report(self, ...):
        report = await self.report_generator.write_report(
            # ... existing params ...
            available_images=self.available_images,  # Pass to report writer
        )
        return report

5. Prompt Updated

File: gpt_researcher/prompts.py

@staticmethod
def generate_report_prompt(..., available_images: List[Dict[str, Any]] = []):
    image_instruction = ""
    if available_images:
        image_list = "\n".join([
            f"- Title: {img.get('title', 'Untitled')}\n  URL: {img['url']}"
            for img in available_images
        ])
        image_instruction = f"""
AVAILABLE IMAGES - Embed where relevant using ![Title](URL):
{image_list}
"""
    
    return f"""...(existing prompt)...
{image_instruction}
"""

---

Testing New Features

# tests/test_my_feature.py
import pytest
from gpt_researcher import GPTResearcher

@pytest.mark.asyncio
async def test_my_feature_disabled():
    """Test that feature is skipped when disabled."""
    researcher = GPTResearcher(query="test")
    # MY_FEATURE_ENABLED defaults to False
    assert researcher.my_feature is None

@pytest.mark.asyncio
async def test_my_feature_enabled(monkeypatch):
    """Test feature execution when enabled."""
    monkeypatch.setenv("MY_FEATURE_ENABLED", "true")
    monkeypatch.setenv("MY_API_KEY", "test-key")
    
    researcher = GPTResearcher(query="test")
    assert researcher.my_feature is not None
    assert researcher.my_feature.is_enabled()

Running Tests

# All tests
python -m pytest tests/

# Specific test
python -m pytest tests/test_my_feature.py -v

# With coverage
python -m pytest tests/ --cov=gpt_researcher

Advanced Patterns Reference

Table of Contents

• Custom Callbacks
• Custom WebSocket Handler
• LangChain Integration
• Search Restrictions
• Error Handling Patterns

---

Custom Callbacks

def cost_callback(cost: float):
    print(f"API call cost: ${cost}")

researcher = GPTResearcher(query="...")
researcher.add_costs = cost_callback  # Override cost tracking

---

Custom WebSocket Handler

class CustomWebSocket:
    def __init__(self):
        self.messages = []
    
    async def send_json(self, data):
        self.messages.append(data)
        if data['type'] == 'logs':
            print(f"Progress: {data['output']}")

researcher = GPTResearcher(query="...", websocket=CustomWebSocket())

---

LangChain Integration

Using with LangChain Documents

from langchain.document_loaders import DirectoryLoader

loader = DirectoryLoader('./docs', glob="**/*.md")
documents = loader.load()

researcher = GPTResearcher(
    query="Summarize the documentation",
    report_source="langchain_documents",
    documents=documents,
)

Using with Vector Store

from langchain.vectorstores import Chroma

vectorstore = Chroma.from_documents(documents, embeddings)

researcher = GPTResearcher(
    query="Find relevant information",
    report_source="langchain_vectorstore",
    vector_store=vectorstore,
    vector_store_filter={"source": "docs"},
)

---

Search Restrictions

Restricting Search Domains

researcher = GPTResearcher(
    query="Company news",
    query_domains=["reuters.com", "bloomberg.com", "wsj.com"],
)

Using Specific Source URLs

researcher = GPTResearcher(
    query="Analyze these articles",
    source_urls=[
        "https://example.com/article1",
        "https://example.com/article2",
    ],
    complement_source_urls=True,  # Also do web search
)

---

Error Handling Patterns

Graceful Degradation

# In skills, always check is_enabled()
async def execute(self, ...):
    if not self.is_enabled():
        logger.warning("Feature not enabled, skipping")
        return []  # Return empty, don't crash
    
    try:
        result = await self.provider.execute(...)
        return result
    except Exception as e:
        logger.error(f"Feature error: {e}")
        await stream_output("logs", "feature_error", f"⚠️ Error: {e}", self.websocket)
        return []  # Graceful degradation

API Rate Limiting

# Providers should handle rate limits
async def execute(self, ...):
    try:
        return await self._call_api(...)
    except RateLimitError as e:
        logger.warning(f"Rate limited, waiting...")
        await asyncio.sleep(60)
        return await self._call_api(...)  # Retry

WebSocket None Check

# Always check websocket before sending
if self.researcher.websocket:
    await stream_output("logs", "event", "message", self.researcher.websocket)

API Reference

Table of Contents

• REST API
• WebSocket API
• Python Client
• Output Files

---

REST API

Base URL: http://localhost:8000

Generate Report

POST `/report/`

{
    "task": "What are the latest AI developments?",
    "report_type": "research_report",
    "report_source": "web",
    "tone": "Objective",
    "source_urls": [],
    "query_domains": [],
    "generate_in_background": false
}

Response:

{
    "report": "# Research Report\n\n...",
    "research_id": "task_1234567890_query",
    "costs": 0.05,
    "pdf_path": "outputs/task_123.pdf",
    "docx_path": "outputs/task_123.docx"
}

Chat with Report

POST `/api/chat`

{
    "report": "The full report text...",
    "messages": [
        {"role": "user", "content": "What are the key findings?"}
    ]
}

Report Management

File Operations

Configuration

---

WebSocket API

Endpoint: ws://localhost:8000/ws

Send Research Request

{
    "task": "Research query",
    "report_type": "research_report",
    "report_source": "web",
    "tone": "Objective",
    "source_urls": [],
    "mcp_enabled": false,
    "mcp_strategy": "fast",
    "mcp_configs": []
}

Message Types (Server → Client)

Message Format

{
    "type": "logs",
    "content": "starting_research",
    "output": "🔍 Starting the research task...",
    "metadata": null
}

Frontend Handler Example

ws.onmessage = (event) => {
    const data = JSON.parse(event.data);
    
    switch (data.type) {
        case 'logs':
            setLogs(prev => [...prev, data]);
            break;
        case 'report':
            setAnswer(prev => prev + data.output);
            break;
        case 'report_complete':
            setAnswer(data.output);
            break;
        case 'path':
            setPaths(prev => ({...prev, [data.content]: data.output}));
            break;
        case 'error':
            setError(data.output);
            break;
    }
};

---

Python Client

Basic Usage

from gpt_researcher import GPTResearcher
import asyncio

async def main():
    researcher = GPTResearcher(
        query="What are the latest AI developments?",
        report_type="research_report",
    )
    
    await researcher.conduct_research()
    report = await researcher.write_report()
    
    print(f"Report: {report}")
    print(f"Costs: ${researcher.get_costs()}")

asyncio.run(main())

With MCP

researcher = GPTResearcher(
    query="Research topic",
    mcp_configs=[{
        "name": "github",
        "command": "npx",
        "args": ["-y", "@modelcontextprotocol/server-github"],
        "env": {"GITHUB_TOKEN": os.getenv("GITHUB_TOKEN")}
    }],
    mcp_strategy="deep",
)

With WebSocket Streaming

class MockWebSocket:
    async def send_json(self, data):
        print(f"[{data['type']}] {data.get('output', '')}")

researcher = GPTResearcher(
    query="Research topic",
    websocket=MockWebSocket(),
)

GPTResearcher Parameters

---

Output Files

outputs/
├── task_{timestamp}_{query}.md
├── task_{timestamp}_{query}.pdf
├── task_{timestamp}_{query}.docx
└── images/
    └── {research_id}/
        └── img_{hash}_{index}.png

---

Error Codes

Architecture Reference

Table of Contents

• System Layers
• Key File Locations

---

System Layers

┌─────────────────────────────────────────────────────────────────────────────┐
│                              USER REQUEST                                    │
│              (query, report_type, report_source, tone, mcp_configs)         │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                         BACKEND API LAYER                                    │
│  ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐          │
│  │  FastAPI Server  │  │ WebSocket Manager│  │  Report Store    │          │
│  │  backend/server/ │  │ Real-time events │  │  JSON persistence│          │
│  │  app.py          │  │ websocket_mgr.py │  │  report_store.py │          │
│  └──────────────────┘  └──────────────────┘  └──────────────────┘          │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                    GPTResearcher (gpt_researcher/agent.py)                   │
│                                                                              │
│  ┌───────────────────────────────────────────────────────────────────────┐  │
│  │                         SKILLS LAYER                                   │  │
│  │  ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐         │  │
│  │  │ ResearchConductor│ │ ReportGenerator │ │ ContextManager  │         │  │
│  │  │ Plan & gather   │ │ Write reports   │ │ Similarity search│         │  │
│  │  │ researcher.py   │ │ writer.py       │ │ context_manager │         │  │
│  │  └─────────────────┘ └─────────────────┘ └─────────────────┘         │  │
│  │  ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐         │  │
│  │  │ BrowserManager  │ │ SourceCurator   │ │ ImageGenerator  │         │  │
│  │  │ Web scraping    │ │ Rank sources    │ │ Gemini images   │         │  │
│  │  │ browser.py      │ │ curator.py      │ │ image_generator │         │  │
│  │  └─────────────────┘ └─────────────────┘ └─────────────────┘         │  │
│  │  ┌─────────────────┐                                                  │  │
│  │  │ DeepResearchSkill│                                                 │  │
│  │  │ Recursive depth │                                                  │  │
│  │  │ deep_research.py│                                                  │  │
│  │  └─────────────────┘                                                  │  │
│  └───────────────────────────────────────────────────────────────────────┘  │
│                                                                              │
│  ┌───────────────────────────────────────────────────────────────────────┐  │
│  │                        ACTIONS LAYER                                   │  │
│  │  ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐         │  │
│  │  │ report_generation│ │ query_processing│ │ web_scraping    │         │  │
│  │  │ LLM report write│ │ Sub-query plan  │ │ URL scraping    │         │  │
│  │  └─────────────────┘ └─────────────────┘ └─────────────────┘         │  │
│  │  ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐         │  │
│  │  │ retriever.py    │ │ agent_creator   │ │ markdown_process│         │  │
│  │  │ Get retrievers  │ │ Choose agent    │ │ Parse markdown  │         │  │
│  │  └─────────────────┘ └─────────────────┘ └─────────────────┘         │  │
│  └───────────────────────────────────────────────────────────────────────┘  │
│                                                                              │
│  ┌───────────────────────────────────────────────────────────────────────┐  │
│  │                       PROVIDERS LAYER                                  │  │
│  │  ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐         │  │
│  │  │ LLM Provider    │ │ Retrievers      │ │ Scrapers        │         │  │
│  │  │ OpenAI,Anthropic│ │ Tavily,Google   │ │ BS4,Playwright  │         │  │
│  │  │ Google,Groq...  │ │ Bing,MCP...     │ │ PDF,DOCX...     │         │  │
│  │  │ llm_provider/   │ │ retrievers/     │ │ scraper/        │         │  │
│  │  └─────────────────┘ └─────────────────┘ └─────────────────┘         │  │
│  │  ┌─────────────────┐                                                  │  │
│  │  │ ImageGenerator  │                                                  │  │
│  │  │ Gemini/Imagen   │                                                  │  │
│  │  │ llm_provider/   │                                                  │  │
│  │  │ image/          │                                                  │  │
│  │  └─────────────────┘                                                  │  │
│  └───────────────────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                        CONFIGURATION LAYER                                   │
│                     gpt_researcher/config/                                   │
│                                                                              │
│     Environment Variables  →  JSON Config File  →  Default Values            │
│           (highest)              (medium)            (lowest)                │
│                                                                              │
│     config.py loads and merges all sources                                   │
│     variables/default.py contains all defaults                               │
│     variables/base.py defines TypedDict for type safety                      │
└─────────────────────────────────────────────────────────────────────────────┘

---

Key File Locations

Core Components & Method Signatures

Table of Contents

• GPTResearcher
• ResearchConductor
• ReportGenerator

---

GPTResearcher

File: gpt_researcher/agent.py

The main orchestrator class. Full initialization signature:

class GPTResearcher:
    def __init__(
        self,
        query: str,                              # Research question (required)
        report_type: str = "research_report",    # research_report, detailed_report, deep, outline_report, resource_report
        report_format: str = "markdown",         # Output format
        report_source: str = "web",              # web, local, hybrid, azure, langchain_documents, langchain_vectorstore
        tone: Tone = Tone.Objective,             # Writing tone (see Tone enum)
        source_urls: list[str] | None = None,    # Specific URLs to research
        document_urls: list[str] | None = None,  # Document URLs to include
        complement_source_urls: bool = False,    # Add web search to source_urls
        query_domains: list[str] | None = None,  # Restrict search to domains
        documents=None,                          # LangChain document objects
        vector_store=None,                       # LangChain vector store
        vector_store_filter=None,                # Filter for vector store
        config_path=None,                        # Path to JSON config file
        websocket=None,                          # WebSocket for streaming
        agent=None,                              # Pre-defined agent type
        role=None,                               # Pre-defined agent role
        parent_query: str = "",                  # Parent query for subtopics
        subtopics: list | None = None,           # Subtopics to research
        visited_urls: set | None = None,         # Already visited URLs
        verbose: bool = True,                    # Verbose logging
        context=None,                            # Pre-loaded context
        headers: dict | None = None,             # HTTP headers
        max_subtopics: int = 5,                  # Max subtopics for detailed
        log_handler=None,                        # Custom log handler
        prompt_family: str | None = None,        # Custom prompt family
        mcp_configs: list[dict] | None = None,   # MCP server configurations
        mcp_max_iterations: int | None = None,   # Deprecated, use mcp_strategy
        mcp_strategy: str | None = None,         # fast, deep, disabled
        **kwargs
    ):

Key Methods

async def conduct_research(self, on_progress=None) -> str:
    """
    Main research orchestration.
    
    1. Selects agent role via LLM (choose_agent)
    2. Delegates to ResearchConductor
    3. Optionally generates images if enabled
    
    Returns: Accumulated research context as string
    """

async def write_report(
    self, 
    existing_headers: list = [],           # Headers to avoid duplication
    relevant_written_contents: list = [],  # Previous content for context
    ext_context=None,                      # External context override
    custom_prompt=""                       # Custom prompt override
) -> str:
    """
    Generate final report from context.
    
    Returns: Markdown report string
    """

def get_costs(self) -> float:
    """Returns total accumulated API costs."""

def add_costs(self, cost: float) -> None:
    """Add to running cost total (used as callback)."""

---

ResearchConductor

File: gpt_researcher/skills/researcher.py

Manages the research process:

class ResearchConductor:
    def __init__(self, researcher: GPTResearcher):
        self.researcher = researcher
        self.logger = logging.getLogger(__name__)

    async def plan_research(self, query: str, query_domains=None) -> list:
        """
        Generate sub-queries from main query using LLM.
        
        1. Gets initial search results
        2. Calls plan_research_outline() to generate sub-queries
        
        Returns: List of sub-query strings
        """

    async def conduct_research(self) -> str:
        """
        Main research execution based on report_source.
        
        Handles: web, local, hybrid, azure, langchain_documents, langchain_vectorstore
        
        For each source type:
        1. Load/search data
        2. Process sub-queries
        3. Combine context
        4. Optionally curate sources
        
        Returns: Combined research context string
        """

    async def _process_sub_query(
        self, 
        sub_query: str, 
        scraped_data: list = [], 
        query_domains: list = []
    ) -> str:
        """
        Process a single sub-query.
        
        1. Get MCP context (if configured, based on strategy)
        2. Scrape URLs from search results
        3. Get similar content via embeddings
        4. Combine MCP + web context
        
        Returns: Combined context for this sub-query
        """

    async def _get_context_by_web_search(
        self, 
        query: str, 
        scraped_data: list = [], 
        query_domains: list = []
    ) -> str:
        """Web-based research with sub-query planning."""

    async def _scrape_data_by_urls(
        self, 
        sub_query: str, 
        query_domains: list = []
    ) -> list:
        """Search and scrape URLs for a sub-query."""

---

ReportGenerator

File: gpt_researcher/skills/writer.py

class ReportGenerator:
    def __init__(self, researcher: GPTResearcher):
        self.researcher = researcher
        self.research_params = {
            "query": researcher.query,
            "agent_role_prompt": researcher.cfg.agent_role or researcher.role,
            "report_type": researcher.report_type,
            "report_source": researcher.report_source,
            "tone": researcher.tone,
            "websocket": researcher.websocket,
            "cfg": researcher.cfg,
            "headers": researcher.headers,
        }

    async def write_report(
        self,
        existing_headers: list = [],
        relevant_written_contents: list = [],
        ext_context=None,
        custom_prompt="",
        available_images: list = [],  # Pre-generated images to embed
    ) -> str:
        """
        Generate report using LLM.
        
        Calls generate_report() action with context and images.
        
        Returns: Markdown report
        """

    async def write_introduction(self, ...) -> str:
        """Write report introduction section."""

    async def write_conclusion(self, ...) -> str:
        """Write report conclusion with references."""

Configuration Reference

Table of Contents

• Required Variables
• LLM Configuration
• Provider API Keys
• Retriever Configuration
• Report Configuration
• Feature Toggles
• Configuration Priority
• Example .env

---

Required Variables

OPENAI_API_KEY=sk-...          # Or another LLM provider key
TAVILY_API_KEY=tvly-...        # Or another retriever key

---

LLM Configuration

LLM_PROVIDER=openai            # openai, anthropic, google, groq, together, etc.
FAST_LLM=gpt-4o-mini           # Quick tasks (summarization)
SMART_LLM=gpt-4o               # Complex reasoning (report writing)
STRATEGIC_LLM=o3-mini          # Planning (agent selection)
TEMPERATURE=0.4                # 0.0-1.0
MAX_TOKENS=4000
REASONING_EFFORT=medium        # For o-series: low, medium, high

---

Provider API Keys

# OpenAI
OPENAI_API_KEY=sk-...
OPENAI_BASE_URL=https://api.openai.com/v1

# Anthropic
ANTHROPIC_API_KEY=sk-ant-...

# Google
GOOGLE_API_KEY=AIza...

# Groq
GROQ_API_KEY=gsk_...

# Azure OpenAI
AZURE_OPENAI_API_KEY=...
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/

---

Retriever Configuration

RETRIEVER=tavily               # Single or comma-separated: tavily,google,mcp
MAX_SEARCH_RESULTS_PER_QUERY=5
MAX_URLS_TO_SCRAPE=10
SIMILARITY_THRESHOLD=0.42

Retriever API Keys

TAVILY_API_KEY=tvly-...
GOOGLE_API_KEY=AIza...
GOOGLE_CX_KEY=...
BING_API_KEY=...
SERPER_API_KEY=...
SERPAPI_API_KEY=...
EXA_API_KEY=...

---

Report Configuration

REPORT_FORMAT=apa              # apa, mla, chicago, harvard, ieee
TOTAL_WORDS=1000
LANGUAGE=english
CURATE_SOURCES=true

---

Feature Toggles

Image Generation

IMAGE_GENERATION_ENABLED=true
GOOGLE_API_KEY=AIza...
IMAGE_GENERATION_MODEL=models/gemini-2.5-flash-image
IMAGE_GENERATION_MAX_IMAGES=3
IMAGE_GENERATION_STYLE=dark    # dark, light, auto

Deep Research

DEEP_RESEARCH_BREADTH=4        # Subtopics per level
DEEP_RESEARCH_DEPTH=2          # Recursion levels
DEEP_RESEARCH_CONCURRENCY=2    # Parallel tasks

MCP

MCP_STRATEGY=fast              # fast, deep, disabled

Local Documents

DOC_PATH=./my-docs
# Supports: PDF, DOCX, TXT, CSV, XLSX, PPTX, MD

Server

HOST=0.0.0.0
PORT=8000
VERBOSE=true

---

Configuration Priority

Environment Variables (highest)
        ↓
JSON Config File (if provided)
        ↓
Default Values (lowest)

Important: Config keys are lowercased when accessed:

# In default.py: "SMART_LLM": "gpt-4o"
# Access as: self.cfg.smart_llm  # lowercase!

---

Example .env

# Required
OPENAI_API_KEY=sk-your-key
TAVILY_API_KEY=tvly-your-key

# LLM
FAST_LLM=gpt-4o-mini
SMART_LLM=gpt-4o

# Report
TOTAL_WORDS=1000
LANGUAGE=english

# Optional: Images
IMAGE_GENERATION_ENABLED=true
GOOGLE_API_KEY=AIza-your-key
IMAGE_GENERATION_STYLE=dark

Deep Research Mode Reference

Table of Contents

• Overview
• Configuration
• DeepResearchSkill
• Usage

---

Overview

Deep Research uses recursive tree-like exploration with configurable depth and breadth.

---

Configuration

DEEP_RESEARCH_BREADTH=4    # Subtopics per level
DEEP_RESEARCH_DEPTH=2      # Recursion levels
DEEP_RESEARCH_CONCURRENCY=2  # Parallel tasks

---

DeepResearchSkill

File: gpt_researcher/skills/deep_research.py

class DeepResearchSkill:
    def __init__(self, researcher):
        self.researcher = researcher
        self.breadth = getattr(researcher.cfg, 'deep_research_breadth', 4)
        self.depth = getattr(researcher.cfg, 'deep_research_depth', 2)
        self.concurrency_limit = getattr(researcher.cfg, 'deep_research_concurrency', 2)
        self.learnings = []
        self.research_sources = []
        self.context = []

    async def deep_research(self, query: str, on_progress=None) -> str:
        """
        Recursive research with depth and breadth.
        
        1. Research main topic
        2. Generate subtopics (breadth)
        3. For each subtopic, recursively research (depth)
        4. Aggregate all findings
        5. Generate comprehensive report
        """

---

Usage

researcher = GPTResearcher(
    query="Comprehensive analysis of quantum computing",
    report_type="deep",  # Triggers deep research
)
await researcher.conduct_research()
report = await researcher.write_report()

Research Tree Structure

Query: "Quantum Computing"
├── Subtopic 1: Hardware (depth 1)
│   ├── Subtopic 1.1: Superconducting qubits (depth 2)
│   └── Subtopic 1.2: Ion traps (depth 2)
├── Subtopic 2: Algorithms (depth 1)
│   ├── Subtopic 2.1: Shor's algorithm (depth 2)
│   └── Subtopic 2.2: Grover's algorithm (depth 2)
├── Subtopic 3: Applications (depth 1)
│   └── ...
└── Subtopic 4: Challenges (depth 1)
    └── ...

With DEEP_RESEARCH_BREADTH=4 and DEEP_RESEARCH_DEPTH=2, this explores 4 subtopics at each level, going 2 levels deep.

Research Flow & Data Flow

Table of Contents

• End-to-End Research Flow
• Data Flow Between Components

---

End-to-End Research Flow

1. Request Entry

File: backend/server/app.py

# REST API endpoint
@app.post("/report/")
async def generate_report(research_request: ResearchRequest, background_tasks: BackgroundTasks):
    research_id = sanitize_filename(f"task_{int(time.time())}_{research_request.task}")
    # Calls write_report() which uses run_agent()

# WebSocket endpoint
@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
    await manager.connect(websocket)
    await handle_websocket_communication(websocket, manager)

2. Agent Runner

File: backend/server/websocket_manager.py

async def run_agent(task, report_type, report_source, source_urls, ...):
    """Main entry point for research execution."""
    # Create logs handler
    logs_handler = CustomLogsHandler(websocket, task)
    
    # Configure MCP if enabled
    if mcp_enabled and mcp_configs:
        os.environ["RETRIEVER"] = f"{current_retriever},mcp"
        os.environ["MCP_STRATEGY"] = mcp_strategy
    
    # Route based on report type
    if report_type == "multi_agents":
        report = await run_research_task(query=task, websocket=logs_handler, ...)
    elif report_type == ReportType.DetailedReport.value:
        researcher = DetailedReport(query=task, ...)
        report = await researcher.run()
    else:
        researcher = BasicReport(query=task, ...)
        report = await researcher.run()
    
    return report

3. Research Phase

File: gpt_researcher/agent.py

async def conduct_research(self, on_progress=None):
    # Handle deep research separately
    if self.report_type == ReportType.DeepResearch.value and self.deep_researcher:
        return await self._handle_deep_research(on_progress)
    
    # Choose agent role via LLM
    if not (self.agent and self.role):
        self.agent, self.role = await choose_agent(
            query=self.query,
            cfg=self.cfg,
            parent_query=self.parent_query,
            cost_callback=self.add_costs,
            headers=self.headers,
            prompt_family=self.prompt_family,
        )
    
    # Conduct research
    self.context = await self.research_conductor.conduct_research()
    
    # Generate images if enabled (pre-generation for seamless UX)
    if self.cfg.image_generation_enabled and self.image_generator:
        self.available_images = await self.image_generator.plan_and_generate_images(
            research_context=self.context,
            research_query=self.query,
            research_id=self.research_id,
            websocket=self.websocket,
        )
    
    return self.context

4. Sub-Query Processing

File: gpt_researcher/skills/researcher.py

async def _process_sub_query(self, sub_query: str, scraped_data: list = [], query_domains: list = []):
    # MCP Strategy handling
    mcp_retrievers = [r for r in self.researcher.retrievers if "mcpretriever" in r.__name__.lower()]
    mcp_strategy = self._get_mcp_strategy()
    
    if mcp_retrievers:
        if mcp_strategy == "fast" and self._mcp_results_cache is not None:
            # Reuse cached MCP results
            mcp_context = self._mcp_results_cache.copy()
        elif mcp_strategy == "deep":
            # Run MCP for every sub-query
            mcp_context = await self._execute_mcp_research_for_queries([sub_query], mcp_retrievers)
    
    # Get web search context
    if not scraped_data:
        scraped_data = await self._scrape_data_by_urls(sub_query, query_domains)
    
    # Get similar content via embeddings
    if scraped_data:
        web_context = await self.researcher.context_manager.get_similar_content_by_query(
            sub_query, scraped_data
        )
    
    # Combine MCP + web context
    combined_context = self._combine_mcp_and_web_context(mcp_context, web_context, sub_query)
    return combined_context

5. Report Generation

File: gpt_researcher/actions/report_generation.py

async def generate_report(
    query: str,
    context: str,
    agent_role_prompt: str,
    report_type: str,
    websocket=None,
    cfg=None,
    tone=None,
    headers=None,
    cost_callback=None,
    prompt_family=None,
    available_images: list = [],
    **kwargs
) -> str:
    """Generate report using LLM."""
    # Get prompt generator
    generate_prompt = prompt_family.get_prompt_by_report_type(report_type)
    
    # Build prompt with context and available images
    content = generate_prompt(
        query, context, report_source,
        report_format=cfg.report_format,
        tone=tone,
        total_words=cfg.total_words,
        language=cfg.language,
        available_images=available_images,
    )
    
    # Call LLM
    report = await create_chat_completion(
        model=cfg.smart_llm,
        messages=[{"role": "user", "content": content}],
        temperature=cfg.temperature,
        llm_provider=cfg.smart_llm_provider,
        max_tokens=cfg.smart_token_limit,
        llm_kwargs=cfg.llm_kwargs,
        cost_callback=cost_callback,
    )
    
    return report

---

Data Flow Between Components

User Query
    │
    ▼
┌─────────────────────────────────────────────────────────────────┐
│ GPTResearcher.__init__()                                         │
│   • Loads Config (env → json → defaults)                        │
│   • Initializes skills: ResearchConductor, ReportGenerator, etc │
│   • Initializes retrievers based on RETRIEVER env var           │
│   • Initializes ImageGenerator if IMAGE_GENERATION_ENABLED      │
└─────────────────────────────────────────────────────────────────┘
    │
    │  researcher.conduct_research()
    ▼
┌─────────────────────────────────────────────────────────────────┐
│ choose_agent()                                                   │
│   Input: query, config                                          │
│   Output: (agent_type: str, role_prompt: str)                   │
│   • LLM selects best agent role for the query                   │
└─────────────────────────────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────────────────────────────┐
│ ResearchConductor.conduct_research()                             │
│   Input: self.researcher (has query, config, retrievers)        │
│   Output: context: str                                          │
│                                                                  │
│   ┌─────────────────────────────────────────────────────────┐   │
│   │ plan_research()                                          │   │
│   │   Input: query                                           │   │
│   │   Output: sub_queries: list[str]                         │   │
│   │   • Calls LLM to generate 3-5 sub-queries                │   │
│   └─────────────────────────────────────────────────────────┘   │
│                          │                                       │
│                          ▼                                       │
│   ┌─────────────────────────────────────────────────────────┐   │
│   │ For each sub_query:                                      │   │
│   │   _process_sub_query()                                   │   │
│   │     Input: sub_query                                     │   │
│   │     Output: sub_context: str                             │   │
│   │                                                          │   │
│   │     1. MCP retrieval (if configured)                     │   │
│   │        → mcp_context: list[dict]                         │   │
│   │                                                          │   │
│   │     2. Web search via retrievers                         │   │
│   │        → search_results: list[dict]                      │   │
│   │                                                          │   │
│   │     3. Scrape URLs                                       │   │
│   │        → scraped_content: list[dict]                     │   │
│   │                                                          │   │
│   │     4. Similarity search via embeddings                  │   │
│   │        → relevant_context: str                           │   │
│   │                                                          │   │
│   │     5. Combine MCP + web context                         │   │
│   │        → combined_context: str                           │   │
│   └─────────────────────────────────────────────────────────┘   │
│                          │                                       │
│                          ▼                                       │
│   Aggregate all sub_contexts → final context: str               │
└─────────────────────────────────────────────────────────────────┘
    │
    │  If IMAGE_GENERATION_ENABLED:
    ▼
┌─────────────────────────────────────────────────────────────────┐
│ ImageGenerator.plan_and_generate_images()                        │
│   Input: context, query, research_id                            │
│   Output: available_images: list[dict]                          │
│     [{"url": "/outputs/images/.../img.png",                     │
│       "title": "...", "description": "..."}]                    │
│                                                                  │
│   1. LLM analyzes context for visual concepts                   │
│   2. Generates 2-3 images in parallel via Gemini                │
│   3. Saves to outputs/images/{research_id}/                     │
└─────────────────────────────────────────────────────────────────┘
    │
    │  researcher.write_report()
    ▼
┌─────────────────────────────────────────────────────────────────┐
│ ReportGenerator.write_report()                                   │
│   Input: context, available_images                              │
│   Output: report: str (markdown)                                │
│                                                                  │
│   → generate_report() action                                    │
│       • Builds prompt with context + image list                 │
│       • LLM generates report with embedded images               │
└─────────────────────────────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────────────────────────────┐
│ Output                                                           │
│   • Streamed via WebSocket (type: "report")                     │
│   • Final via WebSocket (type: "report_complete")               │
│   • Exported to PDF, DOCX, Markdown                             │
│   • Saved to outputs/ directory                                 │
└─────────────────────────────────────────────────────────────────┘

MCP Integration Reference

Table of Contents

• Overview
• Configuration
• Strategy Options
• Processing Logic

---

Overview

MCP (Model Context Protocol) enables research from specialized data sources (GitHub, databases, APIs) alongside web search.

---

Configuration

researcher = GPTResearcher(
    query="...",
    mcp_configs=[
        {
            "name": "github",                    # Server name
            "command": "npx",                    # Command to start
            "args": ["-y", "@modelcontextprotocol/server-github"],
            "env": {"GITHUB_TOKEN": "..."},      # Environment vars
        },
        {
            "name": "filesystem",
            "command": "npx",
            "args": ["-y", "@anthropic/mcp-server-filesystem", "/docs"],
        },
        {
            "name": "remote",
            "connection_url": "ws://server:8080",  # WebSocket connection
            "connection_type": "websocket",
            "connection_token": "auth_token",
        }
    ],
    mcp_strategy="fast",  # fast, deep, disabled
)

---

Strategy Options

---

Processing Logic

File: gpt_researcher/skills/researcher.py

# At start of research (for 'fast' strategy)
if mcp_strategy == "fast":
    mcp_context = await self._execute_mcp_research_for_queries([query], mcp_retrievers)
    self._mcp_results_cache = mcp_context  # Cache for reuse

# During sub-query processing
if mcp_strategy == "fast" and self._mcp_results_cache is not None:
    mcp_context = self._mcp_results_cache.copy()  # Reuse cache
elif mcp_strategy == "deep":
    mcp_context = await self._execute_mcp_research_for_queries([sub_query], mcp_retrievers)

WebSocket Request Example

{
    "task": "Research query",
    "report_type": "research_report",
    "mcp_enabled": true,
    "mcp_strategy": "fast",
    "mcp_configs": [
        {
            "name": "github",
            "command": "npx",
            "args": ["-y", "@modelcontextprotocol/server-github"],
            "env": {"GITHUB_TOKEN": "..."}
        }
    ]
}

Multi-Agent System Reference

Table of Contents

• Overview
• Agent Roles
• Workflow
• Usage

---

Overview

Directory: multi_agents/

LangGraph-based system inspired by STORM paper. Generates 5-6 page reports with multiple agents collaborating.

---

Agent Roles

---

Workflow

1. Browser (GPTResearcher) → Initial research
2. Editor → Plans report outline
3. For each outline topic (parallel):
   a. Researcher → In-depth subtopic research
   b. Reviewer → Validates draft
   c. Revisor → Revises until satisfactory
4. Writer → Compiles final report
5. Publisher → Exports to multiple formats

---

Usage

Via API

report_type = "multi_agents"

Via WebSocket

{
    "task": "Research query",
    "report_type": "multi_agents",
    "tone": "Analytical"
}

Directly in Python

from multi_agents import run_research_task

report = await run_research_task(
    query="Comprehensive analysis of market trends",
    websocket=handler,
    tone=Tone.Analytical,
)

Configuration File

File: multi_agents/task.json

Configure the multi-agent research task parameters and agent behaviors.

Prompt System Reference

Table of Contents

• PromptFamily Class
• Key Prompt Examples

---

PromptFamily Class

File: gpt_researcher/prompts.py

All prompts are centralized in the PromptFamily class. This allows for model-specific prompt variations.

class PromptFamily:
    """
    General purpose class for prompt formatting.
    Can be overwritten with model-specific derived classes.
    """

    def __init__(self, config: Config):
        self.cfg = config

    @staticmethod
    def get_prompt_by_report_type(report_type: str):
        """Returns the appropriate prompt generator for the report type."""
        match report_type:
            case ReportType.ResearchReport.value:
                return PromptFamily.generate_report_prompt
            case ReportType.DetailedReport.value:
                return PromptFamily.generate_report_prompt
            case ReportType.OutlineReport.value:
                return PromptFamily.generate_outline_report_prompt
            # ... etc

---

Key Prompt Examples

Agent Selection Prompt

@staticmethod
def generate_agent_role_prompt(query: str, parent_query: str = "") -> str:
    return f"""Analyze the research query and select the most appropriate agent role.

Query: "{query}"
{f'Parent Query: "{parent_query}"' if parent_query else ''}

Based on the query, determine:
1. The domain expertise needed
2. The research approach required
3. The appropriate agent persona

Return a JSON object with:
- "agent": The agent type (e.g., "Research Analyst", "Technical Writer")
- "role": A detailed role description for how the agent should approach this research
"""

Research Planning Prompt

@staticmethod
def generate_search_queries_prompt(
    query: str,
    parent_query: str = "",
    report_type: str = "",
    max_iterations: int = 3,
    context: str = "",
) -> str:
    return f"""Generate {max_iterations} focused search queries to research: "{query}"

Context from initial search:
{context}

Requirements:
- Each query should explore a different aspect
- Queries should be specific and searchable
- Consider the report type: {report_type}

Return a JSON array of query strings.
"""

Report Generation Prompt (with images)

@staticmethod
def generate_report_prompt(
    question: str,
    context: str,
    report_source: str,
    report_format="apa",
    total_words=1000,
    tone=None,
    language="english",
    available_images: list = [],
) -> str:
    # Build image embedding instruction if images available
    image_instruction = ""
    if available_images:
        image_list = "\n".join([
            f"- Title: {img.get('title')}\n  URL: {img['url']}"
            for img in available_images
        ])
        image_instruction = f"""
AVAILABLE IMAGES (embed where relevant):
{image_list}

Use markdown format: ![Title](URL)
"""

    return f"""Information: "{context}"
---
Using the above information, answer: "{question}" in a detailed report.

- Format: {report_format}
- Length: ~{total_words} words
- Tone: {tone.value if tone else "Objective"}
- Language: {language}
- Include citations for all factual claims
{image_instruction}
"""

MCP Tool Selection Prompt

@staticmethod
def generate_mcp_tool_selection_prompt(query: str, tools_info: list, max_tools: int = 3) -> str:
    return f"""Select the most relevant tools for researching: "{query}"

AVAILABLE TOOLS:
{json.dumps(tools_info, indent=2)}

Select exactly {max_tools} tools ranked by relevance.

Return JSON:
{{
  "selected_tools": [
    {{"index": 0, "name": "tool_name", "relevance_score": 9, "reason": "..."}}
  ]
}}
"""

Retriever System Reference

Table of Contents

• Available Retrievers
• Retriever Selection
• Adding a New Retriever

---

Available Retrievers

Directory: gpt_researcher/retrievers/

---

Retriever Selection

File: gpt_researcher/actions/retriever.py

def get_retriever(retriever: str):
    """Get a retriever class by name."""
    match retriever:
        case "tavily":
            from gpt_researcher.retrievers import TavilySearch
            return TavilySearch
        case "google":
            from gpt_researcher.retrievers import GoogleSearch
            return GoogleSearch
        case "mcp":
            from gpt_researcher.retrievers import MCPRetriever
            return MCPRetriever
        # ... etc

def get_retrievers(retriever_names: str, headers: dict = None) -> list:
    """
    Get multiple retrievers from comma-separated string.
    
    Usage: RETRIEVER=tavily,google,mcp
    """
    retrievers = []
    for name in retriever_names.split(","):
        retriever_class = get_retriever(name.strip())
        if retriever_class:
            retrievers.append(retriever_class)
    return retrievers

---

Adding a New Retriever

Step 1: Create Retriever File

File: gpt_researcher/retrievers/my_retriever/my_retriever.py

class MyRetriever:
    def __init__(self, query: str, headers: dict = None):
        self.query = query
        self.headers = headers
    
    async def search(self, max_results: int = 10) -> list[dict]:
        """
        Returns list of:
        {
            "title": str,
            "href": str,
            "body": str
        }
        """
        # Implementation
        pass

Step 2: Register in retriever.py

File: gpt_researcher/actions/retriever.py

case "my_retriever":
    from gpt_researcher.retrievers.my_retriever import MyRetriever
    return MyRetriever

Step 3: Export in __init__.py

File: gpt_researcher/retrievers/__init__.py

from .my_retriever import MyRetriever
__all__ = [..., "MyRetriever"]

Step 4: Usage

RETRIEVER=tavily,my_retriever

researcher = GPTResearcher(
    query="...",
    # Will use both Tavily and your custom retriever
)