Deep Research

Deep Research Skill

Perform deep research powered by Google Gemini's deep research agent. Upload documents to file search stores for RAG-grounded answers. Manage research sessions with persistent workspace state.

For AI Agents

Get a full capabilities manifest, decision trees, and output contracts:

uv run {baseDir}/scripts/onboard.py --agent

See AGENTS.md for the complete structured briefing.

Security & Transparency

Credentials: This skill requires a Google/Gemini API key (one of GOOGLE_API_KEY, GEMINI_API_KEY, or GEMINI_DEEP_RESEARCH_API_KEY). The key is read from environment variables and passed to the google-genai SDK. It is never logged, written to files, or transmitted anywhere other than the Google Gemini API.

File uploads: The --context flag uploads local files to Google's ephemeral file search stores for RAG grounding. Sensitive files are automatically excluded: .env*, credentials.json, secrets.*, private keys (.pem, .key), and auth tokens (.npmrc, .pypirc, .netrc). Binary files are rejected by MIME type filtering. Build directories (node_modules, __pycache__, .git, dist, build) are skipped. The ephemeral store is auto-deleted after research completes unless --keep-context is specified. Use --dry-run to preview what would be uploaded without sending anything. Only files you explicitly point --context at are uploaded -- no automatic scanning of parent directories or home folders.

Non-interactive mode: When stdin is not a TTY (agent/CI use), confirmation prompts are automatically skipped. This is by design for agent integration but means an autonomous agent with file system access could trigger uploads. Restrict the paths agents can access, or use --dry-run and --max-cost guards.

No obfuscation: All code is readable Python with PEP 723 inline metadata. No binary blobs, no minified scripts, no telemetry, no analytics. The full source is auditable at github.com/24601/agent-deep-research.

Local state: Research session state is written to .gemini-research.json in the working directory. This file contains interaction IDs, store mappings, and upload hashes -- no credentials or research content. Use state.py gc to clean up orphaned stores from crashed runs.

Prerequisites

• A Google API key (GOOGLE_API_KEY or GEMINI_API_KEY environment variable)
• uv installed (see uv install docs)

Quick Start

# Run a deep research query
uv run {baseDir}/scripts/research.py "What are the latest advances in quantum computing?"

# Check research status
uv run {baseDir}/scripts/research.py status <interaction-id>

# Save a completed report
uv run {baseDir}/scripts/research.py report <interaction-id> --output report.md

# Research grounded in local files (auto-creates store, uploads, cleans up)
uv run {baseDir}/scripts/research.py start "How does auth work?" --context ./src --output report.md

# Export as HTML or PDF
uv run {baseDir}/scripts/research.py start "Analyze the API" --context ./src --format html --output report.html

# Auto-detect prompt template based on context files
uv run {baseDir}/scripts/research.py start "How does auth work?" --context ./src --prompt-template auto --output report.md

Environment Variables

Set one of the following (checked in order of priority):

Optional model configuration:

Research Commands

Start Research

uv run {baseDir}/scripts/research.py start "your research question"

The start subcommand is the default, so research.py "question" and research.py start "question" are equivalent.

Important: When --output or --output-dir is used, the command blocks until research completes (2-10+ minutes). Do not background it with &. Use non-blocking mode (omit --output) to get an ID immediately, then poll with status and save with report.

Check Status

uv run {baseDir}/scripts/research.py status <interaction-id>

Returns the current status (in_progress, completed, failed) and outputs if available.

Save Report

uv run {baseDir}/scripts/research.py report <interaction-id>

Structured Output (--output-dir)

When --output-dir is used, results are saved to a structured directory:

<output-dir>/
  research-<id>/
    report.md          # Full final report
    metadata.json      # Timing, status, output count, sizes
    interaction.json   # Full interaction data (all outputs, thinking steps)
    sources.json       # Extracted source URLs/citations

A compact JSON summary (under 500 chars) is printed to stdout:

{
  "id": "interaction-123",
  "status": "completed",
  "output_dir": "research-output/research-interaction-1/",
  "report_file": "research-output/research-interaction-1/report.md",
  "report_size_bytes": 45000,
  "duration_seconds": 154,
  "summary": "First 200 chars of the report..."
}

This is the recommended pattern for AI agent integration -- the agent receives a small JSON payload while the full report is written to disk.

Adaptive Polling

When --output or --output-dir is used, the script polls the Gemini API until research completes. By default, it uses history-adaptive polling that learns from past research completion times:

• Completion times are recorded in .gemini-research.json under researchHistory (last 50 entries, separate curves for grounded vs non-grounded research).
• When 3+ matching data points exist, the poll interval is tuned to the historical distribution:
• Before any research has ever completed: slow polling (30s)
• In the likely completion window (p25-p75): aggressive polling (5s)
• In the tail (past p75): moderate polling (15-30s)
• Unusually long runs (past 1.5x the longest ever): slow polling (60s)
• All intervals are clamped to [2s, 120s] as a fail-safe.

When history is insufficient (<3 data points) or --no-adaptive-poll is passed, a fixed escalating curve is used: 5s (first 30s), 10s (30s-2min), 30s (2-10min), 60s (10min+).

Cost Estimation (--dry-run)

Preview estimated costs before running research:

uv run {baseDir}/scripts/research.py start "Analyze security architecture" --context ./src --dry-run

Outputs a JSON cost estimate to stdout with context upload costs, research query costs, and a total. Estimates are heuristic-based (the Gemini API does not return token counts or billing data) and clearly labeled as such.

After research completes with --output-dir, the metadata.json file includes a usage key with post-run cost estimates based on actual output size and duration.

File Search Store Commands

Manage file search stores for RAG-grounded research and Q&A.

Create a Store

uv run {baseDir}/scripts/store.py create "My Project Docs"

List Stores

uv run {baseDir}/scripts/store.py list

Query a Store

uv run {baseDir}/scripts/store.py query <store-name> "What does the auth module do?"

Delete a Store

uv run {baseDir}/scripts/store.py delete <store-name>

Use --force to skip the confirmation prompt. When stdin is not a TTY (e.g., called by an AI agent), the prompt is automatically skipped.

File Upload

Upload files or entire directories to a file search store.

uv run {baseDir}/scripts/upload.py ./src fileSearchStores/abc123

Hash caches are always saved on successful upload, so a subsequent --smart-sync run will correctly skip unchanged files even if the first upload did not use --smart-sync.

MIME Type Support

36 file extensions are natively supported by the Gemini File Search API. Common programming files (JS, TS, JSON, CSS, YAML, etc.) are automatically uploaded as text/plain via a fallback mechanism. Binary files are rejected. See references/file_search_guide.md for the full list.

File size limit: 100 MB per file.

Session Management

Research IDs and store mappings are cached in .gemini-research.json in the current working directory.

Show Session State

uv run {baseDir}/scripts/state.py show

Show Research Sessions Only

uv run {baseDir}/scripts/state.py research

Show Stores Only

uv run {baseDir}/scripts/state.py stores

JSON Output for Agents

Add --json to any state subcommand to output structured JSON to stdout:

uv run {baseDir}/scripts/state.py --json show
uv run {baseDir}/scripts/state.py --json research
uv run {baseDir}/scripts/state.py --json stores

Clear Session State

uv run {baseDir}/scripts/state.py clear

Use -y to skip the confirmation prompt. When stdin is not a TTY (e.g., called by an AI agent), the prompt is automatically skipped.

Non-Interactive Mode

All confirmation prompts (store.py delete, state.py clear) are automatically skipped when stdin is not a TTY. This allows AI agents and CI pipelines to call these commands without hanging on interactive prompts.

Workflow Example

A typical grounded research workflow:

# 1. Create a file search store
STORE_JSON=$(uv run {baseDir}/scripts/store.py create "Project Codebase")
STORE_NAME=$(echo "$STORE_JSON" | python3 -c "import sys,json; print(json.load(sys.stdin)['name'])")

# 2. Upload your documents
uv run {baseDir}/scripts/upload.py ./docs "$STORE_NAME" --smart-sync

# 3. Query the store directly
uv run {baseDir}/scripts/store.py query "$STORE_NAME" "How is authentication handled?"

# 4. Start grounded deep research (blocking, saves to directory)
uv run {baseDir}/scripts/research.py start "Analyze the security architecture" \
  --store "$STORE_NAME" --output-dir ./research-output --timeout 3600

# 5. Or start non-blocking and check later
RESEARCH_JSON=$(uv run {baseDir}/scripts/research.py start "Analyze the security architecture" --store "$STORE_NAME")
RESEARCH_ID=$(echo "$RESEARCH_JSON" | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])")

# 6. Check progress
uv run {baseDir}/scripts/research.py status "$RESEARCH_ID"

# 7. Save the report when completed
uv run {baseDir}/scripts/research.py report "$RESEARCH_ID" --output-dir ./research-output

Output Convention

All scripts follow a dual-output pattern:

• stderr: Rich-formatted human-readable output (tables, panels, progress bars)
• stdout: Machine-readable JSON for programmatic consumption

This means 2>/dev/null hides the human output, and piping stdout gives clean JSON.

Script

Which script is affected?

• [ ] research.py
• [ ] store.py
• [ ] upload.py
• [ ] state.py

Environment

• Python version:
• uv version:
• OS:

Steps to Reproduce

1. 2. 3.

Expected Behavior

What you expected to happen.

Actual Behavior

What actually happened.

Error Output

Paste any error messages or tracebacks here.

Additional Context

Any other relevant details (API key type, model configuration, file types involved, etc.).

Which script(s) does this relate to?

• [ ] research.py
• [ ] store.py
• [ ] upload.py
• [ ] state.py
• [ ] New script / general

Use Case

Describe the problem you're trying to solve or the workflow you want to enable.

Proposed Solution

Describe how you'd like this to work.

Alternatives Considered

Any alternative approaches you've thought about.

Additional Context

Any other relevant details, mockups, or references.

Description

Brief description of what this PR does.

Related Issue

Closes #

Type of Change

• [ ] Bug fix
• [ ] New feature
• [ ] Documentation
• [ ] Refactor
• [ ] CI/build

Testing

• [ ] python3 -m py_compile scripts/*.py passes
• [ ] uv run scripts/state.py --help runs successfully
• [ ] Manually tested affected script(s)
• [ ] Existing behavior is not broken

Agent Briefing: agent-deep-research

Structured reference for AI agents. Minimal prose, maximum signal.

Quick Start

# Check if configured
uv run {baseDir}/scripts/onboard.py --check

# Get full capabilities manifest (JSON)
uv run {baseDir}/scripts/onboard.py --agent

Capabilities

Decision Tree

"I need to research a topic"

Is there local context (files/code) to ground the research?
  YES -> Do you want to estimate cost first?
    YES -> uv run {baseDir}/scripts/research.py start "question" --context ./path --dry-run
    NO  -> uv run {baseDir}/scripts/research.py start "question" --context ./path --output report.md
  NO  -> uv run {baseDir}/scripts/research.py start "question" --output report.md

"I need to ask about uploaded documents"

Is the question simple/focused?
  YES -> uv run {baseDir}/scripts/store.py query <store-name> "question"
  NO (need deep analysis) -> uv run {baseDir}/scripts/research.py start "question" --store <name> --output report.md

"I want non-blocking research"

1. Start: RESULT=$(uv run {baseDir}/scripts/research.py start "question")
2. Extract ID: ID=$(echo "$RESULT" | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])")
3. Check: uv run {baseDir}/scripts/research.py status "$ID"
4. Save: uv run {baseDir}/scripts/research.py report "$ID" --output-dir ./output

"I need to build a document store"

1. Create: STORE=$(uv run {baseDir}/scripts/store.py create "name")
2. Upload: uv run {baseDir}/scripts/upload.py ./docs <store-name> --smart-sync
3. Query:  uv run {baseDir}/scripts/store.py query <store-name> "question"
4. Research: uv run {baseDir}/scripts/research.py start "question" --store <name>

Common Workflows

1. One-shot research with file context

uv run {baseDir}/scripts/research.py start "How does the auth system work?" \
  --context ./src --output report.md

Context store is created, files uploaded, research grounded, store cleaned up automatically.

2. Cost estimate before committing

uv run {baseDir}/scripts/research.py start "Analyze security architecture" \
  --context ./src --dry-run

Returns JSON cost estimate without starting research.

3. Structured output for downstream processing

uv run {baseDir}/scripts/research.py start "Deep analysis" \
  --output-dir ./research-output 2>/dev/null

Produces research-<id>/ directory with report.md, metadata.json, interaction.json, sources.json. Compact JSON summary on stdout.

4. Follow-up research

uv run {baseDir}/scripts/research.py start "Dive deeper into finding #3" \
  --follow-up <previous-interaction-id> --output followup.md

Important: Blocking Behavior

When --output or --output-dir is used, the command blocks until research completes (typically 2-10 minutes, up to 30+ for deep research). This is by design -- the report is only written after the Gemini API returns results.

DO NOT background the command with shell & -- this detaches the process and you lose the output. Instead:

• Use your agent framework's native background execution (e.g., run_in_background: true in Claude Code's Bash tool)
• Or use non-blocking mode (no --output flag), which returns immediately with {"id": "...", "status": "in_progress"}, then poll with status and retrieve with report --output

Non-blocking pattern (recommended for agent use):

# 1. Start (returns immediately)
RESULT=$(uv run {baseDir}/scripts/research.py start "question" 2>/dev/null)
ID=$(echo "$RESULT" | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])")

# 2. Poll until complete
uv run {baseDir}/scripts/research.py status "$ID" 2>/dev/null

# 3. Save when done
uv run {baseDir}/scripts/research.py report "$ID" --output report.md

Configuration Requirements

API key is checked from these env vars (first found wins): 1. GEMINI_DEEP_RESEARCH_API_KEY 2. GOOGLE_API_KEY 3. GEMINI_API_KEY

Output Contracts

All scripts: stderr = human-readable (Rich), stdout = JSON.

research.py start (non-blocking)

{"id": "interaction-abc123", "status": "in_progress"}

research.py start --output-dir (blocking)

{
  "id": "interaction-abc123",
  "status": "completed",
  "output_dir": "output/research-interaction-a/",
  "report_file": "output/research-interaction-a/report.md",
  "report_size_bytes": 45000,
  "duration_seconds": 154,
  "estimated_cost_usd": 1.22,
  "summary": "First 200 chars..."
}

research.py start --dry-run

{
  "type": "cost_estimate",
  "disclaimer": "Estimates only. Actual costs depend on research complexity, search depth, and API pricing changes.",
  "currency": "USD",
  "estimates": {
    "context_upload": {
      "files": 42,
      "total_bytes": 523000,
      "estimated_tokens": 130750,
      "estimated_cost_usd": 0.02
    },
    "research_query": {
      "estimated_input_tokens": 325000,
      "estimated_output_tokens": 60000,
      "estimated_cost_usd": 1.37,
      "basis": "historical_average | default_estimate"
    },
    "total_estimated_cost_usd": 1.39
  }
}

research.py status

{"id": "interaction-abc123", "status": "completed", "outputCount": 5}

store.py create

{"name": "fileSearchStores/abc123", "displayName": "My Store"}

store.py query

{"store": "fileSearchStores/abc123", "query": "...", "response": "..."}

metadata.json (in --output-dir)

{
  "id": "interaction-abc123",
  "status": "completed",
  "report_file": "output/research-interaction-a/report.md",
  "report_size_bytes": 45000,
  "output_count": 5,
  "source_count": 15,
  "duration_seconds": 154,
  "usage": {
    "disclaimer": "Estimates based on output size and pricing heuristics. Actual billing may differ.",
    "output_bytes": 45000,
    "estimated_output_tokens": 11250,
    "estimated_input_tokens": 250000,
    "estimated_cost_usd": 1.22,
    "context_files_uploaded": 0,
    "context_bytes_uploaded": 0,
    "source_urls_found": 15
  }
}

Error Handling

Common errors:

• No API key: Set env var, run onboard.py --check to verify
• Timeout: Increase --timeout, or use non-blocking mode and poll with status
• Store not found: Run store.py list to find valid store names
• Upload rejected: Check file size (<100 MB) and type (binary files are rejected)
• API rate limit: Wait and retry; the polling loop handles transient errors automatically

Pricing Reference

These are heuristic estimates (Gemini API does not return token counts):

Use --dry-run for per-query estimates.

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[2.1.2] - 2026-02-25

Fixed

• Output path validation: --output and --output-dir paths are now validated before starting research, preventing wasted API spend when the target directory doesn't exist
• Agent blocking behavior: added explicit warnings to AGENTS.md and SKILL.md about --output blocking behavior and the recommended non-blocking pattern for agent callers

[2.1.1] - 2026-02-24

Security

• CRITICAL: SSRF via WeasyPrint -- PDF export now uses a custom url_fetcher that blocks all URL fetching, preventing SSRF and local file exfiltration via malicious markdown (e.g., ![](file:///etc/passwd) or <img src="http://169.254.169.254/">)
• YAML frontmatter quoting -- all SKILL.md values containing colons are now properly double-quoted, and the metadata field is single-quoted. Fixes YAML parse failures in Codex, ClawHub scanner, and strict YAML parsers that caused the "suspicious" classification
• Follow-up sanitization hardened -- now strips ALL XML-like tags from previous research output (was only stripping </previous_findings>), preventing prompt injection via <system>, <instructions>, or delimiter escape attacks

Added

• CLAUDE.md -- project instructions covering security rules, ClawHub compliance, testing checklist, and release process for all future updates

[2.1.0] - 2026-02-24

Security

• Sensitive file filtering: --context uploads now automatically skip .env*, credentials.json, secrets.*, private keys (.pem, .key), auth tokens (.npmrc, .pypirc, .netrc), and build directories (node_modules, __pycache__, .git, dist, build). Applied to both research.py and upload.py. Skipped files are reported to stderr.
• API key echo removed: onboard.py --interactive no longer echoes back user-provided API keys to the terminal
• Cross-validated by Gemini 3.1 Pro and Codex

[2.0.4] - 2026-02-21

Fixed

• ClawHub metadata format: switched from nested YAML to inline JSON string for metadata field, matching the format that ClawHub's parser reliably extracts into structured registry data (bins, env vars, install spec). Nested YAML was parsed but not surfaced to the registry UI.

[2.0.3] - 2026-02-21

Security

• Complete env var declaration: added all 6 env vars the code reads to clawdis.requires.env (was only declaring 3 of 6). Added optional model config vars: GEMINI_DEEP_RESEARCH_AGENT, GEMINI_DEEP_RESEARCH_MODEL, GEMINI_MODEL
• All 3 API key vars added to clawdbot.config.requiredEnv (was only listing 1)
• Compatibility field updated to mention optional model config env vars

[2.0.2] - 2026-02-21

Security

• ClawHub registry metadata: added metadata.clawdis and metadata.clawdbot structured fields that the OpenClaw scanner reads for registry-level declarations: requires.bins (uv), requires.env (all 3 API key vars), primaryEnv (GOOGLE_API_KEY), homepage (GitHub URL), install spec (uv), and clawdbot.config.requiredEnv. Fixes "no required env vars", "no required binaries", "Source: unknown", and "instruction-only" scanner findings.

[2.0.1] - 2026-02-21

Security

• SKILL.md frontmatter: declared all requirements (compatibility, allowed-tools, metadata.required_env, metadata.required_binaries, metadata.primary_credential, metadata.file_upload_behavior, metadata.network_access, metadata.no_obfuscation, metadata.no_telemetry) to resolve OpenClaw "suspicious" classification caused by metadata mismatch
• Security & Transparency section added to SKILL.md body covering credentials handling, file upload scope, non-interactive mode risks, and no-obfuscation guarantee

[2.0.0] - 2026-02-21

Added

• Research depth modes (--depth {quick,standard,deep}) -- control research scope and expected duration; quick targets 2-5 min, deep targets 15-45 min with exhaustive analysis
• Research caching with content-hash deduplication -- identical queries return cached results instantly instead of re-running expensive API calls; cache entries auto-expire after 7 days; bypass with --no-cache
• Progress estimation during polling -- shows "~X% complete" based on adaptive history percentiles (p25/p50/p75) when sufficient data exists
• Cost ceiling guard (--max-cost USD) -- abort before starting if estimated cost exceeds the limit; prevents runaway spending from agent loops
• Query from file (--input-file PATH) -- read long/complex research queries from a file instead of the command line
• Store garbage collection (state.py gc) -- clean up orphaned ephemeral context stores older than 24 hours from crashed runs

Security

• Removed all curl | sh install patterns from documentation (VirusTotal supply chain risk flag)
• Sanitized --follow-up output with data delimiters to mitigate prompt injection from compromised previous research output

[1.3.1] - 2026-02-19

Changed

• Default model updated to gemini-3.1-pro-preview for file search store queries (was models/gemini-flash-latest)
• Documentation updated to reflect current model defaults

[1.3.0] - 2026-02-11

Added

• Output formats (--format {md,html,pdf}) -- export research reports as HTML (dark-themed, styled) or PDF (requires weasyprint). Markdown remains default and canonical format. (Issue #6)
• Prompt templates (--prompt-template {typescript,python,general,auto}) -- domain-specific prompt prefixes that optimize research for TypeScript/JavaScript or Python codebases. Auto-detect mode scans --context file extensions. (Issue #3)
• Edge case tests (tests/test_cost_estimation.py) -- 10 tests covering empty dirs, binary-only dirs, Unicode content, mixed history, zero/negative duration, and more. Run with uv run tests/test_cost_estimation.py. (Issue #5)
• Example research reports (docs/examples/) -- 3 real research outputs demonstrating the tool's capabilities: WebSocket vs SSE comparison, event sourcing patterns, WebAssembly state of the art. (Issue #4)

Changed

• Bumped GitHub Actions: actions/setup-python v5→v6, astral-sh/setup-uv v6→v7, actions/setup-node v4→v6

[1.2.3] - 2026-02-10

Added

• Use cases section in README with 7 domain-specific categories: trading & finance, competitive intelligence, software architecture, security audit prep, design & UX research, academic research & analysis, regulatory compliance -- all with copy-pasteable --context examples

[1.2.2] - 2026-02-09

Changed

• SKILL.md description optimized to ~75 tokens for agent context window efficiency
• Security & Trust section added to README (no obfuscation, no telemetry, fully auditable)
• GitHub topics expanded to 20 (added agent-skill, openclaw-skill, claude-code-skill, autonomous-agent, python, uv, and more)
• Pi agent installation instructions added to README
• ClawHub auto-publish on release via GitHub Actions (uses CLAWHUB_TOKEN secret)
• skills.sh re-index triggered automatically on release

Added

• 4 contributor-friendly GitHub issues seeded (good-first-issue, prompt-engineering, documentation, test-case)
• Pi agent (badlogic/pi-mono) and OpenClaw/ClawHub install instructions in README

[1.2.1] - 2026-02-08

Changed

• Skill renamed from agent-deep-research to deep-research in SKILL.md for better skills.sh search discoverability
• Description clarified: explicitly states Gemini Interactions API usage with no Gemini CLI dependency
• Description enriched: highlights automatic RAG grounding (--context), cost estimation (--dry-run), adaptive polling, and structured output

[1.2.0] - 2026-02-08

Added

• Agent onboarding (scripts/onboard.py) -- interactive setup wizard for humans (--interactive) and JSON capabilities manifest for AI agents (--agent), with quick config check (--check)
• AGENTS.md -- structured agent briefing with capabilities table, decision trees, output contracts, common workflows, and pricing reference
• Cost estimation (--dry-run flag on research.py start) -- preview estimated costs before running research, based on context file size and pricing heuristics
• Post-run usage metadata -- after research completes with --output-dir, metadata.json includes a usage key with estimated tokens, costs, context stats, and source counts
• "For AI Agents" section in SKILL.md pointing to onboard.py and AGENTS.md

[1.1.0] - 2026-02-09

Added

• `--context` flag (scripts/research.py) -- point at a local file or directory to automatically create an ephemeral file search store, upload files, and run RAG-grounded deep research in a single command
• `--context-extensions` flag -- filter which file types to upload from a context path (e.g. --context-extensions py,md)
• `--keep-context` flag -- prevent automatic cleanup of the ephemeral store after research completes, allowing reuse via --store
• Ephemeral context stores are tracked in .gemini-research.json under contextStores for cleanup visibility

[1.0.0] - 2026-02-08

Added

• Deep research (scripts/research.py) -- start background research jobs, check status, save reports via Google Gemini's deep research agent
• File search stores (scripts/store.py) -- create, list, query, and delete stores for RAG-grounded research
• File upload (scripts/upload.py) -- upload files and directories to file search stores with MIME type detection
• Session management (scripts/state.py) -- persistent workspace state for research sessions and store mappings
• Adaptive polling -- history-based poll interval tuning that learns from past research completion times (p25-p75 window targeting, separate curves for grounded vs non-grounded research)
• Structured output (--output-dir) -- save reports, metadata, interaction data, and extracted sources to a structured directory
• Smart sync (--smart-sync) -- hash-based file change detection to skip unchanged uploads
• JSON output (--json) -- machine-readable output on stdout for agent consumption
• Timeout control (--timeout) -- configurable maximum wait time for blocking operations
• Non-interactive mode -- automatic TTY detection to skip confirmation prompts for AI agent and CI integration
• PEP 723 inline metadata -- all scripts declare dependencies inline, run via uv run with zero pre-installation
• skills.sh distribution -- SKILL.md manifest for installation across 30+ AI coding agents
• CI workflow -- SKILL.md validation, py_compile, uv smoke test
• Dependabot -- automated GitHub Actions dependency updates

Changed

• Rebranded from gemini-cli-deep-research (Gemini CLI extension) to agent-deep-research (universal AI agent skill)
• Replaced Node.js MCP server and TOML commands with Python CLI scripts
• License clarified as MIT (was labeled ISC in some places)

Removed

• Node.js MCP server (src/index.ts, package.json, tsconfig.json, etc.)
• Gemini CLI TOML commands (commands/deep-research/*.toml)
• ESLint, Prettier, Jest configuration
• Build infrastructure (build.mjs, release/)

[2.1.2]: https://github.com/24601/agent-deep-research/compare/v2.1.1...v2.1.2 [2.1.1]: https://github.com/24601/agent-deep-research/compare/v2.1.0...v2.1.1 [2.1.0]: https://github.com/24601/agent-deep-research/compare/v2.0.4...v2.1.0 [2.0.4]: https://github.com/24601/agent-deep-research/compare/v2.0.3...v2.0.4 [2.0.3]: https://github.com/24601/agent-deep-research/compare/v2.0.2...v2.0.3 [2.0.2]: https://github.com/24601/agent-deep-research/compare/v2.0.1...v2.0.2 [2.0.1]: https://github.com/24601/agent-deep-research/compare/v2.0.0...v2.0.1 [2.0.0]: https://github.com/24601/agent-deep-research/compare/v1.3.1...v2.0.0 [1.3.1]: https://github.com/24601/agent-deep-research/compare/v1.3.0...v1.3.1 [1.3.0]: https://github.com/24601/agent-deep-research/compare/v1.2.3...v1.3.0 [1.2.3]: https://github.com/24601/agent-deep-research/compare/v1.2.2...v1.2.3 [1.2.2]: https://github.com/24601/agent-deep-research/compare/v1.2.1...v1.2.2 [1.2.1]: https://github.com/24601/agent-deep-research/compare/v1.2.0...v1.2.1 [1.2.0]: https://github.com/24601/agent-deep-research/compare/v1.1.0...v1.2.0 [1.1.0]: https://github.com/24601/agent-deep-research/compare/v1.0.0...v1.1.0 [1.0.0]: https://github.com/24601/agent-deep-research/releases/tag/v1.0.0

Project Instructions

Security Rules (apply to ALL updates)

1. YAML frontmatter quoting: All SKILL.md frontmatter values containing colons MUST be double-quoted. The metadata field MUST be a single-quoted JSON string (not nested YAML). Test with python3 -c "import yaml; yaml.safe_load(open('SKILL.md').read().split('---')[1])" before committing.

2. Sensitive file filtering: The _collect_files function in research.py and collect_files in upload.py MUST skip .env*, credentials.json, secrets.*, private keys (.pem, .key), and auth tokens (.npmrc, .pypirc, .netrc). Never remove this filtering.

3. WeasyPrint SSRF protection: The PDF export (--format pdf) MUST use a custom url_fetcher that blocks all URL fetching. Never pass WeasyPrint an HTML string without this protection -- AI-generated markdown can contain ![](file:///etc/passwd) or SSRF payloads.

4. Follow-up sanitization: The --follow-up feature MUST strip all XML-like tags from previous research output before injecting it into the new query. Use re.sub(r"<[^>]{1,50}>", "", text) to prevent prompt injection via <system>, <tool_call>, or delimiter escape attacks.

5. No API key echo: Never echo user-provided API keys back to the terminal, even in "helpful" export commands. Use <your-key> placeholder instead.

6. No curl|sh patterns: Never use curl ... | sh install patterns in documentation. Link to official install docs pages instead.

7. No telemetry: This project has zero telemetry, analytics, or tracking. Never add any.

ClawHub Compliance

• The metadata field in SKILL.md MUST be a single-quoted JSON string (not nested YAML, not unquoted JSON). ClawHub's getFrontmatterMetadata calls JSON.parse() on string values.
• The JSON MUST include clawdbot (not clawdis) with: primaryEnv, homepage, requires.bins, requires.env, install, and config.requiredEnv.
• All 6 env vars the code reads MUST be declared in requires.env.
• Run npx clawhub inspect agent-deep-research after publishing to verify structured data appears.
• The ClawHub security scanner is LLM-based and reads both the SKILL.md content and the registry metadata. Ensure consistency between what the code does and what the metadata declares.

Testing Checklist (before every release)

1. python3 -m py_compile scripts/research.py scripts/store.py scripts/upload.py scripts/state.py scripts/onboard.py 2. uv run tests/test_cost_estimation.py (10 tests must pass) 3. uv run scripts/research.py start "test" --dry-run (valid JSON on stdout) 4. uv run scripts/research.py start "test" --context ./scripts --dry-run (auto-detect python template) 5. uv run scripts/research.py start "test" --max-cost 0.001 (should abort) 6. uv run scripts/onboard.py --check (should show config status) 7. YAML validation: python3 -c "import yaml; yaml.safe_load(open('SKILL.md').read().split('---')[1])" 8. Verify no curl|sh patterns: grep -r 'curl.*install.*sh' scripts/ *.md 9. Verify sensitive file filter: create a dir with .env + code.py, run --context --dry-run, verify .env is skipped

Release Process

1. Bump version in SKILL.md (inside the metadata JSON string) and scripts/onboard.py 2. Update CHANGELOG.md 3. Run full testing checklist above 4. git commit && git tag vX.Y.Z && git push origin main --tags 5. GHA auto-publishes to GitHub Releases + ClawHub + triggers skills.sh re-index 6. After publish: npx clawhub inspect agent-deep-research to verify 7. Fix symlink if overwritten: rm -rf ~/.agents/skills/deep-research && ln -s $(pwd) ~/.agents/skills/deep-research

Architecture Notes

• All scripts are PEP 723 standalone (inline metadata, run via uv run)
• Dual output convention: stderr = Rich human-readable, stdout = machine-readable JSON
• State file .gemini-research.json is local, contains no credentials
• The deep research agent identifier deep-research-pro-preview-12-2025 is the Interactions API agent name, not a model name
• The default query model in store.py is gemini-3.1-pro-preview

Contributor Covenant Code of Conduct

Our Pledge

We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation.

We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.

Our Standards

Examples of behavior that contributes to a positive environment for our community include:

• Demonstrating empathy and kindness toward other people
• Being respectful of differing opinions, viewpoints, and experiences
• Giving and gracefully accepting constructive feedback
• Accepting responsibility and apologizing to those affected by our mistakes,

and learning from the experience

• Focusing on what is best not just for us as individuals, but for the overall

community

Examples of unacceptable behavior include:

• The use of sexualized language or imagery, and sexual attention or advances of

any kind

• Trolling, insulting or derogatory comments, and personal or political attacks
• Public or private harassment
• Publishing others' private information, such as a physical or email address,

without their explicit permission

• Other conduct which could reasonably be considered inappropriate in a

professional setting

Enforcement Responsibilities

Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful.

Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.

Scope

This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official email address, posting via an official social media account, or acting as an appointed representative at an online or offline event.

Enforcement

Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at [INSERT CONTACT METHOD].

All complaints will be reviewed and investigated promptly and fairly.

All community leaders are obligated to respect the privacy and security of the reporter of any incident.

Enforcement Guidelines

Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:

1. Correction

Community Impact: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.

Consequence: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.

2. Warning

Community Impact: A violation through a single incident or series of actions.

Consequence: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.

3. Temporary Ban

Community Impact: A serious violation of community standards, including sustained inappropriate behavior.

Consequence: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.

4. Permanent Ban

Community Impact: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.

Consequence: A permanent ban from any sort of public interaction within the community.

Attribution

This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.1, available at [https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].

Community Impact Guidelines were inspired by [Mozilla's code of conduct enforcement ladder][Mozilla CoC].

For answers to common questions about this code of conduct, see the FAQ at [https://www.contributor-covenant.org/faq][FAQ]. Translations are available at [https://www.contributor-covenant.org/translations][translations].

[homepage]: https://www.contributor-covenant.org [v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html [Mozilla CoC]: https://github.com/mozilla/diversity [FAQ]: https://www.contributor-covenant.org/faq [translations]: https://www.contributor-covenant.org/translations

Contributing to agent-deep-research

Thanks for your interest in contributing. This document covers the development workflow and conventions.

Prerequisites

• Python 3.10+
• uv (see install docs)
• A Google API key (set GOOGLE_API_KEY or GEMINI_API_KEY)

Development Setup

git clone https://github.com/24601/agent-deep-research.git
cd agent-deep-research

No virtual environment or pip install needed -- all scripts use PEP 723 inline metadata and run directly via uv run.

Running Locally

# Verify syntax
python3 -m py_compile scripts/research.py scripts/store.py scripts/upload.py scripts/state.py

# Smoke test
uv run scripts/state.py --help

# Manual integration tests (requires a Google API key)
uv run scripts/research.py start "test query" --timeout 120
uv run scripts/store.py list

Code Style

• PEP 8 for Python formatting
• PEP 723 inline script metadata (dependencies declared in each script, not a central requirements.txt)
• Dual-output convention: stderr for human-readable output (rich formatting), stdout for machine-readable JSON
• Keep scripts self-contained -- each script in scripts/ should be independently runnable via uv run

Commit Convention

This project uses Conventional Commits:

feat: add store export command
fix: handle empty API response in research polling
docs: update README with new flags
chore: update CI workflow

Prefixes: feat, fix, docs, chore, refactor, test, ci

Pull Request Process

1. Fork the repository and create a feature branch from main 2. Make your changes 3. Verify your changes:

   python3 -m py_compile scripts/*.py
   uv run scripts/state.py --help

4. Commit using the conventional commit format 5. Open a pull request against main 6. Fill out the PR template

Reporting Issues

Use the issue templates to report bugs or request features. For security vulnerabilities, see SECURITY.md.

License

By contributing, you agree that your contributions will be licensed under the MIT License.

Credits

Original Project

This project was forked from allenhutchison/gemini-cli-deep-research by Allen Hutchison.

What Was Inherited

• Gemini API integration concept (deep research + file search)
• MIME type research and documentation (docs/file-search-mime-types.md)
• Original ISC license (relicensed to MIT)

What Is New

The following were built from scratch for the standalone skill:

• Python CLI scripts (scripts/research.py, scripts/store.py, scripts/upload.py, scripts/state.py) -- PEP 723 inline metadata, runs via uv run with zero pre-installation
• SKILL.md packaging -- skills.sh-compatible skill manifest for distribution to 30+ AI agents
• Adaptive history-based polling -- learns from past research completion times to optimize poll intervals (p25-p75 window targeting, separate curves for grounded vs non-grounded)
• Disk output (--output-dir) -- structured directory output with report, metadata, interaction data, and extracted sources
• Smart sync (--smart-sync) -- hash-based file change detection to skip unchanged uploads
• Non-interactive agent mode -- automatic TTY detection to skip confirmation prompts
• JSON output (--json) -- machine-readable output on stdout for agent consumption
• Timeout control (--timeout) -- configurable maximum wait time for blocking operations
• Critique-driven hardening -- 3 independent AI critics + live testing used to identify and fix edge cases

Original Node.js/MCP Artifacts (Removed)

The original project included a Node.js MCP server (src/index.ts), TOML commands for Gemini CLI (commands/), and associated build infrastructure. These were removed during the rebrand to agent-deep-research as the Python CLI scripts provide equivalent functionality with simpler distribution.

<!-- Generated by agent-deep-research --> <!-- Query: Explain the event sourcing pattern in distributed systems. Cover benefits, drawbacks, implementation strategies, and when it's appropriate vs traditional CRUD. --> <!-- Date: 2026-02-11 --> <!-- Model: deep-research-pro-preview-12-2025 -->

Comprehensive Analysis of the Event Sourcing Pattern in Distributed Systems

Key Points

• Fundamental Shift: Event Sourcing shifts data persistence from storing the current state (CRUD) to storing the sequence of events that led to that state.
• Primary Benefits: It provides inherently perfect audit trails, enables temporal queries ("time travel"), and supports complex business logic where the "why" and "how" of a state change are as important as the result.
• Significant Challenges: It introduces complexity regarding eventual consistency, schema evolution (versioning events), and storage management. It often requires the Command Query Responsibility Segregation (CQRS) pattern to be practical.
• Strategic Usage: It is not a "silver bullet." It is best applied to specific "bounded contexts" within a system that require high auditability or complex state transitions, rather than as a monolithic architecture for an entire application.

1. Introduction

In the landscape of distributed systems architecture, data persistence and state management are foundational challenges. The traditional approach, known as CRUD (Create, Read, Update, Delete), treats the database as a repository of the current state of an entity. However, as modern systems require higher levels of auditability, scalability, and historical analysis, an alternative pattern known as Event Sourcing has gained prominence.

Event Sourcing is an architectural pattern where the state of a system is not stored directly. Instead, the system persists the sequence of state-changing events that occurred over time. To obtain the current state, the system replays these events. This approach aligns closely with real-world accounting ledgers, where the current balance is derived from the sum of all past transactions rather than just a single mutable value.

This report provides an exhaustive analysis of the Event Sourcing pattern, exploring its mechanics, the symbiosis with Command Query Responsibility Segregation (CQRS), implementation strategies such as snapshotting and upcasting, and the critical decision framework for choosing between Event Sourcing and traditional CRUD.

2. Core Mechanics of Event Sourcing

2.1 The Immutable Log

At the heart of Event Sourcing is the Event Store, an append-only database. Unlike relational databases in a CRUD model, where rows are overwritten to reflect updates, the Event Store forbids the mutation or deletion of existing records [cite: 1, 2]. Every change to the domain is captured as an immutable event object.

An event represents a fact that has taken place in the past. Therefore, events are always named in the past tense (e.g., OrderPlaced, PaymentReceived, AddressUpdated) [cite: 2, 3].

2.2 Stream-Based Storage

Events are typically organized into streams. A stream corresponds to a specific entity or "Aggregate" in Domain-Driven Design (DDD) terminology. For example, a single bank account would have its own event stream containing all deposits, withdrawals, and transfers associated with that specific account ID [cite: 4, 5].

To reconstruct the state of an entity (a process often called "rehydration"), the system: 1. Queries the Event Store for the stream associated with the entity ID. 2. Loads the history of events in chronological order. 3. Applies each event to a fresh instance of the entity, mutating its in-memory state sequentially. 4. Produces the final, current state ready for processing new commands [cite: 4].

2.3 Atomicity and Concurrency

In distributed systems, handling concurrent updates is critical. Event Sourcing manages this via Optimistic Concurrency Control. When a command attempts to append a new event to a stream, it specifies the expected version (or sequence number) of that stream. If another process has appended an event in the interim, the version numbers will mismatch, and the write will be rejected. This ensures that no data is lost due to race conditions without requiring heavy database locks [cite: 6, 7].

3. Event Sourcing vs. Traditional CRUD

The distinction between Event Sourcing and CRUD is philosophical as well as technical. CRUD focuses on structural data storage, whereas Event Sourcing focuses on behavioral data storage.

3.1 Comparative Analysis

3.2 The Information Gap

A critical deficiency of CRUD is the loss of context. If a user's address changes in a CRUD system, the database reflects the new address. The system loses the information regarding what the old address was, when it was changed, and why (e.g., was it a correction or a move?). Event Sourcing preserves the "customer journey," retaining data that CRUD simply throws away [cite: 1].

As noted in architectural literature, CRUD treats the database like a "blackboard" that is constantly erased and rewritten, whereas Event Sourcing treats it like a "ledger" [cite: 8].

4. Benefits of Event Sourcing

4.1 Auditability and Compliance

For regulated industries (finance, healthcare, legal), Event Sourcing provides a 100% reliable audit log. Since the state is derived from the log, the log cannot arguably be out of sync with the state. This is superior to ad-hoc audit logging in CRUD, where the audit table and the data table are updated separately and can diverge [cite: 9, 10].

4.2 Temporal Queries and "Time Travel"

Event Sourcing enables the reconstruction of the system state at any previous point in time. This is invaluable for:

• Debugging: Developers can copy the production event stream to a test environment and replay it to the exact moment a bug occurred to reproduce the state [cite: 11].
• Retroactive Analysis: Businesses can ask questions that were not anticipated when the data was recorded (e.g., "What would our inventory levels have been last year if we used this new calculation logic?") [cite: 2, 12].

4.3 High-Performance Writes

In high-throughput distributed systems, locking rows for updates can become a bottleneck. Event Sourcing operations are strictly append-only. Appending to a log is a constant-time ($O(1)$) operation that requires minimal locking, allowing for extremely high write throughput [cite: 13, 14].

4.4 Decoupling and Side Effects

Event Sourcing naturally facilitates Event-Driven Architecture (EDA). Because every state change is an event, other microservices can subscribe to these events to trigger their own workflows. For example, an OrderCreated event can be consumed by the Shipping Service, the Notification Service, and the Analytics Service independently, decoupling the Order Service from downstream dependencies [cite: 9, 15].

5. Challenges and Drawbacks

Despite its power, Event Sourcing is widely considered a complex pattern with a steep learning curve [cite: 9, 16].

5.1 Eventual Consistency

Because the write model (Event Store) and the read models (Projections) are often separated (see Section 6.1 on CQRS), there is a latency gap. A user might perform an action and not see the result immediately. This "eventual consistency" requires careful UI design and handling of user expectations, which is significantly more complex than the immediate consistency of ACID-compliant CRUD transactions [cite: 2, 10].

5.2 Versioning and Schema Evolution

In CRUD, if a data model changes, developers run a database migration script to alter the table structure. In Event Sourcing, events are immutable and cannot be changed. If the business logic changes (e.g., splitting a Name field into FirstName and LastName), the system must be able to handle both old events (pre-change) and new events (post-change). This leads to complex versioning strategies (discussed in Section 7) [cite: 17, 18].

5.3 Storage Growth and Replay Performance

The event log grows indefinitely. Replaying a stream with 100 events is fast; replaying one with 1,000,000 events is prohibitively slow. This necessitates the implementation of optimization strategies like snapshots, adding operational overhead [cite: 4]. Furthermore, strict events cannot be easily deleted, which complicates compliance with "Right to be Forgotten" laws (see Section 7.2) [cite: 19].

5.4 Complexity of Tooling

Standard ORMs (like Hibernate or Entity Framework) are designed for CRUD. Event Sourcing often requires specialized databases (EventStoreDB) or custom frameworks. Poor implementation of these tools can lead to disastrous results, as evidenced by engineering teams that have had to rewrite event-sourced systems back to CRUD due to performance and maintenance issues [cite: 16, 20].

6. Implementation Strategies

To make Event Sourcing practical in production distributed systems, several supporting patterns and strategies are required.

6.1 CQRS (Command Query Responsibility Segregation)

Event Stores are excellent for writing (appending) and reading a single stream by ID. They are terrible for complex queries (e.g., "Find all users named John who ordered in the last month"). To solve this, Event Sourcing is almost universally paired with CQRS.

• Write Side: The Event Store. Handles commands and appends events.
• Read Side: One or more Projections (Materialized Views). These are standard databases (SQL, NoSQL, Elasticsearch) that subscribe to the event stream and update their state accordingly.

Queries are executed against the Read Side, which is optimized for retrieval, while writes go to the Write Side, optimized for transactional integrity [cite: 3, 9, 21].

6.2 Snapshotting

To mitigate the performance cost of replaying long event streams, systems utilize Snapshots. A snapshot is a serialized record of an entity's state at a specific version.

• Process: Instead of reading events 0 to 10,000, the system reads the snapshot created at version 9,900 and then replays only events 9,901 to 10,000 [cite: 4].
• Strategy: Snapshots can be taken asynchronously every $N$ events. They are an optimization, not a source of truth; if a snapshot is corrupted, it can be deleted and regenerated from the event log [cite: 5, 22].

6.3 Idempotency

In distributed systems, networks fail, and messages may be delivered more than once (at-least-once delivery). If a "Deposit $100" event is processed twice, the balance is corrupted. Consumers and projections must be idempotent.

• Implementation: Projections track the ID of the last processed event for each aggregate. If an incoming event has an ID lower than or equal to the last processed ID, it is discarded as a duplicate [cite: 3, 23].

6.4 Handling Side Effects During Replay

When replaying events to fix a bug or rebuild a read model, the system must not re-trigger external side effects (e.g., sending emails, charging credit cards).

• Strategy: The domain model should generate events, but not execute side effects directly. Side effects should be handled by "Process Managers" or "Event Handlers" listening to the stream. During a replay, these external handlers are disabled, or the system effectively runs in a "read-only" mode where state is updated but external gateways are mocked or disconnected [cite: 24, 25, 26].

7. Advanced Implementation Topics

7.1 Schema Evolution and Upcasting

Because events are immutable, developers cannot "fix" a bad event schema in the database. Several strategies exist to handle schema changes over time: 1. Multiple Versions: Support UserCreatedV1 and UserCreatedV2 in the code logic. This can lead to bloated code (switch statements) [cite: 27]. 2. Upcasting: A middleware layer transforms old events into the new schema structure on the fly as they are loaded from the database. The application logic only ever deals with the latest version of the event, keeping the domain code clean. The database retains the original event [cite: 17, 28]. 3. Weak Schema: Using flexible formats like JSON allows adding optional fields without breaking deserialization of old events [cite: 17].

7.2 GDPR and Crypto-Shredding

The General Data Protection Regulation (GDPR) includes a "Right to Erasure." This conflicts with the immutability of the Event Store.

• Crypto-Shredding: Personal Identifiable Information (PII) within an event is encrypted using a specific key for that user. When the user requests deletion, the system deletes the key. The data remains in the event log but is cryptographically unreadable, effectively rendering it deleted without breaking the chain of immutability [cite: 19, 29, 30].

8. When to Use Event Sourcing vs. CRUD

Choosing between these patterns is a risk management decision.

8.1 When to Use CRUD

• Simple Domains: Basic Content Management Systems (CMS), To-Do apps, or systems with low business logic complexity [cite: 8, 31].
• Low Value Data: Data where history is irrelevant, such as IoT sensor raw feeds where only the aggregate value matters eventually [cite: 13].
• Small Teams/Tight Deadlines: CRUD is faster to build initially and requires less specialized knowledge [cite: 10, 32].

8.2 When to Use Event Sourcing

• High Audit Requirements: Banking, Insurance, Supply Chain, where proving the history of interactions is a legal or core business requirement [cite: 8].
• Complex Business Logic: Domains where the intent of the user (the "Why") is as important as the data change.
• Collaborative Domains: Systems where multiple users modify the same resources, and conflict resolution is necessary (e.g., Google Docs style collaboration) [cite: 33].
• Analytics and Data Mining: When the business anticipates needing to analyze behavior patterns in the future that are currently undefined [cite: 34, 35].

9. Conclusion

Event Sourcing represents a paradigm shift from data-centric to behavior-centric architecture. It offers unparalleled benefits in terms of auditability, resilience, and business insight, but extracts a high price in terms of complexity and operational overhead. It requires a maturity in distributed systems design, specifically regarding eventual consistency, versioning, and CQRS.

For academic and professional architects, the recommendation is rarely binary. The most effective architectures are often hybrid, applying Event Sourcing strictly to the core "Bounded Contexts" of the system where the business value of the history outweighs the complexity cost, while utilizing simpler CRUD models for supporting sub-domains [cite: 14, 31].

Sources: 1. confluent.io 2. aws.com 3. microsoft.com 4. cqrs.com 5. kurrent.io 6. amazon.com 7. researchgate.net 8. dev.to 9. microservices.io 10. medium.com 11. dev.to 12. youtube.com 13. risingstack.com 14. baytechconsulting.com 15. dzone.com 16. reddit.com 17. codemia.io 18. eventsourcing.ai 19. cqrs.com 20. reddit.com 21. medium.com 22. stackoverflow.com 23. medium.com 24. domaincentric.net 25. eventsourcingdb.io 26. stackexchange.com 27. stackoverflow.com 28. artium.ai 29. patchlevel.de 30. kurrent.io 31. event-driven.io 32. medium.com 33. tamara.co 34. medium.com 35. bemi.io

<!-- Generated by agent-deep-research --> <!-- Query: Analyze the current state of WebAssembly (Wasm) adoption in 2025-2026. Cover the component model, WASI, edge computing use cases, and language support ecosystem. --> <!-- Date: 2026-02-11 --> <!-- Model: deep-research-pro-preview-12-2025 -->

The State of WebAssembly 2025-2026: Standards, Ecosystem, and Adoption Dynamics

Executive Summary

By early 2026, WebAssembly (Wasm) has completed its transition from a browser-centric optimization tool to a universal, polyglot runtime standard for cloud-native and edge computing environments. The period between 2025 and 2026 marked the stabilization of critical infrastructure, most notably the WebAssembly System Interface (WASI) 0.3 and the maturation of the Component Model. While browser adoption has seen steady growth—powering 5.5% of websites visited by Chrome users [cite: 1, 2]—the most transformative developments have occurred "behind the scenes" in serverless architectures, edge AI, and secure plugin ecosystems.

The ecosystem has coalesced around the Component Model as the definitive solution for software composability, effectively solving the "write once, run anywhere" promise that previous technologies struggled to fully realize. With the release of WebAssembly 3.0, including Garbage Collection (WasmGC), managed languages like Kotlin, Dart, and Java have joined Rust and C++ as first-class citizens in the Wasm landscape. This report analyzes the technical milestones, language support maturity, and architectural shifts defining the WebAssembly landscape in the 2025-2026 timeframe.

Key Developments 2025-2026

• WASI 0.3 & Native Async: Released in February 2026, WASI 0.3 introduced native asynchronous I/O support, a critical requirement for high-concurrency server-side workloads, bridging the gap between Wasm and traditional async runtimes like Node.js or Go [cite: 1, 3].
• WasmGC & Wasm 3.0: The standardization of WebAssembly 3.0, specifically Garbage Collection (GC) and Memory64, has removed the need for managed languages to ship their own heavy runtimes, drastically reducing binary sizes and improving performance for languages like Kotlin and C# [cite: 1, 4].
• Edge AI Standardization: WASI-NN (Neural Networks) has emerged as the standard interface for edge inference, allowing Wasm modules to perform tensor operations on diverse hardware backends without platform-specific code [cite: 5, 6].
• Polyglot Components: The "Component Model" has moved from theoretical specs to production reality, enabling "Lego-block" style applications where a Rust component can call a Python component directly, managing memory and types safely across boundaries [cite: 7, 8].

---

1. The Component Model and WASI Evolution

The most significant structural change in the WebAssembly ecosystem during 2025-2026 is the solidification of the Component Model and the WebAssembly System Interface (WASI). These technologies have moved Wasm beyond simple instruction execution into a modular, capability-secure platform.

1.1 The Component Model: Modular Architecture

The Component Model, stabilizing throughout 2025, addresses the "shared-nothing" architecture of Wasm. Unlike traditional linking where libraries share memory space (creating security vulnerabilities), Wasm components communicate through typed interfaces (WIT - WebAssembly Interface Type) while maintaining memory isolation.

• Composability: By 2026, developers are leveraging the Component Model to build polyglot applications. A primary use case involves writing performance-critical logic in Rust and business logic in Python or JavaScript, compiling them into composable Wasm components that interact seamlessly without heavy Foreign Function Interfaces (FFI) [cite: 7, 9].
• WIT (WebAssembly Interface Type): The interface definition language (IDL) for components has matured. Tools like wit-bindgen now automatically generate idiomatic types for host and guest languages, abstracting the complexity of memory pointers and offsets [cite: 10, 11].
• Registry and Distribution: The ecosystem has begun standardizing on WARG (WebAssembly Registry), a protocol for distributing Wasm components. This federated approach supports the "worlds" concept—interface definitions that describe the capabilities a component needs (imports) and provides (exports) [cite: 12, 13].

1.2 WASI 0.2: The Foundation (Stable)

WASI 0.2 (formerly Preview 2), which stabilized in 2024, remains the bedrock for current production deployments in early 2026. It introduced the concept of "Worlds" and standardized interfaces for HTTP, CLI, and filesystem access (wasi-http, wasi-cli). This release successfully decoupled WASI from POSIX-strict adherence, adopting a capability-based security model more appropriate for cloud-native environments [cite: 14, 15].

1.3 WASI 0.3: The Async Revolution (2026)

The release of WASI 0.3 in February 2026 represents the "last lap" in WebAssembly’s race toward ubiquitous server-side adoption.

• Native Async Support: Previous versions of WASI relied on blocking I/O, which severely limited performance in high-concurrency environments (like web servers). WASI 0.3 introduces native future and stream types to the Component Model. This allows Wasm components to handle asynchronous operations natively, yielding control back to the runtime during I/O waits without custom workarounds [cite: 3, 9].
• Standardization of Concurrency: This release enables "composable concurrency," where async components from different languages can interoperate without needing to understand each other's event loops. For example, a Rust component using tokio concepts can seamlessly await a result from a JavaScript component [cite: 7, 15].
• Roadmap to 1.0: Following WASI 0.3, the ecosystem is preparing for WASI 1.0, expected in late 2026 or 2027. This will signal full stability for the entire stack, including threading support which is currently following the async implementation [cite: 2].

---

2. Core WebAssembly Standards (Wasm 3.0)

While WASI defines how Wasm interacts with the world, the Core Wasm specification defines how code executes. The 2025-2026 period saw the arrival of WebAssembly 3.0.

2.1 WebAssembly Garbage Collection (WasmGC)

WasmGC is arguably the most critical feature for expanding language support. Prior to WasmGC, managed languages (Java, Python, C#, Kotlin) had to compile their own garbage collector into the Wasm binary, leading to bloated file sizes and poor performance.

• Browser Support: By 2026, all major browsers (Chrome, Firefox, Safari) fully support WasmGC. Safari’s inclusion of WasmGC and Exception Handling was the final piece of the cross-browser puzzle [cite: 1, 2].
• Impact: Languages like Kotlin have released beta Wasm compilers that leverage the host's (browser's) garbage collector. This results in binaries that are significantly smaller and start faster, as they no longer ship a runtime VM inside the Wasm module [cite: 4, 16].

2.2 Memory64 and Relaxed SIMD

• Memory64: Included in Wasm 3.0, this allows Wasm modules to address more than 4GB of memory (using 64-bit indexes). This is crucial for data-intensive edge workloads, such as running large AI models or in-memory databases, though browser implementations still often cap memory usage for safety [cite: 1, 17].
• Relaxed SIMD: This feature relaxes the strict determinism of Single Instruction, Multiple Data (SIMD) operations to gain performance by using platform-specific vector instructions. This is particularly beneficial for AI inference and multimedia processing in the browser [cite: 2, 16].

---

3. Language Support Ecosystem

The tooling landscape in 2026 is defined by the dichotomy of "native" support versus "adapter" tooling.

3.1 Tier 1: Rust (The Gold Standard)

Rust remains the premier language for WebAssembly development. Its lack of a garbage collector and strong ownership model align perfectly with Wasm’s linear memory.

• Tooling: cargo-component and wasm-bindgen provide a seamless experience. Rust was among the first to support WASI 0.2 and 0.3 features natively [cite: 10, 11].
• Async: Rust's async/await model maps efficiently to the new WASI 0.3 async primitives, making it the language of choice for writing high-performance Wasm microservices [cite: 9].

3.2 Tier 2: JavaScript and TypeScript

Support for JavaScript has evolved from running full JS engines (like QuickJS) inside Wasm to more sophisticated componentization.

• JCO: The jco toolchain (JavaScript Component Tools) allows developers to "transpile" Wasm components into native JavaScript modules for running in browsers or Node.js. Conversely, it can wrap JavaScript code into Wasm components, enabling JS to participate in the Component Model [cite: 9, 18].
• StarlingMonkey: A Wasm-native JS engine that aids in creating efficient components from JS source code, bridging the gap between dynamic typing and Wasm's static interface types [cite: 18].

3.3 Tier 3: Python

Python support has surged due to the demand for AI workloads at the edge.

• Componentize-Py: This tool converts Python applications into WebAssembly components. It bundles a lightweight Python interpreter and the necessary scripts into a single component that targets a specific WIT world [cite: 19, 20].
• Use Cases: Python is primarily used in Wasm for data processing and AI inference (via WASI-NN) rather than high-performance systems programming. The ability to import optimized Rust components into Python Wasm code is a key workflow [cite: 8, 21].

3.4 Tier 4: Go (TinyGo vs. Standard Go)

Go's relationship with Wasm is bifurcated:

• TinyGo: This alternative compiler is the preferred choice for Wasm components. It supports the wasip2 target (WASI 0.2) and produces small, efficient binaries suitable for edge computing and embedded devices [cite: 22, 23].
• Standard Go: While the standard Go compiler supports Wasm, the generated binaries are large (often 2MB+ for "Hello World") because they include the full Go runtime and GC. It is less suitable for the Component Model compared to TinyGo [cite: 9, 24].

3.5 .NET (C#)

Microsoft has invested heavily in Wasm for its Blazor framework and beyond.

• Performance: .NET 10 (released Nov 2025) introduced significant AOT (Ahead-of-Time) compilation improvements, reducing download sizes and improving startup speed. .NET 11 planning focuses on transitioning to the CoreCLR runtime for Wasm [cite: 1, 2].
• Multithreading: Collaboration with the Uno Platform is actively bringing multithreading support to .NET on Wasm, leveraging the new Wasm standards [cite: 1, 25].

---

4. Edge Computing and Serverless Use Cases

By 2026, the "Wasm vs. Containers" debate has settled into a pragmatic coexistence. Wasm is not replacing Docker for long-running, heavy services, but it is dominating the "scale-to-zero" and high-density edge computing space.

4.1 Cold Starts and Density

Wasm's millisecond-level startup times allow edge providers (Cloudflare, Fastly, Fermyon) to run thousands of "sleeping" applications on a single server, waking them only when a request arrives. This density is 10-100x higher than traditional container orchestration [cite: 8, 9].

4.2 Edge AI and WASI-NN

The convergence of AI and Edge computing is a primary driver for Wasm adoption in 2026.

• WASI-NN: The WebAssembly System Interface for Neural Networks (WASI-NN) allows Wasm modules to offload heavy matrix multiplications to the host's hardware acceleration (GPU/TPU) while keeping the application logic portable.
• Inference: Common use cases include deploying privacy-preserving PII redaction, image classification, or voice processing models directly to edge nodes or IoT devices. The Wasm module handles the pre/post-processing, while WASI-NN handles the inference via TensorFlow Lite or OpenVINO [cite: 5, 6].

4.3 Secure Plugins

Beyond the cloud, Wasm is widely adopted as a plugin architecture for SaaS platforms. Companies like Figma (design) and VS Code (Microsoft) use Wasm to safely execute user-submitted code or complex logic (like Rust-based search engines) on the client side, isolating it from the main application thread [cite: 8, 16, 26].

---

5. Adoption Statistics and Trends

• Web Usage: Approximately 5.5% of websites visited by Chrome users employ WebAssembly. While this percentage seems low, it represents high-value, complex applications (e.g., Photoshop Web, Figma, Google Earth) rather than simple content sites [cite: 1, 2].
• Browser Parity: With Safari 18.4, all major browsers now support the critical Wasm feature set, including Exception Handling and Tail Calls, effectively ending the "Safari lag" that hindered adoption in previous years [cite: 2, 25].
• Enterprise Adoption: Financial and healthcare sectors are adopting Wasm for secure data processing "rooms" (enclaves) where code must be verified and sandboxed, leveraging Wasm's capability-based security model [cite: 27].

---

6. Challenges and Future Outlook

Despite the progress, several challenges remain in 2026:

• Threading Complexity: While native async is arriving with WASI 0.3, full multithreading support (shared memory concurrency) is still maturing. Proposals are approved, but widespread implementation across all languages and runtimes is inconsistent [cite: 3, 28].
• Debugging: Debugging has improved with DWARF support and IDE integration (debugging Wasm directly in VS Code), but it remains more complex than debugging native JavaScript or Python [cite: 1, 11].
• GPU Access: While WASI-NN abstracts inference, a direct, standardized interface for general-purpose GPU compute (WebGPU via WASI) is still a work in progress, limiting Wasm's use for training models or heavy graphics rendering outside the browser [cite: 9].

Conclusion: In 2025-2026, WebAssembly has matured into a foundational layer of the internet stack. With the release of WASI 0.3, the establishment of the Component Model, and the standardization of WasmGC, the ecosystem has moved beyond experimentation to robust production usage. It is now the default choice for secure, portable, and high-performance logic at the edge.

Sources: 1. reddit.com 2. platform.uno 3. wasi.dev 4. youtube.com 5. dev.to 6. medium.com 7. medium.com 8. dev.to 9. dev.to 10. youtube.com 11. dev.to 12. github.com 13. warg.io 14. eunomia.dev 15. bytecodealliance.org 16. webassembly.org 17. platform.uno 18. bytecodealliance.org 19. github.com 20. fermyon.com 21. thenewstack.io 22. bytecodealliance.org 23. wasmcloud.com 24. nonstopio.com 25. devclass.com 26. atakinteractive.com 27. techversions.com 28. infoworld.com

<!-- Generated by agent-deep-research --> <!-- Query: Compare WebSocket vs Server-Sent Events (SSE) vs HTTP long-polling for real-time web applications. Cover performance characteristics, browser support, scalability, and when to use each. --> <!-- Date: 2026-02-11 --> <!-- Model: deep-research-pro-preview-12-2025 -->

Comparative Analysis of WebSocket, Server-Sent Events, and HTTP Long-Polling for Real-Time Architectures

Executive Summary

The selection of a real-time transport protocol—WebSocket, Server-Sent Events (SSE), or HTTP Long-Polling—constitutes a critical architectural decision that impacts application latency, scalability, server resource utilization, and client-side power consumption.

• WebSockets provide a full-duplex, bidirectional communication channel over a single TCP connection, characterized by minimal frame overhead (2–14 bytes). They are the industry standard for highly interactive applications such as multiplayer gaming and chat platforms [cite: 1, 2].
• Server-Sent Events (SSE) utilize standard HTTP to establish a persistent, unidirectional (server-to-client) text stream. SSE is often superior for efficient broadcasting (e.g., financial tickers, news feeds) due to its simplicity, built-in automatic reconnection, and compatibility with standard HTTP infrastructure, though it is historically limited by browser connection caps unless HTTP/2 is utilized [cite: 3, 4, 5].
• HTTP Long-Polling acts as a legacy fallback mechanism. It emulates real-time behavior by holding an HTTP request open until data is available, then immediately re-requesting. While universally compatible, it suffers from high latency, significant bandwidth overhead due to repeated header transmission, and poor battery performance on mobile devices [cite: 3, 6, 7].

The evidence suggests that while WebSockets offer the highest raw performance for bidirectional needs, SSE is frequently underutilized and offers a more scalable, firewall-friendly solution for read-heavy streams. Long-polling is now largely obsolete, reserved primarily for legacy system interoperability.

---

1. Introduction to Real-Time Web Communication

The Hypertext Transfer Protocol (HTTP), in its original design, operates on a request-response model: a client initiates a request, and the server provides a definitive response, closing the transaction. This model is inherently antithetical to real-time applications where the server must push data to the client asynchronously without an explicit request.

To bridge this gap, three primary patterns emerged: 1. Polling (Legacy): Simulating real-time by frequently querying the server (Long-Polling). 2. Streaming (Unidirectional): Keeping a connection open to stream data (SSE). 3. Socket Emulation (Bidirectional): Upgrading the connection to a persistent socket (WebSockets).

This report analyzes these technologies through the lenses of protocol mechanics, performance overhead, infrastructure requirements, and client constraints.

---

2. Technical Architectures and Mechanics

2.1 HTTP Long-Polling

Long-polling is arguably a "hack" of the standard HTTP request-response cycle. It is an inversion of the standard polling model (where the client asks repeatedly at fixed intervals).

Mechanism: 1. The client sends a standard XHR/Fetch request to the server. 2. The server does not respond immediately. It holds the request open (pending) until new data is available or a timeout threshold is reached. 3. Upon data availability, the server responds with the payload and closes the connection. 4. The client processes the data and immediately initiates a new request to restart the cycle [cite: 3, 6].

Protocol Characteristics:

• Transport: Standard HTTP/1.1 or HTTP/2.
• Connection State: Ephemeral; connections are constantly created and destroyed.
• Header Overhead: High. Every data packet requires a full HTTP handshake and headers (cookies, authorization tokens, user agents), often ranging from 500 to 1000 bytes per message [cite: 2, 6].

2.2 Server-Sent Events (SSE)

Standardized as part of HTML5, SSE (defined by the EventSource API) formalizes the pattern of keeping an HTTP connection open for streaming text data.

Mechanism: 1. The client initiates a request with the header Accept: text/event-stream. 2. The server responds with Content-Type: text/event-stream and keeps the connection open. 3. The server pushes data as text blocks delimited by newlines (\n\n), often structured with fields like data:, event:, id:, and retry: [cite: 8, 9]. 4. The connection remains open until explicitly closed or interrupted.

Protocol Characteristics:

• Directionality: Unidirectional (Server $\rightarrow$ Client only) [cite: 10].
• Data Format: Strictly UTF-8 Text. Binary data must be Base64 encoded, incurring a ~33% size penalty [cite: 11].
• Reconnection: The EventSource browser API handles reconnection automatically without custom code [cite: 12, 13].

2.3 WebSockets

WebSockets (RFC 6455) provide a true TCP socket experience within the web environment.

Mechanism: 1. Handshake: The client sends an HTTP GET request with Upgrade: websocket and Connection: Upgrade. 2. Upgrade: The server responds with 101 Switching Protocols. 3. Framing: The protocol switches from HTTP to a binary framing protocol. Data is exchanged in "frames" with minimal headers (2 bytes for small payloads) [cite: 1, 2].

Protocol Characteristics:

• Directionality: Full-duplex (Bidirectional).
• Data Format: Supports both UTF-8 strings and Binary (ArrayBuffer/Blob) natively [cite: 2].
• State: Persistent, stateful connection.

---

3. Performance Characteristics

3.1 Latency and Throughput

Performance is generally defined by the time elapsed between an event occurring on the server and its delivery to the client (latency) and the efficiency of data transfer (throughput).

Analysis: For unidirectional data, the performance gap between SSE and WebSockets is negligible. In fact, some benchmarks suggest SSE can achieve higher throughput in specific batching scenarios due to simpler processing [cite: 14]. However, WebSockets dominate in scenarios requiring sub-100ms latency for bidirectional interaction (e.g., gaming), as SSE forces the client to use a separate HTTP channel for upstream communication, incurring standard HTTP latency [cite: 15].

3.2 Mobile Battery Consumption

Battery life on mobile devices is dictated by the state of the cellular radio (Radio Resource Control - RRC states). Frequent network activity prevents the radio from entering a low-power "Idle" state.

• Long-Polling: The most detrimental to battery life. The constant cycle of opening and closing connections keeps the radio in a high-power state (DCH/FACH) continuously [cite: 7].
• WebSockets: Efficient if the connection is idle, but "keep-alive" pings (heartbeats) are required to prevent timeouts by intermediate proxies. If pings are too frequent, they drain the battery.
• SSE: Tests indicate SSE can be more power-efficient than WebSockets on mobile (iOS/Android) for read-only streams (~30% less battery usage in some tests). This is because SSE does not require bidirectional state management or ping/pong frames at the application layer to the same extent, relying instead on the underlying TCP stack managed by the OS [cite: 16, 17].

---

4. Scalability and Infrastructure

4.1 Connection State and Load Balancing

The primary challenge in scaling real-time applications is maintaining thousands of concurrent connections (the C10k problem).

• WebSockets (Stateful):
• Sticky Sessions: Because a WebSocket is a persistent TCP connection to a specific server instance, load balancers cannot simply round-robin individual messages. The connection is "sticky." If the server crashes, the connection is lost.
• Horizontal Scaling: Scaling requires a complex "Backplane" or Pub/Sub broker (e.g., Redis, Kafka, NATS) to synchronize messages across server nodes. If User A is on Server 1 and User B is on Server 2, Server 1 cannot send a message directly to User B without passing it through the broker [cite: 18, 19].
• Server-Sent Events (Stateless/HTTP):
• While technically a persistent connection, SSE operates over standard HTTP. It creates less friction with standard load balancers (Layer 7).
• However, like WebSockets, the server must maintain an open socket for every client.
• Long-Polling (Stateless):
• Easiest to load balance. Every request is a new HTTP request and can be routed to any server (provided the backend data store is shared).

4.2 Browser Connection Limits & HTTP/2

A critical limitation for SSE in legacy environments is the browser's limit on concurrent HTTP connections to a single domain.

• HTTP/1.1 Limitation: Browsers (Chrome, Firefox) limit simultaneous connections to ~6 per domain. If a user opens 6 tabs using SSE, the 7th tab will hang (Head-of-Line Blocking) [cite: 5, 9, 20].
• The HTTP/2 Solution: HTTP/2 supports Multiplexing, allowing multiple logical streams over a single TCP connection. With HTTP/2, the browser connection limit becomes effectively irrelevant (defaults often around 100 streams), allowing SSE to scale across tabs seamlessly [cite: 4, 5, 20].

Note on WebSockets: WebSockets are not subject to the HTTP/1.1 6-connection limit in the same way, as they upgrade the connection, removing it from the HTTP pool [cite: 4].

---

5. Browser Support and Network Intermediaries

5.1 Browser Compatibility

• WebSockets: Universally supported in all modern browsers and most mobile environments.
• SSE: Supported in all modern browsers (Chrome, Firefox, Safari). Historically, Internet Explorer and Edge (pre-Chromium) required polyfills, but this is largely irrelevant in 2024/2025 contexts [cite: 10, 21].
• Long-Polling: Works in every browser ever made, including ancient text-only browsers, as it utilizes standard XHR/Fetch.

5.2 Firewalls and Proxies

Corporate firewalls and proxies often perform Deep Packet Inspection (DPI) or strict port blocking.

• WebSockets: Often blocked or dropped.
• Some proxies do not understand the Upgrade header.
• Idle connections are frequently terminated by aggressive timeouts.
• Non-standard ports are blocked.
• Mitigation: Tunneling over TLS (wss://) usually bypasses these issues as the proxy cannot inspect the encrypted traffic [cite: 12, 22].
• SSE: Highly compatible.
• Appears as a standard HTTP download (long-running).
• "Firewall-friendly" for enterprise environments [cite: 13, 23].
• Long-Polling: Maximum compatibility. Indistinguishable from regular web browsing traffic.

---

6. Synthesis and Decision Matrix

6.1 When to use WebSockets

WebSockets are the gold standard for Event-Driven Architectures requiring high-frequency, two-way interaction.

• Use Cases:
• Multiplayer Games: Latency is critical; state must flow both ways [cite: 24, 25].
• Chat Apps (WhatsApp/Slack style): Typing indicators, read receipts, and instant delivery require bidirectional pipes.
• Collaborative Editing: Google Docs-style operational transformation requires immediate syncing [cite: 25].
• Pros: Lowest latency, full-duplex, binary support.
• Cons: Complex to scale (sticky sessions), manual reconnection logic required, firewall issues without TLS.

6.2 When to use Server-Sent Events (SSE)

SSE is the ideal choice for Data Streaming Architectures where the client is a passive consumer.

• Use Cases:
• Financial Tickers: Stock prices, crypto feeds (Server $\rightarrow$ Client).
• Live Blogging/News Feeds: Breaking news updates.
• System Notifications: "Processing complete" alerts.
• Real-time Dashboards: Analytics updating every few seconds.
• Pros: Native auto-reconnect, works over HTTP/2, firewall-friendly, simpler API (EventSource), lighter on mobile battery for read-only.
• Cons: Unidirectional only, text-only (no binary), requires HTTP/2 for multiple tabs.

6.3 When to use Long-Polling

Long-Polling is strictly a Fallback Strategy.

• Use Cases:
• Legacy Systems: Support for very old browsers (IE6-9).
• Restrictive Networks: Environments blocking both WebSockets and long-running streaming connections.
• Low Frequency: Apps where updates happen so rarely (e.g., once every 10 minutes) that maintaining a persistent socket is wasteful.
• Pros: 100% compatibility, stateless scaling.
• Cons: High latency, high bandwidth/header overhead, battery drain.

6.4 Summary Comparison Table

7. Conclusion

In the modern web landscape, HTTP Long-Polling should generally be avoided unless specific legacy compatibility constraints mandate its use. The choice effectively resides between WebSockets and SSE.

While developers often default to WebSockets for all real-time needs due to their ubiquity, Server-Sent Events often represent a superior architectural choice for read-heavy applications (dashboards, feeds, notifications). SSE leverages the existing HTTP ecosystem (status codes, headers, compression) and provides resilience mechanisms (auto-reconnect) out of the box that must be manually engineered in WebSockets.

However, for applications demanding true interactivity where the client acts as a data producer as well as a consumer—such as gaming or chat—WebSockets remain the unrivaled solution for their low-latency, full-duplex capabilities.

Sources: 1. substack.com 2. medium.com 3. medium.com 4. ycombinator.com 5. medium.com 6. openreplay.com 7. stackoverflow.com 8. javascript.info 9. mozilla.org 10. aklivity.io 11. github.com 12. youtube.com 13. freecodecamp.org 14. timeplus.com 15. stackoverflow.com 16. medium.com 17. youtube.com 18. dev.to 19. medium.com 20. dev.to 21. youtube.com 22. saadkhaleeq.com 23. videosdk.live 24. dev.to 25. medium.com

File Search MIME Type Guide

Condensed reference for Gemini File Search API file type support. For full test methodology and bug details, see docs/file-search-mime-types.md.

Key Facts

• File size limit: 100 MB per file
• Documented types: 180+
• Actually working types: 36 extensions (15.4% of documented)
• Workaround: Text-based files not in the validated list are uploaded as text/plain

Validated MIME Types (36 extensions)

These file types are confirmed to work with the Gemini File Search API.

Application Types

Plain Text

Markup Languages

Programming Languages

Other

Text Fallback (100+ extensions)

Files with these extensions are uploaded as text/plain. Search works correctly despite the generic MIME type.

JavaScript/TypeScript: .js, .mjs, .cjs, .jsx, .ts, .mts, .cts, .tsx, .d.ts, .json, .jsonc, .json5

Web: .css, .scss, .sass, .less, .styl, .vue, .svelte, .astro

Shell/Scripts: .sh, .bash, .zsh, .fish, .ksh, .bat, .cmd, .ps1, .psm1

Config: .yaml, .yml, .toml, .ini, .cfg, .conf, .properties, .editorconfig, .prettierrc, .eslintrc, .babelrc, .npmrc

Other Languages: .rb, .php, .rs, .swift, .scala, .clj, .ex, .hs, .ml, .fs, .r, .jl, .nim, .zig, .dart, .coffee, .elm

Unsupported (Rejected)

Binary files cannot be uploaded:

• Executables: .exe, .dll, .so, .dylib
• Archives: .zip, .tar, .gz, .7z, .rar
• Images: .png, .jpg, .gif, .svg, .webp
• Audio/Video: .mp3, .mp4, .wav, .avi
• Compiled: .class, .pyc, .o, .obj
• Other binary: .wasm, .bin, .dat

Recommendations