Parallel Web

Parallel Web Toolkit

A unified skill for all web-powered tasks: searching, extracting, enriching, and researching — with academic and scientific sources as the default priority.

Routing — pick the right capability

Read the user's request and match it to one of the capabilities below. For web search, extract, enrichment, and deep research, read the corresponding reference file for detailed instructions.

Decision guide

• Default to Web Search for a single lookup, research question, or "what is X?" query. It's fast and cost-effective. When the query touches a scientific or technical topic, include academic domains (see references/web-search.md) to surface peer-reviewed and preprint sources alongside general results.
• Use Web Extract when the user provides a URL or asks you to read/fetch a specific page. Prefer this over the built-in WebFetch tool. Particularly useful for extracting full text from academic PDFs, preprint servers, and journal articles.
• Use Data Enrichment when the user has multiple entities (a CSV, a list of companies/people/products, or even a short inline list) and wants to find or add the same kind of information for each one. The key signal is a repeated lookup across a set of items — e.g., "find the CEO for each of these companies" or "get the founding year for Apple, Stripe, and Anthropic." Even if the user doesn't say "enrich," use parallel-cli enrich whenever the task is the same query applied to multiple entities. Do NOT use Web Search in a loop for this — the enrichment pipeline handles batching, parallelism, and structured output automatically.
• Use Deep Research only when the user explicitly asks for deep, exhaustive, or comprehensive research. It is 10-100x slower and more expensive than Web Search — never default to it. Deep research is especially valuable for literature reviews and multi-paper synthesis.
• If parallel-cli is not found when running any command, follow the Setup section below.

Academic source priority

Across all capabilities, prefer academic and scientific sources when the query is technical or scientific in nature. This means:

• Peer-reviewed journal articles and conference proceedings over blog posts or news articles
• Preprints (arXiv, bioRxiv, medRxiv) when peer-reviewed versions aren't available
• Institutional and government sources (NIH, WHO, NASA, NIST) over commercial sites
• Primary research over secondary summaries

When citing academic sources, include author names and publication year where available (e.g., Smith et al., 2025) in addition to the standard citation format. If a DOI is present, prefer the DOI link.

Context chaining

Several capabilities support multi-turn context via interaction_id. When a research or enrichment task completes, it returns an interaction_id. If the user asks a follow-up question related to that task, pass --previous-interaction-id to carry context forward automatically. This avoids restating what was already found.

---

Setup

If parallel-cli is not installed, install and authenticate:

curl -fsSL https://parallel.ai/install.sh | bash

If unable to install that way, use uv instead:

uv tool install "parallel-web-tools[cli]"

Then authenticate. First, check if a .env file exists in the project root and contains PARALLEL_API_KEY. If so, load it with dotenv:

dotenv -f .env run parallel-cli auth

If dotenv isn't available, install it with pip install python-dotenv[cli] or uv pip install python-dotenv[cli].

If there's no .env file or it doesn't contain the key, fall back to interactive login:

parallel-cli login

Or set the key manually: export PARALLEL_API_KEY="your-key"

Verify with:

parallel-cli auth

If parallel-cli is not found after install, add ~/.local/bin to PATH.

Check task status

parallel-cli research status "$RUN_ID" --json

Report the current status to the user (running, completed, failed, etc.).

Get completed result

parallel-cli research poll "$RUN_ID" --json

Present results in a clear, organized format.

Data Enrichment

Enrich: $ARGUMENTS

Before starting

Inform the user that enrichment may take several minutes depending on the number of rows and fields requested.

Step 1: Start the enrichment

Use ONE of these command patterns (substitute user's actual data):

For inline data:

parallel-cli enrich run --data '[{"company": "Google"}, {"company": "Microsoft"}]' --intent "CEO name and founding year" --target "output.json" --no-wait --json

For CSV file:

parallel-cli enrich run --source-type csv --source "input.csv" --target "/tmp/output.json" --source-columns '[{"name": "company", "description": "Company name"}]' --intent "CEO name and founding year" --no-wait --json

If this is a follow-up to a previous research or enrichment task where you know the interaction_id, add context chaining:

parallel-cli enrich run --data '...' --intent "..." --target "output.json" --no-wait --json --previous-interaction-id "$INTERACTION_ID"

By chaining interaction_id values across requests, each follow-up automatically has the full context of prior turns — so you can enrich entities discovered in earlier research without restating what was already found.

IMPORTANT: Always include --no-wait so the command returns immediately instead of blocking.

Parse the output to extract the taskgroup_id, interaction_id, and monitoring URL. Immediately tell the user:

• Enrichment has been kicked off
• The monitoring URL where they can track progress

Tell them they can background the polling step to continue working while it runs.

Step 2: Poll for results

Choose a short, descriptive filename based on the enrichment task (e.g., companies-ceos, startups-funding). Use lowercase with hyphens, no spaces.

parallel-cli enrich poll "$TASKGROUP_ID" --timeout 540 --json --output "$FILENAME.json"

The --target flag on enrich run does not carry over to the poll — you must pass --output here to save the results. Always use --json to get structured JSON output.

Important:

• Use --timeout 540 (9 minutes) to stay within tool execution limits

If the poll times out

Enrichment of large datasets can take longer than 9 minutes. If the poll exits without completing: 1. Tell the user the enrichment is still running server-side 2. Re-run the same parallel-cli enrich poll command to continue waiting

Response format

After step 1: Share the monitoring URL (for tracking progress).

After step 2: 1. Report number of rows enriched 2. Preview first few rows of the output JSON 3. Tell user the full path to the output JSON file ($FILENAME.json) 4. Share the interaction_id and tell the user they can ask follow-up questions that build on this enrichment

Do NOT re-share the monitoring URL after completion — the results are in the output file.

Remember the `interaction_id` — if the user asks a follow-up question that relates to this enrichment, use it as --previous-interaction-id in the next research or enrichment command.

Deep Research

Research topic: $ARGUMENTS

When to use (vs web search)

ONLY use this capability when the user explicitly requests deep/exhaustive research. Deep research is 10-100x slower and more expensive than web search. For normal "research X" requests, quick lookups, or fact-checking, use web search instead.

Step 1: Start the research

Frame the research objective to prioritize academic literature. If the user's query is scientific or technical, prepend context to the arguments that steers toward scholarly sources — e.g., instead of "effects of sleep deprivation", use "peer-reviewed research and clinical studies on the effects of sleep deprivation". This nudges the research agent toward primary literature without excluding relevant non-academic sources.

parallel-cli research run "$ARGUMENTS" --processor pro-fast --no-wait --json

If this is a follow-up to a previous research or enrichment task where you know the interaction_id, add context chaining:

parallel-cli research run "$ARGUMENTS" --processor lite --no-wait --json --previous-interaction-id "$INTERACTION_ID"

By chaining interaction_id values across requests, each follow-up question automatically has the full context of prior turns — so you can drill deeper into a topic without restating what was already researched. Use --processor lite for follow-ups since the heavy research was already done in the initial turn and the follow-up just needs to build on that context.

This returns instantly. Do NOT omit --no-wait — without it the command blocks for minutes and will time out.

Processor options (choose based on user request):

Parse the JSON output to extract the run_id, interaction_id, and monitoring URL. Immediately tell the user:

• Deep research has been kicked off
• The expected latency for the processor tier chosen (from the table above)
• The monitoring URL where they can track progress

Tell them they can background the polling step to continue working while it runs.

Step 2: Poll for results

Choose a descriptive filename based on the topic (e.g., ai-chip-market-2026, react-vs-vue-comparison). Use lowercase with hyphens, no spaces.

parallel-cli research poll "$RUN_ID" -o "$FILENAME" --timeout 540

Important:

• Use --timeout 540 (9 minutes) to stay within tool execution limits
• Do NOT pass --json — the full output is large and will flood context. The -o flag writes results to files instead.
• The -o flag generates two output files:
• $FILENAME.json — metadata and basis
• $FILENAME.md — formatted markdown report
• The poll command prints an executive summary to stdout when the research completes. Share this executive summary with the user — it gives them a quick overview without having to open the files.

If the poll times out

Higher processor tiers can take longer than 9 minutes. If the poll exits without completing: 1. Tell the user the research is still running server-side 2. Re-run the same parallel-cli research poll command to continue waiting

Response format

After step 1: Share the monitoring URL (for tracking progress only — it is not the final report).

After step 2: 1. Share the executive summary that the poll command printed to stdout 2. Briefly assess the source quality: how many of the cited sources are peer-reviewed journals, preprints, or institutional reports vs. news articles or blog posts. If academic coverage is thin, flag this to the user and suggest a follow-up search targeting specific databases (e.g., "I can search PubMed/arXiv specifically for more primary sources on this"). 3. Tell the user the two generated file paths:

• $FILENAME.md — formatted markdown report
• $FILENAME.json — metadata and basis

4. Share the interaction_id and tell the user they can ask follow-up questions that build on this research (e.g., "drill deeper into X" or "compare that to Y")

Do NOT re-share the monitoring URL after completion — the results are in the files, not at that link.

Ask the user if they would like to read through the files for more detail. Do NOT read the file contents into context unless the user asks.

Remember the `interaction_id` — if the user asks a follow-up question that relates to this research, use it as --previous-interaction-id in the next research or enrichment command.

URL Extraction

Extract content from: $ARGUMENTS

Command

Choose a short, descriptive filename based on the URL or content (e.g., vespa-docs, react-hooks-api). Use lowercase with hyphens, no spaces.

parallel-cli extract "$ARGUMENTS" --json -o "$FILENAME.json"

Options if needed:

• --objective "focus area" to focus on specific content

Academic content handling

When extracting from academic sources (arXiv, PubMed, journal sites, conference proceedings), use --objective to focus on the most valuable sections:

parallel-cli extract "$URL" --json --objective "extract abstract, methodology, key findings, and conclusions" -o "$FILENAME.json"

For arXiv papers, prefer the /abs/ URL (which has structured metadata) over the raw PDF URL when available. If the user provides a PDF link, extract it directly — parallel-cli handles PDFs.

Response format

Return content as:

[Page Title](URL)

For academic papers, include structured metadata when available:

• Authors: list of authors
• Published: date and venue/journal
• DOI: if available
• Abstract: the paper's abstract

Then the extracted content verbatim, with these rules:

• Keep content verbatim - do not paraphrase or summarize
• Parse lists exhaustively - extract EVERY numbered/bulleted item
• Strip only obvious noise: nav menus, footers, ads
• Preserve all facts, names, numbers, dates, quotes
• For academic papers, preserve figure/table captions and references

After the response, mention the output file path ($FILENAME.json) so the user knows it's available for follow-up questions.

Web Search

Search the web for: $ARGUMENTS

Command

Choose a short, descriptive filename based on the query (e.g., ai-chip-news, react-vs-vue). Use lowercase with hyphens, no spaces.

parallel-cli search "$ARGUMENTS" -q "<keyword1>" -q "<keyword2>" --json --max-results 10 --excerpt-max-chars-total 27000 -o "$FILENAME.json"

The first argument is the objective — a natural language description of what you're looking for. It replaces multiple keyword searches with a single call for broad or complex queries. Add -q flags for specific keyword queries to supplement the objective. The -o flag saves the full results to a JSON file for follow-up questions.

Options if needed:

• --after-date YYYY-MM-DD for time-sensitive queries
• --include-domains domain1.com,domain2.com to limit to specific sources

Academic source strategy

For scientific or technical queries, run two searches to ensure academic sources surface alongside general results:

1. Academic-focused search — append --include-domains with scholarly domains:

   parallel-cli search "$ARGUMENTS" -q "<keyword1>" --json --max-results 10 --excerpt-max-chars-total 27000 --include-domains "scholar.google.com,arxiv.org,pubmed.ncbi.nlm.nih.gov,semanticscholar.org,biorxiv.org,medrxiv.org,ncbi.nlm.nih.gov,nature.com,science.org,ieee.org,acm.org,springer.com,wiley.com,cell.com,pnas.org,nih.gov" -o "$FILENAME-academic.json"

2. General search — the standard command without domain restrictions, to catch relevant non-academic sources.

Merge results, leading with academic sources. If only one search is practical (e.g., clearly non-scientific query), skip the academic-focused search.

When to use the two-search pattern: Any query involving scientific claims, medical information, research findings, technical mechanisms, statistical data, or anything where primary literature would be more reliable than secondary reporting.

Parsing results

Do not set max_output_tokens on the command execution — the output is already bounded by --max-results and --excerpt-max-chars-total. Capping output tokens will truncate the JSON and break parsing.

Parse the JSON from stdout. For each result, extract:

• title, url, publish_date
• Useful content from excerpts (skip navigation noise like menus, footers, "Skip to content")

Response format

CRITICAL: Every claim must have an inline citation. Use markdown links pulling only from the JSON output. Never invent or guess URLs.

For academic sources, use author-year citation style where metadata is available:

• Academic: Smith et al., 2025 or Smith & Jones, 2024
• Non-academic: Source Title

Synthesize a response that:

• Leads with findings from peer-reviewed or preprint sources when available
• Clearly distinguishes between claims backed by primary research vs. secondary reporting
• Includes specific facts, names, numbers, dates
• Cites every fact inline — do not leave any claim uncited
• Organizes by theme if multiple topics
• Notes the evidence quality (e.g., "a randomized controlled trial found..." vs. "a blog post reports...")

End with a Sources section listing every URL referenced, grouped by type:

Sources:

Academic / Peer-reviewed:
- [Smith et al., 2025 — Title of Paper](https://doi.org/...) (Nature, 2025)
- [Jones & Lee, 2024 — Title of Paper](https://arxiv.org/...) (arXiv preprint)

Other:
- [Source Title](https://example.com/article) (Feb 2026)

This Sources section is mandatory. Do not omit it. If no academic sources were found, note that and explain why (e.g., the topic is too recent, not yet studied, or inherently non-academic).

After the Sources section, mention the output file path ($FILENAME.json) so the user knows it's available for follow-up questions.