Web Scraper

Name: Web Scraper
Author: tryshift-sh

tryshift-sh/skills-store

27 installs
Updated May 12, 2026
tryshift-sh/skills-store

Helps with ai & agent building tasks.

About

web-scraper is a Claude Code skill for ai & agent building. It helps solo builders move faster with AI-assisted coding.

web-scraper
AI & Agent Building
AI-coding skill

Web Scraper by the numbers

27 all-time installs (skills.sh)
Ranked #9,572 of 16,546 AI & Agent Building skills by installs in the Skillselion catalog
Data as of Jul 27, 2026 (Skillselion catalog sync)

npx skills add https://github.com/tryshift-sh/skills-store --skill web-scraper

Add your badge

Show developers this skill is listed on Skillselion. Paste this into your README.

[![Listed on Skillselion](https://skillselion.com/badge/skills/tryshift-sh/skills-store/web-scraper.svg)](https://skillselion.com/skills/tryshift-sh/skills-store/web-scraper)

Installs	27
Last updated	May 12, 2026
Repository	tryshift-sh/skills-store ↗

What it does

Helps with ai & agent building tasks.

Files

SKILL.mdMarkdownGitHub ↗

Firecrawl Web Scraper

Use this managed skill when the user wants to scrape web pages, crawl websites, discover URLs, search the web, or extract structured data from websites.

This skill uses Shift's local Skill Router. Do not ask the user to paste credentials into chat.

Invocation

Send a POST request to:

${SHIFT_LOCAL_GATEWAY}/skill-router/invoke

Scrape a single page

Extracts content from a URL as clean markdown.

{
  "skillProvider": "firecrawl",
  "skill": "web-scraper",
  "action": "scrape",
  "input": {
    "url": "https://example.com"
  }
}

Optional input fields:

formats: array of output formats, default ["markdown"]. Options: markdown, html, links, screenshot
onlyMainContent: boolean, default true. Exclude nav/footer
includeTags / excludeTags: arrays of HTML tags to include or exclude
waitFor: integer, milliseconds to wait before scraping
timeout: integer, milliseconds, default 30000
mobile: boolean, default false. Emulate mobile viewport
location: object with country (ISO 3166-1 alpha-2) and languages array

Crawl a website (async)

Starts an async crawl job. Returns a job ID to poll with crawl_status.

{
  "skillProvider": "firecrawl",
  "skill": "web-scraper",
  "action": "crawl",
  "input": {
    "url": "https://example.com",
    "limit": 50
  }
}

Optional input fields:

limit: max pages to crawl, default 10
maxDepth: crawl depth
includePaths / excludePaths: regex path filters
allowExternalLinks: boolean, default false
allowSubdomains: boolean, default false
scrapeOptions: object with same options as scrape (e.g. {"formats": ["markdown"]})

Check crawl status

Poll for crawl job results using the ID returned from crawl.

{
  "skillProvider": "firecrawl",
  "skill": "web-scraper",
  "action": "crawl_status",
  "input": {
    "crawlId": "crawl-job-uuid"
  }
}

Status values: scraping, completed, failed.

Map a website's URLs

Discovers all URLs on a site without scraping content. Use this to understand site structure before crawling.

{
  "skillProvider": "firecrawl",
  "skill": "web-scraper",
  "action": "map",
  "input": {
    "url": "https://example.com"
  }
}

Optional input fields:

search: order results by relevance to this query
limit: max URLs, default 5000
includeSubdomains: boolean, default true
ignoreSitemap: boolean, default false

Search the web

Search the web and optionally scrape the result pages.

{
  "skillProvider": "firecrawl",
  "skill": "web-scraper",
  "action": "search",
  "input": {
    "query": "latest AI research papers"
  }
}

Optional input fields:

limit: max results, 1-100, default 5
lang: language code
country: country code, default US
tbs: time filter (qdr:h hour, qdr:d day, qdr:w week, qdr:m month, qdr:y year)
scrapeOptions: object to control content extraction from results

Extract structured data (async)

Extracts structured data from URLs using a prompt and/or JSON schema. Returns a job ID to poll with extract_status.

{
  "skillProvider": "firecrawl",
  "skill": "web-scraper",
  "action": "extract",
  "input": {
    "urls": ["https://example.com/pricing"],
    "prompt": "Extract all pricing plans with name, price, and features"
  }
}

Optional input fields:

schema: JSON Schema defining expected output structure
enableWebSearch: boolean, default false

Check extract status

Poll for extract job results.

{
  "skillProvider": "firecrawl",
  "skill": "web-scraper",
  "action": "extract_status",
  "input": {
    "extractId": "extract-job-uuid"
  }
}

Authentication

This skill requires a Firecrawl API key configured in Shift. Get one at https://www.firecrawl.dev.

Do not ask the user to paste raw credentials into the conversation. Shift handles authentication automatically when the required connection is configured.

Agent behavior

1. Prefer scrape for single pages. Use crawl only when multiple pages are needed. 2. Use map first to discover site structure before starting a large crawl. 3. For async actions (crawl, extract), poll status every few seconds until completed or failed. 4. Default to markdown format unless the user specifically needs HTML or screenshots. 5. When scraping fails, suggest the user check the URL or try with waitFor for JavaScript-heavy pages.

{
  "id": "web-scraper",
  "name": "Web Scraper",
  "description": "Scrape, crawl, map, search, and extract structured data from the web.",
  "actions": {
    "scrape": {
      "upstream": {
        "method": "POST",
        "baseUrl": "https://api.firecrawl.dev",
        "path": "/v1/scrape",
        "auth": { "mode": "bearer" }
      },
      "input": {
        "bodyTemplate": {
          "url": "$input.url",
          "formats": "$input.formats",
          "onlyMainContent": "$input.onlyMainContent",
          "includeTags": "$input.includeTags",
          "excludeTags": "$input.excludeTags",
          "waitFor": "$input.waitFor",
          "timeout": "$input.timeout",
          "mobile": "$input.mobile",
          "location": "$input.location"
        }
      },
      "output": {
        "responseSelect": "data",
        "responseMap": {
          "markdown": "$data.markdown",
          "html": "$data.html",
          "links": "$data.links",
          "screenshot": "$data.screenshot",
          "metadata": {
            "title": "$data.metadata.title",
            "description": "$data.metadata.description",
            "sourceURL": "$data.metadata.sourceURL",
            "statusCode": "$data.metadata.statusCode"
          }
        }
      }
    },
    "crawl": {
      "upstream": {
        "method": "POST",
        "baseUrl": "https://api.firecrawl.dev",
        "path": "/v1/crawl",
        "auth": { "mode": "bearer" }
      },
      "input": {
        "bodyTemplate": {
          "url": "$input.url",
          "limit": "$input.limit",
          "maxDepth": "$input.maxDepth",
          "includePaths": "$input.includePaths",
          "excludePaths": "$input.excludePaths",
          "allowExternalLinks": "$input.allowExternalLinks",
          "allowSubdomains": "$input.allowSubdomains",
          "scrapeOptions": "$input.scrapeOptions"
        }
      },
      "output": {
        "responseMap": {
          "success": "$response.success",
          "id": "$response.id",
          "url": "$response.url"
        }
      }
    },
    "crawl_status": {
      "upstream": {
        "method": "GET",
        "baseUrl": "https://api.firecrawl.dev",
        "path": "/v1/crawl/{crawlId}",
        "auth": { "mode": "bearer" }
      },
      "output": {
        "responseMap": {
          "status": "$response.status",
          "total": "$response.total",
          "completed": "$response.completed",
          "creditsUsed": "$response.creditsUsed",
          "expiresAt": "$response.expiresAt",
          "next": "$response.next",
          "data": {
            "$each": "$response.data",
            "map": {
              "markdown": "$item.markdown",
              "metadata": {
                "title": "$item.metadata.title",
                "sourceURL": "$item.metadata.sourceURL",
                "statusCode": "$item.metadata.statusCode"
              }
            }
          }
        }
      }
    },
    "map": {
      "upstream": {
        "method": "POST",
        "baseUrl": "https://api.firecrawl.dev",
        "path": "/v1/map",
        "auth": { "mode": "bearer" }
      },
      "input": {
        "bodyTemplate": {
          "url": "$input.url",
          "search": "$input.search",
          "limit": "$input.limit",
          "includeSubdomains": "$input.includeSubdomains",
          "ignoreSitemap": "$input.ignoreSitemap"
        }
      },
      "output": {
        "responseMap": {
          "success": "$response.success",
          "links": "$response.links"
        }
      }
    },
    "search": {
      "upstream": {
        "method": "POST",
        "baseUrl": "https://api.firecrawl.dev",
        "path": "/v1/search",
        "auth": { "mode": "bearer" }
      },
      "input": {
        "bodyTemplate": {
          "query": "$input.query",
          "limit": "$input.limit",
          "lang": "$input.lang",
          "country": "$input.country",
          "tbs": "$input.tbs",
          "scrapeOptions": "$input.scrapeOptions"
        }
      },
      "output": {
        "responseMap": {
          "success": "$response.success",
          "data": {
            "$each": "$response.data",
            "map": {
              "title": "$item.title",
              "url": "$item.url",
              "markdown": "$item.markdown",
              "description": "$item.description"
            }
          }
        }
      }
    },
    "extract": {
      "upstream": {
        "method": "POST",
        "baseUrl": "https://api.firecrawl.dev",
        "path": "/v1/extract",
        "auth": { "mode": "bearer" }
      },
      "input": {
        "bodyTemplate": {
          "urls": "$input.urls",
          "prompt": "$input.prompt",
          "schema": "$input.schema",
          "enableWebSearch": "$input.enableWebSearch"
        }
      },
      "output": {
        "responseMap": {
          "success": "$response.success",
          "id": "$response.id"
        }
      }
    },
    "extract_status": {
      "upstream": {
        "method": "GET",
        "baseUrl": "https://api.firecrawl.dev",
        "path": "/v1/extract/{extractId}",
        "auth": { "mode": "bearer" }
      },
      "output": {
        "responseMap": {
          "success": "$response.success",
          "status": "$response.status",
          "data": "$response.data",
          "expiresAt": "$response.expiresAt"
        }
      }
    }
  }
}