Just Scrape
Install when you need search, scrape, crawl, structured JSON extraction, or change monitoring from URLs without wiring ScrapeGraph into app code first.
Overview
just-scrape is an agent skill most often used in Build (also Idea, Grow) that runs the ScrapeGraph AI CLI to search, scrape, crawl, extract JSON, and monitor web pages.
Install
npx skills add https://github.com/scrapegraphai/just-scrape --skill just-scrapeWhat is this skill?
- Global `just-scrape` CLI with `validate` and `credits` for setup checks
- Search, single-page scrape, crawl sections/docs, structured JSON extraction, and page change monitoring
- Request history inspection and credit visibility via CLI
- Explicit split: CLI skill for ops tasks; SDK/API docs for in-product integration
- Requires `SGAI_API_KEY` (env or .env) and Bash-capable agent
Adoption & trust: 133k installs on skills.sh; 22 GitHub stars; 2/3 security scanners passed (skills.sh audits).
What problem does it solve?
You need live web content or structured data from URLs but do not want to hand-roll scrapers or jump straight into SDK integration.
Who is it for?
Solo builders using Claude Code or Cursor who need quick research pulls, doc crawls, or JSON extraction via a maintained CLI.
Skip if: Teams wiring ScrapeGraph directly into app code without the CLI—use the SDK/API path the skill describes instead.
When should I use this skill?
User asks to search the web, scrape a URL, crawl docs, extract JSON, monitor page changes, inspect request history, check credits, or validate ScrapeGraph setup.
What do I get? / Deliverables
After install and `SGAI_API_KEY` validation, your agent can run scoped CLI commands for search, scrape, crawl, extraction, and monitoring with credit and history visibility.
- Scraped or crawled content from target URLs
- Structured JSON extraction output when requested
- Validation/credits status from CLI checks
Recommended Skills
Journey fit
Spans multiple journey phases - primary shelf plus alternate fits below.
Canonical shelf is Build → integrations because the skill centers on the just-scrape CLI and ScrapeGraph API for pulling external web data into your workflow. Integrations is the right subphase: global CLI install, SGAI_API_KEY auth, and commands for scrape/crawl/monitor are external-service wiring, not in-repo frontend or backend code.
Where it fits
Scrape competitor landing and pricing pages into structured notes before you commit to a niche.
Crawl vendor API docs so your agent can answer integration questions with fresh page content.
Pull live FAQ and feature copy from analogous products to sanity-check your positioning scope.
Monitor partner or directory URLs for copy changes that affect your backlinks or listings.
How it compares
CLI operational wrapper for ScrapeGraph, not a generic browser-automation MCP or one-off curl recipes.
Common Questions / FAQ
Who is just-scrape for?
Indie and solo builders who want agent-driven web search, scraping, crawling, and monitoring through the official just-scrape CLI with ScrapeGraph AI.
When should I use just-scrape?
During Idea research (competitor pages, market pages), Build integrations (doc crawl, structured extraction), or Grow/Operate checks (monitor URLs for pricing or content changes)—whenever the task is CLI-shaped, not in-repo SDK wiring.
Is just-scrape safe to install?
Review the Security Audits panel on this Prism page; the skill uses Bash, network calls, and an API key (`SGAI_API_KEY`), so treat keys like production secrets and scope agent permissions accordingly.
SKILL.md
READMESKILL.md - Just Scrape
# just-scrape CLI Search, scrape, crawl, extract structured JSON, and monitor page changes using the just-scrape CLI. Run `just-scrape --help` or `just-scrape <command> --help` for full option details. If the task is to integrate ScrapeGraph AI into application code, add `SGAI_API_KEY` to a project, or choose endpoint usage in product code, inspect the project first and use the ScrapeGraph AI SDK/API docs directly instead of this CLI skill. ## Prerequisites Must be installed and authenticated. Check with `just-scrape validate` and `just-scrape credits`. ```bash which just-scrape || npm install -g just-scrape@latest just-scrape validate just-scrape credits ``` - **API key**: Set `SGAI_API_KEY`, use a `.env` file, use `~/.scrapegraphai/config.json`, or complete the interactive prompt. - **Credits**: Remaining ScrapeGraph AI credits. Each operation consumes credits. Before doing real work, verify the setup with one small request: ```bash mkdir -p .just-scrape just-scrape scrape "https://example.com" --json > .just-scrape/install-check.json ``` ```bash just-scrape search "query" --num-results 3 --json > .just-scrape/search-check.json ``` ## Workflow Follow this escalation pattern: 1. **Search** - No specific URL yet. Find pages, answer questions, discover sources. 2. **Scrape** - Have a URL. Extract markdown, html, screenshots, links, images, summaries, or branding. 3. **Extract** - Need structured JSON from a known URL with an AI prompt and optional schema. 4. **Crawl** - Need bulk content from an entire site section. 5. **Monitor** - Need scheduled page-change tracking with optional webhook notifications. | Need | Command | When | | --------------------------- | ---------- | ------------------------------------------ | | Find pages on a topic | `search` | No specific URL yet | | Get a page's content | `scrape` | Have a URL, need one or more page formats | | AI-powered data extraction | `extract` | Need structured data from a known URL | | Bulk extract a site section | `crawl` | Need many pages or docs sections | | Track changes over time | `monitor` | Need recurring scraping and webhooks | | Inspect prior requests | `history` | Need past request IDs, status, or payloads | | Check credit balance | `credits` | Need remaining API credits | | Validate API setup | `validate` | Need health check and API key validation | For detailed command reference, run `just-scrape <command> --help`. **Scrape vs extract:** - Use `scrape` for raw page formats: `markdown`, `html`, `screenshot`, `branding`, `links`, `images`, `summary`. - Use `scrape -f json -p "<prompt>"` or `extract -p "<prompt>"` for AI-structured output. - Use `extract` when the task is only structured data. Use `scrape` when mixed formats are needed in one call. **Avoid redundant fetches:** - `search -p` can extract structured data from search results. Do not re-scrape those URLs unless results are incomplete. - `crawl` already fetches per-page formats. Do not re-scrape every crawled URL unless a second pass is required. - Check `