General Web Scraper

Name: General Web Scraper
Author: skills.volces.com

skills.volces.com

1 installs
skills.volces.com

Pull links or table data from any URL with CSS selectors and export CSV or JSON for market and competitor research.

About

General-web-scraper is a beginner-friendly Python CLI skill that gives developers and agents a zero-config way to harvest structured data from public web pages. You pass a URL, optionally refine selectors for articles or list items, and receive link lists or table rows as CSV or JSON—ideal for quick competitive scans, directory harvesting, and lightweight content aggregation without standing up Scrapy or paid APIs. Dependencies are limited to requests and BeautifulSoup, so it fits early journey phases when you are proving niches or monitoring rivals. Respect robots.txt and site terms; use it for research and ops-adjacent monitoring, not as a stealth production crawler without rate limits and legal review.

Link scraping via configurable CSS selectors (default: page anchors)
HTML table extraction with --table mode
CSV default export plus optional JSON output
Chinese encoding support for international pages
CLI: python scraper.py URL [selector] [--json] [--table]

General Web Scraper by the numbers

1 all-time installs (skills.sh)
Ranked #1,980 of 2,715 Automation & Workflows skills by installs in the Skillselion catalog
Data as of Jul 7, 2026 (Skillselion catalog sync)

npx skills add https://github.com/skills.volces.com --skill general-web-scraper

Add your badge

Show developers this skill is listed on Skillselion. Paste this into your README.

[![Listed on Skillselion](https://skillselion.com/badge/skills/skills.volces.com/general-web-scraper.svg)](https://skillselion.com/skills/skills.volces.com/general-web-scraper)

Installs	1
Repository	skills.volces.com ↗

What it does

Pull links or table data from any URL with CSS selectors and export CSV or JSON for market and competitor research.

Files

SKILL.mdMarkdownGitHub ↗

Web Scraper — 通用网页数据抓取工具

AI agent 专用的网页数据抓取工具。输入网址和CSS选择器，自动抓取链接或表格数据，导出为CSV或JSON。

功能

链接抓取 — 抓取页面中所有匹配CSS选择器的链接
表格抓取 — 自动提取HTML表格数据
CSV导出 — 默认输出CSV格式
JSON导出 — 支持JSON格式输出
中文友好 — 完整支持中文网页编码

使用方式

# 抓取页面所有链接
python scraper.py https://example.com

# 自定义CSS选择器
python scraper.py https://example.com "a.article-link"

# 导出JSON格式
python scraper.py https://example.com "div.item" --json

# 抓取表格数据
python scraper.py https://example.com "table#data" --table --json