Crawl4ai Skill
- 8 installs
- 33 repo stars
- Updated April 26, 2026
- bighardperson/computer-science-skills-collection
crawl4ai-skill is a skill for web search, scraping and site crawling that outputs LLM-optimized Markdown with no API key required.
About
crawl4ai-skill is a web crawling and scraping tool with LLM-optimized Markdown output. It offers DuckDuckGo search, single-page scraping, full-site crawling and dynamic (JavaScript-rendered) page scraping, needing no API key. A developer uses it to pull clean, token-efficient content from docs sites and web pages for research or LLM ingestion.
- Web search (DuckDuckGo), single-page scraping and full-site crawling with no API key
- LLM-optimized fit_markdown output that removes nav/ads and cuts tokens ~80%
- Handles JavaScript-rendered dynamic pages with wait/delay options
Crawl4ai Skill by the numbers
- 8 all-time installs (skills.sh)
- Ranked #1,522 of 2,719 Automation & Workflows skills by installs in the Skillselion catalog
- Data as of Jul 30, 2026 (Skillselion catalog sync)
crawl4ai-skill capabilities & compatibility
free, no API key required
- Capabilities
- web scraping · web crawling · web search
- Use cases
- web scraping · web search · research · token optimization
- Pricing
- Free
What crawl4ai-skill says it does
Web crawling and scraping tool with LLM-optimized output.
Free, no API key required.
npx skills add https://github.com/bighardperson/computer-science-skills-collection --skill crawl4ai-skillAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 8 |
|---|---|
| repo stars | ★ 33 |
| Last updated | April 26, 2026 |
| Repository | bighardperson/computer-science-skills-collection ↗ |
What it does
Search the web and scrape or crawl pages into clean, token-efficient Markdown with no API key required.
Who is it for?
scraping docs sites and web pages into clean, token-efficient Markdown for LLM use
When should I use this skill?
you need to search the web, scrape a page, or crawl a whole site into Markdown
What you get
Clean fit_markdown that strips nav and ads and cuts tokens by roughly 80%.
By the numbers
- 4 core commands
- ~80% token reduction with fit_markdown
Files
Crawl4AI Skill - Web Crawler & Scraper
Web Crawling 网页爬虫 | Web Scraping 网页爬取 | LLM 优化输出
智能网页爬虫和爬取工具，支持搜索、全站爬取、动态页面抓取。Free web crawler and scraper with LLM-optimized Markdown output.
核心功能 | Core Features
- 🔍 Web Search 网页搜索 - DuckDuckGo search, 免 API key
- 🕷️ Web Crawling 网页爬虫 - Site crawler, spider, sitemap 识别
- 📝 Web Scraping 网页抓取 - Smart scraper, data extraction
- 📄 LLM-Optimized Output - Fit Markdown, 省 Token 80%
- ⚡ Dynamic Page Scraping - JavaScript 渲染页面爬取
---
快速开始 | Quick Start
安装 | Installation
pip install crawl4ai-skillWeb Search | 网页搜索
# Search the web with DuckDuckGo
crawl4ai-skill search "python web scraping"Web Scraping | 单页爬取
# Scrape a single web page
crawl4ai-skill crawl https://example.comWeb Crawling | 全站爬虫
# Crawl entire website / spider
crawl4ai-skill crawl-site https://docs.python.org --max-pages 50---
使用场景 | Use Cases
场景 1：Web Crawler for Documentation | 文档站爬虫
# Crawl documentation site with spider
crawl4ai-skill crawl-site https://docs.fastapi.com --max-pages 100爬虫效果 | Crawler Output:
- ❌ 移除：导航栏、侧边栏、广告
- ✅ 保留：标题、正文、代码块
- 📊 Token：50,000 → 10,000（-80%）
场景 2：Search + Scrape | 搜索+爬取
# Search and scrape top results
crawl4ai-skill search-and-crawl "Vue 3 best practices" --crawl-top 3场景 3：Dynamic Page Scraping | 动态页面抓取
JavaScript 渲染的页面爬取（雪球、知乎等）：
# Scrape JavaScript-heavy pages
crawl4ai-skill crawl https://xueqiu.com/S/BIDU --wait-until networkidle --delay 2---
命令参考 | Commands
| 命令 Command | 说明 Description |
|---|---|
search <query> | Web search 网页搜索 |
crawl <url> | Web scraping 单页爬取 |
crawl-site <url> | Web crawling 全站爬虫 |
search-and-crawl <query> | Search + scrape 搜索并爬取 |
常用参数 | Common Options
# Web Search 搜索
--num-results 10 # Number of results
# Web Scraping 爬取
--format fit_markdown # Output format
--output result.md # Output file
--wait-until networkidle # Wait strategy for dynamic pages
--delay 2 # Additional wait time (seconds)
--wait-for ".selector" # Wait for specific element
# Web Crawling 爬虫
--max-pages 100 # Max pages to crawl
--max-depth 3 # Max crawl depth---
输出格式 | Output Formats
fit_markdown（推荐 Recommended）
智能提取，节省 80% Token。Smart extraction, save 80% tokens.
crawl4ai-skill crawl https://example.com --format fit_markdownraw_markdown
保留完整结构。Preserve full structure.
crawl4ai-skill crawl https://example.com --format raw_markdown---
为什么选择这个爬虫？| Why This Crawler?
✅ 免费爬虫 Free Crawler - 无需 API key，开箱即用 ✅ 智能爬取 Smart Scraper - 自动去噪，提取核心内容 ✅ 全站爬虫 Site Crawler - 支持 sitemap，递归爬取 ✅ 动态爬取 Dynamic Scraping - JavaScript 渲染页面支持 ✅ 搜索集成 Search Integration - DuckDuckGo 搜索内置
---
链接 | Links
{
"ownerId": "kn7a1m1ezm5n3tz3y2n2t9zc3d826658",
"slug": "crawl4ai-skill",
"version": "1.0.10",
"publishedAt": 1773206160408
}{
"version": 1,
"registry": "https://clawhub.ai",
"slug": "crawl4ai-skill",
"installedVersion": "1.0.10",
"installedAt": 1776070446969
}
Related skills
Agent BrowserGive their coding agent reliable, high-fidelity control over any website or Electron desktop app.591k39.6k
Lark ApprovalLet their AI coding agent create, read, update, and approve items in Lark (Feishu) approval workflows without leaving the coding environment.494kLark EventHandle Feishu/Lark bot events, webhooks, and subscription callbacks in agent-driven backend code.493kLark Workflow Meeting SummaryAutomatically generate structured meeting summaries and action items from Lark/Feishu calls without manual note-taking.492kLark Workflow Standup ReportAutomatically generate and post daily standup reports in Lark/Feishu from their workflow and activity data.492k
Twitter AutomationAutomate X/Twitter posts, scheduling, and engagement via X API integration441k669FAQ
Does crawl4ai-skill need an API key?
No, it is free and requires no API key, using DuckDuckGo for search.
How much does the LLM-optimized output save?
The fit_markdown format removes nav/ads and cuts tokens by about 80%.