Openclaw Serper

{
  "owner": "nesdeq",
  "slug": "openclaw-serper",
  "displayName": "openclaw-serper",
  "latest": {
    "version": "3.1.1",
    "publishedAt": 1769899176982,
    "commit": "https://github.com/clawdbot/skills/commit/a305108ce45b0f74b845e3c494cfbb6fb2be68c7"
  },
  "history": []
}

# ============================================================
# Serper 搜索技能环境配置
# ============================================================
# 使用方法：
#   1. 复制本文件为 .env       → cp .env.example .env
#   2. 填入你的真实 API Key
#
# .env 查找顺序（优先级从高到低）：
#   1. 系统环境变量（已设置则不覆盖）
#   2. 本技能文件夹下的 .env
#   3. 上一级目录的 .env（适用于多技能共享同一份密钥）
# ============================================================

# Serper API 密钥（Google 搜索 API，用于网络检索）
# 获取地址：https://serper.dev（注册后免费额度 2500 次）
# 也接受 SERP_API_KEY 作为备用变量名
SERPER_API_KEY=your_serper_api_key_here

#!/usr/bin/env -S python3 -u
"""
Serper — Google search with full page content extraction via trafilatura.

Two search modes:
  - default:  all time web search (5 results, enriched)
  - current:  past week web + news (3 results each, enriched)

Locale is controlled via --gl and --hl flags.
"""

import argparse
import json
import os
import sys
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from typing import Optional, List, Dict, Any

from urllib.request import Request, urlopen
from urllib.error import HTTPError, URLError

try:
    import trafilatura
except ImportError:
    print(json.dumps({
        "error": "trafilatura is required but not installed",
        "fix": "pip install trafilatura",
    }, indent=2), flush=True)
    sys.exit(1)


# =============================================================================
# Auto-load .env from skill directory
# =============================================================================
def _load_env_file():
    env_paths = [
        Path(__file__).parent.parent / ".env",
        Path.home() / ".openclaw" / ".env",
    ]
    for env_path in env_paths:
        if env_path.exists():
            with open(env_path) as f:
                for line in f:
                    line = line.strip()
                    if line and not line.startswith("#") and "=" in line:
                        if line.startswith("export "):
                            line = line[7:]
                        key, _, value = line.partition("=")
                        key = key.strip()
                        value = value.strip().strip('"').strip("'")
                        if key and key not in os.environ:
                            os.environ[key] = value

_load_env_file()


# =============================================================================
# Configuration
# =============================================================================

FETCH_TIMEOUT = 3
USER_AGENT = "Mozilla/5.0 (compatible; Serper/3.0)"

SERP_SEARCH_URL = "https://google.serper.dev/search"
SERP_NEWS_URL = "https://google.serper.dev/news"

# Trafilatura config — shared across all threads
_traf_config = trafilatura.settings.use_config()
_traf_config.set("DEFAULT", "DOWNLOAD_TIMEOUT", str(FETCH_TIMEOUT))


def get_api_key() -> str:
    key = os.environ.get("SERPER_API_KEY") or os.environ.get("SERP_API_KEY")
    if not key:
        print(json.dumps({
            "error": "Missing Serper API key",
            "how_to_fix": [
                "1. Get a free key at https://serper.dev (2,500 queries free)",
                '2. Add SERPER_API_KEY="your-key" to .env in the skill directory',
            ],
        }, indent=2), flush=True)
        sys.exit(1)
    if len(key) < 10:
        print(json.dumps({"error": "Serper API key appears invalid (too short)"}), flush=True)
        sys.exit(1)
    return key


# =============================================================================
# Content extraction via trafilatura
# =============================================================================

def _extract_content(url: str) -> Optional[str]:
    """Fetch a URL and extract clean readable text using trafilatura."""
    try:
        downloaded = trafilatura.fetch_url(url, config=_traf_config)
        if not downloaded:
            return None
        return trafilatura.extract(downloaded, include_links=False, include_images=False,
                                   include_tables=True, deduplicate=True) or None
    except Exception:
        return None


# =============================================================================
# Serper API
# =============================================================================

def _serper_post(endpoint: str, api_key: str, payload: dict) -> dict:
    """POST to Serper API and return parsed JSON."""
    headers = {
        "X-API-KEY": api_key,
        "Content-Type": "application/json",
        "User-Agent": USER_AGENT,
    }
    data = json.dumps(payload).encode("utf-8")
    req = Request(endpoint, data=data, headers=headers, method="POST")
    try:
        with urlopen(req, timeout=10) as resp:
            return json.loads(resp.read().decode("utf-8"))
    except HTTPError as e:
        body = e.read().decode("utf-8", errors="replace") if e.fp else ""
        msgs = {
            401: "Invalid or expired API key.",
            429: "Rate limit exceeded. Wait and retry.",
        }
        raise Exception(msgs.get(e.code, f"Serper HTTP {e.code}: {body[:300]}"))
    except URLError as e:
        raise Exception(f"Network error: {e.reason}")
    except Exception as e:
        raise Exception(f"Request failed: {e}")


def serper_web_search(query: str, api_key: str, num: int = 5,
                      gl: Optional[str] = None, hl: str = "en",
                      tbs: Optional[str] = None) -> List[Dict[str, Any]]:
    """Web search via Serper. Returns list of result dicts."""
    payload: Dict[str, Any] = {"q": query, "num": num, "hl": hl, "autocorrect": True}
    if gl and gl != "world":
        payload["gl"] = gl
    if tbs:
        payload["tbs"] = tbs

    data = _serper_post(SERP_SEARCH_URL, api_key, payload)
    results = []

    kg = data.get("knowledgeGraph")
    if kg and "title" in kg:
        attrs = ""
        if "attributes" in kg:
            attrs = " | ".join(f"{k}: {v}" for k, v in kg["attributes"].items())
        results.append({
            "title": kg["title"],
            "snippet": attrs or kg.get("description", ""),
            "source": "knowledge_graph",
        })

    for item in data.get("organic", [])[:num]:
        r = {
            "title": item.get("title", ""),
            "url": item.get("link", ""),
            "snippet": item.get("snippet", ""),
            "source": "web",
        }
        if item.get("date"):
            r["date"] = item["date"]
        results.append(r)

    return results


def serper_news_search(query: str, api_key: str, num: int = 3,
                       gl: Optional[str] = None, hl: str = "en") -> List[Dict[str, Any]]:
    """News search via Serper. Returns list of result dicts."""
    payload: Dict[str, Any] = {"q": query, "num": num, "hl": hl}
    if gl and gl != "world":
        payload["gl"] = gl

    data = _serper_post(SERP_NEWS_URL, api_key, payload)
    results = []
    for item in data.get("news", [])[:num]:
        r = {
            "title": item.get("title", ""),
            "url": item.get("link", ""),
            "snippet": item.get("snippet", ""),
            "source": "news",
        }
        if item.get("date"):
            r["date"] = item["date"]
        results.append(r)
    return results


# =============================================================================
# Content enrichment — concurrent fetch, streamed as JSON array
# =============================================================================

def enrich_and_stream(results: List[Dict[str, Any]]):
    """Fetch full page content concurrently, print each as JSON array element in order."""
    futures = {}
    pool = ThreadPoolExecutor(max_workers=max(1, len(results)))
    for i, r in enumerate(results):
        if r.get("url"):
            futures[i] = pool.submit(_extract_content, r["url"])

    for i, r in enumerate(results):
        out: Dict[str, Any] = {"title": r["title"]}
        if r.get("url"):
            out["url"] = r["url"]
        out["source"] = r["source"]
        if r.get("date"):
            out["date"] = r["date"]

        if r["source"] == "knowledge_graph":
            out["content"] = r["snippet"]
        else:
            content = None
            if i in futures:
                try:
                    content = futures[i].result(timeout=FETCH_TIMEOUT)
                except Exception:
                    content = None
            out["content"] = content if content else r["snippet"]

        print("," + json.dumps(out, ensure_ascii=False), flush=True)

    pool.shutdown(wait=False)


def search_current(query: str, api_key: str, locale: Dict[str, Optional[str]]) -> List[Dict[str, Any]]:
    """Current/news mode: past week web + news search, 3 results each."""
    all_results = []
    seen_urls = set()

    for r in serper_web_search(query, api_key, num=3, gl=locale["gl"], hl=locale["hl"], tbs="qdr:w"):
        if r["source"] == "knowledge_graph":
            all_results.append(r)
        elif r["url"] not in seen_urls:
            seen_urls.add(r["url"])
            all_results.append(r)

    for r in serper_news_search(query, api_key, num=3, gl=locale["gl"], hl=locale["hl"]):
        url = r.get("url", "")
        if url not in seen_urls:
            seen_urls.add(url)
            all_results.append(r)

    return all_results


# =============================================================================
# CLI
# =============================================================================

def main():
    parser = argparse.ArgumentParser(
        description="Serper — Google search with full content extraction",
    )
    parser.add_argument("--query", "-q", required=True, help="Search query")
    parser.add_argument(
        "--mode", "-m",
        default="default",
        choices=["default", "current"],
        help="Search mode: default (all-time, 5 results) or current (past week + news, 3 each)",
    )
    parser.add_argument("--gl", default="world", help="Country code for Google (e.g. de, us, at, ch). Default: world")
    parser.add_argument("--hl", default="en", help="Language code for results (e.g. en, de)")

    args = parser.parse_args()
    api_key = get_api_key()
    locale = {"gl": args.gl, "hl": args.hl}

    if args.mode == "current":
        results = search_current(args.query, api_key, locale)
    else:
        results = serper_web_search(args.query, api_key, num=5, gl=locale["gl"], hl=locale["hl"])

    if not results:
        print(json.dumps({"error": "No results found", "query": args.query}), flush=True)
        sys.exit(1)

    # JSON array — first element is search metadata
    meta = {
        "query": args.query,
        "mode": args.mode,
        "locale": locale,
        "results": [
            {k: r[k] for k in ("title", "url", "source") if k in r}
            for r in results
        ],
    }
    print("[" + json.dumps(meta, ensure_ascii=False), flush=True)
    enrich_and_stream(results)
    print("]", flush=True)


if __name__ == "__main__":
    main()

---
title: "Specification"
description: "The complete format specification for Agent Skills."
---

This document defines the Agent Skills format.

## Directory structure

A skill is a directory containing at minimum a `SKILL.md` file:

```
skill-name/
└── SKILL.md          # Required
```

<Tip>
You can optionally include [additional directories](#optional-directories) such as `scripts/`, `references/`, and `assets/` to support your skill.
</Tip>

## SKILL.md format

The `SKILL.md` file must contain YAML frontmatter followed by Markdown content.

### Frontmatter (required)

```yaml
---
name: skill-name
description: A description of what this skill does and when to use it.
---
```

With optional fields:

```yaml
---
name: pdf-processing
description: Extract text and tables from PDF files, fill forms, merge documents.
license: Apache-2.0
metadata:
  author: example-org
  version: "1.0"
---
```

| Field | Required | Constraints |
|-------|----------|-------------|
| `name` | Yes | Max 64 characters. Lowercase letters, numbers, and hyphens only. Must not start or end with a hyphen. |
| `description` | Yes | Max 1024 characters. Non-empty. Describes what the skill does and when to use it. |
| `license` | No | License name or reference to a bundled license file. |
| `compatibility` | No | Max 500 characters. Indicates environment requirements (intended product, system packages, network access, etc.). |
| `metadata` | No | Arbitrary key-value mapping for additional metadata. |
| `allowed-tools` | No | Space-delimited list of pre-approved tools the skill may use. (Experimental) |

#### `name` field

The required `name` field:
- Must be 1-64 characters
- May only contain unicode lowercase alphanumeric characters and hyphens (`a-z` and `-`)
- Must not start or end with `-`
- Must not contain consecutive hyphens (`--`)
- Must match the parent directory name

Valid examples:
```yaml
name: pdf-processing
```
```yaml
name: data-analysis
```
```yaml
name: code-review
```

Invalid examples:
```yaml
name: PDF-Processing  # uppercase not allowed
```
```yaml
name: -pdf  # cannot start with hyphen
```
```yaml
name: pdf--processing  # consecutive hyphens not allowed
```

#### `description` field

The required `description` field:
- Must be 1-1024 characters
- Should describe both what the skill does and when to use it
- Should include specific keywords that help agents identify relevant tasks

Good example:
```yaml
description: Extracts text and tables from PDF files, fills PDF forms, and merges multiple PDFs. Use when working with PDF documents or when the user mentions PDFs, forms, or document extraction.
```

Poor example:
```yaml
description: Helps with PDFs.
```

#### `license` field

The optional `license` field:
- Specifies the license applied to the skill
- We recommend keeping it short (either the name of a license or the name of a bundled license file)

Example:
```yaml
license: Proprietary. LICENSE.txt has complete terms
```

#### `compatibility` field

The optional `compatibility` field:
- Must be 1-500 characters if provided
- Should only be included if your skill has specific environment requirements
- Can indicate intended product, required system packages, network access needs, etc.

Examples:
```yaml
compatibility: Designed for Claude Code (or similar products)
```
```yaml
compatibility: Requires git, docker, jq, and access to the internet
```

<Note>
Most skills do not need the `compatibility` field.
</Note>

#### `metadata` field

The optional `metadata` field:
- A map from string keys to string values
- Clients can use this to store additional properties not defined by the Agent Skills spec
- We recommend making your key names reasonably unique to avoid accidental conflicts

Example:
```yaml
metadata:
  author: example-org
  version: "1.0"
```

#### `allowed-tools` field

The optional `allowed-tools` field:
- A space-delimited list of tools that are pre-approved to run
- Experimental. Support for this field may vary between agent implementations

Example:
```yaml
allowed-tools: Bash(git:*) Bash(jq:*) Read
```

### Body content

The Markdown body after the frontmatter contains the skill instructions. There are no format restrictions. Write whatever helps agents perform the task effectively.

Recommended sections:
- Step-by-step instructions
- Examples of inputs and outputs
- Common edge cases

Note that the agent will load this entire file once it's decided to activate a skill. Consider splitting longer `SKILL.md` content into referenced files.

## Optional directories

### scripts/

Contains executable code that agents can run. Scripts should:
- Be self-contained or clearly document dependencies
- Include helpful error messages
- Handle edge cases gracefully

Supported languages depend on the agent implementation. Common options include Python, Bash, and JavaScript.

### references/

Contains additional documentation that agents can read when needed:
- `REFERENCE.md` - Detailed technical reference
- `FORMS.md` - Form templates or structured data formats
- Domain-specific files (`finance.md`, `legal.md`, etc.)

Keep individual [reference files](#file-references) focused. Agents load these on demand, so smaller files mean less use of context.

### assets/

Contains static resources:
- Templates (document templates, configuration templates)
- Images (diagrams, examples)
- Data files (lookup tables, schemas)

## Progressive disclosure

Skills should be structured for efficient use of context:

1. **Metadata** (~100 tokens): The `name` and `description` fields are loaded at startup for all skills
2. **Instructions** (< 5000 tokens recommended): The full `SKILL.md` body is loaded when the skill is activated
3. **Resources** (as needed): Files (e.g. those in `scripts/`, `references/`, or `assets/`) are loaded only when required

Keep your main `SKILL.md` under 500 lines. Move detailed reference material to separate files.

## File references

When referencing other files in your skill, use relative paths from the skill root:

```markdown
See [the reference guide](references/REFERENCE.md) for details.

Run the extraction script:
scripts/extract.py
```

Keep file references one level deep from `SKILL.md`. Avoid deeply nested reference chains.

## Validation

Use the [skills-ref](https://github.com/agentskills/agentskills/tree/main/skills-ref) reference library to validate your skills:

```bash
skills-ref validate ./my-skill
```

This checks that your `SKILL.md` frontmatter is valid and follows all naming conventions.