
Topic Monitor
- 589 installs
- 635 repo stars
- Updated March 7, 2026
- sundial-org/awesome-openclaw-skills
topic-monitor is an agent skill that runs scheduled topic watches with AI importance scoring and Telegram, Discord, or email alerts for developers who need proactive news surfacing instead of manual polling.
About
topic-monitor is an OpenClaw agent skill that turns a coding assistant from reactive to proactive by scheduling automated web searches on topics you define. Each run applies AI importance scoring to classify findings as immediate alerts, weekly digest items, or noise to ignore, with rate limiting to reduce alert fatigue. Alerts can route through Telegram, Discord, or email, and memory integration lets notifications reference prior conversation context. Developers reach for topic-monitor when they want continuous domain monitoring—security advisories, competitor moves, framework releases—without writing custom cron jobs or RSS parsers. Weekly digests summarize interesting findings that did not warrant instant alerts.
- Interactive `setup.py` wizard or manual `config.json` from template
- `manage_topics.py` for add, test, keywords, frequency, and importance
- AI importance scoring: alert versus digest versus ignore
- Multi-channel delivery: Telegram, Discord, email
- Weekly digests, memory integration, and rate limiting against alert fatigue
Topic Monitor by the numbers
- 589 all-time installs (skills.sh)
- +3 installs in the week ending Jul 27, 2026 (Skillselion tracking)
- Ranked #387 of 2,742 Automation & Workflows skills by installs in the Skillselion catalog
- Security screen: HIGH risk (skills.sh audit)
- Data as of Jul 27, 2026 (Skillselion catalog sync)
npx skills add https://github.com/sundial-org/awesome-openclaw-skills --skill topic-monitorAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 589 |
|---|---|
| repo stars | ★ 635 |
| Security audit | 0 / 3 scanners passed |
| Last updated | March 7, 2026 |
| Repository | sundial-org/awesome-openclaw-skills ↗ |
How do you automate topic monitoring with smart alerts?
Run scheduled topic watches with AI importance scoring and Telegram, Discord, or email alerts so your assistant surfaces news instead of waiting for prompts.
Who is it for?
Developers running OpenClaw assistants who need hands-off monitoring of technical topics with filtered, multi-channel alerts.
Skip if: Teams that only need one-off web searches at prompt time and do not want scheduled background jobs or notification channels.
When should I use this skill?
The user asks to watch, track, or get alerted on topics, news, or domains on a recurring schedule with digest or instant notification routing.
What you get
Scheduled watch jobs, scored alert/digest/ignore decisions, and multi-channel notification payloads
- Scheduled topic watch configuration
- Scored alert or digest messages
Files
Topic Monitor
Monitor what matters. Get notified when it happens.
Topic Monitor transforms your assistant from reactive to proactive by continuously monitoring topics you care about and intelligently alerting you only when something truly matters.
Core Capabilities
1. Topic Configuration - Define subjects with custom parameters 2. Scheduled Monitoring - Automated searches at configurable intervals 3. AI Importance Scoring - Smart filtering: immediate alert vs digest vs ignore 4. Contextual Summaries - Not just links—meaningful summaries with context 5. Weekly Digest - Low-priority findings compiled into readable reports 6. Memory Integration - References your past conversations and interests
First Run
When you first use Topic Monitor, run the interactive setup wizard:
python3 scripts/setup.pyThe wizard will guide you through:
1. Topics - What subjects do you want to monitor? 2. Search queries - How to search for each topic 3. Keywords - What terms indicate relevance 4. Frequency - How often to check (hourly/daily/weekly) 5. Importance threshold - When to send alerts (low/medium/high) 6. Weekly digest - Compile non-urgent findings into a summary
The wizard creates config.json with your preferences. You can always edit it later or use manage_topics.py to add/remove topics.
Example session:
🔍 Topic Monitor - Setup Wizard
What topics do you want to monitor?
> AI Model Releases
> Security Vulnerabilities
>
--- Topic 1/2: AI Model Releases ---
Search query for 'AI Model Releases' [AI Model Releases news updates]: new AI model release announcement
Keywords to watch for in 'AI Model Releases'?
> GPT, Claude, Llama, release
--- Topic 2/2: Security Vulnerabilities ---
Search query for 'Security Vulnerabilities' [Security Vulnerabilities news updates]: CVE critical vulnerability patch
Keywords to watch for in 'Security Vulnerabilities'?
> CVE, vulnerability, critical, patch
How often should I check for updates?
1. hourly
2. daily *
3. weekly
✅ Setup Complete!Quick Start
Already know what you're doing? Here's the manual approach:
# Initialize config from template
cp config.example.json config.json
# Add a topic
python3 scripts/manage_topics.py add "Product Updates" \
--keywords "release,update,patch" \
--frequency daily \
--importance medium
# Test monitoring (dry run)
python3 scripts/monitor.py --dry-run
# Set up cron for automatic monitoring
python3 scripts/setup_cron.pyTopic Configuration
Each topic has:
- name - Display name (e.g., "AI Model Releases")
- query - Search query (e.g., "new AI model release announcement")
- keywords - Relevance filters (["GPT", "Claude", "Llama", "release"])
- frequency -
hourly,daily,weekly - importance_threshold -
high(alert immediately),medium(alert if important),low(digest only) - channels - Where to send alerts (["telegram", "discord"])
- context - Why you care (for AI contextual summaries)
Example config.json
{
"topics": [
{
"id": "ai-models",
"name": "AI Model Releases",
"query": "new AI model release GPT Claude Llama",
"keywords": ["GPT", "Claude", "Llama", "release", "announcement"],
"frequency": "daily",
"importance_threshold": "high",
"channels": ["telegram"],
"context": "Following AI developments for work",
"alert_on": ["model_release", "major_update"]
},
{
"id": "tech-news",
"name": "Tech Industry News",
"query": "technology startup funding acquisition",
"keywords": ["startup", "funding", "Series A", "acquisition"],
"frequency": "daily",
"importance_threshold": "medium",
"channels": ["telegram"],
"context": "Staying informed on tech trends",
"alert_on": ["major_funding", "acquisition"]
},
{
"id": "security-alerts",
"name": "Security Vulnerabilities",
"query": "CVE critical vulnerability security patch",
"keywords": ["CVE", "vulnerability", "security", "patch", "critical"],
"frequency": "hourly",
"importance_threshold": "high",
"channels": ["telegram", "email"],
"context": "DevOps security monitoring",
"alert_on": ["critical_cve", "zero_day"]
}
],
"settings": {
"digest_day": "sunday",
"digest_time": "18:00",
"max_alerts_per_day": 5,
"deduplication_window_hours": 72,
"learning_enabled": true
}
}Scripts
manage_topics.py
Manage research topics:
# Add topic
python3 scripts/manage_topics.py add "Topic Name" \
--query "search query" \
--keywords "word1,word2" \
--frequency daily \
--importance medium \
--channels telegram
# List topics
python3 scripts/manage_topics.py list
# Edit topic
python3 scripts/manage_topics.py edit eth-price --frequency hourly
# Remove topic
python3 scripts/manage_topics.py remove eth-price
# Test topic (preview results without saving)
python3 scripts/manage_topics.py test eth-pricemonitor.py
Main monitoring script (run via cron):
# Normal run (alerts + saves state)
python3 scripts/monitor.py
# Dry run (no alerts, shows what would happen)
python3 scripts/monitor.py --dry-run
# Force check specific topic
python3 scripts/monitor.py --topic eth-price
# Verbose logging
python3 scripts/monitor.py --verboseHow it works: 1. Reads topics due for checking (based on frequency) 2. Searches using web-search-plus or built-in web_search 3. Scores each result with AI importance scorer 4. High-importance → immediate alert 5. Medium-importance → saved for digest 6. Low-importance → ignored 7. Updates state to prevent duplicate alerts
digest.py
Generate weekly digest:
# Generate digest for current week
python3 scripts/digest.py
# Generate and send
python3 scripts/digest.py --send
# Preview without sending
python3 scripts/digest.py --previewOutput format:
# Weekly Research Digest - [Date Range]
## 🔥 Highlights
- **AI Models**: Claude 4.5 released with improved reasoning
- **Security**: Critical CVE patched in popular framework
## 📊 By Topic
### AI Model Releases
- [3 findings this week]
### Security Vulnerabilities
- [1 finding this week]
## 💡 Recommendations
Based on your interests, you might want to monitor:
- "Kubernetes security" (mentioned 3x this week)setup_cron.py
Configure automated monitoring:
# Interactive setup
python3 scripts/setup_cron.py
# Auto-setup with defaults
python3 scripts/setup_cron.py --auto
# Remove cron jobs
python3 scripts/setup_cron.py --removeCreates cron entries:
# Topic Monitor - Hourly topics
0 * * * * cd /path/to/skills/topic-monitor && python3 scripts/monitor.py --frequency hourly
# Topic Monitor - Daily topics
0 9 * * * cd /path/to/skills/topic-monitor && python3 scripts/monitor.py --frequency daily
# Topic Monitor - Weekly digest
0 18 * * 0 cd /path/to/skills/topic-monitor && python3 scripts/digest.py --sendAI Importance Scoring
The scorer uses multiple signals to decide alert priority:
Scoring Signals
HIGH priority (immediate alert):
- Major breaking news (detected via freshness + keyword density)
- Price changes >10% (for finance topics)
- Product releases matching your exact keywords
- Security vulnerabilities in tools you use
- Direct answers to specific questions you asked
MEDIUM priority (digest-worthy):
- Related news but not urgent
- Minor updates to tracked products
- Interesting developments in your topics
- Tutorial/guide releases
- Community discussions with high engagement
LOW priority (ignore):
- Duplicate news (already alerted)
- Tangentially related content
- Low-quality sources
- Outdated information
- Spam/promotional content
Learning Mode
When enabled (learning_enabled: true), the system: 1. Tracks which alerts you interact with 2. Adjusts scoring weights based on your behavior 3. Suggests topic refinements 4. Auto-adjusts importance thresholds
Learning data stored in .learning_data.json (privacy-safe, never shared).
Memory Integration
Topic Monitor connects to your conversation history:
Example alert:
🔔 Dirac Live Update
>
Version 3.8 released with the room correction improvements you asked about last week.
>
Context: You mentioned struggling with bass response in your studio. This update includes new low-frequency optimization.
>
[Link] | [Full details]
How it works: 1. Reads references/memory_hints.md (create this file) 2. Scans recent conversation logs (if available) 3. Matches findings to past context 4. Generates personalized summaries
memory_hints.md (optional)
Help the AI connect dots:
# Memory Hints for Topic Monitor
## AI Models
- Using Claude for coding assistance
- Interested in reasoning improvements
- Comparing models for different use cases
## Security
- Running production Kubernetes clusters
- Need to patch critical CVEs quickly
- Interested in zero-day disclosures
## Tech News
- Following startup ecosystem
- Interested in developer tools space
- Tracking potential acquisition targetsAlert Channels
Telegram
Requires OpenClaw message tool:
{
"channels": ["telegram"],
"telegram_config": {
"chat_id": "@your_username",
"silent": false,
"effects": {
"high_importance": "🔥",
"medium_importance": "📌"
}
}
}Discord
Webhook-based:
{
"channels": ["discord"],
"discord_config": {
"webhook_url": "https://discord.com/api/webhooks/...",
"username": "Research Bot",
"avatar_url": "https://..."
}
}SMTP or API:
{
"channels": ["email"],
"email_config": {
"to": "you@example.com",
"from": "research@yourdomain.com",
"smtp_server": "smtp.gmail.com",
"smtp_port": 587
}
}Advanced Features
Alert Conditions
Fine-tune when to alert:
{
"alert_on": [
"price_change_10pct",
"keyword_exact_match",
"source_tier_1",
"high_engagement"
],
"ignore_sources": [
"spam-site.com",
"clickbait-news.io"
],
"boost_sources": [
"github.com",
"arxiv.org",
"official-site.com"
]
}Regex Patterns
Match specific patterns:
{
"patterns": [
"version \\d+\\.\\d+\\.\\d+",
"\\$\\d{1,3}(,\\d{3})*",
"CVE-\\d{4}-\\d+"
]
}Rate Limiting
Prevent alert fatigue:
{
"settings": {
"max_alerts_per_day": 5,
"max_alerts_per_topic_per_day": 2,
"quiet_hours": {
"start": "22:00",
"end": "08:00"
}
}
}State Management
.research_state.json
Tracks:
- Last check time per topic
- Alerted URLs (deduplication)
- Importance scores history
- Learning data (if enabled)
Example:
{
"topics": {
"eth-price": {
"last_check": "2026-01-28T22:00:00Z",
"last_alert": "2026-01-28T15:30:00Z",
"alerted_urls": [
"https://example.com/eth-news-1"
],
"findings_count": 3,
"alerts_today": 1
}
},
"deduplication": {
"url_hash_map": {
"abc123": "2026-01-28T15:30:00Z"
}
}
}.findings/ directory
Stores digest-worthy findings:
.findings/
├── 2026-01-22_eth-price.json
├── 2026-01-24_fm26-patches.json
└── 2026-01-27_ai-breakthroughs.jsonBest Practices
1. Start conservative - Set importance_threshold: medium initially, adjust based on alert quality 2. Use context field - Helps AI generate better summaries 3. Refine keywords - Add negative keywords to filter noise: "keywords": ["AI", "-clickbait", "-spam"] 4. Enable learning - Improves over time based on your behavior 5. Review digest weekly - Don't ignore the digest—it surfaces patterns 6. Combine with personal-analytics - Get topic recommendations based on your chat patterns
Integration with Other Skills
web-search-plus
Automatically uses intelligent routing:
- Product/price topics → Serper
- Research topics → Tavily
- Company/startup discovery → Exa
personal-analytics
Suggests topics based on conversation patterns:
"You've asked about Rust 12 times this month. Want me to monitor 'Rust language updates'?"
Privacy & Security
- All data local - No external services except search APIs
- State files gitignored - Safe to use in version-controlled workspace
- Memory hints optional - You control what context is shared
- Learning data stays local - Never sent to APIs
Troubleshooting
No alerts being sent:
- Check cron is running:
crontab -l - Verify channel config (Telegram chat ID, Discord webhook)
- Run with
--dry-run --verboseto see scoring
Too many alerts:
- Increase
importance_threshold - Add rate limiting
- Refine keywords (add negative filters)
- Enable learning mode
Missing important news:
- Decrease
importance_threshold - Increase check frequency
- Broaden keywords
- Check
.research_state.jsonfor deduplication issues
Digest not generating:
- Verify
.findings/directory exists and has content - Check digest cron schedule
- Run manually:
python3 scripts/digest.py --preview
Example Workflows
Track Product Release
python3 scripts/manage_topics.py add "iPhone 17 Release" \
--query "iPhone 17 announcement release date" \
--keywords "iPhone 17,Apple event,September" \
--frequency daily \
--importance high \
--channels telegram \
--context "Planning to upgrade from iPhone 13"Monitor Competitor
python3 scripts/manage_topics.py add "Competitor Analysis" \
--query "CompetitorCo product launch funding" \
--keywords "CompetitorCo,product,launch,Series,funding" \
--frequency weekly \
--importance medium \
--channels discord,emailResearch Topic
python3 scripts/manage_topics.py add "Quantum Computing Papers" \
--query "quantum computing arxiv" \
--keywords "quantum,qubit,arxiv" \
--frequency weekly \
--importance low \
--channels emailCredits
Built for ClawHub. Uses web-search-plus skill for intelligent search routing.
{
"topics": [
{
"id": "example-topic",
"name": "Example Topic",
"query": "search query here",
"keywords": ["keyword1", "keyword2", "keyword3"],
"frequency": "daily",
"importance_threshold": "medium",
"channels": ["telegram"],
"context": "Why this topic matters to you",
"alert_on": ["keyword_exact_match"],
"ignore_sources": [],
"boost_sources": []
}
],
"settings": {
"digest_day": "sunday",
"digest_time": "18:00",
"max_alerts_per_day": 5,
"max_alerts_per_topic_per_day": 2,
"deduplication_window_hours": 72,
"learning_enabled": true,
"quiet_hours": {
"enabled": false,
"start": "22:00",
"end": "08:00"
}
},
"channels": {
"telegram": {
"enabled": true,
"chat_id": null,
"silent": false,
"effects": {
"high_importance": "🔥",
"medium_importance": "📌"
}
},
"discord": {
"enabled": false,
"webhook_url": null,
"username": "Research Bot",
"avatar_url": null
},
"email": {
"enabled": false,
"to": null,
"from": "research@yourdomain.com",
"smtp_server": "smtp.gmail.com",
"smtp_port": 587,
"smtp_user": null,
"smtp_password": null
}
}
}
Topic Monitor
Never miss what matters. Get alerted when it happens.
Topic Monitor transforms your assistant from reactive to proactive by continuously monitoring topics you care about and intelligently alerting you only when something truly important occurs.
Features
- 🔍 Automated Monitoring - Scheduled web searches for your topics
- 🧠 AI Importance Scoring - Smart filtering: alert vs digest vs ignore
- 📱 Multi-Channel Alerts - Telegram, Discord, Email
- 📊 Weekly Digests - Curated summaries of interesting findings
- 🧩 Memory Integration - Contextual alerts referencing your past conversations
- ⚡ Rate Limiting - Prevent alert fatigue
- 🎯 Custom Conditions - Fine-tune when to alert
Quick Start
Option 1: Interactive Setup (Recommended)
Run the setup wizard for a guided experience:
python3 scripts/setup.pyThe wizard asks friendly questions about what you want to monitor and creates your config automatically.
Option 2: Manual Setup
# 1. Copy the template
cp config.example.json config.json
# 2. Add your first topic
python3 scripts/manage_topics.py add "AI Models" \
--query "new AI model release announcement" \
--keywords "GPT,Claude,Llama,release" \
--frequency daily \
--importance high \
--channels telegram
# 3. Test it
python3 scripts/manage_topics.py test ai-models
# 4. Set up automated monitoring
python3 scripts/setup_cron.pyUse Cases
📈 Price Monitoring
Track product prices, SaaS pricing changes, or market trends with alerts on significant changes.
🔧 Product Updates
Monitor software releases, patches, and feature announcements.
📰 News Tracking
Stay updated on specific topics without drowning in noise.
🏢 Competitor Analysis
Track competitor product launches, funding, and news.
🎓 Research Papers
Monitor arXiv, GitHub, or academic publications in your field.
How It Works
1. Configure Topics - Define what to monitor and when to alert 2. Scheduled Checks - Cron jobs run searches at your chosen frequency 3. AI Scoring - Each result is scored for importance 4. Smart Alerting - High priority → immediate alert, Medium → digest, Low → ignore 5. Deduplication - Never get the same alert twice
Configuration
See SKILL.md for complete documentation.
Example Topic
{
"id": "ai-breakthroughs",
"name": "AI Research Breakthroughs",
"query": "artificial intelligence breakthrough research",
"keywords": ["AI", "LLM", "transformer", "AGI"],
"frequency": "daily",
"importance_threshold": "medium",
"channels": ["telegram"],
"context": "Following AI developments for work",
"alert_on": ["major_paper", "model_release"]
}Commands
Manage Topics
# Add topic
python3 scripts/manage_topics.py add "Topic Name" \
--query "search query" \
--keywords "word1,word2" \
--frequency daily
# List topics
python3 scripts/manage_topics.py list
# Edit topic
python3 scripts/manage_topics.py edit topic-id --frequency hourly
# Remove topic
python3 scripts/manage_topics.py remove topic-id
# Test topic
python3 scripts/manage_topics.py test topic-idMonitor
# Manual check (dry run)
python3 scripts/monitor.py --dry-run --verbose
# Check specific topic
python3 scripts/monitor.py --topic ai-models
# Check all hourly topics
python3 scripts/monitor.py --frequency hourlyDigest
# Preview this week's digest
python3 scripts/digest.py --preview
# Generate and send
python3 scripts/digest.py --sendCron Setup
# Interactive setup
python3 scripts/setup_cron.py
# Auto-setup
python3 scripts/setup_cron.py --auto
# Remove cron jobs
python3 scripts/setup_cron.py --removeIntegration
Works With
- web-search-plus - Intelligent search routing (Serper, Tavily, Exa)
- personal-analytics - Get topic recommendations from your chat patterns
- OpenClaw message tool - Send alerts via Telegram, Discord
Channel Setup
Telegram
Configure in config.json:
{
"channels": {
"telegram": {
"enabled": true,
"chat_id": "@your_username"
}
}
}Discord
Add webhook URL:
{
"channels": {
"discord": {
"enabled": true,
"webhook_url": "https://discord.com/api/webhooks/..."
}
}
}Privacy
- All data stored locally
- No external services except search APIs
- Learning data stays on your machine
- State files are gitignored
Requirements
- Python 3.8+
- Optional: web-search-plus skill (for better search)
- Cron (for automated monitoring)
License
MIT
Credits
Built for ClawHub.
#!/usr/bin/env python3
"""
Configuration loader for proactive-research skill.
"""
import json
from pathlib import Path
from typing import Dict, List, Optional
SKILL_DIR = Path(__file__).parent.parent
CONFIG_FILE = SKILL_DIR / "config.json"
STATE_FILE = SKILL_DIR / ".research_state.json"
FINDINGS_DIR = SKILL_DIR / ".findings"
def load_config() -> Dict:
"""Load configuration from config.json."""
if not CONFIG_FILE.exists():
raise FileNotFoundError(
f"Config file not found: {CONFIG_FILE}\n"
"Copy config.example.json to config.json and customize it."
)
with open(CONFIG_FILE) as f:
return json.load(f)
def save_config(config: Dict):
"""Save configuration to config.json."""
with open(CONFIG_FILE, 'w') as f:
json.dump(config, f, indent=2)
def load_state() -> Dict:
"""Load state from .research_state.json."""
if STATE_FILE.exists():
with open(STATE_FILE) as f:
return json.load(f)
return {
"topics": {},
"deduplication": {"url_hash_map": {}},
"learning": {"interactions": []}
}
def save_state(state: Dict):
"""Save state to .research_state.json."""
with open(STATE_FILE, 'w') as f:
json.dump(state, f, indent=2)
def get_topics() -> List[Dict]:
"""Get all topics from config."""
config = load_config()
return config.get("topics", [])
def get_topic(topic_id: str) -> Optional[Dict]:
"""Get a specific topic by ID."""
topics = get_topics()
for topic in topics:
if topic.get("id") == topic_id:
return topic
return None
def get_settings() -> Dict:
"""Get global settings."""
config = load_config()
return config.get("settings", {})
def get_channel_config(channel: str) -> Dict:
"""Get channel-specific configuration."""
config = load_config()
channels = config.get("channels", {})
return channels.get(channel, {})
def ensure_findings_dir():
"""Ensure .findings directory exists."""
FINDINGS_DIR.mkdir(exist_ok=True)
def get_findings_file(topic_id: str, date_str: str) -> Path:
"""Get path to findings file for topic and date."""
ensure_findings_dir()
return FINDINGS_DIR / f"{date_str}_{topic_id}.json"
def save_finding(topic_id: str, date_str: str, finding: Dict):
"""Save a finding to the findings directory."""
findings_file = get_findings_file(topic_id, date_str)
# Load existing findings
findings = []
if findings_file.exists():
with open(findings_file) as f:
findings = json.load(f)
# Append new finding
findings.append(finding)
# Save
with open(findings_file, 'w') as f:
json.dump(findings, f, indent=2)
def load_findings(topic_id: str, date_str: str) -> List[Dict]:
"""Load findings for a topic and date."""
findings_file = get_findings_file(topic_id, date_str)
if findings_file.exists():
with open(findings_file) as f:
return json.load(f)
return []
#!/usr/bin/env python3
"""
Generate and send weekly research digest.
Compiles medium-priority findings from the week into a readable report.
"""
import sys
import json
import argparse
from datetime import datetime, timedelta
from pathlib import Path
from collections import defaultdict
sys.path.insert(0, str(Path(__file__).parent))
from config import load_config, get_topic, FINDINGS_DIR
def get_week_range(offset_weeks: int = 0) -> tuple[datetime, datetime]:
"""Get start and end of current week (or offset)."""
today = datetime.now()
# Find most recent Sunday
days_since_sunday = (today.weekday() + 1) % 7
sunday = today - timedelta(days=days_since_sunday)
# Apply offset
start = sunday - timedelta(weeks=offset_weeks)
end = start + timedelta(days=6, hours=23, minutes=59, seconds=59)
return start, end
def load_week_findings(start: datetime, end: datetime) -> dict:
"""Load all findings from the week."""
if not FINDINGS_DIR.exists():
return {}
findings_by_topic = defaultdict(list)
# Scan findings directory
for findings_file in FINDINGS_DIR.glob("*.json"):
# Parse filename: YYYY-MM-DD_topic-id.json
parts = findings_file.stem.split("_", 1)
if len(parts) != 2:
continue
date_str, topic_id = parts
try:
file_date = datetime.strptime(date_str, "%Y-%m-%d")
except ValueError:
continue
# Check if in week range
if not (start <= file_date <= end):
continue
# Load findings
with open(findings_file) as f:
findings = json.load(f)
findings_by_topic[topic_id].extend(findings)
return dict(findings_by_topic)
def generate_digest(findings_by_topic: dict, start: datetime, end: datetime) -> str:
"""Generate digest markdown."""
config = load_config()
# Header
digest = f"# 📊 Weekly Research Digest\n\n"
digest += f"**{start.strftime('%B %d')} - {end.strftime('%B %d, %Y')}**\n\n"
digest += "---\n\n"
# Summary stats
total_findings = sum(len(f) for f in findings_by_topic.values())
topic_count = len(findings_by_topic)
if total_findings == 0:
digest += "No new findings this week.\n"
return digest
digest += f"📈 **Summary:** {total_findings} findings across {topic_count} topic(s)\n\n"
digest += "---\n\n"
# Highlights (highest scored findings)
all_findings = []
for topic_id, findings in findings_by_topic.items():
topic = get_topic(topic_id)
if not topic:
continue
for finding in findings:
all_findings.append({
"topic": topic,
"finding": finding
})
# Sort by score
all_findings.sort(key=lambda x: x["finding"].get("score", 0), reverse=True)
if len(all_findings) >= 3:
digest += "## 🔥 Top Highlights\n\n"
for item in all_findings[:3]:
topic = item["topic"]
finding = item["finding"]
result = finding.get("result", {})
score = finding.get("score", 0)
digest += f"### {topic.get('name')} ({score:.2f})\n\n"
digest += f"**{result.get('title', 'Untitled')}**\n\n"
digest += f"{result.get('snippet', '')}\n\n"
digest += f"🔗 [{result.get('url', '')}]({result.get('url', '')})\n\n"
digest += "---\n\n"
# Findings by topic
digest += "## 📚 Findings by Topic\n\n"
for topic_id, findings in sorted(findings_by_topic.items()):
topic = get_topic(topic_id)
if not topic:
continue
topic_name = topic.get("name", topic_id)
topic_emoji = topic.get("emoji", "📌")
digest += f"### {topic_emoji} {topic_name}\n\n"
digest += f"**{len(findings)} finding(s) this week**\n\n"
# Sort findings by score
sorted_findings = sorted(findings, key=lambda x: x.get("score", 0), reverse=True)
for finding in sorted_findings[:5]: # Top 5 per topic
result = finding.get("result", {})
score = finding.get("score", 0)
reason = finding.get("reason", "")
digest += f"- **{result.get('title', 'Untitled')}** ({score:.2f})\n"
digest += f" {result.get('snippet', '')[:150]}...\n"
digest += f" 🔗 {result.get('url', '')}\n"
if reason:
digest += f" _Reason: {reason}_\n"
digest += "\n"
if len(sorted_findings) > 5:
digest += f"_...and {len(sorted_findings) - 5} more_\n\n"
digest += "\n"
# Recommendations (future enhancement)
digest += "---\n\n"
digest += "## 💡 Recommendations\n\n"
digest += "_Feature coming soon: AI-powered topic suggestions based on your findings_\n\n"
return digest
def send_digest(digest: str, dry_run: bool = False):
"""Send digest via configured channels."""
config = load_config()
settings = config.get("settings", {})
# For now, just print (would integrate with message tool in real environment)
if dry_run:
print("\n" + "="*60)
print("DIGEST PREVIEW:")
print("="*60 + "\n")
print(digest)
print("\n" + "="*60)
else:
# Would send via Telegram, Discord, Email
print("📧 Sending digest...")
print(digest)
print("\n✅ Digest sent")
def main():
parser = argparse.ArgumentParser(description="Generate weekly research digest")
parser.add_argument("--preview", action="store_true", help="Preview without sending")
parser.add_argument("--send", action="store_true", help="Generate and send")
parser.add_argument("--week-offset", type=int, default=0,
help="Week offset (0=current, 1=last week, etc.)")
args = parser.parse_args()
# Get week range
start, end = get_week_range(args.week_offset)
print(f"📊 Generating digest for {start.strftime('%Y-%m-%d')} to {end.strftime('%Y-%m-%d')}")
# Load findings
findings_by_topic = load_week_findings(start, end)
if not findings_by_topic:
print("⚠️ No findings for this period")
return
# Generate digest
digest = generate_digest(findings_by_topic, start, end)
# Send or preview
if args.send:
send_digest(digest, dry_run=False)
else:
send_digest(digest, dry_run=True)
if __name__ == "__main__":
main()
#!/usr/bin/env python3
"""
AI-powered importance scoring for research findings.
Scores findings as:
- HIGH: Immediate alert
- MEDIUM: Include in digest
- LOW: Ignore
"""
import re
from typing import Dict, List, Tuple
from datetime import datetime, timedelta
class ImportanceScorer:
"""Score research findings for importance."""
def __init__(self, topic: Dict, settings: Dict):
self.topic = topic
self.settings = settings
self.learning_enabled = settings.get("learning_enabled", False)
def score(self, result: Dict) -> Tuple[str, float, str]:
"""
Score a search result.
Returns:
(priority, score, reason)
priority: "high", "medium", "low"
score: 0.0-1.0
reason: Human-readable explanation
"""
signals = []
total_score = 0.0
# Extract fields
title = result.get("title", "")
snippet = result.get("snippet", "")
url = result.get("url", "")
published = result.get("published_date", "")
content = f"{title} {snippet}".lower()
# Signal 1: Keyword matching (0.0 - 0.3)
keyword_score, keyword_reason = self._score_keywords(content)
signals.append(("keyword_match", keyword_score, keyword_reason))
total_score += keyword_score
# Signal 2: Freshness (0.0 - 0.2)
freshness_score, freshness_reason = self._score_freshness(published)
signals.append(("freshness", freshness_score, freshness_reason))
total_score += freshness_score
# Signal 3: Source quality (0.0 - 0.2)
source_score, source_reason = self._score_source(url)
signals.append(("source_quality", source_score, source_reason))
total_score += source_score
# Signal 4: Alert conditions (0.0 - 0.3)
condition_score, condition_reason = self._score_conditions(content, title)
signals.append(("alert_conditions", condition_score, condition_reason))
total_score += condition_score
# Determine priority
threshold = self.topic.get("importance_threshold", "medium")
if threshold == "high":
# Only very high scores alert
if total_score >= 0.8:
priority = "high"
elif total_score >= 0.5:
priority = "medium"
else:
priority = "low"
elif threshold == "medium":
# Balanced
if total_score >= 0.6:
priority = "high"
elif total_score >= 0.3:
priority = "medium"
else:
priority = "low"
else: # low
# Almost everything goes to digest
if total_score >= 0.4:
priority = "high"
elif total_score >= 0.1:
priority = "medium"
else:
priority = "low"
# Build reason
top_signals = sorted(signals, key=lambda x: x[1], reverse=True)[:2]
reason_parts = [s[2] for s in top_signals if s[2]]
reason = " + ".join(reason_parts) if reason_parts else "low_relevance"
return priority, total_score, reason
def _score_keywords(self, content: str) -> Tuple[float, str]:
"""Score based on keyword matching."""
keywords = self.topic.get("keywords", [])
if not keywords:
return 0.0, ""
matches = 0
exact_matches = 0
for keyword in keywords:
keyword_lower = keyword.lower()
# Check for negation (keywords starting with -)
if keyword.startswith("-"):
negative_keyword = keyword_lower[1:]
if negative_keyword in content:
return 0.0, f"contains_excluded_{negative_keyword}"
continue
# Exact match (whole word)
if re.search(r'\b' + re.escape(keyword_lower) + r'\b', content):
exact_matches += 1
matches += 1
# Partial match
elif keyword_lower in content:
matches += 1
if exact_matches >= 2:
return 0.3, f"exact_match_{exact_matches}_keywords"
elif exact_matches == 1:
return 0.2, "exact_match_1_keyword"
elif matches >= 2:
return 0.15, f"partial_match_{matches}_keywords"
elif matches == 1:
return 0.1, "partial_match_1_keyword"
else:
return 0.0, "no_keyword_match"
def _score_freshness(self, published: str) -> Tuple[float, str]:
"""Score based on recency."""
if not published:
return 0.0, ""
try:
# Try parsing ISO format
if "T" in published:
pub_date = datetime.fromisoformat(published.replace("Z", "+00:00"))
else:
pub_date = datetime.strptime(published, "%Y-%m-%d")
age = datetime.now() - pub_date.replace(tzinfo=None)
if age < timedelta(hours=6):
return 0.2, "very_fresh_<6h"
elif age < timedelta(days=1):
return 0.15, "fresh_<24h"
elif age < timedelta(days=3):
return 0.1, "recent_<3d"
else:
return 0.05, "older_>3d"
except:
return 0.0, ""
def _score_source(self, url: str) -> Tuple[float, str]:
"""Score based on source quality."""
# Check boost sources
boost_sources = self.topic.get("boost_sources", [])
for source in boost_sources:
if source in url:
return 0.2, f"boosted_source_{source}"
# Check ignore sources
ignore_sources = self.topic.get("ignore_sources", [])
for source in ignore_sources:
if source in url:
return -1.0, f"ignored_source_{source}"
# Default trusted sources
trusted = ["github.com", "arxiv.org", "news.ycombinator.com",
"techcrunch.com", "theverge.com", "arstechnica.com"]
for source in trusted:
if source in url:
return 0.15, f"trusted_source_{source}"
return 0.05, "standard_source"
def _score_conditions(self, content: str, title: str) -> Tuple[float, str]:
"""Score based on alert conditions."""
alert_on = self.topic.get("alert_on", [])
for condition in alert_on:
if condition == "price_change_10pct":
if self._detect_price_change(content, threshold=0.10):
return 0.3, "price_change_>10%"
elif condition == "keyword_exact_match":
# Already handled in keyword scoring, but boost it
keywords = self.topic.get("keywords", [])
for kw in keywords:
if re.search(r'\b' + re.escape(kw.lower()) + r'\b', content):
return 0.2, "exact_keyword_in_condition"
elif condition == "major_paper":
if "arxiv" in content or "paper" in title.lower():
return 0.25, "academic_paper_detected"
elif condition == "model_release":
if re.search(r'(release|launch|announce).*\b(model|gpt|llm)\b', content, re.I):
return 0.3, "model_release_detected"
elif condition == "patch_release":
if re.search(r'(patch|update|version|release).*\d+\.\d+', content, re.I):
return 0.25, "patch_release_detected"
elif condition == "major_bug_fix":
if re.search(r'(fix|patch|solve).*(critical|major|bug)', content, re.I):
return 0.2, "major_bug_fix_detected"
elif condition == "high_engagement":
# Would need engagement data from API
pass
elif condition == "source_tier_1":
# Already handled in source scoring
pass
return 0.0, ""
def _detect_price_change(self, content: str, threshold: float = 0.10) -> bool:
"""Detect significant price changes."""
# Look for percentage patterns
pct_pattern = r'(\d+(?:\.\d+)?)\s*%'
matches = re.findall(pct_pattern, content)
for match in matches:
pct = float(match)
if pct >= threshold * 100: # Convert to percentage
return True
# Look for price change keywords
change_keywords = ["surge", "plunge", "jump", "drop", "spike", "crash"]
for keyword in change_keywords:
if keyword in content.lower():
return True
return False
def score_result(result: Dict, topic: Dict, settings: Dict) -> Tuple[str, float, str]:
"""
Score a single search result.
Convenience function for scoring without creating scorer instance.
"""
scorer = ImportanceScorer(topic, settings)
return scorer.score(result)
#!/usr/bin/env python3
"""
Topic management CLI for proactive-research.
Usage:
python3 manage_topics.py add "Topic Name" --query "search" --keywords "a,b,c"
python3 manage_topics.py list
python3 manage_topics.py edit <id> --frequency hourly
python3 manage_topics.py remove <id>
python3 manage_topics.py test <id>
"""
import sys
import argparse
import re
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent))
from config import load_config, save_config, get_topic
from monitor import search_topic, monitor_topic, load_state, get_settings
def generate_id(name: str) -> str:
"""Generate topic ID from name."""
# Convert to lowercase, replace spaces with hyphens, remove special chars
topic_id = name.lower()
topic_id = re.sub(r'[^\w\s-]', '', topic_id)
topic_id = re.sub(r'[-\s]+', '-', topic_id)
return topic_id.strip('-')
def add_topic(args):
"""Add a new topic."""
config = load_config()
# Generate ID
topic_id = args.id or generate_id(args.name)
# Check for duplicates
existing_ids = [t.get("id") for t in config.get("topics", [])]
if topic_id in existing_ids:
print(f"❌ Topic with ID '{topic_id}' already exists", file=sys.stderr)
sys.exit(1)
# Build topic
topic = {
"id": topic_id,
"name": args.name,
"query": args.query,
"keywords": args.keywords.split(",") if args.keywords else [],
"frequency": args.frequency,
"importance_threshold": args.importance,
"channels": args.channels.split(",") if args.channels else ["telegram"],
"context": args.context or "",
"alert_on": args.alert_on.split(",") if args.alert_on else [],
"ignore_sources": [],
"boost_sources": []
}
# Add to config
if "topics" not in config:
config["topics"] = []
config["topics"].append(topic)
save_config(config)
print(f"✅ Added topic: {args.name} ({topic_id})")
print(f" Query: {args.query}")
print(f" Frequency: {args.frequency}")
print(f" Importance: {args.importance}")
def list_topics(args):
"""List all topics."""
config = load_config()
topics = config.get("topics", [])
if not topics:
print("No topics configured")
return
print(f"\n📋 Configured Topics ({len(topics)})\n")
for topic in topics:
print(f"{'='*60}")
print(f"ID: {topic.get('id')}")
print(f"Name: {topic.get('name')}")
print(f"Query: {topic.get('query')}")
print(f"Keywords: {', '.join(topic.get('keywords', []))}")
print(f"Frequency: {topic.get('frequency')}")
print(f"Importance: {topic.get('importance_threshold')}")
print(f"Channels: {', '.join(topic.get('channels', []))}")
if topic.get('context'):
print(f"Context: {topic.get('context')}")
print()
def edit_topic(args):
"""Edit an existing topic."""
config = load_config()
topics = config.get("topics", [])
# Find topic
topic_idx = None
for idx, topic in enumerate(topics):
if topic.get("id") == args.topic_id:
topic_idx = idx
break
if topic_idx is None:
print(f"❌ Topic '{args.topic_id}' not found", file=sys.stderr)
sys.exit(1)
topic = topics[topic_idx]
# Update fields
if args.name:
topic["name"] = args.name
if args.query:
topic["query"] = args.query
if args.keywords:
topic["keywords"] = args.keywords.split(",")
if args.frequency:
topic["frequency"] = args.frequency
if args.importance:
topic["importance_threshold"] = args.importance
if args.channels:
topic["channels"] = args.channels.split(",")
if args.context:
topic["context"] = args.context
if args.alert_on:
topic["alert_on"] = args.alert_on.split(",")
# Save
topics[topic_idx] = topic
config["topics"] = topics
save_config(config)
print(f"✅ Updated topic: {topic.get('name')} ({args.topic_id})")
def remove_topic(args):
"""Remove a topic."""
config = load_config()
topics = config.get("topics", [])
# Find and remove
new_topics = [t for t in topics if t.get("id") != args.topic_id]
if len(new_topics) == len(topics):
print(f"❌ Topic '{args.topic_id}' not found", file=sys.stderr)
sys.exit(1)
config["topics"] = new_topics
save_config(config)
print(f"✅ Removed topic: {args.topic_id}")
def test_topic(args):
"""Test a topic (search without saving)."""
topic = get_topic(args.topic_id)
if not topic:
print(f"❌ Topic '{args.topic_id}' not found", file=sys.stderr)
sys.exit(1)
print(f"🧪 Testing topic: {topic.get('name')}\n")
# Run monitor in dry-run mode
state = load_state()
settings = get_settings()
monitor_topic(topic, state, settings, dry_run=True, verbose=True)
def main():
parser = argparse.ArgumentParser(description="Manage research topics")
subparsers = parser.add_subparsers(dest="command", required=True)
# Add command
add_parser = subparsers.add_parser("add", help="Add a new topic")
add_parser.add_argument("name", help="Topic name")
add_parser.add_argument("--id", help="Custom topic ID (auto-generated if not provided)")
add_parser.add_argument("--query", required=True, help="Search query")
add_parser.add_argument("--keywords", help="Comma-separated keywords")
add_parser.add_argument("--frequency", choices=["hourly", "daily", "weekly"],
default="daily", help="Check frequency")
add_parser.add_argument("--importance", choices=["high", "medium", "low"],
default="medium", help="Importance threshold")
add_parser.add_argument("--channels", default="telegram",
help="Comma-separated channels (telegram,discord,email)")
add_parser.add_argument("--context", help="Why this topic matters to you")
add_parser.add_argument("--alert-on", help="Comma-separated alert conditions")
add_parser.set_defaults(func=add_topic)
# List command
list_parser = subparsers.add_parser("list", help="List all topics")
list_parser.set_defaults(func=list_topics)
# Edit command
edit_parser = subparsers.add_parser("edit", help="Edit a topic")
edit_parser.add_argument("topic_id", help="Topic ID to edit")
edit_parser.add_argument("--name", help="New name")
edit_parser.add_argument("--query", help="New query")
edit_parser.add_argument("--keywords", help="New keywords (comma-separated)")
edit_parser.add_argument("--frequency", choices=["hourly", "daily", "weekly"])
edit_parser.add_argument("--importance", choices=["high", "medium", "low"])
edit_parser.add_argument("--channels", help="New channels (comma-separated)")
edit_parser.add_argument("--context", help="New context")
edit_parser.add_argument("--alert-on", help="New alert conditions (comma-separated)")
edit_parser.set_defaults(func=edit_topic)
# Remove command
remove_parser = subparsers.add_parser("remove", help="Remove a topic")
remove_parser.add_argument("topic_id", help="Topic ID to remove")
remove_parser.set_defaults(func=remove_topic)
# Test command
test_parser = subparsers.add_parser("test", help="Test a topic")
test_parser.add_argument("topic_id", help="Topic ID to test")
test_parser.set_defaults(func=test_topic)
args = parser.parse_args()
try:
args.func(args)
except FileNotFoundError as e:
print(f"❌ {e}", file=sys.stderr)
sys.exit(1)
if __name__ == "__main__":
main()
#!/usr/bin/env python3
"""
Proactive Research Monitor
Checks topics due for monitoring, scores findings, and sends alerts.
Run via cron for automated monitoring.
"""
import sys
import json
import hashlib
import argparse
from datetime import datetime, timedelta
from pathlib import Path
from typing import Dict, List, Tuple
# Add parent dir to path for imports
sys.path.insert(0, str(Path(__file__).parent))
from config import (
load_config, load_state, save_state, get_settings,
get_channel_config, save_finding
)
from importance_scorer import score_result
def hash_url(url: str) -> str:
"""Create hash of URL for deduplication."""
return hashlib.md5(url.encode()).hexdigest()
def is_duplicate(url: str, state: Dict, dedup_hours: int = 72) -> bool:
"""Check if URL was recently alerted."""
url_hash = hash_url(url)
dedup_map = state.get("deduplication", {}).get("url_hash_map", {})
if url_hash not in dedup_map:
return False
# Check if within deduplication window
last_seen_str = dedup_map[url_hash]
last_seen = datetime.fromisoformat(last_seen_str)
age = datetime.now() - last_seen
return age < timedelta(hours=dedup_hours)
def mark_as_seen(url: str, state: Dict):
"""Mark URL as seen in deduplication map."""
url_hash = hash_url(url)
if "deduplication" not in state:
state["deduplication"] = {"url_hash_map": {}}
state["deduplication"]["url_hash_map"][url_hash] = datetime.now().isoformat()
def search_topic(topic: Dict, dry_run: bool = False) -> List[Dict]:
"""
Search for topic using available search tools.
In real OpenClaw environment, this would use web_search tool.
For standalone testing, returns mock results.
"""
query = topic.get("query", "")
# Try to use web-search-plus if available
web_search_plus = Path(__file__).parent.parent.parent / "web-search-plus" / "scripts" / "search.py"
if web_search_plus.exists():
import subprocess
try:
result = subprocess.run(
["python3", str(web_search_plus), "--query", query, "--max-results", "5"],
capture_output=True,
text=True,
timeout=30
)
if result.returncode == 0:
data = json.loads(result.stdout)
return data.get("results", [])
except Exception as e:
print(f"⚠️ web-search-plus failed: {e}", file=sys.stderr)
# Fallback: Return mock results for testing
if dry_run:
return [
{
"title": f"Mock result for {query}",
"url": f"https://example.com/mock-{hashlib.md5(query.encode()).hexdigest()[:8]}",
"snippet": f"This is a test result for query: {query}",
"published_date": datetime.now().isoformat()
}
]
return []
def should_check_topic(topic: Dict, state: Dict, force: bool = False) -> bool:
"""Determine if topic should be checked now."""
if force:
return True
topic_id = topic.get("id")
frequency = topic.get("frequency", "daily")
# Get last check time
topic_state = state.get("topics", {}).get(topic_id, {})
last_check_str = topic_state.get("last_check")
if not last_check_str:
return True # Never checked
last_check = datetime.fromisoformat(last_check_str)
now = datetime.now()
# Check frequency
if frequency == "hourly":
return (now - last_check) >= timedelta(hours=1)
elif frequency == "daily":
return (now - last_check) >= timedelta(days=1)
elif frequency == "weekly":
return (now - last_check) >= timedelta(weeks=1)
return False
def check_rate_limits(topic: Dict, state: Dict, settings: Dict) -> bool:
"""Check if we've hit rate limits."""
topic_id = topic.get("id")
max_per_day = settings.get("max_alerts_per_day", 5)
max_per_topic_per_day = settings.get("max_alerts_per_topic_per_day", 2)
topic_state = state.get("topics", {}).get(topic_id, {})
alerts_today = topic_state.get("alerts_today", 0)
# Check topic limit
if alerts_today >= max_per_topic_per_day:
return False
# Check global limit (count across all topics)
total_alerts_today = sum(
s.get("alerts_today", 0)
for s in state.get("topics", {}).values()
)
if total_alerts_today >= max_per_day:
return False
return True
def send_alert(topic: Dict, result: Dict, priority: str, score: float, reason: str, dry_run: bool = False):
"""Send alert via configured channels."""
channels = topic.get("channels", [])
# Build message
emoji_map = {"high": "🔥", "medium": "📌", "low": "📝"}
emoji = emoji_map.get(priority, "📌")
topic_name = topic.get("name", "Research Alert")
topic_emoji = topic.get("emoji", "🔍")
context = topic.get("context", "")
message = f"{emoji} **{topic_name}** {topic_emoji}\n\n"
message += f"**{result['title']}**\n\n"
message += f"{result.get('snippet', '')}\n\n"
if context:
message += f"💡 *Context:* {context}\n\n"
message += f"🔗 {result['url']}\n\n"
message += f"_Score: {score:.2f} | {reason}_"
if dry_run:
print(f"\n{'='*60}")
print(f"DRY RUN - Would send alert:")
print(f"Channels: {', '.join(channels)}")
print(f"Priority: {priority.upper()}")
print(f"\n{message}")
print(f"{'='*60}\n")
return
# Send to channels
for channel in channels:
if channel == "telegram":
send_telegram(message, priority)
elif channel == "discord":
send_discord(message, priority)
elif channel == "email":
send_email(message, priority, topic_name)
def send_telegram(message: str, priority: str):
"""Send via Telegram (requires OpenClaw message tool)."""
# In real environment, this would use OpenClaw's message tool
# For now, just log
print(f"📱 [TELEGRAM] {priority.upper()}: {message[:100]}...")
def send_discord(message: str, priority: str):
"""Send via Discord webhook."""
import requests
from config import get_channel_config
discord_config = get_channel_config("discord")
webhook_url = discord_config.get("webhook_url")
if not webhook_url:
print("⚠️ Discord webhook not configured", file=sys.stderr)
return
payload = {
"username": discord_config.get("username", "Research Bot"),
"avatar_url": discord_config.get("avatar_url"),
"content": message
}
try:
requests.post(webhook_url, json=payload, timeout=10)
print(f"✅ Sent to Discord")
except Exception as e:
print(f"❌ Discord send failed: {e}", file=sys.stderr)
def send_email(message: str, priority: str, subject: str):
"""Send via email."""
print(f"📧 [EMAIL] {priority.upper()}: {subject}")
def monitor_topic(topic: Dict, state: Dict, settings: Dict, dry_run: bool = False, verbose: bool = False):
"""Monitor a single topic."""
topic_id = topic.get("id")
topic_name = topic.get("name")
if verbose:
print(f"\n🔍 Checking topic: {topic_name} ({topic_id})")
# Search
results = search_topic(topic, dry_run=dry_run)
if verbose:
print(f" Found {len(results)} results")
# Score and filter
dedup_hours = settings.get("deduplication_window_hours", 72)
high_priority = []
medium_priority = []
for result in results:
url = result.get("url", "")
# Check deduplication
if is_duplicate(url, state, dedup_hours):
if verbose:
print(f" ⏭️ Skipping duplicate: {url}")
continue
# Score
priority, score, reason = score_result(result, topic, settings)
if verbose:
print(f" {priority.upper():6} ({score:.2f}) - {result.get('title', '')[:50]}...")
# Categorize
if priority == "high":
high_priority.append((result, score, reason))
elif priority == "medium":
medium_priority.append((result, score, reason))
# Mark as seen
mark_as_seen(url, state)
# Send high priority alerts
for result, score, reason in high_priority:
if check_rate_limits(topic, state, settings):
send_alert(topic, result, "high", score, reason, dry_run=dry_run)
# Increment alert counter
if not dry_run:
if "topics" not in state:
state["topics"] = {}
if topic_id not in state["topics"]:
state["topics"][topic_id] = {}
state["topics"][topic_id]["alerts_today"] = \
state["topics"][topic_id].get("alerts_today", 0) + 1
else:
if verbose:
print(f" ⚠️ Rate limit reached, skipping alert")
# Save medium priority to findings
date_str = datetime.now().strftime("%Y-%m-%d")
for result, score, reason in medium_priority:
if not dry_run:
save_finding(topic_id, date_str, {
"result": result,
"score": score,
"reason": reason,
"timestamp": datetime.now().isoformat()
})
if verbose:
print(f" 💾 Saved to digest: {result.get('title', '')[:50]}...")
# Update topic state
if not dry_run:
if "topics" not in state:
state["topics"] = {}
if topic_id not in state["topics"]:
state["topics"][topic_id] = {}
state["topics"][topic_id]["last_check"] = datetime.now().isoformat()
state["topics"][topic_id]["last_results_count"] = len(results)
state["topics"][topic_id]["findings_count"] = \
state["topics"][topic_id].get("findings_count", 0) + len(medium_priority)
def main():
parser = argparse.ArgumentParser(description="Monitor research topics")
parser.add_argument("--dry-run", action="store_true", help="Don't send alerts or save state")
parser.add_argument("--verbose", "-v", action="store_true", help="Verbose output")
parser.add_argument("--topic", help="Check specific topic by ID")
parser.add_argument("--force", action="store_true", help="Force check even if not due")
parser.add_argument("--frequency", choices=["hourly", "daily", "weekly"],
help="Only check topics with this frequency")
args = parser.parse_args()
# Load config and state
try:
config = load_config()
except FileNotFoundError as e:
print(f"❌ {e}", file=sys.stderr)
sys.exit(1)
state = load_state()
settings = get_settings()
topics = config.get("topics", [])
if not topics:
print("⚠️ No topics configured", file=sys.stderr)
sys.exit(0)
# Filter topics
topics_to_check = []
for topic in topics:
# Filter by specific topic
if args.topic and topic.get("id") != args.topic:
continue
# Filter by frequency
if args.frequency and topic.get("frequency") != args.frequency:
continue
# Check if due
if should_check_topic(topic, state, force=args.force):
topics_to_check.append(topic)
if not topics_to_check:
if args.verbose:
print("✅ No topics due for checking")
sys.exit(0)
print(f"🔍 Monitoring {len(topics_to_check)} topic(s)...")
# Monitor each topic
for topic in topics_to_check:
try:
monitor_topic(topic, state, settings, dry_run=args.dry_run, verbose=args.verbose)
except Exception as e:
print(f"❌ Error monitoring {topic.get('name')}: {e}", file=sys.stderr)
if args.verbose:
import traceback
traceback.print_exc()
# Save state
if not args.dry_run:
save_state(state)
print("✅ State saved")
print("✅ Monitoring complete")
if __name__ == "__main__":
main()
#!/usr/bin/env python3
"""
Setup cron jobs for proactive research monitoring.
Creates cron entries for:
- Hourly topic checks
- Daily topic checks
- Weekly digest
"""
import sys
import subprocess
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent))
from config import load_config, get_settings
SKILL_DIR = Path(__file__).parent.parent
MONITOR_SCRIPT = SKILL_DIR / "scripts" / "monitor.py"
DIGEST_SCRIPT = SKILL_DIR / "scripts" / "digest.py"
def get_current_crontab() -> str:
"""Get current user's crontab."""
try:
result = subprocess.run(
["crontab", "-l"],
capture_output=True,
text=True
)
return result.stdout if result.returncode == 0 else ""
except Exception:
return ""
def set_crontab(content: str):
"""Set user's crontab."""
subprocess.run(
["crontab", "-"],
input=content,
text=True,
check=True
)
def remove_old_entries(crontab: str) -> str:
"""Remove old proactive-research cron entries."""
lines = crontab.split("\n")
filtered = [
line for line in lines
if "proactive-research" not in line.lower()
]
return "\n".join(filtered)
def generate_cron_entries(settings: dict) -> list[str]:
"""Generate cron entries based on settings."""
entries = []
# Header
entries.append("# Proactive Research - Auto-generated")
# Hourly check (every hour)
entries.append(
f"0 * * * * cd {SKILL_DIR} && /usr/bin/python3 {MONITOR_SCRIPT} --frequency hourly"
)
# Daily check (9 AM)
entries.append(
f"0 9 * * * cd {SKILL_DIR} && /usr/bin/python3 {MONITOR_SCRIPT} --frequency daily"
)
# Weekly check (Sunday 9 AM)
entries.append(
f"0 9 * * 0 cd {SKILL_DIR} && /usr/bin/python3 {MONITOR_SCRIPT} --frequency weekly"
)
# Weekly digest
digest_day = settings.get("digest_day", "sunday")
digest_time = settings.get("digest_time", "18:00")
# Parse time
hour, minute = digest_time.split(":")
# Parse day
day_map = {
"sunday": "0", "monday": "1", "tuesday": "2", "wednesday": "3",
"thursday": "4", "friday": "5", "saturday": "6"
}
day_num = day_map.get(digest_day.lower(), "0")
entries.append(
f"{minute} {hour} * * {day_num} cd {SKILL_DIR} && /usr/bin/python3 {DIGEST_SCRIPT} --send"
)
return entries
def setup_cron(auto: bool = False):
"""Setup cron jobs."""
print("🔧 Setting up proactive research cron jobs...\n")
# Load config
try:
config = load_config()
settings = get_settings()
except FileNotFoundError as e:
print(f"❌ {e}", file=sys.stderr)
sys.exit(1)
# Get current crontab
current = get_current_crontab()
# Remove old entries
cleaned = remove_old_entries(current)
# Generate new entries
new_entries = generate_cron_entries(settings)
# Preview
print("The following cron jobs will be added:\n")
for entry in new_entries:
if not entry.startswith("#"):
print(f" {entry}")
print()
# Confirm
if not auto:
response = input("Continue? [y/N]: ").strip().lower()
if response not in ("y", "yes"):
print("Aborted")
sys.exit(0)
# Build new crontab
new_crontab = cleaned.strip()
if new_crontab:
new_crontab += "\n\n"
new_crontab += "\n".join(new_entries) + "\n"
# Set crontab
try:
set_crontab(new_crontab)
print("✅ Cron jobs installed successfully")
print("\nTo verify, run: crontab -l")
except Exception as e:
print(f"❌ Failed to install cron jobs: {e}", file=sys.stderr)
sys.exit(1)
def remove_cron():
"""Remove all proactive-research cron jobs."""
print("🗑️ Removing proactive research cron jobs...\n")
# Get current crontab
current = get_current_crontab()
# Remove entries
cleaned = remove_old_entries(current)
if cleaned == current:
print("⚠️ No proactive-research cron jobs found")
return
# Set crontab
try:
set_crontab(cleaned)
print("✅ Cron jobs removed successfully")
except Exception as e:
print(f"❌ Failed to remove cron jobs: {e}", file=sys.stderr)
sys.exit(1)
def main():
import argparse
parser = argparse.ArgumentParser(description="Setup cron jobs for proactive research")
parser.add_argument("--auto", action="store_true", help="Auto-setup without confirmation")
parser.add_argument("--remove", action="store_true", help="Remove cron jobs")
args = parser.parse_args()
if args.remove:
remove_cron()
else:
setup_cron(auto=args.auto)
if __name__ == "__main__":
main()
#!/usr/bin/env python3
"""
Interactive onboarding wizard for topic-monitor skill.
Runs on first use when no config.json exists.
"""
import json
import sys
from pathlib import Path
SKILL_DIR = Path(__file__).parent.parent
CONFIG_FILE = SKILL_DIR / "config.json"
CONFIG_EXAMPLE = SKILL_DIR / "config.example.json"
def print_welcome():
"""Print welcome message."""
print()
print("=" * 55)
print(" 🔍 Topic Monitor - Setup Wizard")
print("=" * 55)
print()
print("Welcome! Let's set up your personal topic monitoring.")
print("I'll ask a few questions to understand what you want to track.")
print()
print("You can always edit config.json later to fine-tune things.")
print()
def prompt(question: str, default: str = None) -> str:
"""Prompt user for input with optional default."""
if default:
question = f"{question} [{default}]: "
else:
question = f"{question}: "
response = input(question).strip()
return response if response else (default or "")
def prompt_yes_no(question: str, default: bool = True) -> bool:
"""Prompt for yes/no answer."""
default_hint = "Y/n" if default else "y/N"
response = input(f"{question} ({default_hint}): ").strip().lower()
if not response:
return default
return response in ('y', 'yes', 'ja', 'si', 'oui')
def prompt_choice(question: str, choices: list, default: str = None) -> str:
"""Prompt for choice from list."""
print(f"\n{question}")
for i, choice in enumerate(choices, 1):
marker = " *" if choice == default else ""
print(f" {i}. {choice}{marker}")
while True:
response = input(f"\nEnter number or value [{default or choices[0]}]: ").strip()
if not response:
return default or choices[0]
# Check if it's a number
try:
idx = int(response)
if 1 <= idx <= len(choices):
return choices[idx - 1]
except ValueError:
pass
# Check if it's a valid choice
response_lower = response.lower()
for choice in choices:
if choice.lower() == response_lower:
return choice
print(f" Please enter a number 1-{len(choices)} or a valid option.")
def prompt_multiline(question: str, hint: str = None) -> list:
"""Prompt for multiple lines of input."""
print(f"\n{question}")
if hint:
print(f" {hint}")
print(" (Enter each item on a new line. Empty line when done)")
print()
items = []
while True:
line = input(" > ").strip()
if not line:
break
items.append(line)
return items
def prompt_keywords(topic_name: str) -> list:
"""Prompt for keywords for a topic."""
print(f"\n Keywords to watch for in '{topic_name}'?")
print(" (comma-separated, e.g.: release, update, announcement)")
response = input(" > ").strip()
if not response:
return []
keywords = [k.strip() for k in response.split(",") if k.strip()]
return keywords
def create_topic_id(name: str) -> str:
"""Create a safe ID from topic name."""
# Convert to lowercase, replace spaces with hyphens, remove special chars
topic_id = name.lower()
topic_id = topic_id.replace(" ", "-")
topic_id = "".join(c for c in topic_id if c.isalnum() or c == "-")
topic_id = "-".join(filter(None, topic_id.split("-"))) # Remove duplicate hyphens
return topic_id[:30] # Limit length
def gather_topics() -> list:
"""Gather topic configurations interactively."""
topics = []
print("-" * 55)
print("📋 STEP 1: Topics to Monitor")
print("-" * 55)
topic_names = prompt_multiline(
"What topics do you want to monitor?",
"Examples: AI Models, Security Alerts, Product Updates"
)
if not topic_names:
print("\n⚠️ No topics entered. You can add them later with manage_topics.py")
return []
print(f"\nGreat! Let's configure each of your {len(topic_names)} topics.\n")
for i, name in enumerate(topic_names, 1):
print(f"\n--- Topic {i}/{len(topic_names)}: {name} ---")
# Search query
default_query = f"{name} news updates"
query = prompt(f" Search query for '{name}'", default_query)
# Keywords
keywords = prompt_keywords(name)
topic = {
"id": create_topic_id(name),
"name": name,
"query": query,
"keywords": keywords,
"frequency": None, # Will be set globally
"importance_threshold": None, # Will be set globally
"channels": ["telegram"],
"context": "",
"alert_on": ["keyword_exact_match"],
"ignore_sources": [],
"boost_sources": []
}
topics.append(topic)
return topics
def gather_settings() -> dict:
"""Gather global settings interactively."""
print()
print("-" * 55)
print("⚙️ STEP 2: Monitoring Settings")
print("-" * 55)
# Frequency
frequency = prompt_choice(
"How often should I check for updates?",
["hourly", "daily", "weekly"],
default="daily"
)
# Importance threshold
print("\nImportance threshold determines when you get alerts:")
print(" • high = Only alert for major news")
print(" • medium = Alert for moderately important updates")
print(" • low = Alert for anything relevant")
importance = prompt_choice(
"\nImportance threshold for alerts?",
["low", "medium", "high"],
default="medium"
)
# Weekly digest
print()
print("-" * 55)
print("📊 STEP 3: Weekly Digest")
print("-" * 55)
print("\nThe weekly digest compiles lower-priority findings")
print("so you don't miss interesting but non-urgent updates.")
digest_enabled = prompt_yes_no("\nEnable weekly digest?", default=True)
digest_day = "sunday"
if digest_enabled:
digest_day = prompt_choice(
"Which day should I send the digest?",
["monday", "tuesday", "wednesday", "thursday", "friday", "saturday", "sunday"],
default="sunday"
)
settings = {
"frequency": frequency,
"importance_threshold": importance,
"digest_enabled": digest_enabled,
"digest_day": digest_day,
"digest_time": "18:00",
"max_alerts_per_day": 5,
"max_alerts_per_topic_per_day": 2,
"deduplication_window_hours": 72,
"learning_enabled": True,
"quiet_hours": {
"enabled": False,
"start": "22:00",
"end": "08:00"
}
}
return settings
def build_config(topics: list, settings: dict) -> dict:
"""Build the final config object."""
# Apply global frequency and importance to topics
for topic in topics:
topic["frequency"] = settings.pop("frequency")
topic["importance_threshold"] = settings.pop("importance_threshold")
# Load channel config from example
channels = {
"telegram": {
"enabled": True,
"chat_id": None,
"silent": False,
"effects": {
"high_importance": "🔥",
"medium_importance": "📌"
}
},
"discord": {
"enabled": False,
"webhook_url": None,
"username": "Topic Monitor",
"avatar_url": None
},
"email": {
"enabled": False,
"to": None,
"from": "monitor@yourdomain.com",
"smtp_server": "smtp.gmail.com",
"smtp_port": 587,
"smtp_user": None,
"smtp_password": None
}
}
config = {
"topics": topics,
"settings": settings,
"channels": channels
}
return config
def save_config(config: dict):
"""Save config to file."""
with open(CONFIG_FILE, 'w') as f:
json.dump(config, f, indent=2)
def print_summary(config: dict):
"""Print configuration summary."""
print()
print("=" * 55)
print(" ✅ Setup Complete!")
print("=" * 55)
print()
topics = config.get("topics", [])
settings = config.get("settings", {})
print(f"📋 Topics configured: {len(topics)}")
for topic in topics:
print(f" • {topic['name']}")
print(f" Query: {topic['query'][:40]}...")
print(f" Keywords: {', '.join(topic['keywords'][:3])}{'...' if len(topic['keywords']) > 3 else ''}")
print(f" Frequency: {topic['frequency']}, Threshold: {topic['importance_threshold']}")
print()
print(f"⚙️ Settings:")
print(f" • Weekly digest: {'Enabled' if settings.get('digest_enabled', True) else 'Disabled'}")
if settings.get('digest_enabled', True):
print(f" • Digest day: {settings.get('digest_day', 'sunday').capitalize()}")
print(f" • Max alerts/day: {settings.get('max_alerts_per_day', 5)}")
print(f" • Learning mode: {'Enabled' if settings.get('learning_enabled', True) else 'Disabled'}")
print()
print("📁 Config saved to: config.json")
print()
print("-" * 55)
print("Next Steps:")
print("-" * 55)
print()
print("1. Test your topics:")
print(" python3 scripts/monitor.py --dry-run --verbose")
print()
print("2. Set up automated monitoring:")
print(" python3 scripts/setup_cron.py")
print()
print("3. Manage topics later:")
print(" python3 scripts/manage_topics.py list")
print(" python3 scripts/manage_topics.py add \"New Topic\" --query \"...\"")
print()
print("Happy monitoring! 🔍")
print()
def main():
"""Main entry point."""
# Check if config already exists
if CONFIG_FILE.exists():
print()
print("⚠️ config.json already exists!")
print()
overwrite = prompt_yes_no("Do you want to start fresh and overwrite it?", default=False)
if not overwrite:
print("\nKeeping existing config. Use manage_topics.py to edit topics.")
print("Or delete config.json and run setup again.")
sys.exit(0)
print()
print_welcome()
try:
# Gather information
topics = gather_topics()
settings = gather_settings()
# Build and save config
config = build_config(topics, settings)
save_config(config)
# Show summary
print_summary(config)
except KeyboardInterrupt:
print("\n\n⚠️ Setup cancelled. No changes made.")
sys.exit(1)
except EOFError:
print("\n\n⚠️ Input ended. No changes made.")
sys.exit(1)
if __name__ == "__main__":
main()
{
"name": "topic-monitor",
"version": "1.1.0",
"description": "Monitor topics of interest and get proactive alerts when important developments occur",
"author": "robbyczgw-cla",
"license": "MIT",
"keywords": [
"monitoring",
"alerts",
"research",
"proactive",
"news",
"tracking"
],
"repository": {
"type": "git",
"url": "https://github.com/robbyczgw-cla/skills"
},
"files": [
"SKILL.md",
"README.md",
"config.example.json",
"scripts/"
],
"scripts": {
"setup": "python3 scripts/setup.py",
"monitor": "python3 scripts/monitor.py",
"digest": "python3 scripts/digest.py",
"topics": "python3 scripts/manage_topics.py"
},
"dependencies": {
"python": ">=3.8"
},
"optional_dependencies": {
"skills": ["web-search-plus"]
}
}
Related skills
How it compares
Choose topic-monitor over manual search skills when you need recurring watches with scored filtering and push notifications rather than on-demand queries.
FAQ
What channels does topic-monitor support for alerts?
topic-monitor supports Telegram, Discord, and email for outbound alerts. The skill scores each finding and can also compile weekly digests instead of sending every low-signal hit immediately.
How does topic-monitor avoid alert fatigue?
topic-monitor applies AI importance scoring to classify each finding as alert, digest, or ignore, and includes rate limiting so only high-signal events trigger instant notifications.
Is Topic Monitor safe to install?
skills.sh reports 0 of 3 security scanners passed. Review the Security Audits panel on this page before installing in production.