Dual Channel Watchexec

Dual-Channel Watchexec Notifications

Send reliable notifications to both Telegram and Pushover when watchexec detects file changes or process crashes.

Self-Evolving Skill: This skill improves through use. If instructions are wrong, parameters drifted, or a workaround was needed — fix this file immediately, don't defer. Only update for real, reproducible issues.

When to Use This Skill

Use this skill when:

• Setting up file change monitoring with notifications
• Implementing process crash alerting via Telegram and Pushover
• Creating watchexec wrappers with dual-channel notification support
• Formatting messages for both HTML (Telegram) and plain text (Pushover)
• Troubleshooting notification delivery or formatting issues

Core Pattern

watchexec wrapper script → detect event → notify-script → Telegram + Pushover

# wrapper.sh - Monitors process and detects restart reasons
watchexec --restart -- python bot.py

# On event, call:
notify-script.sh <reason> <exit_code> <watchexec_info_file> <crash_context>

---

Critical Rule: Format Differences

Telegram: HTML mode ONLY

MESSAGE="<b>Alert</b>: <code>file.py</code>"
# Escape 3 chars: & → &amp;, < → &lt;, > → &gt;

Pushover: Plain text ONLY

/usr/bin/env bash << 'SKILL_SCRIPT_EOF'
# Strip HTML tags before sending
MESSAGE_PLAIN=$(echo "$MESSAGE_HTML" | sed 's/<[^>]*>//g')
SKILL_SCRIPT_EOF

Why HTML for Telegram:

• Markdown requires escaping 40+ chars (., -, _, etc.)
• HTML only requires escaping 3 chars (&, <, >)
• Industry best practice

---

Quick Reference

Send to Both Channels

/usr/bin/env bash << 'SKILL_SCRIPT_EOF_2'
# 1. Build HTML message for Telegram
MESSAGE_HTML="<b>File</b>: <code>handler_classes.py</code>"

# 2. Strip HTML for Pushover
MESSAGE_PLAIN=$(echo "$MESSAGE_HTML" | sed 's/<[^>]*>//g')

# 3. Send to Telegram with HTML
curl -s -d "chat_id=$CHAT_ID" \
  -d "text=$MESSAGE_HTML" \
  -d "parse_mode=HTML" \
  https://api.telegram.org/bot$BOT_TOKEN/sendMessage

# 4. Send to Pushover with plain text
curl -s --form-string "message=$MESSAGE_PLAIN" \
  https://api.pushover.net/1/messages.json
SKILL_SCRIPT_EOF_2

Execution Pattern

# Fire-and-forget background notifications (don't block restarts)
"$NOTIFY_SCRIPT" "crash" "$EXIT_CODE" "$INFO_FILE" "$CONTEXT_FILE" &

---

Validation Checklist

Before deploying:

• [ ] Using HTML parse mode for Telegram (not Markdown)
• [ ] HTML tags stripped for Pushover (plain text only)
• [ ] HTML escaping applied to all dynamic content (&, <, >)
• [ ] Credentials loaded from env vars/Doppler (not hardcoded)
• [ ] Message archiving enabled for debugging
• [ ] File detection uses stat (not find -newermt)
• [ ] Heredocs use unquoted delimiters for variable expansion
• [ ] Notifications run in background (fire-and-forget)
• [ ] Tested with files containing special chars (_, ., -)
• [ ] Both Telegram and Pushover successfully receiving

---

Summary

Key Lessons:

1. Always use HTML mode for Telegram (simpler escaping) 2. Always strip HTML tags for Pushover (plain text only) 3. Escape only 3 chars in HTML: & → &, < → <, > → > 4. Archive messages before sending for debugging 5. Use stat for file detection on macOS (not find -newermt) 6. Load credentials from env vars/Doppler (never hardcode) 7. Fire-and-forget background notifications (don't block restarts)

---

Reference Documentation

For detailed information, see:

• Telegram HTML - HTML mode formatting and message templates
• Pushover Integration - API calls and priority levels
• Credential Management - Doppler, env vars, and keychain patterns
• Watchexec Patterns - File detection and restart reason detection
• Common Pitfalls - HTML tags in Pushover, escaping issues, macOS compatibility

Bundled Examples:

• examples/notify-restart.sh - Complete dual-channel notification script
• examples/bot-wrapper.sh - watchexec wrapper with restart detection
• examples/setup-example.sh - Setup guide and installation steps

---

Troubleshooting

Post-Execution Reflection

After this skill completes, check before closing:

1. Did the command succeed? — If not, fix the instruction or error table that caused the failure. 2. Did parameters or output change? — If the underlying tool's interface drifted, update Usage examples and Parameters table to match. 3. Was a workaround needed? — If you had to improvise (different flags, extra steps), update this SKILL.md so the next invocation doesn't need the same workaround.

Only update if the issue is real and reproducible — not speculative.

Implementation Reference

Detailed implementation notes for dual-channel watchexec notifications.

Directory Structure

your-project/
├── notify-restart.sh      # Dual-channel notification script
├── bot-wrapper.sh         # watchexec wrapper with restart detection
├── your-app.py            # Your main application
└── logs/
    ├── bot-notifications.log              # Notification execution log
    └── notification-archive/              # Pre-send message archives
        └── YYYYMMDD-HHMMSS-reason-PID.txt

Complete Example Scripts

All examples are available in this skill's examples/ directory:

• examples/notify-restart.sh - Dual-channel notification script
• examples/bot-wrapper.sh - watchexec wrapper
• examples/setup-example.sh - Complete setup guide

notify-restart.sh Deep Dive

Script Architecture

Arguments → Parse watchexec info → Build HTML message → Archive → Send (Telegram + Pushover)

Key Implementation Details

1. HTML Escaping Function

/usr/bin/env bash << 'REFERENCE_SCRIPT_EOF'
# Escape only 3 characters for HTML: & < >
ESCAPED=$(echo "$text" | sed 's/&/\&amp;/g; s/</\&lt;/g; s/>/\&gt;/g')
REFERENCE_SCRIPT_EOF

Why this works:

• HTML only treats &, <, > as special
• Markdown requires escaping 40+ chars (., -, _, *, etc.)
• Simpler = more reliable

2. Heredoc Variable Expansion

WRONG (literal $MESSAGE sent):

cat > "$FILE" <<'MSGEOF'
$MESSAGE
MSGEOF

CORRECT (variable expanded):

cat > "$FILE" <<MSGEOF
$MESSAGE
MSGEOF

Rule: Unquoted heredoc delimiter (<<MSGEOF) allows variable expansion, quoted (<<'MSGEOF') treats as literal.

3. Python Telegram API Call

# Read message from temp file (avoids shell escaping hell)
with open(sys.argv[1], 'r') as f:
    message = f.read()

data = {
    'chat_id': chat_id,
    'text': message,
    'parse_mode': 'HTML'  # Key: HTML mode
}

req = urllib.request.Request(
    url,
    data=json.dumps(data).encode('utf-8'),
    headers={'Content-Type': 'application/json'}
)

Why Python + temp file:

• Avoids bash quote escaping complexity
• Handles Unicode properly
• Reliable JSON encoding

4. Priority and Sound Mapping

case "$REASON" in
    startup)
        EMOJI="🚀"
        PUSHOVER_SOUND="cosmic"
        PUSHOVER_PRIORITY=0  # Normal
        ;;
    code_change)
        EMOJI="🔄"
        PUSHOVER_SOUND="bike"
        PUSHOVER_PRIORITY=0  # Normal
        ;;
    crash)
        EMOJI="💥"
        PUSHOVER_SOUND="siren"
        PUSHOVER_PRIORITY=1  # High (bypasses quiet hours)
        ;;
esac

Pushover Priority Levels:

• 0: Normal (respects quiet hours, default sound)
• 1: High (bypasses quiet hours, requires acknowledgment)
• 2: Emergency (repeats until acknowledged)

5. Message Archiving

/usr/bin/env bash << 'REFERENCE_SCRIPT_EOF_2'
MESSAGE_ARCHIVE_FILE="$MESSAGE_ARCHIVE_DIR/$(date '+%Y%m%d-%H%M%S')-$REASON-$PID.txt"

cat > "$MESSAGE_ARCHIVE_FILE" <<ARCHIVE_EOF
========================================================================
Notification Archive
========================================================================
Timestamp: $TIMESTAMP
Reason: $REASON

--- TELEGRAM MESSAGE ---
$MESSAGE

--- WATCHEXEC INFO FILE ---
$(cat "$WATCHEXEC_INFO_FILE" 2>/dev/null || echo "Not available")

--- CRASH CONTEXT FILE ---
$(cat "$CRASH_CONTEXT_FILE" 2>/dev/null || echo "Not available")
========================================================================
ARCHIVE_EOF
REFERENCE_SCRIPT_EOF_2

Why archive:

• Post-mortem debugging (what was actually sent?)
• Audit trail
• Reproducing Telegram 400 errors
• ~5ms overhead per notification

bot-wrapper.sh Deep Dive

Restart Detection Logic

/usr/bin/env bash << 'REFERENCE_SCRIPT_EOF_3'
FIRST_RUN_MARKER="/tmp/watchexec_first_run_$$"

if [[ ! -f "$FIRST_RUN_MARKER" ]]; then
    REASON="startup"
    touch "$FIRST_RUN_MARKER"
else
    REASON="code_change"
fi

# Run process
EXIT_CODE=0
python3 "$MAIN_SCRIPT" || EXIT_CODE=$?

# Update reason if crashed
if [[ $EXIT_CODE -ne 0 ]]; then
    REASON="crash"
    # Send crash notification
    "$NOTIFY_SCRIPT" "crash" "$EXIT_CODE" "$INFO_FILE" "$CRASH_CONTEXT" &
fi
REFERENCE_SCRIPT_EOF_3

State transitions:

1. First run: startup → notify, create marker 1. watchexec restart (exit=0): code_change → notify 1. Process crash (exit≠0): crash → notify with context

File Change Detection (macOS Compatible)

Problem: find -newermt syntax differs on BSD (macOS) vs GNU (Linux)

Solution: Use stat to check modification time directly

/usr/bin/env bash << 'REFERENCE_SCRIPT_EOF_4'
NOW=$(date +%s)
FILE_MTIME=$(stat -f %m "$file" 2>/dev/null || stat -c %Y "$file" 2>/dev/null)
AGE=$((NOW - FILE_MTIME))

if [[ $AGE -lt 60 ]]; then
    echo "File modified ${AGE}s ago"
fi
REFERENCE_SCRIPT_EOF_4

Platform compatibility:

• macOS: stat -f %m (BSD stat)
• Linux: stat -c %Y (GNU stat)
• Fallback with || ensures portability

Crash Context Capture

/usr/bin/env bash << 'REFERENCE_SCRIPT_EOF_5'
CRASH_CONTEXT="/tmp/crash_context_$$.txt"

# Last 20 lines of main log
tail -20 "$BOT_LOG" > "$CRASH_CONTEXT"

# Last 10 lines of stderr
if [[ -f "$CRASH_LOG" ]]; then
    echo "--- STDERR ---" >> "$CRASH_CONTEXT"
    tail -10 "$CRASH_LOG" >> "$CRASH_CONTEXT"
fi

# Send in background (non-blocking)
"$NOTIFY_SCRIPT" "crash" "$EXIT_CODE" "$INFO_FILE" "$CRASH_CONTEXT" &
REFERENCE_SCRIPT_EOF_5

Why last N lines:

• Crash often has error at end of log
• Keeps notification message short
• Full logs available on server for deep debugging

Credential Management Patterns

Pattern 1: Environment Variables (Simple)

/usr/bin/env bash << 'REFERENCE_SCRIPT_EOF_6'
# ~/.bashrc or ~/.zshrc
export TELEGRAM_BOT_TOKEN="1234567890:ABC..."
export TELEGRAM_CHAT_ID="-1001234567890"
export PUSHOVER_APP_TOKEN="azGDORePK8gMa..."
export PUSHOVER_USER_KEY="uQiRzpo4DXghD..."

# In script
if [[ -z "${TELEGRAM_BOT_TOKEN:-}" ]]; then
    echo "Error: TELEGRAM_BOT_TOKEN not set"
    exit 1
fi
REFERENCE_SCRIPT_EOF_6

Pros: Simple, works everywhere Cons: Visible in ps, stored in shell history

Pattern 2: Doppler (Recommended for Production)

For Pushover (notifications/dev):

/usr/bin/env bash << 'SETUP_EOF'
# Install Doppler CLI
brew install dopplerhq/cli/doppler

# Load Pushover credentials from dedicated project
export PUSHOVER_APP_TOKEN=$(doppler secrets get PUSHOVER_APP_TOKEN \
  --project notifications \
  --config dev \
  --plain)
export PUSHOVER_USER_KEY=$(doppler secrets get PUSHOVER_USER_KEY \
  --project notifications \
  --config dev \
  --plain)

# Run with Doppler
doppler run --project notifications --config dev -- \
  watchexec --restart -- ./bot-wrapper.sh
SETUP_EOF

For Telegram (generic):

/usr/bin/env bash << 'DOPPLER_EOF'
# Set secrets
doppler secrets set TELEGRAM_BOT_TOKEN --value "..."
doppler secrets set TELEGRAM_CHAT_ID --value "..."

# Load in script
export TELEGRAM_BOT_TOKEN=$(doppler secrets get TELEGRAM_BOT_TOKEN --plain)
export TELEGRAM_CHAT_ID=$(doppler secrets get TELEGRAM_CHAT_ID --plain)
DOPPLER_EOF

Pros: Encrypted, team sync, audit trail, rotation Cons: Requires Doppler account

Pattern 3: macOS Keychain

/usr/bin/env bash << 'REFERENCE_SCRIPT_EOF_7'
# Store secret
security add-generic-password \
    -s 'telegram-bot-token' \
    -a "$USER" \
    -w 'your_token_here'

# Load in script
TELEGRAM_BOT_TOKEN=$(security find-generic-password \
    -s 'telegram-bot-token' \
    -a "$USER" \
    -w)
REFERENCE_SCRIPT_EOF_7

Pros: OS-level encryption, native macOS Cons: macOS only, no team sync

Pattern 4: systemd Environment File (Linux)

# /etc/systemd/system/myapp.service.d/env.conf
[Service]
EnvironmentFile=/etc/myapp/secrets.env

# /etc/myapp/secrets.env (chmod 600)
TELEGRAM_BOT_TOKEN=1234567890:ABC...
TELEGRAM_CHAT_ID=-1001234567890

Pros: systemd integration, file permissions Cons: Linux only, manual rotation

watchexec Configuration

Basic Usage

# Watch ./src directory, restart on .py file changes
watchexec --restart --watch ./src --exts py -- ./bot-wrapper.sh

Advanced Options

# Watch multiple directories
watchexec \
    --restart \
    --watch ./src \
    --watch ./lib \
    --watch ./config \
    --exts py,yaml \
    --ignore '*.pyc' \
    --ignore '__pycache__' \
    -- ./bot-wrapper.sh

Diagnostic Output

# Export watchexec events to JSON (for debugging)
watchexec \
    --restart \
    --watch ./src \
    --emit-events-to json \
    -- ./bot-wrapper.sh

With Delay (Debouncing)

# Wait 2s after file change before restarting (debounce rapid edits)
watchexec \
    --restart \
    --watch ./src \
    --debounce 2000 \
    -- ./bot-wrapper.sh

HTML Message Construction

HTML Tags Supported by Telegram

Not supported: <h1>, <div>, <span>, CSS, JavaScript

HTML Entity Escaping

/usr/bin/env bash << 'REFERENCE_SCRIPT_EOF_8'
# Required escaping
&  → &amp;   # Must be first to avoid double-escaping
<  → &lt;
>  → &gt;

# Escaping function
escape_html() {
    echo "$1" | sed 's/&/\&amp;/g; s/</\&lt;/g; s/>/\&gt;/g'
}

# Usage
FILENAME=$(basename "$file" | escape_html)
MESSAGE="Modified: <code>$FILENAME</code>"
REFERENCE_SCRIPT_EOF_8

Order matters: Always escape & first, otherwise you'll double-escape the & in < and >.

Message Template

MESSAGE="$EMOJI <b>Service $STATUS</b>

<b>Host</b>: <code>$HOSTNAME</code>
<b>Time</b>: $TIMESTAMP
<b>Exit Code</b>: $EXIT_CODE

<b>Trigger</b>: <code>$TRIGGER_PATH</code>
<b>Action</b>: $CHANGED_FILES

<i>Monitoring: watchexec</i>"

Pushover Message Format

Pushover uses plain text (no HTML or Markdown):

PUSHOVER_MESSAGE="Host: $HOSTNAME
Time: $TIMESTAMP
Exit: $EXIT_CODE
File: $CHANGED_FILE"

To strip HTML tags from Telegram message:

/usr/bin/env bash << 'REFERENCE_SCRIPT_EOF_9'
# Remove all <tag> and </tag>
PLAIN_TEXT=$(echo "$HTML_MESSAGE" | sed 's/<[^>]*>//g')
REFERENCE_SCRIPT_EOF_9

Testing Procedures

1. Test Notification Script Directly

# Startup notification
./notify-restart.sh startup 0

# Code change notification
./notify-restart.sh code_change 0

# Crash notification (with fake context)
echo "Error: Something went wrong" > /tmp/crash_context.txt
./notify-restart.sh crash 1 "" /tmp/crash_context.txt

2. Test HTML Rendering

# Message with special characters
MESSAGE="<b>Test</b>: <code>handler_classes.py</code> & <i>special_chars</i>"

# Should render correctly with underscores visible

3. Test watchexec Integration

# Start watchexec
watchexec --restart --watch ./src --exts py -- ./bot-wrapper.sh

# In another terminal, trigger change
touch ./src/test.py

# Check logs
tail -f ./logs/bot-notifications.log

4. Test Crash Handling

# Create script that crashes
cat > ./crash-test.py <<EOF
import sys
print("About to crash...")
sys.exit(1)
EOF

# Run wrapper
MAIN_SCRIPT=./crash-test.py ./bot-wrapper.sh

# Should send crash notification with exit code 1

Troubleshooting

Telegram 400 Bad Request

Symptoms: HTTP 400 error, no message received

Common causes:

1. Unescaped HTML entities (&, <, >) 1. Unclosed HTML tags (<b>text without </b>) 1. Unsupported HTML tags (<div>, <h1>) 1. Message too long (>4096 chars)

Debug:

/usr/bin/env bash << 'PREFLIGHT_EOF'
# Check archived message
cat logs/notification-archive/$(ls -t logs/notification-archive/ | head -1)

# Validate HTML structure
echo "$MESSAGE" | grep -E '<[^>]*$'  # Check for unclosed tags
PREFLIGHT_EOF

File Detection Not Working

Symptoms: Empty Trigger/Action fields

Check:

# Test stat command
stat -f %m ./src/test.py  # macOS
stat -c %Y ./src/test.py  # Linux

# Check watchexec info file
cat /tmp/watchexec_info_*.json

# Verify time window (default 60s)
# File must be modified within last 60s

Credentials Not Loading

Check environment:

# Are variables set?
echo "$TELEGRAM_BOT_TOKEN"
env | grep TELEGRAM

# Test Doppler
doppler secrets get TELEGRAM_BOT_TOKEN --plain

# Test keychain
security find-generic-password -s 'telegram-bot-token' -a "$USER" -w

No Notifications Received

Check:

# Telegram bot token valid?
curl "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/getMe"

# Chat ID correct?
# Should be negative for groups: -1001234567890

# Pushover credentials valid?
curl -s \
    --form-string "token=$PUSHOVER_APP_TOKEN" \
    --form-string "user=$PUSHOVER_USER_KEY" \
    --form-string "message=Test" \
    https://api.pushover.net/1/messages.json

Performance Metrics

From production deployment (2025-10-29):

Fire-and-forget: Notifications run in background (&), so process restart not delayed.

Security Best Practices

1. Never log credentials: Secrets should only exist in memory 1. Restrict archive permissions: chmod 700 logs/notification-archive/ 1. No secrets in filenames: File paths appear in messages 1. Use read-only API scopes: Limit bot permissions 1. Rotate credentials: Use Doppler or similar for automated rotation 1. Validate inputs: Sanitize any user-provided data before archiving

Real-World Output Example

Telegram Message (HTML Rendered)

🔄 Service Restarted (code change)

Host: myserver
Time: 2025-10-29 22:58:21 PDT
PID: 31307
Exit Code: 0

Trigger: /app/lib/format_utils.py
Action: Modified: format_utils.py

Monitoring: watchexec

Archived Message File

========================================================================
Notification Archive
========================================================================
Timestamp: 2025-10-29 22:58:21 PDT
Reason: code_change
Exit Code: 0
Host: myserver
PID: 31307

--- TELEGRAM MESSAGE ---
🔄 <b>Service Restarted (code change)</b>

<b>Host</b>: <code>myserver</code>
<b>Time</b>: 2025-10-29 22:58:21 PDT
<b>PID</b>: 31307
<b>Exit Code</b>: 0

<b>Trigger</b>: <code>/app/lib/format_utils.py</code>
<b>Action</b>: Modified: <code>format_utils.py</code>

<i>Monitoring: watchexec</i>

--- WATCHEXEC INFO FILE ---
{
  "timestamp": "2025-10-30T05:58:21Z",
  "watchexec": {
    "written_path": "/app/lib/format_utils.py"
  }
}
========================================================================

Success Metrics

Production deployment results:

• ✅ 100+ notifications sent successfully
• ✅ 0 formatting errors (after HTML migration)
• ✅ 100% dual-channel delivery
• ✅ File detection: 95% accuracy (5% missing due to rapid restart \<60s window)
• ✅ Average latency: 400ms per notification
• ✅ Zero blocking (fire-and-forget background execution)

Further Reading

• Telegram Bot API: https://core.telegram.org/bots/api#html-style
• Pushover API: https://pushover.net/api
• watchexec: https://github.com/watchexec/watchexec
• Doppler: https://docs.doppler.com/docs/cli

Skill: Dual-Channel Watchexec Notifications

Common Pitfalls

Pitfall 1: Pushover Shows HTML Tags (CRITICAL)

Problem: Pushover displays literal <code>, <b>, </code> in notifications

Cause: Pushover uses plain text only - does NOT interpret HTML

Solution: Strip HTML tags before sending to Pushover

/usr/bin/env bash << 'COMMON_PITFALLS_SCRIPT_EOF'
# ❌ WRONG - Sends HTML to Pushover
PUSHOVER_MESSAGE="Modified: <code>handler_classes.py</code>"
# User sees: Modified: <code>handler_classes.py</code>

# ✅ CORRECT - Strip HTML tags
CHANGED_FILES_PLAIN=$(echo "$CHANGED_FILES" | sed 's/<[^>]*>//g')
PUSHOVER_MESSAGE="Modified: $CHANGED_FILES_PLAIN"
# User sees: Modified: handler_classes.py
COMMON_PITFALLS_SCRIPT_EOF

Remember: Telegram = HTML, Pushover = Plain Text

Pitfall 2: Markdown Escaping Hell

Problem: Files with underscores (handler_classes.py) display as handlerclasses.py

Cause: Markdown treats _ as italic marker

Solution: Use HTML mode, wrap in <code> tags

/usr/bin/env bash << 'COMMON_PITFALLS_SCRIPT_EOF_2'
# ❌ WRONG (Markdown)
MESSAGE="Modified: handler_classes.py"  # Renders: handlerclasses.py

# ✅ CORRECT (HTML)
FILENAME=$(basename "$file" | sed 's/&/\&amp;/g; s/</\&lt;/g; s/>/\&gt;/g')
MESSAGE="Modified: <code>$FILENAME</code>"  # Renders: handler_classes.py
COMMON_PITFALLS_SCRIPT_EOF_2

Pitfall 3: Literal Variable Names Sent

Problem: Telegram receives literal text "$MESSAGE" instead of content

Cause: Heredoc with quotes prevents variable expansion

Solution: Use heredoc WITHOUT quotes

# ❌ WRONG
cat > "$FILE" <<'MSGEOF'
$MESSAGE
MSGEOF

# ✅ CORRECT
cat > "$FILE" <<MSGEOF
$MESSAGE
MSGEOF

Pitfall 4: macOS File Detection Failures

Problem: Empty Trigger/Action fields, no file detected

Cause: find -newermt syntax differs on BSD (macOS) vs GNU (Linux)

Solution: Use stat instead of find -newermt

/usr/bin/env bash << 'COMMON_PITFALLS_SCRIPT_EOF_3'
# ✅ CORRECT (portable)
FILE_MTIME=$(stat -f %m "$file" 2>/dev/null || echo "0")  # macOS
# For Linux: stat -c %Y "$file"
COMMON_PITFALLS_SCRIPT_EOF_3

Pitfall 5: Telegram 400 Bad Request

Problem: HTTP 400 errors with "Bad Request"

Causes:

1. Missing HTML escaping (&, <, >) 2. Unclosed HTML tags 3. Invalid HTML structure

Solution: Always escape special chars, validate HTML structure

# Test message before sending
echo "$MESSAGE" | grep -E '<[^>]*$'  # Check for unclosed tags

Pitfall 6: Hardcoded Credentials

Problem: Secrets leaked in git, exposed in logs

Solution: Use Doppler (canonical), env vars, or keychain

/usr/bin/env bash << 'VALIDATE_EOF'
# ❌ WRONG - Hardcoded secrets
PUSHOVER_APP_TOKEN="aej7osoja3x8nvxgi96up2poxdjmfj"
TELEGRAM_BOT_TOKEN="1234567890:ABC..."

# ✅ CORRECT - Load from Doppler (canonical source)
# For Pushover (notifications/dev):
PUSHOVER_APP_TOKEN=$(doppler secrets get PUSHOVER_APP_TOKEN \
  --project notifications --config dev --plain)
PUSHOVER_USER_KEY=$(doppler secrets get PUSHOVER_USER_KEY \
  --project notifications --config dev --plain)

# For Telegram (claude-config/dev):
TELEGRAM_BOT_TOKEN=$(doppler secrets get TELEGRAM_BOT_TOKEN \
  --project claude-config --config dev --plain)

# ✅ ALSO CORRECT - Validate env vars are set
TELEGRAM_BOT_TOKEN="${TELEGRAM_BOT_TOKEN:-}"
if [[ -z "$TELEGRAM_BOT_TOKEN" ]]; then
    echo "Error: TELEGRAM_BOT_TOKEN not set"
    exit 1
fi
VALIDATE_EOF

See: credential-management.md for complete patterns

Skill: Dual-Channel Watchexec Notifications

Credential Management

Pattern 1: Doppler (Recommended)

For Pushover (notifications/dev):

/usr/bin/env bash << 'CONFIG_EOF'
# Load Pushover credentials from Doppler
export PUSHOVER_APP_TOKEN=$(doppler secrets get PUSHOVER_APP_TOKEN \
  --project notifications \
  --config dev \
  --plain)
export PUSHOVER_USER_KEY=$(doppler secrets get PUSHOVER_USER_KEY \
  --project notifications \
  --config dev \
  --plain)
CONFIG_EOF

For Telegram (generic example):

/usr/bin/env bash << 'DOPPLER_EOF'
# Load from Doppler project (use bash wrapper for zsh compatibility)
/usr/bin/env bash -c 'export TELEGRAM_BOT_TOKEN=$(doppler secrets get TELEGRAM_BOT_TOKEN --plain) && export TELEGRAM_CHAT_ID=$(doppler secrets get TELEGRAM_CHAT_ID --plain)'
DOPPLER_EOF

Pattern 2: Environment Variables

/usr/bin/env bash << 'CREDENTIAL_MANAGEMENT_SCRIPT_EOF'
# From shell environment
if [[ -n "${TELEGRAM_BOT_TOKEN:-}" ]] && [[ -n "${TELEGRAM_CHAT_ID:-}" ]]; then
    # Send notification
fi
CREDENTIAL_MANAGEMENT_SCRIPT_EOF

Pattern 3: Keychain (macOS)

/usr/bin/env bash << 'CREDENTIAL_MANAGEMENT_SCRIPT_EOF_2'
/usr/bin/env bash -c 'PUSHOVER_TOKEN=$(security find-generic-password -s "pushover-app-token" -a "username" -w 2>/dev/null)'
CREDENTIAL_MANAGEMENT_SCRIPT_EOF_2

Security: Never hardcode credentials in scripts or skill files!

Evolution Log

Convention: Reverse chronological order (newest on top, oldest at bottom). Prepend new entries.

---

2026-02-26: Initial Evolution Log

Status: Skill is in use and maintained. Track improvements here.

Purpose

This evolution log tracks updates to the skill. Each entry should note:

• What changed (content, structure, tooling)
• Why it changed (bug fix, feature request, best practice)
• Files affected

How to Use

1. When updating SKILL.md or references, add an entry here with the date 2. Keep entries reverse-chronological (newest first) 3. Link to ADRs or GitHub issues when relevant 4. Reference specific line changes when helpful

---

Skill: Dual-Channel Watchexec Notifications

Credential Loading (Canonical Source)

Load from Doppler (notifications/dev):

/usr/bin/env bash << 'CONFIG_EOF'
# Canonical source for Pushover credentials
export PUSHOVER_APP_TOKEN=$(doppler secrets get PUSHOVER_APP_TOKEN \
  --project notifications \
  --config dev \
  --plain)
export PUSHOVER_USER_KEY=$(doppler secrets get PUSHOVER_USER_KEY \
  --project notifications \
  --config dev \
  --plain)
CONFIG_EOF

See: credential-management.md for fallback patterns (JSON config, local override)

---

API Call Pattern

curl -s \
  --form-string "token=$PUSHOVER_APP_TOKEN" \
  --form-string "user=$PUSHOVER_USER_KEY" \
  --form-string "device=device_name" \
  --form-string "title=$TITLE" \
  --form-string "message=$MESSAGE" \
  --form-string "sound=$SOUND" \
  --form-string "priority=$PRIORITY" \
  https://api.pushover.net/1/messages.json

Priority Levels:

• 0: Normal (default sound, respects quiet hours)
• 1: High (bypasses quiet hours, alert sound)

Sounds: cosmic, bike, siren, etc.

CRITICAL: Pushover Does NOT Support HTML

Pushover uses plain text only - MUST strip HTML tags before sending:

/usr/bin/env bash << 'PUSHOVER_INTEGRATION_SCRIPT_EOF'
# ❌ WRONG - Pushover will display literal HTML tags
PUSHOVER_MESSAGE="<b>Alert</b>: <code>file.py</code>"
# User sees: <b>Alert</b>: <code>file.py</code>

# ✅ CORRECT - Strip HTML tags for plain text
CHANGED_FILES_PLAIN=$(echo "$CHANGED_FILES" | sed 's/<[^>]*>//g')
PUSHOVER_MESSAGE="Alert: $CHANGED_FILES_PLAIN"
# User sees: Alert: file.py
PUSHOVER_INTEGRATION_SCRIPT_EOF

Why This Matters:

• Telegram uses HTML mode for formatting
• Pushover does NOT interpret HTML
• Sending HTML to Pushover shows ugly <code>, <b> tags in notification
• Always strip tags: sed 's/<[^>]*>//g'

Pattern: Build message in HTML for Telegram, then strip tags for Pushover:

/usr/bin/env bash << 'PUSHOVER_INTEGRATION_SCRIPT_EOF_2'
# 1. Build HTML message for Telegram
MESSAGE_HTML="<b>File</b>: <code>handler_classes.py</code>"

# 2. Strip HTML for Pushover
MESSAGE_PLAIN=$(echo "$MESSAGE_HTML" | sed 's/<[^>]*>//g')
# Result: "File: handler_classes.py"
PUSHOVER_INTEGRATION_SCRIPT_EOF_2

Skill: Dual-Channel Watchexec Notifications

Telegram: Use HTML Mode (NOT Markdown)

Why HTML Mode

Industry Best Practice:

• Markdown/MarkdownV2 requires escaping 40+ special characters (., -, _, etc.)
• HTML only requires escaping 3 characters: &, <, >
• More reliable, simpler, less error-prone

HTML Formatting

# Python API call
data = {
    'chat_id': chat_id,
    'text': message,
    'parse_mode': 'HTML'  # NOT 'Markdown' or 'MarkdownV2'
}

HTML Tags:

• Bold: <b>text</b>
• Code: <code>text</code>
• Italic: <i>text</i>
• Code blocks: <pre>text</pre>

HTML Escaping (Bash):

/usr/bin/env bash << 'TELEGRAM_HTML_SCRIPT_EOF'
# Escape special chars before sending
ESCAPED=$(echo "$text" | sed 's/&/\&amp;/g; s/</\&lt;/g; s/>/\&gt;/g')
MESSAGE="<b>Alert</b>: <code>$ESCAPED</code>"
TELEGRAM_HTML_SCRIPT_EOF

Message Template

Simplified format:

/usr/bin/env bash << 'TELEGRAM_HTML_SCRIPT_EOF_2'
# Build session debug line
SESSION_DEBUG_LINE="session=$CLAUDE_SESSION_ID | debug=~/.claude/debug/\${session}.txt"

# Normal restart (code change or startup)
MESSAGE="$EMOJI <b>Bot $STATUS</b>

<b>Directory</b>: <code>$WORKING_DIR</code>
<b>Branch</b>: <code>$GIT_BRANCH</code>
<code>$SESSION_DEBUG_LINE</code>
$WATCHEXEC_DETAILS"

# Crash (includes exit code and error details)
MESSAGE="$EMOJI <b>Bot Crashed</b>

<b>Directory</b>: <code>$WORKING_DIR</code>
<b>Branch</b>: <code>$GIT_BRANCH</code>
<code>$SESSION_DEBUG_LINE</code>

<b>Exit Code</b>: $EXIT_CODE
$CRASH_INFO"
TELEGRAM_HTML_SCRIPT_EOF_2

Why this format:

• Consistent with other Telegram messages (workflow completions, notifications)
• Removes unnecessary info (host, monitoring system, timestamp)
• Adds context (session ID, branch, directory)
• Exit code only shown for crashes (not for normal restarts with exit code 0)

Skill: Dual-Channel Watchexec Notifications

watchexec Integration

File Change Detection (macOS Compatible)

DO (works on macOS):

/usr/bin/env bash << 'PREFLIGHT_EOF'
# Use stat to check modification time
NOW=$(date +%s)
FILE_MTIME=$(stat -f %m "$file" 2>/dev/null || echo "0")
AGE=$((NOW - FILE_MTIME))

if [[ $AGE -lt 60 ]]; then
    echo "File modified ${AGE}s ago"
fi
PREFLIGHT_EOF

DON'T (broken on macOS):

# find -newermt has different syntax on BSD/macOS
find . -newermt "60 seconds ago"  # ❌ Fails on macOS

Restart Reason Detection

/usr/bin/env bash << 'WATCHEXEC_PATTERNS_SCRIPT_EOF'
# Determine why process restarted
if [[ ! -f "$FIRST_RUN_MARKER" ]]; then
    REASON="startup"
    touch "$FIRST_RUN_MARKER"
elif [[ $EXIT_CODE -ne 0 ]]; then
    REASON="crash"
else
    REASON="code_change"
fi
WATCHEXEC_PATTERNS_SCRIPT_EOF

Message Archiving (Debugging)

Always save messages before sending for post-mortem debugging:

/usr/bin/env bash << 'WATCHEXEC_PATTERNS_SCRIPT_EOF_2'
MESSAGE_ARCHIVE_DIR="/path/to/logs/notification-archive"
mkdir -p "$MESSAGE_ARCHIVE_DIR"
MESSAGE_FILE="$MESSAGE_ARCHIVE_DIR/$(date '+%Y%m%d-%H%M%S')-$REASON-$PID.txt"

cat > "$MESSAGE_FILE" <<ARCHIVE_EOF
========================================================================
Timestamp: $TIMESTAMP
Reason: $REASON
Exit Code: $EXIT_CODE

--- TELEGRAM MESSAGE ---
$MESSAGE

--- CONTEXT ---
$(cat "$WATCHEXEC_INFO_FILE" 2>/dev/null || echo "Not available")
========================================================================
ARCHIVE_EOF
WATCHEXEC_PATTERNS_SCRIPT_EOF_2