Linux Service Triage

Linux & service basics: logs, systemd/PM2, permissions, Nginx reverse proxy, DNS checks

PURPOSE

Diagnoses common Linux service issues using logs, systemd/PM2, file permissions, Nginx reverse proxy checks, and DNS sanity checks.

WHEN TO USE

• TRIGGERS:
• Show me why this service is failing using logs, then give the exact fix commands.
• Restart this app cleanly and confirm it is listening on the right port.
• Fix the permissions on this folder so the service can read and write safely.
• Set up Nginx reverse proxy for this port and verify DNS and TLS are sane.
• Create a systemd service for this script and make it survive reboots.
• DO NOT USE WHEN…
• You need kernel debugging or deep performance profiling.
• You want to exploit systems or bypass access controls.

INPUTS

• REQUIRED:
• Service type: systemd unit name or PM2 process name.
• Observed symptom: error message, status output, or logs (pasted by user).
• OPTIONAL:
• Nginx config snippet, domain name, expected upstream port.
• Filesystem paths used by the service.
• EXAMPLES:
• systemctl status myapp output + journalctl excerpt
• Nginx server block + domain + upstream port

OUTPUTS

• Default: triage report (likely cause, evidence from logs, minimal fix plan).
• If explicitly requested and safe: exact shell commands to apply the fix.

Success = service runs, listens on expected port, and reverse proxy/DNS path is correct.

WORKFLOW

1. Confirm scope and safety:

• identify service name and whether changes are permitted.

2. Gather evidence:

• status output + recent logs (see references/triage-commands.md).

3. Classify failure:

• config error, dependency missing, permission denied, port conflict, upstream unreachable, DNS mismatch.

4. Propose minimal fix + verification steps. 5. Validate network path (if web service):

• app listens → Nginx proxies → DNS resolves → (TLS sanity if applicable).

6. Provide restart/reload plan and confirm health checks. 7. STOP AND ASK THE USER if:

• logs/status output are missing,
• actions require privileged access not confirmed,
• TLS/cert management is required but setup is unknown.

OUTPUT FORMAT

TRIAGE REPORT
- Symptom:
- Evidence (what you provided):
- Most likely cause:
- Fix plan (minimal steps):
- Exact commands (ONLY if user approved changes):
- Verification:
- Rollback:

SAFETY & EDGE CASES

• Read-only by default: diagnose from provided outputs; do not assume you can run commands.
• Avoid destructive changes; require explicit confirmation for anything risky.
• Prefer nginx -t before reload and verify ports with ss.

EXAMPLES

• Input: “journal shows permission denied on /var/app/uploads.”

Output: path permission analysis + safe chown/chmod plan + verification.

• Input: “App works locally but domain returns 502.”

Output: upstream port checks + nginx error log interpretation + proxy_pass fix plan.

linux-service-triage 集成思路

与 OpenClaw 系统的深度集成

1. Heartbeat 增强

在 HEARTBEAT.md 的基础上，加入服务健康检查：

# 每6小时执行
check_critical_services() {
    services=(
        "openclaw-gateway"
        "nginx"
        "cron"
    )

    for service in "${services[@]}"; do
        if ! systemctl is-active --quiet "$service"; then
            # 收集诊断信息
            journalctl -u "$service" -n 50 > /tmp/service-triage-$service.log
            # 发送告警
            ./skills/feishu-automation/feishu-send.sh "⚠️ 服务 $service 异常，已收集诊断信息"
        fi
    done
}

2. 快速诊断命令包装

创建便捷命令供用户直接调用：

# ~/clawd/scripts/triage.sh
#!/bin/bash
# Linux Service 快速诊断

SERVICE=${1:-"openclaw-gateway"}

echo "=== TRIAGE REPORT: $SERVICE ==="
echo "[1] Service Status:"
systemctl status $SERVICE --no-pager

echo -e "\n[2] Recent Logs:"
journalctl -u $SERVICE -n 50 --no-pager

echo -e "\n[3] Port Listening:"
if [ "$SERVICE" = "nginx" ]; then
    ss -ltnp | grep :80
    ss -ltnp | grep :443
elif [ "$SERVICE" = "openclaw-gateway" ]; then
    ss -ltnp | grep :3000
fi

echo -e "\n[4] Configuration Test:"
if [ "$SERVICE" = "nginx" ]; then
    nginx -t
fi

3. Auto-Triage Agent

结合 multi-agent-architecture，创建专门的诊断 Agent：

// skills/linux-service-triage/agent.js
{
  name: "triage-agent",
  trigger: "service failure detected",
  actions: [
    "collect_evidence",
    "analyze_logs",
    "propose_fix",
    "verify_resolution"
  ]
}

常见问题知识库

OpenClaw Gateway 故障模式

Nginx 502 模式

诊断速查卡

创建 /home/aa/clawd/skills/linux-service-triage/references/quick-reference.md：

# Linux Service 故障速查卡

## 🚨 紧急诊断三步法
1. systemctl status <service>
2. journalctl -u <service> -n 100
3. ss -ltnp | grep <port>

## 🔍 权限问题

namei -l /path/to/file # 逐级检查权限 ls -la /path/to/file # 查看文件权限

## 🌐 网络问题

ss -ltnp # 查看监听端口 curl localhost:PORT # 测试本地连接 dig +short domain.com # 测试 DNS 解析

## 📋 Nginx 问题

nginx -t # 测试配置 tail -f /var/log/nginx/error.log # 实时错误日志

## 🔥 进程问题

ps aux | grep name # 查找进程 kill -9 PID # 强制杀死

自动化改进

预防性监控

# 添加到 crontab
*/30 * * * * /home/aa/clawd/scripts/service-health-check.sh

自愈脚本

# services/autoheal.sh
#!/bin/bash
# 服务自动恢复（需谨慎使用）

SERVICE=$1
if ! systemctl is-active --quiet "$SERVICE"; then
    logger "Auto-healing: attempting to restart $SERVICE"
    systemctl restart "$SERVICE"
    sleep 5
    if systemctl is-active --quiet "$SERVICE"; then
        logger "Auto-heal successful: $SERVICE is running"
        ./skills/feishu-automation/feishu-send.sh "✅ $SERVICE 自动恢复成功"
    else
        logger "Auto-heal failed: $SERVICE still down"
        ./skills/feishu-automation/feishu-send.sh "🚨 $SERVICE 自动恢复失败，需要人工介入"
    fi
fi

与其他 Skill 的协同

+ healthcheck

• healthcheck: 配置审计、安全加固
• triage: 故障诊断、服务恢复
• 协同: 定期审计 → 发现配置问题 → triage 修复

+ docker-deployment

• 容器服务诊断
• 网络模式桥接问题
• 卷挂载权限问题

+ security-audit

• 服务权限最小化
• 日志审计
• 异常检测

持续改进

学习反馈循环

1. 记录每次故障案例 2. 提取通用模式 3. 更新知识库 4. 优化诊断脚本 5. 提高自动化程度

度量指标

• 平均诊断时间 (MTTD)
• 平均修复时间 (MTTR)
• 自动恢复成功率
• 重复故障模式识别

Linux Service Triage Commands (safe first)

Logs

• systemd: journalctl -u <service> -n 200 --no-pager
• live: journalctl -u <service> -f
• PM2: pm2 logs <name> --lines 200

Status

• systemd: systemctl status <service> --no-pager
• ports: ss -ltnp | grep <port>
• processes: ps aux | grep <name>

Permissions

• ls -la <path>
• namei -l <path> (checks each directory in path)

Nginx

• config test: nginx -t
• reload: systemctl reload nginx
• logs: /var/log/nginx/access.log, /var/log/nginx/error.log

DNS sanity

• dig +short <host>
• dig +trace <host>