Alibabacloud Ecs Reboot Or Crash Diagnosis

ECS Instance Reboot/Crash Diagnosis

Diagnose root cause of ECS instance unexpected reboot or crash. Uses standard workflow: check platform maintenance events first, then check internal system logs. Supports both Linux and Windows systems.

Required Parameters

Before starting diagnosis, must obtain the following parameters from user:

If user does not provide any of the above parameters, must ask user first. Do not start diagnosis.

Mandatory Execution Rules

1. Must obtain parameters first — Instance ID and Region ID are required. Must ask user if missing. 2. Standard workflow cannot be skipped — Must execute in order: Maintenance Event Check → OSType Detection → System Log Check 3. Must check Cloud Assistant status before diagnostics — Before executing Step 3A/3B, must verify Cloud Assistant is running via DescribeCloudAssistantStatus. If not running, provide alternative diagnostic approaches. 4. All diagnostic conclusions must be based on actual data — No fabrication, speculation, or assumptions 5. Output format must be strictly followed — After diagnosis, must read the complete template in `references/output-format.md`, output strictly according to template structure. No free-form output, no omitted sections, no changed hierarchy. Every placeholder {...} in the template must be filled with actual data.

---

Prerequisites

CLI Tools

• aliyun-cli 3.3.3+ (required) — For calling Alibaba Cloud API
• Installation & configuration: see CLI Installation Guide

AI-Mode Configuration (Required)

Before using aliyun CLI commands, must configure AI-Mode:

# Enable AI-Mode
aliyun configure ai-mode enable

# Set user-agent for skill identification
aliyun configure ai-mode set-user-agent --user-agent "AlibabaCloud-Agent-Skills/alibabacloud-ecs-reboot-or-crash-diagnosis"

# Update plugins
aliyun plugin update

After diagnosis complete, disable AI-Mode:

aliyun configure ai-mode disable

Alibaba Cloud Credentials

Credentials must be pre-configured outside of agent session. Agent only verifies:

aliyun configure list

Instance Requirements

• Cloud Assistant client must be installed and running on the instance
• Alibaba Cloud Linux: Pre-installed by default
• Ubuntu/CentOS/Other: May require manual installation, check with DescribeCloudAssistantStatus API
• Installation guide: https://help.aliyun.com/document_detail/64930.html
• Instance status must be Running
• Note: If Cloud Assistant is not running, diagnostic commands cannot be executed remotely. Must provide manual diagnostic steps to user.

---

Required RAM Permissions

See [RAM Policies](references/ram-policies.md) for the complete permission list and custom policy example.

---

Step 1: Confirm Instance Information (Cannot Skip)

Verify instance exists and get basic information:

aliyun ecs describe-instances \
  --biz-region-id <REGION_ID> \
  --region <REGION_ID> \
  --instance-ids '["<INSTANCE_ID>"]'

Confirm from returned JSON:

• RegionId — Region ID (matches user provided)
• Status — Instance status (Running/Stopped)
• InstanceName — Instance name
• OSType — Operating system type (windows / linux)

Record OSType for Step 3 branch selection.

---

Step 2: Check ECS Maintenance Events

Query instance historical system events to determine if platform maintenance caused reboot:

aliyun ecs describe-instance-history-events \
  --biz-region-id <REGION_ID> \
  --region <REGION_ID> \
  --instance-id <INSTANCE_ID> \
  --event-cycle-status Executed

Event Analysis:

Important Notes for InstanceFailure.Reboot:

• This event indicates the platform detected an instance-level anomaly and triggered automatic recovery
• Common causes: kernel panic, OOM, system hang, critical process failure
• Must execute Step 3 to check system logs for root cause
• Even if no obvious errors in logs, the instance may have been unresponsive at kernel level

If maintenance event found:

• Clearly inform user of reboot cause (event type, time, reason)
• Provide handling suggestions
• End diagnosis flow

If no maintenance event found:

• Continue to Step 3, check internal system logs based on OSType

---

Step 3A: Linux System Diagnosis (Execute when OSType is linux)

Step 3A.1: Check Cloud Assistant Status (Mandatory)

Before executing diagnostic commands, verify Cloud Assistant is running:

aliyun ecs describe-cloud-assistant-status \
  --biz-region-id <REGION_ID> \
  --region <REGION_ID> \
  --instance-id <INSTANCE_ID>

Check the response:

{
  "InstanceCloudAssistantStatusSet": {
    "InstanceCloudAssistantStatus": [
      {
        "InstanceId": "i-xxx",
        "RegionId": "cn-xxx",
        "CloudAssistantStatus": "true",
        "LastHeartbeatTime": "2026-04-09T07:26:58Z"
      }
    ]
  }
}

Important Notes:

• CloudAssistantStatus is a string ("true"/"false"), not boolean
• Check LastHeartbeatTime to ensure it's recent (within last few minutes)
• Even if status is "true", RunCommand may still fail if service is unstable
• Always check RunCommand execution result and handle failures gracefully
• Ubuntu vs RHEL differences:
• RHEL/CentOS/Alibaba Cloud Linux: Service name is kdump, crash files named vmcore-*
• Ubuntu/Debian: Service name is kdump-tools, crash files named dump.* and dmesg.*
• Diagnostic script now checks both service names and all crash file types

If CloudAssistantStatus is false or command fails:

• Cloud Assistant is not installed or not running on the instance
• Cannot proceed with remote diagnostic commands
• Alternative approaches:

1. Guide user to SSH into the instance and check logs manually 2. Provide manual diagnostic commands for user to execute 3. Suggest installing Cloud Assistant: Installation Guide 4. Check instance monitoring data via CloudMonitor API

If CloudAssistantStatus is true:

• Proceed to Step 3A.2

Step 3A.2: Execute Linux Diagnostic Script

Execute Linux diagnostic script via Cloud Assistant to check:

• System reboot records (last reboot, /var/log/messages or /var/log/syslog)
• Kernel Panic records (dmesg)
• OOM records and vm.panic_on_oom configuration
• Kdump configuration and crash dump file status
• Crash dump files: vmcore (RHEL/CentOS) or dump./dmesg. (Ubuntu/Debian)

Complete diagnostic commands: see [diagnostic-commands.md](references/diagnostic-commands.md#linux-system-diagnosis)

Linux Result Analysis:

---

Step 3B: Windows System Diagnosis (Execute when OSType is windows)

Step 3B.1: Check Cloud Assistant Status (Mandatory)

Before executing diagnostic commands, verify Cloud Assistant is running:

aliyun ecs describe-cloud-assistant-status \
  --biz-region-id <REGION_ID> \
  --region <REGION_ID> \
  --instance-id <INSTANCE_ID>

Check the response:

• CloudAssistantStatus: true — Cloud Assistant is running, proceed to Step 3B.2
• CloudAssistantStatus: false — Cloud Assistant is not running
• Cannot proceed with remote diagnostic commands
• Guide user to SSH/RDP into instance and run diagnostics manually
• Suggest reinstalling Cloud Assistant: Windows Installation Guide

Step 3B.2: Execute Windows Diagnostic Script

Execute Windows diagnostic script via Cloud Assistant to check:

• System uptime and unexpected shutdown events (Event ID 41, 1074, 6008, 6006)
• Memory dump configuration and pagefile settings
• MEMORY.DMP and minidump files existence
• BSOD events and application crashes

Complete diagnostic commands: see [diagnostic-commands.md](references/diagnostic-commands.md#windows-system-diagnosis)

Windows Result Analysis:

---

Step 3.5: Get Cloud Assistant Command Output (Required after Step 3)

After executing diagnostic script via RunCommand, query the execution result:

aliyun ecs describe-invocations \
  --biz-region-id <REGION_ID> \
  --region <REGION_ID> \
  --instance-id <INSTANCE_ID> \
  --invoke-id <INVOKE_ID>

Important Notes:

• Use --instance-id (not --instance-id.1) for describe-invocations API
• The InvokeId is returned by the RunCommand API call
• Decode the Output field from Base64 to get diagnostic results
• Check InvokeStatus to ensure command execution completed successfully

---

Step 4: Analyze Crash Dump Files

If Step 3 found crash dump files (vmcore on Linux, MEMORY.DMP/minidump on Windows), perform preliminary analysis.

Complete analysis commands: see [diagnostic-commands.md](references/diagnostic-commands.md#crash-dump-analysis)

Important: If Linux vmcore files need deep analysis or Windows dump files (MEMORY.DMP/minidump) are found, recommend the user contact Alibaba Cloud technical support team for professional crash dump analysis assistance.

---

Step 5: Recommend Kdump Configuration (If Not Configured)

If Step 3A found Kernel Panic records but no vmcore files, must advise user to configure Kdump.

When to Recommend Kdump Configuration

• Kernel panic records found in dmesg or system logs, but /var/crash has no vmcore files
• Kdump service status shows inactive or failed
• /proc/cmdline does not contain crashkernel= parameter

Key Points to Communicate

1. Why Kdump is needed: Without Kdump, kernel crashes will not generate vmcore files, making root cause analysis impossible.

2. Configuration requirements:

• Reserve memory for crash kernel via crashkernel= kernel parameter
• Enable and start the kdump (RHEL/CentOS) or kdump-tools (Ubuntu/Debian) service
• Ensure sufficient disk space in /var/crash (or configured path)

3. Configuration reference: Provide guidance from diagnostic-commands.md

Kdump Configuration Steps Summary

RHEL/CentOS/Alibaba Cloud Linux: 1. Install: yum install -y kexec-tools 2. Add crashkernel=auto to kernel parameters in /etc/default/grub 3. Run grub2-mkconfig -o /boot/grub2/grub.cfg 4. Reboot the instance 5. Enable: systemctl enable --now kdump

Ubuntu/Debian: 1. Install: apt-get install -y kdump-tools 2. Set USE_KDUMP=1 in /etc/default/kdump-tools 3. Run update-grub (crashkernel parameter usually auto-added) 4. Reboot the instance 5. Verify: systemctl status kdump-tools

Windows Memory Dump Configuration

If Step 3B found BSOD events but no dump files: 1. Verify pagefile is configured and has sufficient size 2. Enable memory dump: System Properties → Advanced → Startup and Recovery → Settings 3. Select "Automatic memory dump" or "Kernel memory dump" 4. Ensure CrashDumpEnabled registry value is not 0

---

Final Output (Must execute after diagnosis complete)

After all diagnostic steps complete, must do both of the following:

1. Read `references/output-format.md` — Get complete output format template 2. Output strictly according to template structure — Choose corresponding template based on actual result

---

References

• [Output Format](references/output-format.md) — Diagnostic result output template
• [Common Scenarios](references/scenarios.md) — Typical problem diagnosis examples
• [Diagnostic Commands](references/diagnostic-commands.md) — Complete diagnostic scripts and analysis commands

Diagnostic Commands Reference

本文档提供诊断检查项和命令参考。Agent 应根据实际操作系统类型和版本，动态生成适配的诊断命令。

重要原则：

- 不同 Linux 发行版的服务名称、日志路径、工具命令可能不同

- 先通过 DescribeInstances 获取 OSType 和 OSName，再生成适配的命令

- 命令应包含错误处理，避免因路径不存在或命令不可用而中断

---

Linux 系统诊断

检查项清单

操作系统差异对照表

命令示例参考

以下命令仅供参考，Agent 应根据实际操作系统动态调整。

1. 获取系统信息

# 操作系统版本
cat /etc/os-release

# 内核版本
uname -r

# 系统运行时间
uptime

2. 系统重启历史

# 重启历史记录
last reboot | head -10

# 最近一次启动时间
who -b

3. 系统日志检查

RHEL/CentOS/Alibaba Cloud Linux:

# 重启/关机相关日志
grep -i "reboot\|shutdown\|restart" /var/log/messages | tail -20

# Kernel Panic 记录
dmesg | grep -i "panic\|oops" | tail -20
grep -i "kernel panic" /var/log/messages | tail -10

# OOM 记录
grep -i "out of memory\|oom\|kill process" /var/log/messages | tail -20

Ubuntu/Debian:

# 重启/关机相关日志
grep -i "reboot\|shutdown\|restart" /var/log/syslog | tail -20

# Kernel Panic 记录
dmesg | grep -i "panic\|oops" | tail -20
grep -i "kernel panic" /var/log/syslog | tail -10

# OOM 记录
grep -i "out of memory\|oom\|kill process" /var/log/syslog | tail -20

4. OOM Panic 配置

# 检查 OOM 时是否触发 panic
sysctl -n vm.panic_on_oom

# 返回值说明:
# 0 - OOM 只杀死进程，系统继续运行
# 1 - OOM 触发 kernel panic，导致系统重启

5. Kdump 状态检查

RHEL/CentOS/Alibaba Cloud Linux:

# 检查 kdump 服务状态
systemctl status kdump

# 检查是否已配置
cat /etc/kdump.conf

# 获取转储路径 (默认 /var/crash)
grep "^path" /etc/kdump.conf

Ubuntu/Debian:

# 检查 kdump-tools 服务状态
systemctl status kdump-tools

# 检查是否已配置
cat /etc/default/kdump-tools

# 检查内核 crashkernel 参数
cat /proc/cmdline | grep crashkernel

6. 崩溃转储文件检查

# 检查转储目录
ls -la /var/crash/

# RHEL/CentOS: 查找 vmcore 文件
find /var/crash -name "vmcore*" -type f -exec ls -lh {} \;

# Ubuntu/Debian: 查找 dump 和 dmesg 文件
find /var/crash -name "dump.*" -type f -exec ls -lh {} \;
find /var/crash -name "dmesg.*" -type f -exec ls -lh {} \;

# 检查最近 7 天的转储文件
find /var/crash -type f \( -name "vmcore*" -o -name "dump.*" -o -name "dmesg.*" \) -mtime -7

---

Kdump 配置建议

何时需要建议配置 Kdump

以下情况应建议用户配置 Kdump：

1. 检测到 Kernel Panic 迹象但无 vmcore 文件

• dmesg 或系统日志中有 panic 记录
• 但 /var/crash 目录为空或不存在

2. Kdump 服务未运行

• systemctl status kdump 显示 inactive/failed
• systemctl status kdump-tools 显示 inactive/failed

3. 内核未配置 crashkernel 参数

• /proc/cmdline 中没有 crashkernel= 参数

Kdump 配置参考

RHEL/CentOS/Alibaba Cloud Linux:

# 1. 安装 kexec-tools (如未安装)
yum install -y kexec-tools

# 2. 配置 /etc/kdump.conf
# 默认配置通常可用，关键配置项：
# path /var/crash          # 转储文件存储路径
# core_collector makedumpfile -l --message-level 1 -d 31  # 压缩转储

# 3. 在 /etc/default/grub 的 GRUB_CMDLINE_LINUX 中添加 crashkernel 参数
# crashkernel=auto  或  crashkernel=128M

# 4. 更新 grub 配置
grub2-mkconfig -o /boot/grub2/grub.cfg

# 5. 重启系统使 crashkernel 生效
reboot

# 6. 启用并启动 kdump 服务
systemctl enable kdump
systemctl start kdump
systemctl status kdump

Ubuntu/Debian:

# 1. 安装 kdump-tools
apt-get install -y kdump-tools

# 2. 配置 /etc/default/kdump-tools
# USE_KDUMP=1

# 3. 更新 grub 配置 (安装时通常会自动添加 crashkernel 参数)
update-grub

# 4. 重启系统
reboot

# 5. 验证服务状态
systemctl status kdump-tools

Kdump 配置验证

# 验证 crashkernel 参数已生效
cat /proc/cmdline | grep crashkernel

# 验证 kdump 服务状态
systemctl status kdump  # RHEL/CentOS
systemctl status kdump-tools  # Ubuntu/Debian

---

Windows 系统诊断

检查项清单

事件 ID 说明

内存转储类型

PowerShell 命令示例

# 系统信息
Get-ComputerInfo | Select-Object WindowsProductName, WindowsVersion, OsArchitecture, CsName

# 系统运行时间
$os = Get-CimInstance Win32_OperatingSystem
$uptime = (Get-Date) - $os.LastBootUpTime
Write-Host "Last boot: $($os.LastBootUpTime)"
Write-Host "Uptime: $($uptime.Days) days, $($uptime.Hours) hours"

# 意外关机事件
Get-WinEvent -FilterHashtable @{LogName="System"; ID=41,1074,6008,6006} -ErrorAction SilentlyContinue | 
    Select-Object TimeCreated, Id, Message -First 10

# 内存转储配置
$crashControl = Get-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\CrashControl"
Write-Host "CrashDumpEnabled: $($crashControl.CrashDumpEnabled)"
Write-Host "DumpFile: $($crashControl.DumpFile)"
Write-Host "MinidumpDir: $($crashControl.MinidumpDir)"

# 页面文件配置
Get-CimInstance Win32_PageFileUsage | Select-Object Name, AllocatedBaseSize, CurrentUsage

# 检查 MEMORY.DMP
$dumpFile = $crashControl.DumpFile
if (-not $dumpFile) { $dumpFile = "C:\Windows\MEMORY.DMP" }
if (Test-Path $dumpFile) {
    $fileInfo = Get-Item $dumpFile
    Write-Host "MEMORY.DMP found: Size=$([math]::Round($fileInfo.Length/1GB,2)) GB, Modified=$($fileInfo.LastWriteTime)"
}

# 检查 Minidump 文件
$minidumpDir = $crashControl.MinidumpDir
if (-not $minidumpDir) { $minidumpDir = "C:\Windows\Minidump" }
if (Test-Path $minidumpDir) {
    Get-ChildItem -Path $minidumpDir -Filter "*.dmp" | Sort-Object LastWriteTime -Descending | Select-Object -First 5
}

# BSOD 事件
Get-WinEvent -FilterHashtable @{LogName="System"; ProviderName="Microsoft-Windows-WER-SystemErrorReporting"} -ErrorAction SilentlyContinue | 
    Select-Object TimeCreated, Id, Message -First 10

Windows 内存转储配置建议

当检测到 BSOD 事件但无转储文件时，建议配置内存转储：

1. 通过系统属性配置：

• 右键"此电脑" → 属性 → 高级系统设置
• 启动和故障恢复 → 设置
• 选择"自动内存转储"或"内核内存转储"
• 确保页面文件大小足够（至少内存大小 + 1MB）

2. PowerShell 配置：

# 设置自动内存转储
Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\CrashControl" -Name "CrashDumpEnabled" -Value 7

# 确保页面文件存在且大小足够
# 通常由系统自动管理，检查方法：
$cs = Get-CimInstance Win32_ComputerSystem
if ($cs.AutomaticManagedPagefile) {
    Write-Host "Pagefile is automatically managed"
} else {
    # 手动配置页面文件
    # 需要重启生效
}

---

崩溃转储文件分析

Linux vmcore 分析

如果找到 vmcore 文件，可读取 vmcore-dmesg.txt 进行初步分析：

# 查看 vmcore-dmesg.txt 内容
cat /var/crash/127.0.0.1-*/vmcore-dmesg.txt

# 关键信息搜索
grep -i "kernel panic" /var/crash/*/vmcore-dmesg.txt
grep -i "RIP:" /var/crash/*/vmcore-dmesg.txt
grep -i "Call Trace" /var/crash/*/vmcore-dmesg.txt

关键信息解读：

注意：深度 vmcore 分析需要使用 crash 工具和调试符号包，建议联系阿里云技术支持获取专业分析。

Windows 转储文件分析

使用 WinDbg 或 BlueScreenView 工具分析：

# WinDbg 命令
!analyze -v          # 自动分析崩溃原因
k                    # 查看调用栈
.bugcheck            # 查看 bugcheck 代码

注意：深度 dump 分析建议联系阿里云技术支持。

---

通过云助手执行命令

诊断命令通过阿里云云助手远程执行：

Linux 命令执行

aliyun ecs run-command \
  --biz-region-id <REGION_ID> \
  --region <REGION_ID> \
  --type RunShellScript \
  --instance-id <INSTANCE_ID> \
  --timeout 3600 \
  --command-content '<SCRIPT_CONTENT>'

Windows 命令执行

aliyun ecs run-command \
  --biz-region-id <REGION_ID> \
  --region <REGION_ID> \
  --type RunPowerShellScript \
  --instance-id <INSTANCE_ID> \
  --timeout 3600 \
  --command-content '<SCRIPT_CONTENT>'

获取命令执行结果

aliyun ecs describe-invocations \
  --biz-region-id <REGION_ID> \
  --region <REGION_ID> \
  --instance-id <INSTANCE_ID> \
  --invoke-id <INVOKE_ID>

注意：Output 字段为 Base64 编码，需要解码后查看。

Output Format Requirements

After diagnosis is complete, output results according to the following structure.

Table of Contents

1. Linux Diagnosis Result 2. Windows Diagnosis Result

---

Linux Diagnosis Result

## Diagnostic Progress

### Step 1: Confirm Instance Information
> First need to confirm instance basic info and region.

- Instance ID: {instance_id}
- Region: {region_id}
- OS Type: Linux
- Current Status: {status}

### Step 2: Check Maintenance Events
> Check if platform maintenance events caused reboot.

**Findings:**
- {event_query_result}

### Step 3A: Linux System Diagnosis (if needed)
> No maintenance events found, checking internal restart or panic records.

**Cloud Assistant Status Check:**
- Cloud Assistant Running: {yes/no}
- If no: {explain why cannot proceed and provide alternative approaches}

**Findings:**
- {cloud_assistant_execution_result}

**Kdump Configuration Status:**
- Service Status: {active/inactive/active (kdump-tools)}
- Service Type: {kdump (RHEL/CentOS) / kdump-tools (Ubuntu/Debian)}
- Crash Dump Path: {configured_path}

**OOM Panic Configuration:**
- vm.panic_on_oom: {0/1}
- Impact: {OOM kills process only / OOM triggers kernel panic and reboot}

**Crash Dump File Check:**
- Found crash dumps: {yes/no}
- Dump type: {vmcore (RHEL) / dump.*+dmesg.* (Ubuntu)}
- Latest dump: {file_path, size, time}
- Panic reason (from dmesg): {panic_message_if_available}

**Alternative Diagnostic Approaches (if Cloud Assistant not available):**

Provide these commands to user for manual execution via SSH

ssh root@{instance_public_ip}

Check reboot history

last reboot

Check system logs

grep -i "reboot\|shutdown\|panic\|oom" /var/log/syslog | tail -50

Check dmesg for errors

dmesg | grep -i "panic\|oom\|error" | tail -20

Check kdump

systemctl status kdump ls -lh /var/crash/

### Step 4A: vmcore-dmesg.txt Analysis (if vmcore found)
> Found vmcore file, reading vmcore-dmesg.txt for preliminary analysis.

**Panic Reason:**
- {panic_specific_reason}

**Crash Location:**
- RIP: {function_and_address_at_crash}
- Involved Modules: {related_kernel_modules}

**Call Stack:**

{key_call_stack_fragment}

### Step 5: Kdump Configuration Recommendation (if no vmcore and kdump not configured)
> Kernel panic detected but no crash dump file found. Kdump is not properly configured.

**Current Kdump Status:**
- Service Status: {kdump_service_status}
- crashkernel Parameter: {present/absent in /proc/cmdline}
- Config File Exists: {yes/no}

**Why Kdump is Needed:**
Without Kdump configured, kernel crashes will not generate vmcore files, making root cause analysis impossible for future occurrences.

**Configuration Steps for {OS_Type}:**

{configuration_steps_based_on_os}

---

## Diagnostic Conclusion

- **Root Cause Analysis**: {root_cause}
- **Impact Scope**: {impact_scope}

---

## Recommendations

1. {recommendation_1}
2. {recommendation_2}

---

Windows Diagnosis Result

## Diagnostic Progress

### Step 1: Confirm Instance Information
> First need to confirm instance basic info and region.

- Instance ID: {instance_id}
- Region: {region_id}
- OS Type: Windows
- Current Status: {status}

### Step 2: Check Maintenance Events
> Check if platform maintenance events caused reboot.

**Findings:**
- {event_query_result}

### Step 3B: Windows System Diagnosis (if needed)
> No maintenance events found, checking Windows crash dump and event logs.

**System Uptime:**
- Last Boot Time: {last_boot_time}
- Uptime: {days} days, {hours} hours

**Unexpected Shutdown Events:**
- {shutdown_event_summary}

**Memory Dump Configuration:**
- CrashDumpEnabled: {0/1/2/3/7}
- Dump Type: {None/Complete/Kernel/Small/Automatic}
- Dump File Path: {dump_file_path}
- Pagefile: {configured/not configured}

**Memory Dump File Check:**
- Memory dump file: {found/not found}
- File size: {size}
- Last modified: {timestamp}

**Minidump Files:**
- Count: {count}
- Latest: {filename, timestamp}

**BSOD Events:**
- {bsod_event_summary}

---

## Diagnostic Conclusion

- **Root Cause Analysis**: {root_cause}
- **Impact Scope**: {impact_scope}

---

## Recommendations

1. {recommendation_1}
2. {recommendation_2}

RAM 权限清单

本 Skill 执行所需的 RAM 权限（最小权限原则）:

必需权限

ecs:DescribeInstances — 确认实例存在并获取基本信息（状态、名称、操作系统类型）

ecs:DescribeInstanceAttribute — 获取实例详细属性，用于操作系统类型检测和分支选择

ecs:DescribeInstanceHistoryEvents — 查询实例历史维护事件，判断是否为平台触发重启

ecs:DescribeCloudAssistantStatus — 验证云助手运行状态，确保远程诊断命令可执行

ecs:RunCommand — 通过云助手执行诊断脚本（Linux Shell 或 Windows PowerShell）

ecs:DescribeInvocations — 获取云助手命令执行结果，提取诊断输出

权限说明

• 权限范围: 仅包含诊断所需的只读和命令执行权限
• 写操作: 无（本 Skill 不修改实例配置）
• 通配符: 未使用（遵循最小权限原则）

自定义策略示例

{
  "Version": "1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecs:DescribeInstances",
        "ecs:DescribeInstanceAttribute",
        "ecs:DescribeInstanceHistoryEvents",
        "ecs:DescribeCloudAssistantStatus",
        "ecs:RunCommand",
        "ecs:DescribeInvocations"
      ],
      "Resource": "*"
    }
  ]
}

使用场景

此权限配置适用于 ECS 实例故障诊断场景:

• 检查平台维护事件
• 通过云助手远程执行诊断命令
• 获取系统日志和崩溃转储文件信息
• 分析重启/崩溃根因

Common Diagnostic Scenarios

This document lists common diagnostic scenarios and expected outputs for the ecs-reboot-or-crash skill.

---

Linux Scenarios

Scenario 1: System Maintenance Reboot

Diagnosis Result:
- Found event: SystemMaintenance.Reboot
- Event time: 2025-03-20 10:00:00
- Reason: Planned system maintenance

Conclusion: Instance reboot was caused by Alibaba Cloud platform maintenance, which is normal ops activity.

---

Scenario 2: Kernel Panic + vmcore Available

Diagnosis Result:
- No maintenance events found
- Cloud Assistant found: "Kernel panic - not syncing" record in dmesg
- Kdump service status: active
- Found vmcore: /var/crash/127.0.0.1-2025-03-20-10:30:00/vmcore (2.5G)
- vmcore time: 2025-03-20 10:30:00

vmcore-dmesg.txt analysis:
- Panic reason: Kernel panic - not syncing: Fatal exception in interrupt
- Crash location: RIP: 0010:nvme_queue_rq+0x1a2/0x4d0 [nvme]
- Involved module: nvme driver
- Call stack: nvme_queue_rq -> blk_mq_dispatch_rq_list -> ...

Conclusion: Instance rebooted due to NVMe driver abnormality causing kernel crash, vmcore dump file generated.
Suggestion: Check NVMe driver version, upgrade driver or kernel if needed. Use crash tool for deeper vmcore analysis.

---

Scenario 3: Kernel Panic + No vmcore

Diagnosis Result:
- No maintenance events found
- Cloud Assistant found: "Kernel panic" record in dmesg
- Kdump service status: inactive
- Found vmcore: No

Conclusion: Instance rebooted due to kernel crash, but kdump not configured or not working, unable to capture vmcore.
Suggestion: Configure kdump service to capture vmcore on next crash for root cause analysis.

---

Scenario 4: OOM with panic_on_oom Enabled

Diagnosis Result:
- No maintenance events found
- Cloud Assistant found: "Out of memory: Kill process" record in /var/log/messages
- vm.panic_on_oom: 1 (OOM triggers kernel panic)
- Kdump service status: active
- Found vmcore: Yes

Conclusion: OOM event triggered kernel panic because vm.panic_on_oom=1, causing system reboot.
Suggestions:
1. Disable panic_on_oom: sysctl -w vm.panic_on_oom=0 (add to /etc/sysctl.conf for persistence)
2. Optimize application memory usage or upgrade instance type
3. Review OOM killed processes to identify memory-hungry applications

---

Scenario 5: OOM Killer Only

Diagnosis Result:
- No maintenance events found
- Cloud Assistant found: "Out of memory: Kill process" record in /var/log/messages
- vm.panic_on_oom: 0 (OOM only kills process, no panic)
- No kernel panic records found

Conclusion: Instance triggered OOM Killer due to insufficient memory, some processes were terminated but system continued running.
Suggestion: Optimize application memory usage, or upgrade instance type.

---

Windows Scenarios

Scenario 6: BSOD with Memory Dump

Diagnosis Result:
- No maintenance events found
- Unexpected shutdown event: Event ID 41 (Kernel-Power)
- BSOD events found in WER logs
- Memory dump configuration: Automatic memory dump (CrashDumpEnabled=7)
- Memory dump file found: C:\Windows\MEMORY.DMP (4.2 GB)
- Dump time: 2025-03-20 10:30:00

Conclusion: Windows BSOD crash occurred, memory dump captured.
Suggestion: Download MEMORY.DMP and analyze with WinDbg:
1. Install Windows Debugging Tools
2. Open dump file in WinDbg
3. Run: !analyze -v

---

Scenario 7: BSOD without Dump (Not Configured)

Diagnosis Result:
- No maintenance events found
- Unexpected shutdown event: Event ID 41 (Kernel-Power)
- BSOD events found in WER logs
- Memory dump configuration: None (CrashDumpEnabled=0)
- Memory dump file: Not found

Conclusion: Windows BSOD crash occurred but memory dump was not configured.
Suggestions:
1. Enable memory dump: System Properties > Advanced > Startup and Recovery > Settings
2. Select "Automatic memory dump" or "Kernel memory dump"
3. Ensure pagefile is configured and has sufficient space

---

Scenario 8: BSOD without Dump (Pagefile Issue)

Diagnosis Result:
- No maintenance events found
- Unexpected shutdown event: Event ID 41 (Kernel-Power)
- Memory dump configuration: Automatic memory dump (CrashDumpEnabled=7)
- Pagefile: Not configured
- Memory dump file: Not found

Conclusion: Windows BSOD crash occurred but memory dump was not captured because pagefile is not configured.
Suggestions:
1. Configure pagefile: System Properties > Advanced > Performance > Settings > Advanced > Virtual memory
2. Set pagefile size to at least RAM size + 1MB
3. Reboot for pagefile changes to take effect

---

Scenario 9: Minidump Available

Diagnosis Result:
- No maintenance events found
- Unexpected shutdown event: Event ID 41 (Kernel-Power)
- Memory dump configuration: Small memory dump (CrashDumpEnabled=3)
- Minidump files found in C:\Windows\Minidump:
  - 032025-12345-01.dmp (128 KB, 2025-03-20 10:30:00)

Conclusion: Windows crash occurred, minidump captured.
Suggestion: Analyze minidump with WinDbg Preview (Microsoft Store) or BlueScreenView tool.

---

Scenario 10: Application Crash Causing Instability

Diagnosis Result:
- No maintenance events found
- No unexpected shutdown events
- Application crash events found: Multiple crashes of {application_name}.exe
- No system crash dump files

Conclusion: Application crashes detected but no system-level crash. System remained running.
Suggestions:
1. Check application logs for crash details
2. Verify application compatibility with Windows version
3. Check for application updates or known issues