Alibabacloud Elasticsearch Instance Diagnose

Alibaba Cloud Elasticsearch Instance Diagnosis

Collect signals from Alibaba Cloud OpenAPI (control plane) and the Elasticsearch REST API (data plane), combine them with the SOP knowledge base under references/, and produce root-cause analysis, an evidence chain, prioritized remediation guidance, and—when multiple dimensions fire—a recency-ordered incident timeline (severity vs time in window; see Timeline and recency (MUST) in §5 Step 4).

Architecture: Alibaba Cloud Elasticsearch OpenAPI + Alibaba CloudMonitor (CMS) + Elasticsearch REST API + diagnostic SOPs

Closure: If MUST applies and ES_* is set, finish authenticated ES API evidence before the final report (see Feasibility order in §5).

---

1. Prerequisites

1.1 Aliyun CLI

Pre-check: Aliyun CLI >= 3.3.1 required (for RAM permission checks and OpenAPI CLI fallback)

Run aliyun version to verify the version is >= 3.3.1. If the CLI is missing or too old, see references/cli-installation-guide.md.

After installation, run aliyun configure set --auto-plugin-install true to enable automatic plugin installation (do not pass plaintext AccessKey pairs on this command line; see §1.2).

1.2 Alibaba Cloud account authentication and security (MUST)

Security rules (mandatory):

- NEVER read, echo, or print AccessKey ID or AccessKey Secret values.

- NEVER prompt or ask the user to paste plaintext AccessKeys in the conversation.

- NEVER embed AccessKeys in scripts, CLI arguments, or curl URLs.

- NEVER use aliyun configure set (or similar) to pass literal AccessKey ID/Secret on the command line.

- NEVER accept AccessKeys that the user pastes into the chat, even if offered voluntarily.

- ONLY use configured CLI profiles (aliyun configure) or environment variables such as ALIBABA_CLOUD_ACCESS_KEY_ID / ALIBABA_CLOUD_ACCESS_KEY_SECRET that the user has set in their local shell (the agent must not echo those values in the session).

⚠️ If the user provides AccessKeys in the chat (e.g. “my AK is xxx”)

>

1. Stop immediately: do not run any Alibaba Cloud command that requires credentials.

2. Decline politely and give only the names of approved configuration methods (do not repeat any secret the user may have leaked):

- Recommended: run aliyun configure in a local terminal and enter credentials when prompted; credentials are stored in the local profile file.

- Alternatively: set ALIBABA_CLOUD_ACCESS_KEY_ID / ALIBABA_CLOUD_ACCESS_KEY_SECRET in the local shell (the user types values only in the terminal, not in chat).

3. Resume the diagnosis request only after credentials are configured correctly.

Verify credentials without exposing secrets:

>

```bash

aliyun configure list

aliyun --profile <profile_name> sts get-caller-identity

```

>

Credential policy:

1. Prefer an aliyun configure profile (default or --profile).

2. If there is no valid identity (configure list / get-caller-identity fails), STOP and guide the user to configure locally; do not guess or fabricate credentials.

3. Never pass plaintext AccessKeys through the conversation.

1.3 Elasticsearch direct-connect credential boundary

- NEVER ask the user to paste ES_PASSWORD in chat; NEVER echo, print, or log the password; NEVER copy a password from chat into commands, hooks, or repo files.

- Shell expansion for curl -u "$ES_USERNAME:$ES_PASSWORD" (or equivalent) is allowed when vars are pre-exported in the user’s local shell; NEVER put the secret as a literal in chat, scripts checked into repos, or command output.

- If the user tries to send a password in chat: STOP as well and ask them to set ES_PASSWORD only locally via export (see §2.2).

---

2. Environment setup

2.1 Control plane OpenAPI (via Aliyun CLI)

All control-plane and CMS data collection for this skill uses the Aliyun CLI.

[MUST] `elasticsearch` / `cms` — plugin-mode shell only (avoid legacy CLI)

Whenever the agent emits executable aliyun lines (chat, reproducibility exports, or copy-paste steps), use plugin subcommands (lowercase-hyphenated) and kebab-case flags — the same shape as scripts/openapi_cli_collect.py and references/verification-method.md.

- Do not use legacy POP-style invocations: a PascalCase verb immediately after elasticsearch or cms on the same aliyun line (the old “action name = subcommand” style), or CamelCase flags like --InstanceId, --Namespace, --StartTime in new commands. Use plugin verbs only (describe-instance, describe-metric-list, …).

- Naming split: DescribeInstance, ListSearchLog, DescribeMetricList, etc. are OpenAPI action names (PascalCase — docs, RAM, console). The token after aliyun elasticsearch or aliyun cms in a shell must be the CLI plugin name (describe-instance, list-search-log, describe-metric-list, …).

- Prefer python3 scripts/check_es_instance_health.py for the standard control-plane + CMS bundle so subprocess calls stay aligned with this repo.

- CLI references: Elasticsearch CLI 中心, 云监控 CLI 中心.

AI-Mode and plugin baseline (required) — wrap every diagnosis session that runs aliyun OpenAPI/CMS commands:

aliyun configure ai-mode enable
aliyun configure ai-mode set-user-agent --user-agent "AlibabaCloud-Agent-Skills/alibabacloud-elasticsearch-instance-diagnose"
aliyun plugin update
# … diagnosis: aliyun / python3 scripts/check_es_instance_health.py …
aliyun configure ai-mode disable

`configure ai-mode` missing or failing: Skip the wrapper above; use `ALIBABA_CLOUD_USER_AGENT` (next block). Log the CLI failure (e.g. subcommand unavailable). Whether the profile is valid is determined only by `aliyun configure list` and `sts get-caller-identity` — write valid / validity, not vaild.

User-Agent (required): set a User-Agent for Alibaba Cloud API calls:

export ALIBABA_CLOUD_USER_AGENT="AlibabaCloud-Agent-Skills/alibabacloud-elasticsearch-instance-diagnose"

CLI hardening (recommended): when authoring raw aliyun commands, use §2.1 MUST plugin shape first, then add `--connect-timeout 3 --read-timeout 10` (increase read-timeout for large responses or CMS), consistent with the instance-management skill examples, to avoid indefinite hangs on network faults. If the global User-Agent is not set, add `--user-agent AlibabaCloud-Agent-Skills/alibabacloud-elasticsearch-instance-diagnose` per invocation. For optional Elasticsearch probes inside check_es_instance_health.py (when ES_* is set), the same knobs exist as `--connect-timeout` / `--read-timeout` on that script — they map to curl for engine calls only, not to the Aliyun OpenAPI client.

Run before diagnosis:

aliyun version
aliyun configure list
aliyun --profile <profile_name> sts get-caller-identity

2.2 Elasticsearch API direct access (curl)

Have the user set connection variables in a local terminal after you confirm the Elasticsearch endpoint (VPC or public) and admin credentials—do not hardcode user-specific values in chat:

export ES_ENDPOINT="http://<elasticsearch-endpoint-ip>:9200"
export ES_USERNAME="elastic"
export ES_PASSWORD="<elasticsearch-admin-password>"

Public access and `http` vs `https`: From `DescribeInstance`, use `publicDomain` / `domain` and the reported `protocol`. When `protocol` is `HTTP` (typical public listener), set `ES_ENDPOINT` to `http://<publicDomain>:9200`. Using `https://` against an HTTP-only endpoint causes TLS errors (e.g. `WRONG_VERSION_NUMBER`). Use `https://` only when `protocol` is `HTTPS` (or TLS is actually enabled on the port you use), and supply CA / fingerprint options as in HTTPS options below.

>

If `http://` “does not work” — when to try `https://`: Treat `DescribeInstance` `protocol` as the source of truth for the REST listener. `000`, timeouts, or connection refused on `http://` usually mean network path / allowlist / security group / wrong host or port — not “try HTTPS next” when `protocol` is still `HTTP`. Do switch to `https://` when `protocol` is `HTTPS` (or the console / product doc states TLS on that endpoint) and the failure on http:// is a TLS or scheme symptom (e.g. `WRONG_VERSION_NUMBER`, `error:0A00010B`, immediate SSL alert while probing with the wrong scheme). If `protocol` is `HTTP` and only plain TCP is advertised, HTTPS is not a fallback for reachability.

Credential safety

- NEVER echo, print, or log ES_PASSWORD; NEVER copy credentials from chat into shell history or saved files.

- NEVER ask the user to paste the password in plaintext in chat.

- ONLY use the following checks to verify that variables are set:

```bash

[[ -n "$ES_ENDPOINT" ]] && echo "ES_ENDPOINT: $ES_ENDPOINT" || echo "ES_ENDPOINT: NOT SET"

[[ -n "$ES_PASSWORD" ]] && echo "ES_PASSWORD: SET" || echo "ES_PASSWORD: NOT SET"

```

Network connectivity and access control

>

| Issue | How to check | Mitigation |

|------|--------------|------------|

| Public network access disabled | Elasticsearch console → Network | Enable public access or use the VPC endpoint |

| Public access allowlist | Console → Security → Public access allowlist | Add the agent host’s public IP |

| VPC isolation | e.g. telnet <ES_IP> 9200 | VPC peering, Express Connect, or equivalent |

| Security group | Inbound rules on the ECS/security group hosting Elasticsearch | Allow TCP 9200 (or the configured port) |

Connectivity probe: curl -sS -o /dev/null -w "%{http_code}" --connect-timeout 5 "${ES_ENDPOINT}" — HTTP code 000 usually means the path is unreachable. `401` without `-u` is normal (auth required); if ES_PASSWORD is SET, proceed to authenticated GET /_cluster/health (§7). `401` with `-u` → wrong credentials. `000` / refused / timeout → network, allowlist, or TLS/scheme mismatch.

HTTPS — prerequisites (what must be true)

1. Listener: The Elasticsearch HTTP port you call (9200 unless changed) must actually speak TLS — align with `DescribeInstance` `protocol` (`HTTPS`) or console/network documentation.

2. URL: `https://<host>:<port>` with the same host (e.g. `publicDomain`) you would use for HTTP.

3. Client trust of the server certificate: Your client must trust the cluster’s certificate chain (cluster / cloud CA PEM, or corporate proxy CA if TLS is intercepted). `curl`: prefer `curl --cacert /path/to/ca.crt ...`; `-k` / `--insecure` only for short, non-production diagnosis.

4. Auth: Same `ES_USERNAME` / `ES_PASSWORD` as for HTTP (Basic auth over TLS).

>

HTTPS — how this skill documents it

- Manual `curl` (§7 and [es-api-call-failures.md](references/es-api-call-failures.md)): Add `--cacert` (or `-k` for testing) to every `curl` when using `https://` if the default trust store does not include your cluster CA.

- `check_es_instance_health.py` optional ES probes: They invoke `curl` with `-u` only; they do not read `ES_CA_CERTS` / `ES_SSL_FINGERPRINT` / `ES_VERIFY_CERTS` (those names are common for Python Elasticsearch clients). For HTTPS instances, use §7 `curl` with `--cacert` for deep checks, or extend the script later to pass `--cacert` from an env var.

- Python-style env vars (reference for other tooling): ES_CA_CERTS, ES_SSL_FINGERPRINT, ES_VERIFY_CERTS=false (testing only) — not wired into this repo’s optional `curl` path today.

---

3. RAM permission check

[MUST] RAM permission pre-check

>

Before running this skill, verify the principal has the required RAM permissions.

See references/ram-policies.md for the full list.

If the user reports insufficient permissions, direct them to attach the corresponding policies in the RAM console.

---

4. Parameter confirmation

IMPORTANT: Parameter confirmation

Confirm the following with the user before any command or API call.

Do not assume undeclared defaults or hardcode user-specific parameters.

Boundary controls (MUST)

- Region and `instance-id` must not be guessed or taken from unverified defaults; if they disagree with DescribeInstance or the user’s explicit statement, reconfirm.

- Do not apply metrics, logs, or DescribeInstance conclusions from instance A to instance B; ES_ENDPOINT must match the instance under diagnosis (see Pre-flight validation for Elasticsearch API below).

- This skill is read-only diagnosis: do not invoke mutating control-plane APIs (create, resize, restart, delete instance, etc.). If the user requests a change, provide recommendations only; execution belongs in the console or an approved change workflow.

---

5. End-to-end diagnostic workflow

Agent hard rules (non-negotiable)

Aliyun CLI shape: For `aliyun elasticsearch` and `aliyun cms`, follow §2.1 MUST (plugin mode only) in every new executable command — do not resurrect legacy DescribeInstance / ListSearchLog-as-subcommand lines or --InstanceId-style flags in session exports or user-facing step lists (they drift from openapi_cli_collect.py and fail static checks).

OpenAPI/CMS cannot replace MUST engine APIs. For any §5 MUST table row or `check_es_instance_health.py` rule-engine MUST, Alibaba Cloud OpenAPI and CloudMonitor do not replace the listed Elasticsearch REST calls for engine-level root cause—when feasibility holds, run those curl endpoints (see §7); they are complementary layers, not interchangeable.

>

Feasibility is decided only by checks, not by assumption. Whether the agent may call Elasticsearch must be determined by actually running the Feasibility order (§5): at minimum verify ES_ENDPOINT / ES_PASSWORD per §2.2, align ES_ENDPOINT with DescribeInstance, then authenticated GET /_cluster/health. Do not assume ES_* is unset or the path is unreachable without performing these steps in the session.

For Elasticsearch incidents, follow these four steps; each has a distinct role.

Execution strategy (root-cause driven)

Full policy: es-api-diagnosis-strategy.md

Data-plane curl collection requires both:

1. Feasibility: ES_ENDPOINT and ES_PASSWORD are set and the network path works. 2. Necessity: root-cause analysis needs data-plane evidence that the control plane or CMS cannot establish alone.

For endpoints listed under a fired MUST table row or rule-engine MUST, necessity for those calls is already satisfied by the trigger—still require feasibility (Feasibility order). For optional engine curl not in those lists, apply feasibility and necessity per es-api-diagnosis-strategy.md.

MUST triggers (if any CMS condition below holds, collect the listed Elasticsearch evidence):

Thread-pool row: interpret search vs write / bulk using sop-query-thread-pool.md vs sop-write-performance.md (see also Write-path / bulk saturation below).

Rule-engine MUST: If check_es_instance_health.py prints a §5 MUST / §5–§7 callout for this run, treat it like a row above—collect that listed ES evidence when feasibility holds.

Binding rule (MUST triggers): If any MUST-trigger row or the rule-engine MUST line above applies, necessity is satisfied for that evidence set—OpenAPI/CMS cannot replace those calls for engine-level root cause (cluster-health: allocation/explain + _cat/shards for Yellow/Red). Confirm feasibility per Feasibility order below. If reachable with auth, run the MUST-listed endpoints in Step 2 in parallel with control-plane collection. If still blocked after authenticated GET /_cluster/health, lead with blocking reason: unset ES_*; transport failure (000, refused, timeout); 401 with `-u`; scheme/TLS mismatch—not 401 on an unauthenticated probe when ES_PASSWORD is SET.

Write-path / bulk saturation

If `ThreadPool.WriteRejected` or `write` pool stress matches high-QPS bulk indexing, read and follow `references/sop-write-performance.md` — §2, subsection “Evidence interpretation: bulk QPS → write pool” for the evidence chain, `rejected` semantics (cumulative since node start), report ordering vs Old GC / heap (causal chain or dual P0 — write path before JVM-only headline), per-node `rejected`/`completed` numbers (reject share), per-node asymmetry, and write-only vs search. Do not lead with a JVM-only narrative when that subsection applies. For write-queue–style acceptance prompts, the opening conclusion should read as write-capacity (data-plane counters + optional CMS rule names), not only a GC/heap headline.

Search-primary vs write (both pools show cumulative rejected)

When `_nodes/stats/thread_pool` shows `search.rejected` ≫ `write.rejected` on the same node(s) and `ThreadPool.SearchRejected` / query-driven overload applies, lead the executive summary and P0 ordering with `search` (high concurrent query / terms / slow query; hot index when verified) — not `write` first. `write.rejected` may remain P0/P1 as parallel or secondary (bulk, catch-up); Old GC / CPU / node disconnect stay co-stress or cascade. Checker listing order is not proof of narrative order — see acceptance-criteria.md §6.5 and sop-query-thread-pool.md Report narrative.

>

Recency overrides this magnitude default when time-resolved evidence exists: do not rank the opening story by `search.rejected` vs `write.rejected` alone — cumulative counters lack timestamps. Full rubric: acceptance-criteria.md §6.5 (P0 / executive order vs `search` ≫ `write`: unless write dominated by time) and §6.6 (Executive order, No false recency from counters). Binding: Timeline and recency (MUST) below (same skill).

activating / change workflow stuck (cross-layer root cause)

When an instance stays in `activating`, a change is unfinished, and Red or unassigned shards coexist, follow `references/sop-activating-change-stuck.md` end-to-end (MUST includes ListActionRecords, DescribeInstance before/after remediation, collection order section 3.1, reporting section 4).

Pre-flight validation for Elasticsearch API

[IMPORTANT] `ES_ENDPOINT` must match the diagnosed instance

>

Compare publicDomain / domain and `protocol` from DescribeInstance with ES_ENDPOINT.

If they differ, warn: ⚠️ ES_ENDPOINT does not match the current instance; run export ES_ENDPOINT="http://{publicDomain}:9200" when `protocol` is `HTTP`, or https://… only when `protocol` is `HTTPS` (adjust host/port to match the deployment).

When Elasticsearch credentials are missing or connections fail

[CRITICAL] Guide the user to fix connectivity explicitly; classify failure modes (do not default persistent timeouts to “allowlist only”). Do not imply the agent “forgot” Elasticsearch — if the first answer is CMS/OpenAPI-heavy, give the blocking reason per Feasibility order below: unset ES_*; transport errors; 401 with valid `-u`; TLS/scheme—not 401 on a probe without -u when ES_PASSWORD is SET (use authenticated curl first).

Progressive playbook (read in order): references/es-api-call-failures.md (sections 1 → 4). MUST / strategy context: references/es-api-diagnosis-strategy.md (sections 1–3 and 3.5 summary table).

Mandatory warning when MUST applies but Elasticsearch is not configured

[CRITICAL] If a MUST trigger fires but data-plane evidence is missing, put a warning at the top of the report: follow section 4 of references/es-api-call-failures.md (blocking reason first, then MUST list, missing evidence; if ES_* unset, pointer to section 2.2 of this SKILL; if vars are set, use es-api-call-failures sections 1–2 for auth vs transport).

Step 1: Quick health scan (initial signals)

Run the lightweight rules engine (17 metric rules) to list P0 / P1 / P2 findings and steer deeper collection:

python3 scripts/check_es_instance_health.py -i <InstanceId> -r <RegionId> [--window <minutes, default 60>] [--profile <profile_name>]

Feasibility order (agent)

1. Run §2.2 ES_* checks (password = SET only)—do not skip; never infer feasibility without this step. 2. ES_ENDPOINT matches DescribeInstance domain / publicDomain (scheme/port). 3. Authenticated GET /_cluster/health—do not stop at 401 on an unauthenticated probe if ES_PASSWORD is SET. 4. MUST scope: table rows and/or rule-engine MUST line in §5.

Step 2: Collect evidence in parallel

Based on Step 1, run collection in parallel (prioritize dimensions with signals). If a MUST-trigger row or rule-engine MUST applies: run Feasibility order, then run that Required Elasticsearch evidence via curl in the same round (see §7). If no MUST applies, add optional data-plane curl only when feasibility and necessity both hold per the strategy doc.

Re-run `check_es_instance_health.py` with the same invocation pattern as Step 1; for this parallel round, `--window 120` and explicit `--profile <profile_name>` are common.

To backfill control-plane evidence (DescribeInstance, ListSearchLog, CMS-style calls), use `aliyun` patterns in references/verification-method.md (epoch times, profiles, namespaces).

Note: data-plane access still requires ES_ENDPOINT / ES_PASSWORD; the Aliyun CLI cannot replace curl to the cluster.

>

For MUST-trigger rows, necessity for the listed endpoints is already established—do not skip them when feasibility including reachability holds. Outside those rows, avoid unrelated bulk curl solely because ES_* is set; use the strategy doc’s feasibility + necessity test instead.

Step 3: Read SOPs by signal

Map signals to SOPs and read for deeper reasoning. With multiple signals, process P0 → P1 → P2 for severity, then apply Timeline and recency (MUST) in Step 4 so the narrative order matches when signals mattered in the window—not only static rule-engine print order.

Step 4: Synthesize and write the structured report

Acceptance-style optional checklists: references/acceptance-criteria.md §6.1–§6.6 — Red/Yellow; read-heavy CPU + search pool (+ CMS alignment); JVM / breakers / fielddata; write-queue vs GC + `rejected`/`completed`; read-heavy search pool vs GC-only headline (expand in sop-query-thread-pool.md Report narrative: search pool vs GC / CPU headlines); timeline/recency. Bulk/write: references/sop-write-performance.md §2. Shard `reroute`: references/sop-node-load-imbalance.md §1.3 (allocator / change control only).

[CRITICAL] Remediation must match the diagnosed root cause — avoid generic templates. Wrong breaker or concurrency fixes (e.g. in_flight_requests vs request, “split query” when concurrency is the issue) → see `sop-memory-gc.md` and the fired signal’s SOP.

`activating` + data-plane anomaly: include the one-line cross-layer root cause; see references/sop-activating-change-stuck.md section 4.

Report skeleton (copy/fill): references/report-template.md.

Timeline and recency (MUST for synthesized reports)

Problem: check_es_instance_health.py and P0/P1/P2 bands express severity, not when a signal mattered most within the analysis window. Cumulative engine counters (search.rejected, write.rejected) do not encode recency—write and search issues can both be “real” while only one path dominated the recent past (e.g. search pressure closer to window end than write pressure).

Binding rules for the agent:

1. Two axes — Treat severity (P0/P1/P2) and temporal relevance (proximity to window end / “now”) as orthogonal. Do not infer recency from priority alone (e.g. “write is P0 so it must be the current headline”) when time-resolved evidence says otherwise. 2. Mandatory human-facing section — When more than one major finding fires (e.g. write pool + search pool + GC/CPU), the synthesized report must include an `### Incident timeline (recency-ordered)` (or equivalent) block before or immediately after the executive summary, unless the user explicitly asks for a minimal report. In that block:

• Order bullets or rows by time (earlier → later), or state which signal cluster peaked / persisted in the latter portion of `{begin} ~ {end}`.
• Call out divergence: e.g. “write-path stress earlier in window; search-path / CPU more recent” when CMS or logs support it.

3. Evidence for recency (use what exists; do not invent timestamps):

• CloudMonitor: per-metric time series — note peak timestamp or sustained-high interval for NodeCPUUtilization, NodeHeapMemoryUtilization, GC-related metrics, ThreadPool.* if exposed as rates or non-cumulative series in the collected JSON.
• Slow logs / `ListSearchLog`: correlate query vs index slow entries to minutes.
• Engine (optional): two _nodes/stats/thread_pool samples at known times to show delta on rejected / completed; or `_tasks` / `hot_threads` for current skew vs historical cumulative counters.

4. Executive summary ordering — The opening 2–4 sentences should reflect recency-weighted user impact: if search pressure is closer to current than write pressure, lead with search/query concurrency and co-stress (GC/CPU) as appropriate, and place historical write saturation as context or second wave—without dropping P0 write findings if they remain valid for remediation backlog. 5. Explicit uncertainty — If only cumulative counters exist and no time series differentiates paths, state one line: recency is undifferentiated; recommend narrower window, slow logs, or delta sampling for the next run.

---

6. Data collection details (CLI OpenAPI + injected input)

One-shot entry

Use the same `check_es_instance_health.py` command as §5 Step 1 (optional `--window` / `--profile`; default window 60 minutes if omitted).

Injected input mode (paired with CLI)

check_es_instance_health.py accepts external JSON to avoid duplicate calls:

python3 scripts/check_es_instance_health.py \
  -i <InstanceId> -r <RegionId> \
  --data-source input \
  --input-json-file /path/to/diag-input.json

Input JSON shape:

{
  "status_info": {},
  "metrics": {},
  "events": [],
  "logs": []
}

--data-source modes:

• auto: prefer injected fields; backfill gaps via Aliyun CLI.
• cli: ignore injection; fetch everything via CLI.
• input: injection only; no OpenAPI calls.

Manual control-plane CLI backfill

For additional OpenAPI examples, see references/verification-method.md.

---

7. Elasticsearch direct API access (data-plane deep dive)

When feasibility holds (including reachability), execute the REST calls required by any MUST-trigger row (§5). For endpoints not listed in a fired MUST row, call them only when feasibility and necessity both hold per the strategy doc.

ES_ENDPOINT may be host:port or a full URL. For the samples below, normalize to http://${ES_ENDPOINT#http://} (use https:// consistently when the cluster serves TLS).

>

Timeouts: every curl must use --connect-timeout 10 --max-time 30.

Red / Yellow (MUST) — recommended set

Scope: The cluster-health MUST row uses ClusterStatus max ≥ Yellow (includes Red). Use this set for unassigned / misallocated shard root cause on the engine.

curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \
  "http://${ES_ENDPOINT#http://}/_cluster/health?pretty"

curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \
  -H "Content-Type: application/json" \
  -X POST "http://${ES_ENDPOINT#http://}/_cluster/allocation/explain?pretty" \
  -d '{}'

curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \
  "http://${ES_ENDPOINT#http://}/_cat/shards?v&h=index,shard,prirep,state,node,unassigned.reason&s=state"

curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \
  "http://${ES_ENDPOINT#http://}/_cluster/pending_tasks?pretty"

curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \
  "http://${ES_ENDPOINT#http://}/_nodes/stats/thread_pool?pretty"

Query / write performance (MUST) — recommended set

Include `_cluster/settings` when heap / GC / breaker rules fired in Step 1 or `_nodes/stats/breaker` shows concern — read transient and persistent indices.breaker.* / network.breaker.*.

curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \
  "http://${ES_ENDPOINT#http://}/_nodes/hot_threads?threads=3"

curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \
  "http://${ES_ENDPOINT#http://}/_nodes/stats/breaker?pretty"

curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \
  "http://${ES_ENDPOINT#http://}/_cluster/settings?include_defaults=true&pretty"

`/_cluster/pending_tasks` and `GET /_nodes/stats/thread_pool` are also listed under Red / Yellow (MUST) above—one call each per session when both sections apply. If you run only this performance block, add those two curl lines from that block.

Resource anomalies without a closed loop (SHOULD) — recommended set

curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \
  "http://${ES_ENDPOINT#http://}/_cat/nodes?v&s=cpu:desc&h=name,ip,cpu,heap.percent,ram.percent,load_1m"

curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \
  "http://${ES_ENDPOINT#http://}/_nodes/stats/jvm?pretty"

curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \
  "http://${ES_ENDPOINT#http://}/_cat/allocation?v&bytes=gb"

`GET /_cluster/settings?include_defaults=true` also appears under Query / write performance (MUST) above—reuse one response when both blocks apply. If you run only this SHOULD block, add the same curl line from that block.

Protocol sanity (avoid WRONG_VERSION_NUMBER): usually http/https scheme mismatch on ES_ENDPOINT — fix scheme/port and retry.

Scenario → endpoint index: references/es-api-catalog.md.

---

8. Diagnostic coverage

The knowledge base covers 48+ health-event-style rules and chained scenarios (e.g. disk pressure → allocation → Red). Per-category counts, P0/P1/P2 mix, and event codes: references/health-events-catalog.md — scenario runbooks: references/sop-*.md (index: references/README.md).

---

9. Best practices

Read-only: no mutating control-plane APIs; no teardown.

1. Layered + evidence-bound: scan → SOP depth; every conclusion cites metrics/logs/events; if ES is unreachable, state limits (es-api-call-failures.md). 2. Priority vs narrative: P0→P2 for urgency; Incident timeline when multiple dimensions differ in time (Step 4). Credentials / TLS / parameters: §1–2 and §4. 3. Green is not “all clear” — watermarks, blocks, mis-set limits still matter; MUST + reachable ES: do not skip §5/§7 evidence because the cluster is Green or OpenAPI “explains” symptoms. 4. Thread-pool `rejected`: cumulative unless you show a delta — sop-query-thread-pool.md §1–2; write/bulk: sop-write-performance.md §2.

---

10. Reference links

• references/verification-method.md — Verification (how to validate diagnosis; metrics, APIs, workflows)
• references/report-template.md — Structured diagnosis report skeleton
• references/README.md — Language map (reference assets and sop-*.md runbooks; English in this repo)
• references/ram-policies.md — RAM policy list
• references/acceptance-criteria.md — Correct/incorrect patterns and acceptance (includes credential and safety anti-patterns)
• references/cli-installation-guide.md — Aliyun CLI installation
• references/es-api-catalog.md — Elasticsearch REST API catalog
• references/health-events-catalog.md — Health event catalog
• references/sop-*.md — Scenario SOPs (e.g. sop-activating-change-stuck.md for activating / change stuck, cross-layer root cause)
• references/es-api-diagnosis-strategy.md — Elasticsearch API diagnosis strategy

Acceptance criteria: alibabacloud-elasticsearch-instance-diagnosis

Scenario: Elasticsearch instance diagnosis Purpose: Skill test / acceptance checklist

---

Version notes (2026-03)

- Control-plane OpenAPI collection uses the aliyun CLI only.

- Health diagnosis entry point: python3 scripts/check_es_instance_health.py ....

- Engine-level deep collection uses curl against ES REST APIs (no invoke_es_api.py).

1. Environment dependencies

1.1 CLI dependency

✅ CORRECT

aliyun version

❌ INCORRECT

# Error: aliyun CLI not available
aliyun: command not found

---

2. Credential configuration

2.1 OpenAPI credentials (CLI profile)

✅ CORRECT

aliyun configure list
aliyun --profile <profile_name> sts get-caller-identity

❌ INCORRECT

# Error: profile does not exist
aliyun --profile not-exist sts get-caller-identity

2.2 Direct ES credentials

✅ CORRECT

[[ -n "$ES_ENDPOINT" ]] && echo "ES_ENDPOINT: SET" || echo "ES_ENDPOINT: NOT SET"
[[ -n "$ES_PASSWORD" ]] && echo "ES_PASSWORD: SET" || echo "ES_PASSWORD: NOT SET"

❌ INCORRECT

# Error: ES password not configured
unset ES_PASSWORD

---

3. Running the diagnosis

3.1 Main health-check entry

✅ CORRECT

python3 scripts/check_es_instance_health.py \
  -i es-cn-xxxxx -r cn-hangzhou \
  --data-source cli \
  --profile <profile_name>

Expected output:

• Structured report (P0/P1/P2, evidence, remediation)
• When multiple major dimensions fire, an incident timeline (recency-ordered) section aligning narrative with when signals peaked in the window (see §6.6)
• Summary of key monitoring metrics
• No dependency on deprecated scripts

3.2 Injected-input + auto backfill

✅ CORRECT

python3 scripts/check_es_instance_health.py \
  -i es-cn-xxxxx -r cn-hangzhou \
  --data-source auto \
  --input-json-file /path/to/diag-input.json \
  --profile <profile_name>

Expected output:

• Input JSON fields take precedence when present
• Missing fields are backfilled via CLI OpenAPI

---

4. OpenAPI coverage

4.1 Elasticsearch OpenAPI

✅ CORRECT

aliyun --profile <profile_name> elasticsearch describe-instance \
  --region cn-hangzhou \
  --instance-id es-cn-xxxxx

aliyun --profile <profile_name> elasticsearch list-search-log \
  --region cn-hangzhou \
  --instance-id es-cn-xxxxx \
  --type INSTANCELOG \
  --query "*" \
  --begin-time <epoch_ms> \
  --end-time <epoch_ms>

aliyun --profile <profile_name> elasticsearch list-action-records \
  --region cn-hangzhou \
  --instance-id es-cn-xxxxx

aliyun --profile <profile_name> elasticsearch list-all-node \
  --region cn-hangzhou \
  --instance-id es-cn-xxxxx

❌ INCORRECT (legacy CLI — do not use for this skill)

Do not copy old tutorials that put a PascalCase OpenAPI action name as the aliyun <product> <verb> verb, or use CamelCase query flags. The ✅ CORRECT block above is the only canonical shell shape for this skill.

4.2 CMS OpenAPI

✅ CORRECT

aliyun --profile <profile_name> cms describe-metric-list \
  --region cn-hangzhou \
  --namespace acs_elasticsearch \
  --metric-name ClusterStatus \
  --dimensions '[{"clusterId":"es-cn-xxxxx"}]' \
  --start-time <epoch_ms> \
  --end-time <epoch_ms> \
  --period 300

aliyun --profile <profile_name> cms describe-system-event-attribute \
  --region cn-hangzhou \
  --product elasticsearch \
  --search-keywords es-cn-xxxxx \
  --start-time <epoch_ms> \
  --end-time <epoch_ms>

aliyun --profile <profile_name> cms describe-metric-meta-list \
  --region cn-hangzhou \
  --namespace acs_elasticsearch

Expected output:

• Success shape (Code=200 or Success=true)
• Key fields present (instance status, logs, metrics, events, nodes, change records)

---

5. Engine-level ES API checks (curl)

If calls fail or return 401 / timeouts, classify the failure using [es-api-call-failures.md](es-api-call-failures.md) before judging PASS / PARTIAL.

5.1 Calls

✅ CORRECT

curl -sS -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \
  "http://${ES_ENDPOINT#http://}/_cluster/health?pretty"

curl -sS -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" -H "Content-Type: application/json" \
  -X POST "http://${ES_ENDPOINT#http://}/_cluster/allocation/explain?pretty" -d '{}'

curl -sS -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \
  "http://${ES_ENDPOINT#http://}/_nodes/stats/thread_pool?pretty"

❌ INCORRECT

# Error: wrong URL path (missing leading /)
curl -sS -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \
  "http://${ES_ENDPOINT#http://}_cluster/allocation/explain"

---

6. Report format

✅ CORRECT report shape

## Diagnosis summary

**Instance**: es-cn-xxxxx (cn-hangzhou)
**Window**: 2026-03-24 10:00 ~ 2026-03-24 12:00

### Incident timeline (recency-ordered)

- Optional in minimal reports; **required** when search vs write vs GC show **different** emphasis over time (SKILL Step 4).

### Findings (by priority)

#### P0 - Critical (immediate action)
- [HealthCheck.ClusterUnhealthy] Cluster status Red
  - Evidence: ClusterStatus=2, UNASSIGNED primary shards
  - Likely cause: shard allocation failure
  - Immediate action: run `POST /_cluster/allocation/explain`

6.1 Cluster Red / Yellow — optional “full checklist” (acceptance-style)

Use when you want the report to align with structured acceptance rubrics (control plane vs data plane already correct). These are additive, not a substitute for allocation/explain + sop-cluster-health.md.

6.2 Read-heavy CPU + search pool (optional checklist)

Optional lines when query-driven overload is present (e.g. _tasks shows *search* on a hot index with match_all-style bodies):

6.3 JVM / fielddata / circuit breakers (optional checklist)

Use when heap, GC, breaker, or fielddata signals appear — cluster may still be Green.

6.4 Write-path saturation + CPU + JVM (optional checklist)

Use when ingest / bulk, `ThreadPool.WriteRejected`, CPU peaks, and/or Old GC / heap signals co-occur — for example sustained high-QPS bulk indexing that stresses the write pool.

Write-queue / bulk prompts (rubric alignment): Graders often treat `ThreadPool.WriteRejected` + `_nodes/stats/thread_pool` as the primary storyline. A conclusion that only headlines Old GC / heap while relegating write-pool saturation to “P1 / participating factor” can read as a JVM/GC report, not a write-capacity diagnosis. Prefer one of:

• Dual P0 (parallel): Opening treats quantified write-pool stress (`write` `rejected` and `completed`, or reject share per node or cluster) and Old GC wall-clock / `GCTimeRatioTooHigh` as co-equal P0-class signals (see health-events-catalog.md `ThreadPool.WriteRejected` / `HealthCheck.ThreadPoolSaturation`).
• Causal chain (upstream first): First sentence states bulk / ingest saturates the `write` pool → merges / heap → Old GC + CPU — so `write.rejected` is in the lead, not buried below a GC headline.

Add at least one line of numbers: e.g. `rejected` / `completed` on the busiest node(s), optional reject share (approximately rejected / (rejected + completed)) for the same pool, and rule / event names when citing CMS (e.g. `ThreadPool.WriteRejected`).

Align structured reports with [health-events-catalog.md](health-events-catalog.md): CMS P0 for write rejects usually requires sustained reject rate + traffic, not cumulative counters alone — human-written reports should still quote cumulative `rejected`/`completed` with context so the data-plane story is concrete.

Engine snapshot (`check_es_instance_health.py`): `rejected` > 0 is emitted as P1 when `JVMMemory.GCTimeRatioTooHigh` is not in the same finding set. When `GCTimeRatioTooHigh` (P0) does fire, the script promotes `ThreadPool.WriteRejected` from P1 → P0 so severity bands match a dual-P0 or causal-chain narrative — see table below. Co-occurrence is not proof of causation; use `hot_threads` / `_tasks` to confirm an ingest-heavy path before asserting write overload → GC as the only story.

6.5 Read-heavy scenarios (search pool vs GC-only headline)

Use when the incident is query-driven (high concurrent search, terms / QPS patterns, “search reject → cascade” timelines). Section 6.4 already guards write-path vs GC-only headlines; section 6.2 covers search pool + CPU alignment — this subsection fixes the remaining gap: Old GC / CPU as the only lead when `search` pool saturation is the primary plausible storyline. Long-form narrative, Chinese templates, and evidence closure: sop-query-thread-pool.md Report narrative: search pool vs GC / CPU headlines (keep this section as a short rubric; avoid duplicating those paragraphs here).

Search-pool-primary vs write (both pools have cumulative `rejected`): For query-saturation cases where the catalog targets `ThreadPool.SearchRejected` and `_nodes/stats/thread_pool` shows `search.rejected` ≫ `write.rejected` on the same data nodes, the executive lead and first P0 bullets must not read like a write-primary report with search as a footnote — put `search` pool / query concurrency first (or before write in the same band). `write.rejected` may remain P0/P1 as parallel or secondary (bulk, catch-up, historical ingest); say so explicitly so readers do not infer “mainly an ingest/write incident.” Tie the search lead to high concurrent query / terms / slow query and hot index name when verified.

6.6 Timeline vs severity (recency-ordered narrative)

Use when multiple of: `ThreadPool.WriteRejected`, `ThreadPool.SearchRejected`, GC / heap / CPU appear together. Severity bands (P0/P1) are not a substitute for when each path stressed the cluster inside {begin} ~ {end}.

---

7. Common mistakes

Aliyun CLI Installation & Configuration Guide

Complete guide for installing and configuring Aliyun CLI.

Aliyun CLI 3.3.1+: Supports installing and using all published Alibaba Cloud product plugins. Make sure to upgrade to 3.3.1 or later for full plugin ecosystem coverage.

Installation

macOS

Using Homebrew (Recommended)

brew install aliyun-cli
# Upgrade to latest
brew upgrade aliyun-cli

# Verify version (>= 3.3.1)
aliyun version

Using Binary

# Download
wget https://aliyuncli.alicdn.com/aliyun-cli-macosx-latest-amd64.tgz

# Extract
tar -xzf aliyun-cli-macosx-latest-amd64.tgz

# Move to PATH
sudo mv aliyun /usr/local/bin/

# Verify
aliyun version

Linux

Debian/Ubuntu

# Download
wget https://aliyuncli.alicdn.com/aliyun-cli-linux-latest-amd64.tgz

# Extract and install
tar -xzf aliyun-cli-linux-latest-amd64.tgz
sudo mv aliyun /usr/local/bin/

# Verify
aliyun version

ARM64 Architecture

# Download ARM64 version
wget https://aliyuncli.alicdn.com/aliyun-cli-linux-latest-arm64.tgz

# Extract and install
tar -xzf aliyun-cli-linux-latest-arm64.tgz
sudo mv aliyun /usr/local/bin/

Windows

1. Download from: https://aliyuncli.alicdn.com/aliyun-cli-windows-latest-amd64.zip 2. Extract the ZIP file 3. Add the directory to your PATH environment variable 4. Verify: aliyun version

Configuration

Quick Start

aliyun configure set \
  --mode AK \
  --access-key-id <your-access-key-id> \
  --access-key-secret <your-access-key-secret> \
  --region cn-hangzhou

Environment Variables

export ALIBABA_CLOUD_ACCESS_KEY_ID=your_access_key_id
export ALIBABA_CLOUD_ACCESS_KEY_SECRET=your_access_key_secret
export ALIBABA_CLOUD_REGION_ID=cn-hangzhou

Verification

# Check configuration
aliyun configure list

# Test authentication
aliyun ecs describe-regions

References

• Official Documentation: https://help.aliyun.com/zh/cli/
• Access Key Management: https://ram.console.aliyun.com/manage/ak

Elasticsearch REST API call failures (progressive guide)

Use this document in order when curl to the data plane fails or returns no useful JSON. For when engine APIs are mandatory vs optional, see es-api-diagnosis-strategy.md (sections 1–3: feasibility, necessity, MUST triggers).

Security: never print ES_PASSWORD or ask for it in chat — only SKILL.md section 2.2 patterns (local export in the same shell the agent uses).

---

0. Mandatory order (agents and operators)

Do not infer “this environment has no Elasticsearch credentials” from a lone unauthenticated curl (for example a probe that omits -u), even if the response is 401. 401 without `Authorization` is expected on secured clusters.

Always run §1.1 first in the same shell that will execute the diagnosis curl chain (for Cursor: the integrated terminal session the agent uses). Then:

This order matches actual runs: many failures are “agent skipped §1.1” or “wrong http/https,” not missing cloud APIs.

---

1. Preconditions — why the first answer might skip ES JSON

Engine curl needs both:

If Executable fails, the agent is not “forgetting” Elasticsearch: calls either were not possible (no env), were sent without auth and got 401, used the wrong auth or URL scheme, or hit transport errors — so _cluster/health, /_cluster/allocation/explain, etc. do not yield business JSON.

1.1 Default habit (same shell as the agent)

Run presence only (never echo secrets):

[[ -n "$ES_ENDPOINT" ]] && echo "ES_ENDPOINT: SET" || echo "ES_ENDPOINT: NOT SET"
[[ -n "$ES_PASSWORD" ]] && echo "ES_PASSWORD: SET" || echo "ES_PASSWORD: NOT SET"
[[ -n "$ES_USERNAME" ]] && echo "ES_USERNAME: SET" || echo "ES_USERNAME: NOT SET"

Interpretation:

• `ES_ENDPOINT` + `ES_PASSWORD` both SET → proceed to §1.3 immediately (minimal authenticated checks). Optional: print only the host/scheme for sanity (echo "$ES_ENDPOINT" is OK if it contains no secrets; do not echo password).
• Either NOT SET → state clearly: this shell has no engine credentials, not “Elasticsearch is unreachable.” Point to SKILL.md section 2.2; user export in the integrated terminal (same shell), then you re-run §1.1 then §1.3 — do not treat an earlier unauthenticated 401 as proof that credentials are absent.

1.2 HTTP 401 Unauthorized (after you know whether auth was sent)

Classify 401:

Principle: “We saw 401” is not actionable until the table above is satisfied.

1.3 Minimal authenticated checks (run as soon as §1.1 passes)

Use timeouts on every call (see SKILL.md §7). Normalize base URL once:

• If ES_ENDPOINT is host:port with no scheme, prepend `http://` or `https://` consistently with the cluster (Alibaba Cloud DescribeInstance reports protocol for the ES endpoint).
• Strip trailing / before appending paths.

Example pattern (do not log password):

BASE="${ES_ENDPOINT}"
case "$BASE" in
  http://*|https://*) ;;
  *) BASE="http://${BASE}" ;;
esac
BASE="${BASE%/}"
curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" "${BASE}/_cluster/health?pretty"

For `https://`, add `--cacert /path/to/ca.pem` (or `-k` for short, non-production tests) if the system trust store does not trust the cluster certificate. Do not switch from `http://` to `https://` as a blind fix for `000`/timeout when `DescribeInstance` `protocol` is `HTTP` — prefer allowlist / SG / path checks; see SKILL.md §2.2 (If `http://` does not work — when to try `https://` and HTTPS — prerequisites).

When MUST triggers require allocation evidence and the cluster is Red, prefer targeted explain after a quick shard listing — see §1.4 (empty explain body can pick a replica while the business-critical failure is an unassigned primary).

1.4 POST /_cluster/allocation/explain — practical pitfall

POST /_cluster/allocation/explain with an empty body {} returns one explanatory shard, which may be a replica ("primary" : false). For Red clusters, the P0 signal is usually an unassigned primary.

Runbook:

1. GET _cat/shards?v&h=index,shard,prirep,state,node,unassigned.reason&s=state — find UNASSIGNED rows with `p` (primary). 2. POST /_cluster/allocation/explain with body, for example:

{
  "index": "<index_from_cat>",
  "shard": 0,
  "primary": true
}

Then read allocate_explanation, unassigned_info, and node_allocation_decisions[].deciders[] (see sop-cluster-health.md).

---

2. Transport and TLS outcomes (classify before blaming “network”)

2.1 Persistent timeout (exit 28) — context-aware

Do not equate exit 28 with “allowlist only.” Exit 28 means the TCP connection did not complete in time — triangulate with control plane + CMS.

Suggested order:

1. Allowlist: DescribeInstance publicIpWhitelist / networkConfig.whiteIpGroupList (whiteIpType=PUBLIC_ES) vs client egress IP. 2. If allowlist looks OK but still timing out:

Principle: exit 28 is a transport-timeout signal, not proof of a firewall drop. If CMS shows resource pressure, prefer overload before “pure network.”

2.2 Intermittent timeouts

When some curl calls succeed and others time out in the same session:

1. Do not label as “preconditions not met” if ES_* is set. 2. Must cross-check CMS NodeCPUUtilization. 3. If CPU > 80%, treat as suspected meltdown — sop-service-avalanche.md. 4. Use successful responses as partial evidence; state coverage limits in the report.

Principle: flaky connectivity is itself a signal — often overload with a live process, not a simple misconfig.

2.3 Flaky API collection tactics

When the cluster is hot: longer --connect-timeout / --max-time, start with light APIs (_cat/nodes, _cluster/health), retry _nodes/hot_threads a few times — see sop-service-avalanche.md (intermittent timeout entry).

---

3. Evidence boundary (one screen)

• With only control plane + CMS, or with no successful authenticated engine JSON, you may still state high-confidence facts such as: CMS `ClusterStatus` = Red ⇒ at least one primary shard is unassigned.
• You cannot uniquely pin allocation-explain-class causes (e.g. bad routing.allocation.require._name, exact decider text) without at least one authenticated successful /_cluster/allocation/explain targeting the relevant shard (or equivalent) — see §1.4.
• Say explicitly: the gap is missing authenticated data-plane evidence or wrong shard in explain output, not “Elasticsearch cannot be diagnosed.”

---

4. Report checklist (when MUST data is missing)

Put at the top of the report:

1. One line blocking reason — only after §1.1: e.g. ES_PASSWORD NOT SET in this shell, `401` with authenticated curl (then scheme/user/password), connection refused, persistent vs intermittent timeout — before a long generic “evidence boundary” paragraph. 2. Which MUST conditions fired (from CMS / script). 3. Which engine evidence types would be collected after auth/path works (allocation/explain with primary shard if Red, hot_threads, …). 4. Pointer to SKILL.md section 2.2 only when §1.1 shows vars not set; if vars are set but calls fail, point to §1.2 / §2 instead.

---

5. Related references

Elasticsearch REST API catalog (curl)

Use curl directly against the Elasticsearch data plane.

export ES_ENDPOINT="<host:9200 or http://host:9200>"
export ES_USERNAME="elastic"
export ES_PASSWORD="<elasticsearch-admin-password>"

# Generic template (HTTP)
curl -sS -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \
  "http://${ES_ENDPOINT#http://}/<endpoint>?pretty"

If the cluster only serves HTTPS, switch to https:// and add CA options or -k for testing only.

---

Cluster layer

---

Node layer

Version note

- The ?local flag on GET /_cat/nodes was deprecated in 7.x and removed in 8.0 — do not use it.

- Other node APIs are compatible across 6.x / 7.x / 8.x.

---

Index and shard layer

---

Task layer

---

Snapshots and ILM

---

Version compatibility

Notes

• _ilm/status: ILM is not available on 6.x.
• Do not use ?local on _cat/nodes on 8.x.
• _cat/fielddata on 6.x may require additional settings to return data.

---

Scenario quick map

Elasticsearch REST API diagnosis strategy (MUST rules)

This note captures Q&A from skill usage: when SKILL.md says ES APIs are MUST, why some paths still rely on control plane + CMS only, and how skill text relates to tooling.

When `curl` fails (401, timeouts, refused): use the progressive guide es-api-call-failures.md first, then return here for MUST / SHOULD rules.

SKILL alignment: MUST-trigger workflow (reachability including ES_* and endpoint match, then required `curl` per fired row) is normative in SKILL.md §5 (Binding rule (MUST triggers)), Step 2, and §7. The CMS tables in this file are supplementary; if workflow nuance differs, follow `SKILL.md` for agent execution.

---

1. Decision rules in the skill (evidence-driven)

Calling engine-layer curl (REST) needs both:

Summary

• MUST (run ES APIs)
• Cluster Red/Yellow: unassigned shard reasons, allocation filters.
• Search/write performance: thread pools, rejections, hot threads, breakers, task backlog.
• Cases where control plane looks fine but the engine is not.

• SHOULD (prefer ES APIs)
• CPU/memory/load anomalies that disagree with control-plane signals — cross-check.
• Post-change incidents — confirm stuck recovery or long-running tasks on the engine.

• CAN SKIP
• Root cause is already closed on the control plane and engine data would not change the answer (e.g. clear RAM denial, instance in a controlled change window, billing/resource state making the instance unavailable).

---

2. MUST trigger signals (CMS)

Treat as MUST when any of the following CMS signals appear:

Mandatory hint: If a MUST situation is detected and ES_ENDPOINT is not set, the report must start with a warning listing the missing engine-layer evidence.

---

2.5 ALWAYS checks (when ES is reachable)

These do not wait on CMS thresholds; run whenever `ES_ENDPOINT` (and password) is available:

Rationale (watermark edge case): Low disk utilization and cluster Green can still coexist with transient absolute-byte watermarks that block writes. CMS-only views miss that class of misconfiguration.

---

3. Why sometimes only control plane + CMS?

MUST addresses necessity: in the situations above, methodology says you should use engine APIs for deep proof.

But the skill also states executable preconditions. If ES_ENDPOINT / ES_PASSWORD are missing, or TCP to 9200 fails, curl cannot run. Then you can only:

• Use aliyun CLI + CMS + logs;
• State the evidence boundary in the report: no ES REST path — no engine-level root cause.

So: not “MUST can be skipped on purpose”, but “MUST cannot run until preconditions hold.”

---

3.5 Connection failures — differentiate patterns

Full playbooks (credential probe, HTTP 401 meaning, exit 28 triage, intermittent timeouts, evidence boundary, report checklist): es-api-call-failures.md

Summary table (read the linked doc for steps):

---

4. Documentation vs check_es_instance_health.py today

When ES_ENDPOINT and ES_PASSWORD are set, _check_cluster_config_optional automatically calls:

Finding remediation text may recommend /_cluster/allocation/explain, /_cat/shards, /_nodes/hot_threads, etc. Those are operator / agent next steps — the script does not HTTP-fetch hot_threads or allocation/explain bodies by itself. When rule findings imply a §5 MUST row, check_es_instance_health.py prints a short §5 MUST — engine APIs footer (deduped paths); the agent must still execute those calls after §2.2 reachability.

Skill / agent workflow: for Red/Yellow, thread-pool saturation, and other MUST rows in SKILL.md, the agent should still run the listed curl (or Console) steps to collect engine evidence beyond what the script auto-pulls.

Planned vs implemented

Older drafts described stderr banners such as [MUST采集] … auto allocation/explain. That auto-fetch is not in the current script — allocation/explain and hot_threads bodies remain manual MUST. The script may print a §5 MUST — engine APIs footer (stdout) when findings map to SKILL.md §5; that is a checklist, not an HTTP substitute.

---

5. Summary

---

6. Related references

• Main skill: ../SKILL.md (workflow, credentials, direct ES access)
• REST call failures (progressive): es-api-call-failures.md
• Cluster health SOP: sop-cluster-health.md
• Performance SOPs: sop-write-performance.md, sop-query-thread-pool.md

Elasticsearch health event catalog

18 event codes / 49 reason codes across 8 categories. Each row: reason code under its event code.

---

Quick nav: TOP 15 by case frequency

---

1. Cluster health (3 reason codes)

Event code: HealthCheck.ClusterUnhealthy

Log-based (no CMS rule): Cluster.UnavailableShards — UnavailableShardsException in logs, primary inactive; same event code HealthCheck.ClusterUnhealthy.

Metric source: ClusterStatus, ClusterDisconnectedNodeCount from CMS (fetch_metrics_batch).

---

2. Master stability (3 reason codes)

Event code: HealthCheck.MasterStabilityRisk

Metric status: ClusterPendingTasksCount, MasterElectionCount not in CMS yet — collect via ES API (GET _cluster/pending_tasks, GET _cat/master).

---

3. Service availability (1 reason code)

Event code: HealthCheck.ClusterRequestError

Historical typo Cluster.RequstErrorTooHigh may appear in older exports; canonical name: `Cluster.RequestErrorTooHigh`.

---

4. Resource anomalies (17 reason codes)

4.1 JVM memory pressure

Event code: HealthCheck.JVMMemoryPressure

4.2 High CPU

Event code: HealthCheck.CPULoadHigh

CPU.PersistUsageHigh uses dual thresholds: avg > 70% → P0, 60–70% → P1 (same reason_code, different priority).

4.3 Disk utilization

Event code: HealthCheck.DiskUsageHigh

4.4 Disk IO

Event code: HealthCheck.DiskIOBottleneck

IO rule priority was P1 → updated to P0 (rule #19).

4.5 File descriptors

Event code: HealthCheck.SystemFileDescriptorsHigh

---

5. Performance bottlenecks (10 reason codes)

5.1 Thread pool saturation

Event code: HealthCheck.ThreadPoolSaturation

5.2 High latency

Event code: HealthCheck.LatencyHigh

5.3 Slow recovery

Event code: HealthCheck.RecoverySlow

---

6. Capacity planning (6 reason codes)

6.1 Shard misconfiguration

Event code: HealthCheck.ShardMisconfigured

6.2 Load imbalance

Event code: HealthCheck.LoadUnbalanced

Imbalance spans traffic, data placement, and resource levels. See sop-node-load-imbalance.md.

---

7. High-availability risk (4 reason codes)

7.1 Cluster scale

Event code: HealthCheck.ClusterScaleLow

7.2 Missing replication

Event code: HealthCheck.ReplicationMissing

7.3 Backup failure

Event code: HealthCheck.BackupFailure

---

8. Configuration risk (3 reason codes)

Event code: HealthCheck.ConfigurationRisk

---

Appendix A: Threshold cheat sheet (health script subset)

Thresholds implemented in scripts/check_es_instance_health.py (baseline 20260318) may differ slightly from CMS-only rule numbers below — use the script THRESHOLDS for ground truth.

---

Appendix B: How metrics are collected

Available via CMS fetch_metrics_batch

References: Basic metrics (Alibaba Cloud Help) |

Cluster metrics guide

ClusterStatus                   Cluster health (0=Green, 1=Yellow, 2=Red)
ClusterAutoSnapshotLatestStatus Snapshot status (-1=none/0=ok/1=running/2=failed)
ClusterDisconnectedNodeCount    Disconnected node count
ClusterNodeCount                Node count
ClusterShardCount               Total shard count
ClusterQueryQPS                 Cluster search QPS
ClusterIndexQPS                 Cluster indexing QPS
ClusterSearchLatency            Avg search latency (ms)
ClusterIndexingLatency          Avg indexing latency (ms)
ClusterSlowSearchingCount       Slow query count
NodeCPUUtilization              Node CPU (%)
NodeHeapMemoryUtilization       Node heap (%)
NodeDiskUtilization             Node disk (%)
NodeFreeStorageSpace            Free storage (MiB)
NodeLoad_1m                     1m load
NodeStatsDataDiskUtil           Disk IO utilization (%)
NodeStatsDataDiskRm             Disk read bandwidth (MiB/s)
NodeStatsDataDiskWm             Disk write bandwidth (MiB/s)
NodeStatsDataDiskR              Read IOPS
NodeStatsDataDiskW              Write IOPS
JVMGCOldCollectionCount         Old GC count (per sample bucket)
JVMGCOldCollectionDuration      Old GC duration ms (per sample bucket)
NodeStatsFullGcCollectionCount  Full GC count
NodeStatsExceptionLogCount      Exception log count
...

Requires additional ES API (for deeper diagnosis)

# Thread pools (ThreadPool.*)
GET _nodes/stats/thread_pool?filter_path=nodes.*.thread_pool

# Breakers (JVMMemory.BreakerTripped)
GET _nodes/stats/breaker

# Heap / mem (JVMMemory.FielddataCacheTooLarge)
GET _nodes/stats/jvm?filter_path=nodes.*.jvm.mem

# Running tasks (Latency.SearchTaskRunningLong)
GET _tasks?detailed=true

# Pending tasks (Master.TasksPending*)
GET _cluster/pending_tasks

# Shard states (Cluster.StatusRed/Yellow)
GET _cat/shards?v&s=state&h=index,shard,prirep,state,node,unassigned.reason

# Unassigned reason
GET _cluster/allocation/explain

# Recovery (Recovery.SlowWarning)
GET _cat/recovery?v&active_only=true

# Snapshots (Backup.*)
GET _cat/snapshots?v

# ILM (often tied to Master.TasksPending*)
GET _ilm/status

# Read-only blocks (Disk.IndexReadOnly / watermarks)
GET _all/_settings?filter_path=*.settings.index.blocks
# or
GET _cat/indices?v&h=index,status,health

# Watermarks (Disk.WatermarkAbsolute*)
GET _cluster/settings?include_defaults=true&filter_path=**.watermark,**.flood_stage
GET _cat/allocation?format=json&bytes=b

RAM permission list

Minimum read-only RAM permissions required for this skill (no write actions):

Alibaba Cloud Elasticsearch OpenAPI

• elasticsearch:DescribeInstance — Instance details (including clusterTasks)
• elasticsearch:ListInstance — List instances
• elasticsearch:ListSearchLog — Instance runtime logs
• elasticsearch:ListActionRecords — Instance change / action records
• elasticsearch:ListAllNode — Cluster node information

Cloud Monitor (CMS)

• cms:DescribeMetricList — Time-series metrics (CPU, memory, disk, load, cluster health)
• cms:DescribeSystemEventAttribute — System events (control-plane changes, restarts, etc.)
• cms:DescribeMetricMetaList — Metric metadata (available metric catalog)

Optional but recommended

• sts:GetCallerIdentity — Validate the CLI profile (aliyun sts get-caller-identity)

Runtime dependencies

aliyun CLI >= 3.3.1
curl

References (references/)

Operational knowledge for the Alibaba Cloud Elasticsearch instance diagnosis skill: APIs, verification, event catalog, acceptance rules, and scenario SOPs (sop-*.md).

Discoverability: skill trigger keywords (including 中文 for Chinese-speaking users) live in repo root `SKILL.md` frontmatter and `metadata.yaml` (triggers). Markdown here stays English; triggers bridge user language to the skill.

---

Index

APIs and access

Verification, strategy, and quality bar

Tooling

Scenario SOPs (sop-*.md)

---

For agents

1. Follow the workflow in the repo root [`SKILL.md`](../SKILL.md) (especially the signal → SOP routing table). 2. Prefer `SKILL.md` over ad-hoc reading order; open SOPs only when the observed signal matches. 3. Report text and evidence keys in `scripts/check_es_instance_health.py` are English — keep doc examples aligned (e.g. cluster_status_latest, affected_nodes, configured_watermark_low).

---

Contributing

Edits to references should stay consistent with `SKILL.md` and the checker script’s field names. Prefer small, accurate updates over duplicating long procedures that already live in a SOP.

Related APIs

Complete list of Alibaba Cloud OpenAPIs and Elasticsearch REST APIs used by this skill.

---

Control plane (OpenAPI)

OpenAPI action names follow the official Alibaba Cloud API (PascalCase), e.g. Elasticsearch API and CloudMonitor (CMS) API. Use the Aliyun CLI plugin subcommands (lowercase-hyphenated) when showing shell examples.

Elasticsearch OpenAPI

Alibaba Cloud Monitor (CMS) OpenAPI

---

Data plane (Elasticsearch REST API)

Cluster health and state

Nodes

Indices and shards

Tasks and recovery

Snapshots and ILM

---

Dependencies

aliyun CLI >= 3.3.1
elasticsearch>=7,<9   # client library if used by tooling (optional for curl-only path)

---

Permissions

See ram-policies.md.

Structured diagnosis report (skeleton)

Use with [acceptance-criteria.md](acceptance-criteria.md) (§6.x) and repo root [SKILL.md](../SKILL.md) §5 Step 4. Copy the block below and fill placeholders.

Before publish: Reconcile ClusterStatus wording if CMS window max (e.g. Yellow) and a single /_cluster/health snapshot (e.g. green) both appear — qualify time vs aggregation (acceptance-criteria.md §6.1). If per-node CPU % seems to contradict which node was `search` pool–bound, add one sentence on sampling / window mean vs spike (acceptance-criteria.md §6.2). If slow-log node name ≠ node named in pool-rejection / INSTANCELOG lines, add one sentence on routing / phase / time (acceptance-criteria.md §6.2). If CMS `ClusterShardCount` (or similar) jumps (e.g. half then back), do not imply shard loss without `_cat/shards` + ops record cross-check (acceptance-criteria.md §6.1).

## Diagnosis summary

**Instance**: {instance_id} ({region_id})
**Analysis window**: {begin} ~ {end}

### Cross-layer root cause (required when `activating` coexists with Red / unassigned)

**One-line root cause**: {Chain per `sop-activating-change-stuck.md` section 4: change waiting for recovery ← Red ← allocation explain}

(Omit or mark “N/A” if there is no `activating` or the cluster is Green.)

### Incident timeline (recency-ordered)

{Earlier → later, or “latter-window emphasis”: which dimensions (search / write / GC / CPU / disk) peaked or persisted when; cite CMS peak times or log alignment}

### Findings (by priority)

#### P0 - Critical (immediate)
- [Event code] Description
  - Evidence: {metrics / log keywords / events}
  - Root-cause reasoning: {analysis}
  - Immediate actions: {commands or steps}

#### P1 - Warning (within 30 minutes)
...

### Root-cause chain diagram
{Propagation path, e.g. disk full → shards cannot allocate → cluster Red}

### Open questions / follow-ups
{Uncertainties and next checks}

SOP: Instance activating / change stuck

Covers: ManagementPlane.ActivatingStuck; often appears together with HealthCheck.ClusterUnhealthy (cluster Red) and unassigned shards on the engine.

Related: For engine-side Red / allocation, follow sop-cluster-health.md (Section 1 — cluster Red). This SOP explains the cross-layer causality between control-plane lifecycle activating and engine health — so the report does not only list “one control-plane line + one engine line” without a closed loop.

---

1. What activating means

activating is not a random standalone fault. It is the control-plane lifecycle state while a change has not finished (console / OpenAPI). Typical causes:

• Rolling orchestration after RestartInstance and similar;
• Tasks still in updating, e.g. plugin install/remove or config changes.

User experience: “The instance has been changing / activating for hours.”

---

2. Cross-layer causality (closed-loop root cause)

2.1 Control-plane facts

• DescribeInstance: status == activating (or equivalent), often with updatedAt to detect “stuck too long” (aligned with rule engine ManagementPlane.ActivatingStuck).
• ListActionRecords: change type (e.g. rolling restart, plugin op), phase, timeline.
• ListAllNode: node roles, whether under rolling, abnormal nodes.

2.2 Orchestration constraint (why “change never finishes” shows as activating)

Many changes require the cluster to reach an acceptable health state (e.g. rolling waits for shard assignment, waits until the cluster is servicable) before the next step. If that condition cannot be met for a long time, the orchestration task stays in updating and the instance stays `activating`.

2.3 Engine-side closure (why “the cluster never recovers”)

Do not duplicate /_cluster/allocation/explain, _cat/shards, unassigned.reason playbooks here — use sop-cluster-health.md (Section 1 — cluster Red).

In the “long activating” context, remember: orchestration is waiting for “cluster can recover / shards can allocate”; the engine Red root cause still comes from allocation explain (e.g. allocation filter to a non-existent node, disk, allocation disabled). Until that is fixed, “wait for cluster recovery” may never succeed → instance stays activating.

Takeaway: On the control plane, activating means “change not finished”. The technical closure that explains why it cannot finish is still `allocation/explain`, not a paraphrase of activating text instead of engine evidence.

2.4 Remediation order

1. When `activating` is confirmed, complete control-plane evidence before touching the engine: at least `DescribeInstance` + `ListActionRecords` (MUST), and `ListAllNode` is recommended. ListActionRecords gives change type (e.g. RestartInstance), phase, rolling progress, stuck node / pendingOperation, etc. Reporting only “activating + RestartInstance” without this API is an incomplete control-plane chain (a common gap when reviews expect full change-task evidence). 2. Then follow sop-cluster-health.md (Red / allocation section) for engine explain / shard root cause and fixes (delete index, change settings, etc.). 3. Before destructive or recovery actions (e.g. DELETE index, PUT allocation-related settings): keep a pre-action snapshot — same-moment DescribeInstance + ListActionRecords summary (or explicit timestamps and key fields) so the report can state “control-plane state while the change was stuck”. 4. After engine blockers are cleared, expect intermediate Yellow, throttling, same_shard, CMS lag, etc. while moving toward Green (see cluster-health SOP) — that is normal. Re-check control plane after remediation: `DescribeInstance` again (and `ListActionRecords` if needed) to see whether activating ended and to validate “engine recovered → change completed → lifecycle normalized”; if still activating, continue from the new records.

---

3. Evidence checklist (MUST / SHOULD)

3.1 Recommended collection order

1. DescribeInstance → confirm activating and instance fields. 2. `ListActionRecords` → task detail (distinguish from “we guessed RestartInstance”: must come from API). 3. ListAllNode (recommended) → corroborate with change records. 4. Engine: allocation/explain, _cat/shards, etc. (cluster-health Red section). 5. Before risky remediation: another snapshot of DescribeInstance + ListActionRecords if time has passed since steps 1–2 or you are about to delete indices. 6. Execute remediation → watch _cluster/health / CMS for intermediate states. 7. After remediation: DescribeInstance (+ optional ListActionRecords) → confirm activating cleared.

---

4. Report expectations (skill template)

When both `activating` (or `ManagementPlane.ActivatingStuck`) and engine Red / unassigned primary exist:

• Do not write only “control plane and engine in parallel” with no causal link.
• Evidence must include a `ListActionRecords` summary (change type, phase/progress, abnormal or pending nodes), cross-checked with `DescribeInstance`; if remediation ran, add before/after control-plane comparison (Section 3.1 recommended collection order).
• Must add a one-line cross-layer root cause that states explicitly:

Change waiting for cluster recovery ← cluster stays Red ← concrete reason from allocation explain (e.g. require points to a non-existent node)

Example sentence (replace index name and explain conclusion with real values):

The rolling change on the control plane waits for cluster health; on the engine, index {index} has index.routing.allocation.require._name pointing at a non-existent node, so primaries cannot allocate and the cluster stays Red — orchestration cannot proceed → instance remains activating.

If there is no activating or the engine is already Green, write “N/A” or omit this subsection.