Now liveThe Skillselion MCP - thousands of ranked skills, loaded into your agent mid-task. No install.Get it →
elastic avatar

Observability K8s Investigation

  • 1.4k installs
  • 546 repo stars
  • Updated July 22, 2026
  • elastic/agent-skills

observability-k8s-investigation is an agent skill that diagnoses Kubernetes pod, node, and control-plane failures using OTel EDOT telemetry and ES|QL on kube-stack indices.

About

The observability-k8s-investigation skill diagnoses Kubernetes workload, node, and control-plane issues using OTel telemetry from EDOT and the kube-stack collector. It scopes OTel-receiver indices such as metrics-kubeletstatsreceiver.otel-*, metrics-k8sclusterreceiver.otel-*, logs-k8seventsreceiver.otel-*, and logs-k8sobjectsreceiver.otel-* with k8s.pod.name and related semantic conventions. Legacy ECS kubernetes.* fields are explicitly out of scope. A failure-mode taxonomy covers OOMKilled, CPU throttling, CrashLoopBackOff, ImagePullBackOff, node NotReady cascades, admission webhook blocks, and HPA unready-pod dampening. Investigation flows orient on pod and namespace, characterize restarts and utilization, classify via pivotal signals, corroborate with K8s events and app logs, and synthesize with confidence calibration. Guidelines forbid confabulating from empty log results, treat restart count as boolean, and prefer reporting uncertainty over false confidence.

  • OTel-only indices - legacy ECS kubernetes.* out of scope.
  • Failure-mode taxonomy for workload, node, and control-plane layers.
  • ES|QL gotchas for VALUES scalar vs array and histogram percentiles.
  • Synthesis template with hypothesis, evidence, and confidence note.
  • Empty logs reported as no_logs_available, not invented causes.

Observability K8s Investigation by the numbers

  • 1,376 all-time installs (skills.sh)
  • +160 installs in the week ending Aug 5, 2026 (Skillselion tracking)
  • Ranked #159 of 1,435 DevOps & CI/CD skills by installs in the Skillselion catalog
  • Data as of Aug 5, 2026 (Skillselion catalog sync)
At a glance

observability-k8s-investigation capabilities & compatibility

Capabilities
otel k8s index es|ql queries · failure mode taxonomy classification · k8s event corroboration · baseline comparison for utilization findings · confidence calibrated synthesis output
Use cases
debugging · devops
npx skills add https://github.com/elastic/agent-skills --skill observability-k8s-investigation

Add your badge

Show developers this skill is listed on Skillselion. Paste this into your README.

Listed on Skillselion
Installs1.4k
repo stars546
Last updatedJuly 22, 2026
Repositoryelastic/agent-skills

Why is my Kubernetes pod CrashLoopBackOff, OOMKilling, or failing after a deploy?

Diagnose Kubernetes pod failures, node pressure, and control-plane issues using OTel EDOT telemetry and ES|QL across kubeletstats and k8sevents indices.

Who is it for?

SREs investigating K8s incidents with Elastic Observability OTel ingest on kubeletstats and k8sevents indices.

Skip if: Skip for legacy Elastic Agent kubernetes.* indices, cluster provisioning, or APM-only service SLO analysis.

When should I use this skill?

User reports pod failures, node pressure, rollout stuck, or asks what broke in a Kubernetes cluster.

What you get

A hypothesis-driven RCA with evidence from K8s metrics and events, confidence level, and recommended next steps.

  • incident investigation reports
  • correlated log findings
  • cluster signal summaries

Files

SKILL.mdMarkdownGitHub ↗

Cloud Environment Setup

Configure Elastic Cloud authentication and preferences. All other cloud/* skills depend on this setup.

Workflow

Setup Progress:
- [ ] Step 1: Verify API key
- [ ] Step 2: Set defaults
- [ ] Step 3: Validate connection

Step 1: Verify API key

Check whether EC_API_KEY is already set:

echo "${EC_API_KEY:?Not set}"

If not set, instruct the user to set it. Never ask the user to paste an API key into the chat — secrets must not appear in conversation history.

If the user indicates they do not have an Elastic Cloud account yet, propose starting a free trial at Elastic Cloud free trial. The trial provides 14 days of full access to Elastic Cloud Serverless with no credit card required. Once the user has registered and logged in, proceed with API key generation below.

Direct the user to:

1. Generate a key at Elastic Cloud API keys. Only Organization owners can create and manage Cloud API keys. 1. When creating this key, include Project Admin privileges or higher (Org Owner) so it can create and manage serverless projects. 1. Create a .env file in the project root (recommended — works in sandboxed agent shells):

EC_API_KEY=your-api-key

All cloud/* scripts auto-load .env from the working directory — no manual sourcing needed.

Alternatively, export directly in the terminal:

export EC_API_KEY="your-api-key"

Terminal exports might not be visible to sandboxed agents running in a separate shell session. Prefer the .env file when working with an agent.

Remind the user that storing secrets in local files is acceptable for development, but for production or shared environments, use a centralized secrets manager (for example, HashiCorp Vault, AWS Secrets Manager, 1Password CLI) to avoid secrets sprawl.

Step 2: Set defaults

Export the base URL and default region:

export EC_BASE_URL="https://api.elastic-cloud.com"
export EC_REGION="gcp-us-central1"

Ask the user if they want a different region. To list available regions:

curl -s -H "Authorization: ApiKey ${EC_API_KEY}" \
  "${EC_BASE_URL}/api/v1/serverless/regions" | python3 -m json.tool

Step 3: Validate connection

Confirm the API key works by calling the regions endpoint:

curl -sf -H "Authorization: ApiKey ${EC_API_KEY}" \
  "${EC_BASE_URL}/api/v1/serverless/regions" > /dev/null && echo "Authenticated." || echo "Authentication failed."

If validation fails, check:

  • The API key is valid and not expired
  • Network connectivity to api.elastic-cloud.com

Examples

First-time setup

User: set up my cloud environment
Agent: Check if EC_API_KEY is set in your terminal. If not, generate a key at
       https://cloud.elastic.co/account/keys and run:
       export EC_API_KEY="your-key"
       Then confirm and I'll validate the connection.

Setup with custom region

User: set up cloud with eu region
Agent: [runs setup, sets EC_REGION to user's preferred EU region]

Guidelines

  • Never receive, echo, or log API keys, passwords, or any credentials in the chat. Instruct the user to manage secrets

in their terminal or using files directly.

  • Always validate the connection after setting the key.
  • Default region is gcp-us-central1 — only change if the user requests a different region.
  • This skill is a prerequisite. Other cloud skills should refer here when EC_API_KEY is missing.

Environment variables

VariableRequiredDescription
EC_API_KEYYesElastic Cloud API key
EC_BASE_URLNoCloud API base URL (default: https://api.elastic-cloud.com)
EC_REGIONNoDefault region (default: gcp-us-central1)

Troubleshooting

ProblemFix
401 UnauthorizedAPI key is invalid or expired — generate a new one
connection refusedCheck network access to api.elastic-cloud.com

Related skills

How it compares

Choose observability-k8s-investigation for Elastic-native K8s signal triage rather than generic kubectl debugging without observability correlation.

FAQ

Which telemetry does observability-k8s-investigation use?

OTel EDOT indices like metrics-kubeletstatsreceiver.otel-* and logs-k8seventsreceiver.otel-* - not legacy kubernetes.* fields.

How does it handle missing application logs?

It reports no_logs_available and does not invent dependency failure modes from empty results.

What is a common CPU throttling misdiagnosis?

Error exits blamed on app bugs when cpu_limit_utilization above 1.0 from CFS throttling caused liveness probe timeouts.

DevOps & CI/CDmonitoringinfra

This week in AI coding

Five minutes, every Monday - the tools, releases and tactics for developers.

unsubscribe anytime.