
Observability K8s Investigation
- 1.4k installs
- 546 repo stars
- Updated July 22, 2026
- elastic/agent-skills
observability-k8s-investigation is an agent skill that diagnoses Kubernetes pod, node, and control-plane failures using OTel EDOT telemetry and ES|QL on kube-stack indices.
About
The observability-k8s-investigation skill diagnoses Kubernetes workload, node, and control-plane issues using OTel telemetry from EDOT and the kube-stack collector. It scopes OTel-receiver indices such as metrics-kubeletstatsreceiver.otel-*, metrics-k8sclusterreceiver.otel-*, logs-k8seventsreceiver.otel-*, and logs-k8sobjectsreceiver.otel-* with k8s.pod.name and related semantic conventions. Legacy ECS kubernetes.* fields are explicitly out of scope. A failure-mode taxonomy covers OOMKilled, CPU throttling, CrashLoopBackOff, ImagePullBackOff, node NotReady cascades, admission webhook blocks, and HPA unready-pod dampening. Investigation flows orient on pod and namespace, characterize restarts and utilization, classify via pivotal signals, corroborate with K8s events and app logs, and synthesize with confidence calibration. Guidelines forbid confabulating from empty log results, treat restart count as boolean, and prefer reporting uncertainty over false confidence.
- OTel-only indices - legacy ECS kubernetes.* out of scope.
- Failure-mode taxonomy for workload, node, and control-plane layers.
- ES|QL gotchas for VALUES scalar vs array and histogram percentiles.
- Synthesis template with hypothesis, evidence, and confidence note.
- Empty logs reported as no_logs_available, not invented causes.
Observability K8s Investigation by the numbers
- 1,376 all-time installs (skills.sh)
- +160 installs in the week ending Aug 5, 2026 (Skillselion tracking)
- Ranked #159 of 1,435 DevOps & CI/CD skills by installs in the Skillselion catalog
- Data as of Aug 5, 2026 (Skillselion catalog sync)
observability-k8s-investigation capabilities & compatibility
- Capabilities
- otel k8s index es|ql queries · failure mode taxonomy classification · k8s event corroboration · baseline comparison for utilization findings · confidence calibrated synthesis output
- Use cases
- debugging · devops
npx skills add https://github.com/elastic/agent-skills --skill observability-k8s-investigationAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 1.4k |
|---|---|
| repo stars | ★ 546 |
| Last updated | July 22, 2026 |
| Repository | elastic/agent-skills ↗ |
Why is my Kubernetes pod CrashLoopBackOff, OOMKilling, or failing after a deploy?
Diagnose Kubernetes pod failures, node pressure, and control-plane issues using OTel EDOT telemetry and ES|QL across kubeletstats and k8sevents indices.
Who is it for?
SREs investigating K8s incidents with Elastic Observability OTel ingest on kubeletstats and k8sevents indices.
Skip if: Skip for legacy Elastic Agent kubernetes.* indices, cluster provisioning, or APM-only service SLO analysis.
When should I use this skill?
User reports pod failures, node pressure, rollout stuck, or asks what broke in a Kubernetes cluster.
What you get
A hypothesis-driven RCA with evidence from K8s metrics and events, confidence level, and recommended next steps.
- incident investigation reports
- correlated log findings
- cluster signal summaries
Files
Cloud Environment Setup
Configure Elastic Cloud authentication and preferences. All other cloud/* skills depend on this setup.
Workflow
Setup Progress:
- [ ] Step 1: Verify API key
- [ ] Step 2: Set defaults
- [ ] Step 3: Validate connectionStep 1: Verify API key
Check whether EC_API_KEY is already set:
echo "${EC_API_KEY:?Not set}"If not set, instruct the user to set it. Never ask the user to paste an API key into the chat — secrets must not appear in conversation history.
If the user indicates they do not have an Elastic Cloud account yet, propose starting a free trial at Elastic Cloud free trial. The trial provides 14 days of full access to Elastic Cloud Serverless with no credit card required. Once the user has registered and logged in, proceed with API key generation below.
Direct the user to:
1. Generate a key at Elastic Cloud API keys. Only Organization owners can create and manage Cloud API keys. 1. When creating this key, include Project Admin privileges or higher (Org Owner) so it can create and manage serverless projects. 1. Create a .env file in the project root (recommended — works in sandboxed agent shells):
EC_API_KEY=your-api-keyAll cloud/* scripts auto-load .env from the working directory — no manual sourcing needed.
Alternatively, export directly in the terminal:
export EC_API_KEY="your-api-key"Terminal exports might not be visible to sandboxed agents running in a separate shell session. Prefer the .env file when working with an agent.
Remind the user that storing secrets in local files is acceptable for development, but for production or shared environments, use a centralized secrets manager (for example, HashiCorp Vault, AWS Secrets Manager, 1Password CLI) to avoid secrets sprawl.
Step 2: Set defaults
Export the base URL and default region:
export EC_BASE_URL="https://api.elastic-cloud.com"
export EC_REGION="gcp-us-central1"Ask the user if they want a different region. To list available regions:
curl -s -H "Authorization: ApiKey ${EC_API_KEY}" \
"${EC_BASE_URL}/api/v1/serverless/regions" | python3 -m json.toolStep 3: Validate connection
Confirm the API key works by calling the regions endpoint:
curl -sf -H "Authorization: ApiKey ${EC_API_KEY}" \
"${EC_BASE_URL}/api/v1/serverless/regions" > /dev/null && echo "Authenticated." || echo "Authentication failed."If validation fails, check:
- The API key is valid and not expired
- Network connectivity to
api.elastic-cloud.com
Examples
First-time setup
User: set up my cloud environment
Agent: Check if EC_API_KEY is set in your terminal. If not, generate a key at
https://cloud.elastic.co/account/keys and run:
export EC_API_KEY="your-key"
Then confirm and I'll validate the connection.Setup with custom region
User: set up cloud with eu region
Agent: [runs setup, sets EC_REGION to user's preferred EU region]Guidelines
- Never receive, echo, or log API keys, passwords, or any credentials in the chat. Instruct the user to manage secrets
in their terminal or using files directly.
- Always validate the connection after setting the key.
- Default region is
gcp-us-central1— only change if the user requests a different region. - This skill is a prerequisite. Other cloud skills should refer here when
EC_API_KEYis missing.
Environment variables
| Variable | Required | Description |
|---|---|---|
EC_API_KEY | Yes | Elastic Cloud API key |
EC_BASE_URL | No | Cloud API base URL (default: https://api.elastic-cloud.com) |
EC_REGION | No | Default region (default: gcp-us-central1) |
Troubleshooting
| Problem | Fix |
|---|---|
401 Unauthorized | API key is invalid or expired — generate a new one |
connection refused | Check network access to api.elastic-cloud.com |
Related skills
How it compares
Choose observability-k8s-investigation for Elastic-native K8s signal triage rather than generic kubectl debugging without observability correlation.
FAQ
Which telemetry does observability-k8s-investigation use?
OTel EDOT indices like metrics-kubeletstatsreceiver.otel-* and logs-k8seventsreceiver.otel-* - not legacy kubernetes.* fields.
How does it handle missing application logs?
It reports no_logs_available and does not invent dependency failure modes from empty results.
What is a common CPU throttling misdiagnosis?
Error exits blamed on app bugs when cpu_limit_utilization above 1.0 from CFS throttling caused liveness probe timeouts.