Holmesgpt

HolmesGPT Skill

AI-powered troubleshooting for Kubernetes and cloud-native environments.

Overview

HolmesGPT is a CNCF Sandbox project that connects AI models with live observability data to investigate infrastructure problems, find root causes, and suggest remediations. It operates with read-only access and respects RBAC permissions, making it safe for production environments.

Quick Reference

Key Features

• Root Cause Analysis: Investigates alerts and cluster issues
• Multi-Source Integration: 30+ toolsets (K8s, Prometheus, Grafana)
• Alert Integration: AlertManager, PagerDuty, OpsGenie, Jira, Slack
• Interactive Mode: Troubleshooting with /run, /show, /clear
• Custom Toolsets: Extend with proprietary tools via YAML configuration
• CI/CD Integration: Automated deployment failure investigation

Installation Quick Start

CLI (Homebrew)

brew tap robusta-dev/homebrew-holmesgpt
brew install holmesgpt
export ANTHROPIC_API_KEY="your-key"  # or OPENAI_API_KEY
holmes ask "what pods are unhealthy?"

Kubernetes (Helm)

helm repo add robusta https://robusta-charts.storage.googleapis.com
helm repo update
helm install holmesgpt robusta/holmes -f values.yaml

Docker

docker run -it --net=host \
  -e OPENAI_API_KEY="your-key" \
  -v ~/.kube/config:/root/.kube/config \
  us-central1-docker.pkg.dev/genuine-flight-317411/devel/holmes \
  ask "what pods are crashing?"

Essential Commands

# Basic investigation
holmes ask "what pods are unhealthy and why?"
holmes ask "why is my deployment failing?"

# Interactive mode
holmes ask "investigate issue" --interactive

# Alert investigation
holmes investigate alertmanager --alertmanager-url http://localhost:9093
holmes investigate pagerduty --pagerduty-api-key <KEY> --update

# With file context
holmes ask "summarize the key points" -f ./logs.txt

# CI/CD integration
holmes ask "why did deployment fail?" --destination slack --slack-token <TOKEN>

Supported AI Providers

Basic Helm Values Structure

# values.yaml for Kubernetes deployment
image:
  repository: robustadev/holmes
  tag: latest

env:
  - name: ANTHROPIC_API_KEY
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: anthropic-api-key

# Model configuration
modelList:
  sonnet:
    api_key: "{{ env.ANTHROPIC_API_KEY }}"
    model: anthropic/claude-sonnet-4-20250514
    temperature: 0

# Toolsets to enable
toolsets:
  kubernetes/core:
    enabled: true
  kubernetes/logs:
    enabled: true
  prometheus/metrics:
    enabled: true

# Resources
resources:
  requests:
    memory: "1024Mi"
    cpu: "100m"
  limits:
    memory: "1024Mi"

# RBAC (read-only by default)
createServiceAccount: true

Interactive Mode Commands

Custom Toolset Example

# custom-toolset.yaml
toolsets:
  my-custom-tool:
    description: "Custom diagnostic tool"
    tools:
      - name: check_service_health
        description: "Check health of a specific service"
        command: |
          curl -s http://{{ service_name }}.{{ namespace }}.svc.cluster.local/health
        parameters:
          - name: service_name
            description: "Name of the service"
          - name: namespace
            description: "Kubernetes namespace"

Use with: holmes ask "check health" -t custom-toolset.yaml

Kubernetes Annotations for Integration

# Add to Services/Deployments for HolmesGPT context
metadata:
  annotations:
    holmesgpt.dev/runbook: |
      This service handles payment processing.
      Common issues: database connectivity, API rate limits.
      Check: kubectl logs -l app=payment-service

Environment Variables Reference

Best Practices

1. Use Specific Queries: Include namespace, deployment name, symptoms 2. Start with Claude Sonnet 4.0/4.5: Best accuracy for complex investigations 3. Enable Relevant Toolsets: Only enable what you need to reduce noise 4. Use Interactive Mode: For complex multi-step investigations 5. Set Up Runbooks: Provide context for known alert types 6. CI/CD Integration: Automate deployment failure analysis

Security Considerations

• HolmesGPT uses read-only access (get, list, watch only)
• Respects existing RBAC permissions
• Never modifies, creates, or deletes resources
• API keys stored in Kubernetes Secrets
• Data not used for model training

Official Resources

• Documentation: <https://holmesgpt.dev/>
• GitHub: <https://github.com/robusta-dev/holmesgpt>
• Helm Chart: <https://github.com/robusta-dev/holmesgpt/tree/master/helm/holmes>
• Slack Community: Cloud Native Slack

---

Gotchas

• Read-only RBAC means HolmesGPT can't see Secrets by default: Investigations involving misconfigured Secret refs return "no permission to read" even though the agent flags it as a possible cause. Either grant secrets:get on a specific namespace or accept the blind spot — don't broaden cluster-wide.
• Toolset enablement is cumulative and noisy at scale: Enabling all 30+ toolsets makes the LLM scan irrelevant data and dilutes accuracy. Enable only the toolsets matching your stack — every extra one costs tokens and adds noise to root-cause analysis.
• Model temperature MUST be 0 for reproducible investigations: Default Helm values sometimes ship with temperature > 0; same alert gives different root causes across runs. Pin temperature: 0 in modelList or compare results between runs and lose trust.
• AlertManager URL must be reachable from the HolmesGPT pod, not the CLI: holmes investigate alertmanager --alertmanager-url http://localhost:9093 works from a laptop but fails inside the cluster — use the in-cluster service DNS (http://kube-prometheus-stack-alertmanager.monitoring:9093).
• `/clear` doesn't reset toolset context, only conversation history: Cached tool outputs from prior investigation persist within the session. Long interactive sessions accumulate stale Prometheus data that contaminates new questions. Restart the CLI between unrelated incidents.
• Anthropic model names in `modelList` need the `anthropic/` prefix: model: claude-sonnet-4-20250514 fails silently with provider-not-found; correct form is model: anthropic/claude-sonnet-4-20250514. LiteLLM error message says "model not found" without naming the missing prefix.

HolmesGPT Commands Reference

Complete CLI command documentation with examples.

Command Overview

holmes ask

The primary command for interactive troubleshooting.

Basic Syntax

holmes ask "<query>" [options]

Options

Examples

Basic Queries

# Simple troubleshooting
holmes ask "what pods are unhealthy and why?"

# Namespace-specific
holmes ask "why are pods crashing in production namespace?"

# Specific workload
holmes ask "investigate high memory usage in payment-service deployment"

# Check cluster health
holmes ask "summarize the overall health of my cluster"

With File Context

# Include log file
holmes ask "analyze these logs for errors" -f /tmp/app-logs.txt

# Include multiple files
holmes ask "compare these configurations" -f config1.yaml -f config2.yaml

# From prompt file (for complex queries)
holmes ask --prompt-file ~/prompts/investigation.txt

Model Selection

# Use specific model
holmes ask "complex analysis needed" --model opus

# Use faster model for simple queries
holmes ask "list namespaces" --model gpt4o

Interactive Mode

# Start interactive session
holmes ask "investigate alert" --interactive

# Start with initial context
holmes ask "payment service issues" --interactive

CI/CD Integration

# Send to Slack
holmes ask "why did deployment fail?" \
  --destination slack \
  --slack-token xoxb-your-token \
  --slack-channel "#deployments"

# With specific namespace context
holmes ask "analyze deployment failure in staging namespace for app-v2" \
  --destination slack \
  --slack-token $SLACK_TOKEN \
  --slack-channel "#alerts"

Custom Toolsets

# Use custom toolset
holmes ask "check database health" -t ~/toolsets/database.yaml

# Multiple toolsets
holmes ask "full system check" \
  -t ~/toolsets/database.yaml \
  -t ~/toolsets/messaging.yaml

holmes investigate

Automated alert investigation from various sources.

Investigate Syntax

holmes investigate <source> [options]

Alert Sources

AlertManager

# Basic AlertManager investigation
holmes investigate alertmanager --alertmanager-url http://localhost:9093

# With authentication
holmes investigate alertmanager \
  --alertmanager-url http://alertmanager:9093 \
  --alertmanager-username admin \
  --alertmanager-password secret

# Filter by alertname
holmes investigate alertmanager \
  --alertmanager-url http://localhost:9093 \
  --alertname "KubePodCrashLooping"

# With runbooks
holmes investigate alertmanager \
  --alertmanager-url http://localhost:9093 \
  -r ~/runbooks/kubernetes.yaml

PagerDuty

# Investigate PagerDuty incidents
holmes investigate pagerduty --pagerduty-api-key $PAGERDUTY_KEY

# Update incident with analysis
holmes investigate pagerduty \
  --pagerduty-api-key $PAGERDUTY_KEY \
  --update

# Specific incident
holmes investigate pagerduty \
  --pagerduty-api-key $PAGERDUTY_KEY \
  --incident-id P123ABC

OpsGenie

# Investigate OpsGenie alerts
holmes investigate opsgenie --opsgenie-api-key $OPSGENIE_KEY

# Update with findings
holmes investigate opsgenie \
  --opsgenie-api-key $OPSGENIE_KEY \
  --update

Jira

# Investigate Jira issues
holmes investigate jira \
  --jira-url https://company.atlassian.net \
  --jira-username user@company.com \
  --jira-api-token $JIRA_TOKEN

# Specific project
holmes investigate jira \
  --jira-url https://company.atlassian.net \
  --jira-project OPS \
  --jira-username user@company.com \
  --jira-api-token $JIRA_TOKEN

GitHub Issues

# Investigate GitHub issues
holmes investigate github \
  --github-token $GITHUB_TOKEN \
  --github-repo owner/repo

# With labels filter
holmes investigate github \
  --github-token $GITHUB_TOKEN \
  --github-repo owner/repo \
  --github-labels "bug,priority:high"

Common Options

Interactive Mode Slash Commands

When in interactive mode (--interactive), these slash commands are available:

Navigation & Session Management

Toolset Management

Output Control

Modal Navigation (vim-style keys):

• j/k - Move down/up
• g/G - Jump to top/bottom
• d/u - Half-page down/up
• w - Toggle word wrap
• q or Esc - Close modal

Execution Commands

Interactive Mode Examples

# Start session
$ holmes ask "investigate payment failures" --interactive

> AI begins investigation...

# Clear context for new topic
/clear

# Run custom command and share results
/run kubectl get pods -n production -o wide

# Show full output of last tool call
/show

# Review what AI knows
/context

# Exit session
/exit

Human-in-the-Loop Workflow

$ holmes ask "investigate network issues" --interactive

> AI: I see connectivity problems. Can you run a network test?

# Run command AI can't access
/run ssh node1 ping -c 3 node2

> Pinging node2 (10.0.1.5):
> 64 bytes: icmp_seq=1 ttl=64 time=0.5ms
> 64 bytes: icmp_seq=2 ttl=64 time=0.4ms
> ...

> AI: Network connectivity looks good. Let me check DNS...

Query Best Practices

Effective Queries

# Good: Specific and contextual
holmes ask "why is payment-service pod restarting in production namespace?"

# Good: Clear scope
holmes ask "analyze CPU usage for deployments in monitoring namespace"

# Good: Actionable
holmes ask "investigate OOMKilled events in the last hour"

Ineffective Queries

# Bad: Too vague
holmes ask "why is my pod not working?"

# Bad: No context
holmes ask "what's wrong?"

# Bad: Too broad
holmes ask "check everything"

Query Patterns by Use Case

Troubleshooting Crashes

holmes ask "why is <deployment> crashing in <namespace>?"
holmes ask "analyze CrashLoopBackOff for <pod-name>"
holmes ask "what caused the last restart of <deployment>?"

Performance Issues

holmes ask "why is <service> slow?"
holmes ask "investigate high latency in <namespace>"
holmes ask "analyze memory usage trends for <deployment>"

Deployment Failures

holmes ask "why did <deployment> rollout fail?"
holmes ask "analyze pending pods in <namespace>"
holmes ask "check image pull errors for <deployment>"

Resource Issues

holmes ask "which pods are consuming most memory in <namespace>?"
holmes ask "identify resource bottlenecks in cluster"
holmes ask "check node capacity and utilization"

Network Issues

holmes ask "investigate service connectivity issues for <service>"
holmes ask "check network policies affecting <pod>"
holmes ask "analyze DNS resolution failures"

Output Formats

Default Output

Human-readable analysis with sections:

• Alert Explanation
• Key Findings
• Root Causes
• Next Steps

JSON Output

# For programmatic processing
holmes ask "query" --output json

Slack Output

# Formatted for Slack
holmes ask "query" --destination slack --slack-token $TOKEN --slack-channel "#alerts"

Environment Variables for Commands

# Set defaults via environment
export HOLMES_MODEL="sonnet"
export HOLMES_LOG_LEVEL="DEBUG"
export PROMETHEUS_URL="http://prometheus:9090"
export ALERTMANAGER_URL="http://alertmanager:9093"

# Run with defaults
holmes ask "check cluster health"
holmes investigate alertmanager

Scripting Examples

Bash Script for CI/CD

#!/bin/bash
# investigate-failure.sh

DEPLOYMENT_NAME=$1
NAMESPACE=$2

# Check if deployment failed
ROLLOUT_CMD="kubectl rollout status deployment/$DEPLOYMENT_NAME"
if ! $ROLLOUT_CMD -n $NAMESPACE --timeout=300s; then
    # Trigger HolmesGPT investigation
    holmes ask "why did $DEPLOYMENT_NAME fail in $NAMESPACE?" \
        --destination slack \
        --slack-token "$SLACK_TOKEN" \
        --slack-channel "#deployments"
fi

GitHub Actions Integration

- name: Investigate Deployment Failure
  if: failure()
  run: |
    pip install holmesgpt
    REPO="${{ github.event.repository.name }}"
    holmes ask "analyze deployment failure for $REPO" \
      --destination slack \
      --slack-token ${{ secrets.SLACK_TOKEN }} \
      --slack-channel "#ci-alerts"
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Cron Job for Health Checks

#!/bin/bash
# daily-health-check.sh

export ANTHROPIC_API_KEY="your-key"

REPORT=$(holmes ask "summarize cluster health and any issues from the past 24 hours")

# Send to Slack if issues found
if echo "$REPORT" | grep -q "Warning\|Critical\|Error"; then
    holmes ask "detail the issues found in daily health check" \
        --destination slack \
        --slack-token "$SLACK_TOKEN" \
        --slack-channel "#daily-reports"
fi

HolmesGPT Configuration Reference

Complete configuration options for all deployment methods.

Robusta Platform Integration

HolmesGPT is a core component of the Robusta Kubernetes observability platform. When deployed alongside Robusta, additional features are available:

• Automatic Alert Enrichment: Robusta forwards alerts with full context
• Unified Dashboard: View investigations in Robusta UI
• Playbook Integration: Trigger HolmesGPT from Robusta automation playbooks
• Bi-directional Sync: Investigation results feed back into Robusta

Standalone vs Robusta-Integrated

Robusta Helm Values

# In Robusta's values.yaml
enableHolmesGPT: true
holmes:
  additionalEnvVars:
    - name: ROBUSTA_ACCOUNT_ID
      valueFrom:
        secretKeyRef:
          name: robusta-secrets
          key: account-id
    - name: ROBUSTA_API_KEY
      valueFrom:
        secretKeyRef:
          name: robusta-secrets
          key: api-key

# HolmesGPT specific settings
holmesConfig:
  modelList:
    sonnet:
      api_key: "{{ env.ANTHROPIC_API_KEY }}"
      model: anthropic/claude-sonnet-4-20250514
      temperature: 0

Enable Robusta Toolset (Standalone)

# When running HolmesGPT standalone but connecting to Robusta
toolsets:
  robusta:
    enabled: true
env:
  - name: ROBUSTA_API_KEY
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: robusta-api-key

Environment Variables

AI Provider Configuration

HolmesGPT Configuration

Data Source Configuration

LLM Tool Calling

Config File Structure

Location: ~/.holmes/config.yaml

# Default model to use
model: sonnet

# Model definitions (alternative to environment-based)
modelList:
  sonnet:
    api_key: "${ANTHROPIC_API_KEY}"
    model: anthropic/claude-sonnet-4-20250514
    temperature: 0
  gpt4:
    api_key: "${OPENAI_API_KEY}"
    model: openai/gpt-4.1
    temperature: 0.1

# Default toolsets to enable
toolsets:
  - kubernetes/core
  - kubernetes/logs
  - prometheus/metrics

# Custom toolset files
custom_toolsets:
  - ~/toolsets/my-custom-toolset.yaml

# Runbook directories
runbooks:
  - ~/runbooks/
  - /etc/holmes/runbooks/

# Logging
log_level: INFO

# Cache settings
cache_dir: ~/.holmes/cache

# Prometheus configuration
prometheus:
  url: http://prometheus:9090
  auth:
    username: admin
    password: "${PROMETHEUS_PASSWORD}"

# Slack integration
slack:
  token: "${SLACK_BOT_TOKEN}"
  default_channel: "#alerts"

Helm Chart Configuration

Complete values.yaml Reference

# Image configuration
image:
  repository: robustadev/holmes
  tag: latest
  pullPolicy: IfNotPresent

# Registry (if using private registry)
registry: robustadev

# Replica count
replicaCount: 1

# Environment variables
env:
  - name: ANTHROPIC_API_KEY
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: anthropic-api-key
  - name: PROMETHEUS_URL
    value: "http://prometheus-server.monitoring.svc.cluster.local"
  - name: HOLMES_LOG_LEVEL
    value: "INFO"

# Model configuration
modelList:
  sonnet:
    api_key: "{{ env.ANTHROPIC_API_KEY }}"
    model: anthropic/claude-sonnet-4-20250514
    temperature: 0
  gpt4:
    api_key: "{{ env.OPENAI_API_KEY }}"
    model: openai/gpt-4.1
    temperature: 0

# Toolset configuration
toolsets:
  kubernetes/core:
    enabled: true
  kubernetes/logs:
    enabled: true
  prometheus/metrics:
    enabled: true
  robusta:
    enabled: false
  internet:
    enabled: false
  github:
    enabled: false

# Custom MCP servers (Model Context Protocol)
mcpServers: {}

# Logging and telemetry
logLevel: INFO  # DEBUG, INFO, WARN, ERROR
enableTelemetry: false
sentryDSN: ""

# Post-processing
enablePostProcessing: false
postProcessingPrompt: ""

# Resource limits
resources:
  requests:
    cpu: 100m
    memory: 1024Mi
  limits:
    cpu: 500m
    memory: 2048Mi

# Service account configuration
createServiceAccount: true
customServiceAccountName: ""
customClusterRoleRules: []

# Image pull secrets
imagePullSecrets: []

# Scheduling
nodeSelector: {}
tolerations: []
affinity: {}
priorityClassName: ""

# OpenShift compatibility
openshift: false

# Additional volumes
additionalVolumes: []
additionalVolumeMounts: []

# Pod annotations
podAnnotations: {}

# Pod security context
podSecurityContext:
  runAsNonRoot: true
  runAsUser: 1000
  fsGroup: 1000

# Container security context
securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop:
      - ALL

Model Configuration

Model List YAML Format

# model_list.yaml
modelList:
  # Anthropic models
  sonnet:
    api_key: "${ANTHROPIC_API_KEY}"
    model: anthropic/claude-sonnet-4-20250514
    temperature: 0
    max_tokens: 4096

  opus:
    api_key: "${ANTHROPIC_API_KEY}"
    model: anthropic/claude-opus-4-20250514
    temperature: 0

  # OpenAI models
  gpt4:
    api_key: "${OPENAI_API_KEY}"
    model: openai/gpt-4.1
    temperature: 0.1

  gpt4o:
    api_key: "${OPENAI_API_KEY}"
    model: openai/gpt-4o
    temperature: 0

  # Azure OpenAI
  azure-gpt4:
    api_key: "${AZURE_API_KEY}"
    model: azure/gpt-4-deployment
    api_base: "${AZURE_API_BASE}"
    api_version: "2024-02-15-preview"
    temperature: 0

  # AWS Bedrock
  bedrock-claude:
    model: bedrock/anthropic.claude-3-sonnet-20240229-v1:0
    aws_access_key_id: "${AWS_ACCESS_KEY_ID}"
    aws_secret_access_key: "${AWS_SECRET_ACCESS_KEY}"
    aws_region: "${AWS_DEFAULT_REGION}"
    temperature: 0

  # Google Gemini
  gemini:
    api_key: "${GEMINI_API_KEY}"
    model: gemini/gemini-1.5-pro
    temperature: 0

  # Ollama (local)
  ollama-llama:
    model: ollama/llama3.1:70b
    api_base: "http://localhost:11434"
    temperature: 0

Use with:

export MODEL_LIST_FILE_LOCATION="/path/to/model_list.yaml"
holmes ask "query" --model=sonnet

Provider-Specific Features

Anthropic (Claude)

Prompt Caching: Reduces costs by caching system prompts.

modelList:
  sonnet:
    api_key: "${ANTHROPIC_API_KEY}"
    model: anthropic/claude-sonnet-4-20250514
    temperature: 0
    # Prompt caching is automatic for supported models

Recommended Models:

• claude-sonnet-4-20250514 - Best balance of performance and cost
• claude-opus-4-20250514 - Highest capability for complex analysis

OpenAI

Reasoning Effort Levels (GPT-5 and O-series models):

modelList:
  gpt5-high:
    api_key: "${OPENAI_API_KEY}"
    model: openai/gpt-5
    temperature: 0
    reasoning_effort: high  # Options: minimal, low, medium, high

Reasoning Effort Options:

AWS Bedrock

Extended Context Window (up to 1M tokens):

modelList:
  bedrock-claude-extended:
    model: bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
    aws_access_key_id: "${AWS_ACCESS_KEY_ID}"
    aws_secret_access_key: "${AWS_SECRET_ACCESS_KEY}"
    aws_region: "us-east-1"
    temperature: 0
    # Supports up to 1M token context window

IAM Policy Required:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream"],
    "Resource": "arn:aws:bedrock:*:*:foundation-model/*"
  }]
}

Ollama (Local LLMs)

Important Limitations:

• Tool-calling support varies by model
• Recommended models with tool support: llama3.1:70b, mistral, qwen2.5
• Models without tool support will have degraded functionality

modelList:
  ollama-local:
    model: ollama/llama3.1:70b
    api_base: "http://localhost:11434"
    temperature: 0

  # For Kubernetes deployment, use service DNS
  ollama-cluster:
    model: ollama/llama3.1:70b
    api_base: "http://ollama.ollama.svc.cluster.local:11434"
    temperature: 0

Ollama Prerequisites:

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model with tool support
ollama pull llama3.1:70b

# Verify it's running
curl http://localhost:11434/api/tags

Google Gemini / Vertex AI

Gemini Configuration:

modelList:
  gemini:
    api_key: "${GEMINI_API_KEY}"
    model: gemini/gemini-1.5-pro
    temperature: 0

Important: For Gemini compatibility, set:

export TOOL_SCHEMA_NO_PARAM_OBJECT_IF_NO_PARAMS=true

Vertex AI Configuration:

modelList:
  vertex-gemini:
    model: vertex_ai/gemini-1.5-pro
    vertex_project: "${VERTEXAI_PROJECT}"
    vertex_location: "${VERTEXAI_LOCATION}"
    temperature: 0

Vertex AI Prerequisites:

export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"
export VERTEXAI_PROJECT="your-gcp-project"
export VERTEXAI_LOCATION="us-central1"

Azure OpenAI

modelList:
  azure-gpt4:
    api_key: "${AZURE_API_KEY}"
    model: azure/your-deployment-name
    api_base: "${AZURE_API_BASE}"
    api_version: "2024-02-15-preview"
    temperature: 0

Environment Variables:

export AZURE_API_KEY="your-azure-key"
export AZURE_API_BASE="https://your-resource.openai.azure.com"
export AZURE_API_VERSION="2024-02-15-preview"

Custom Runbooks

Runbook Format

# runbooks/kubernetes-alerts.yaml
runbooks:
  - alert_name: "KubePodCrashLooping"
    instructions: |
      ## Investigation Steps
      1. Check pod logs: `kubectl logs <pod> --previous`
      2. Check pod events: `kubectl describe pod <pod>`
      3. Check resource limits and requests
      4. Look for OOMKilled events

      ## Common Causes
      - Application startup failures
      - Missing configuration/secrets
      - Resource exhaustion
      - Dependency unavailable

  - alert_name: "KubeDeploymentReplicasMismatch"
    instructions: |
      ## Investigation Steps
      1. Check deployment status: `kubectl get deployment <name>`
      2. Check ReplicaSet: `kubectl get rs`
      3. Check pending pods: `kubectl get pods | grep Pending`

      ## Common Causes
      - Insufficient cluster resources
      - Node affinity/taints preventing scheduling
      - PVC binding issues

  - alert_name: "HighMemoryUsage"
    instructions: |
      ## Investigation Steps
      1. Identify top memory consumers
      2. Check for memory leaks
      3. Review application heap settings

      ## Remediation
      - Increase memory limits
      - Restart affected pods
      - Scale horizontally

Use with:

holmes investigate alertmanager -r ~/runbooks/
# or in config.yaml
runbooks:
  - ~/runbooks/

CLI Configuration Options

Global Flags

Ask Command Flags

Investigate Command Flags

Multi-Cluster Support

HolmesGPT can investigate multiple Kubernetes clusters from a single deployment.

CLI: Multiple Contexts

# Specify kubeconfig context
holmes ask "check pods in production" --context prod-cluster

# Use different kubeconfig file
holmes ask "investigate alert" --kubeconfig ~/.kube/prod-config

# Environment variable
export KUBECONFIG=~/.kube/prod-config:~/.kube/staging-config
holmes ask "compare deployments across clusters"

Helm: Multi-Cluster Configuration

# Mount multiple kubeconfig files
additionalVolumes:
  - name: kubeconfigs
    secret:
      secretName: cluster-kubeconfigs

additionalVolumeMounts:
  - name: kubeconfigs
    mountPath: /etc/kubernetes/
    readOnly: true

env:
  - name: KUBECONFIG
    value: "/etc/kubernetes/prod.kubeconfig:/etc/kubernetes/staging.kubeconfig"

Create Multi-Cluster Secret

# Combine kubeconfigs into single secret
kubectl create secret generic cluster-kubeconfigs \
  --namespace holmesgpt \
  --from-file=prod.kubeconfig=/path/to/prod-kubeconfig \
  --from-file=staging.kubeconfig=/path/to/staging-kubeconfig

Hub-and-Spoke Pattern

For large deployments, use a central HolmesGPT instance that connects to multiple clusters:

# Central hub values.yaml
replicaCount: 2

# Service account with cross-cluster RBAC
createServiceAccount: true
customClusterRoleRules:
  - apiGroups: [""]
    resources: ["pods", "services", "events", "nodes"]
    verbs: ["get", "list", "watch"]

# Environment for each cluster
env:
  - name: PROD_CLUSTER_URL
    value: "https://prod-api.example.com"
  - name: STAGING_CLUSTER_URL
    value: "https://staging-api.example.com"

Cost Optimization

AI API costs can accumulate quickly. Use these strategies to optimize spending.

Model Selection Strategy

Configure Model Tiers

modelList:
  # Cheap tier for simple queries
  fast:
    api_key: "${OPENAI_API_KEY}"
    model: openai/gpt-4o-mini
    temperature: 0

  # Standard tier for most investigations
  default:
    api_key: "${ANTHROPIC_API_KEY}"
    model: anthropic/claude-sonnet-4-20250514
    temperature: 0

  # Premium tier for complex analysis
  deep:
    api_key: "${ANTHROPIC_API_KEY}"
    model: anthropic/claude-opus-4-20250514
    temperature: 0

Usage:

holmes ask "list unhealthy pods" --model fast
holmes ask "investigate crash" --model default
holmes ask "deep root cause analysis" --model deep

Reduce Token Usage

1. Be Specific in Queries

   # Bad: Explores entire cluster
   holmes ask "what's wrong?"

   # Good: Targeted, uses fewer tokens
   holmes ask "why is payment-service crashing in prod namespace?"

2. Limit Toolsets

   # Only enable what you need
   toolsets:
     kubernetes/core:
       enabled: true
     kubernetes/logs:
       enabled: true
     # Disable unused toolsets
     internet:
       enabled: false
     confluence:
       enabled: false

3. Use Prompt Caching (Anthropic)

Anthropic models automatically cache system prompts, reducing costs by up to 90% for repeated queries with the same context.

Monitor Costs

# Enable verbose logging to track token usage
env:
  - name: HOLMES_LOG_LEVEL
    value: "INFO"  # Logs include token counts

Log output includes:

INFO: Query completed. Tokens used: input=1500, output=800, total=2300
INFO: Estimated cost: $0.023 (claude-sonnet-4)

Budget Alerts

Set up alerts in your AI provider dashboard:

• Anthropic: Console → Usage → Set spending limit
• OpenAI: Settings → Billing → Usage limits
• Azure: Cost Management → Budgets

Cost Estimation

Prices as of early 2025. Check provider websites for current pricing.

HolmesGPT Data Sources & Toolsets

Complete reference for built-in and custom data source integrations.

Built-in Toolsets Overview

HolmesGPT includes 30+ pre-built integrations organized by category.

Kubernetes & Cloud Infrastructure

Monitoring & Observability

Prometheus Integration Details

HolmesGPT can query Prometheus metrics to diagnose performance issues.

Enable Prometheus Toolset:

toolsets:
  prometheus/metrics:
    enabled: true

env:
  - name: PROMETHEUS_URL
    value: "http://prometheus-server.monitoring.svc.cluster.local"
  # Optional: For authenticated Prometheus
  - name: PROMETHEUS_USERNAME
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: prometheus-username
  - name: PROMETHEUS_PASSWORD
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: prometheus-password

Example Prometheus Queries:

HolmesGPT translates natural language to PromQL automatically:

# Memory analysis
holmes ask "show memory usage trend for payment-service over last hour"
# Uses: rate(container_memory_usage_bytes{pod=~"payment.*"}[1h])

# CPU throttling detection
holmes ask "which pods have the highest CPU throttling?"
# Uses: rate(container_cpu_cfs_throttled_seconds_total[5m])

# Error rate analysis
holmes ask "what's the error rate for api-gateway?"
# Uses: rate(http_requests_total{status=~"5.."}[5m])

# Resource saturation
holmes ask "are any nodes running low on resources?"
# Uses: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes

# Pod restarts
holmes ask "which pods have restarted most in the last 24 hours?"
# Uses: increase(kube_pod_container_status_restarts_total[24h])

Prometheus-Specific Runbook:

runbooks:
  - alert_name: "HighMemoryUsage"
    instructions: |
      ## Prometheus Queries to Run
      1. Current usage: `container_memory_usage_bytes{pod="<pod>"}`
      2. Usage trend: `rate(container_memory_usage_bytes{pod="<pod>"}[1h])`
      3. Limit comparison: `container_memory_usage_bytes / container_spec_memory_limit_bytes`

      ## Thresholds
      - Warning: > 80% of limit
      - Critical: > 90% of limit

  - alert_name: "HighLatency"
    instructions: |
      ## Prometheus Queries to Run
      1. P99 latency: `histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))`
      2. Request rate: `rate(http_requests_total[5m])`
      3. Error rate: `rate(http_requests_total{status=~"5.."}[5m])`

Grafana Loki Integration

toolsets:
  loki:
    enabled: true

env:
  - name: LOKI_URL
    value: "http://loki.monitoring.svc.cluster.local:3100"

Example Loki Queries:

# Search for errors
holmes ask "show errors from payment-service in the last 30 minutes"

# Analyze log patterns
holmes ask "what are the most common error messages in production?"

# Correlate with events
holmes ask "show logs around the time of the last pod restart"

Data & Messaging

Infrastructure & DevOps

ArgoCD Integration Details

HolmesGPT can investigate ArgoCD sync failures and application health issues.

Enable ArgoCD Toolset:

toolsets:
  argocd:
    enabled: true

env:
  - name: ARGOCD_SERVER
    value: "argocd-server.argocd.svc.cluster.local"
  - name: ARGOCD_AUTH_TOKEN
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: argocd-token

Create ArgoCD Token:

# Generate token for HolmesGPT service account
argocd account generate-token --account holmesgpt --id holmesgpt-token

# Store in secret
kubectl create secret generic holmesgpt-secrets \
  --namespace holmesgpt \
  --from-literal=argocd-token="<generated-token>"

Example ArgoCD Queries:

# Investigate sync failures
holmes ask "why is application payment-service out of sync?"

# Check application health
holmes ask "what ArgoCD applications are degraded?"

# Analyze sync history
holmes ask "show recent sync failures for production apps"

# Investigate specific error
holmes ask "analyze ArgoCD sync error: ComparisonError for deployment/api"

ArgoCD-Specific Runbook:

runbooks:
  - alert_name: "ArgoCDAppOutOfSync"
    instructions: |
      ## Investigation Steps
      1. Check app sync status: `argocd app get <app-name>`
      2. Review sync history: `argocd app history <app-name>`
      3. Check for resource conflicts
      4. Verify Git repository accessibility

      ## Common Causes
      - Manual changes to cluster resources
      - Git repository access issues
      - Resource conflicts with other controllers
      - Invalid manifests in Git

Documentation & Collaboration

Utilities

Enabling Toolsets

CLI Usage

# Use default toolsets
holmes ask "what pods are crashing?"

# Add custom toolset
holmes ask "query" -t /path/to/custom-toolset.yaml

# Multiple toolsets
holmes ask "query" -t toolset1.yaml -t toolset2.yaml

Helm Configuration

toolsets:
  # Kubernetes (recommended baseline)
  kubernetes/core:
    enabled: true
  kubernetes/logs:
    enabled: true

  # Monitoring
  prometheus/metrics:
    enabled: true
  grafana:
    enabled: false
  loki:
    enabled: false

  # DevOps tools
  argocd:
    enabled: true
  helm:
    enabled: true

  # Documentation
  confluence:
    enabled: false
  github:
    enabled: false

  # Utilities
  internet:
    enabled: false  # Disable for security
  bash:
    enabled: false  # Disable for security

Config File

# ~/.holmes/config.yaml
toolsets:
  - kubernetes/core
  - kubernetes/logs
  - prometheus/metrics
  - argocd

custom_toolsets:
  - ~/toolsets/my-custom.yaml

Custom Toolset Development

Basic Structure

# custom-toolset.yaml
toolsets:
  my-toolset:
    description: "Description for the LLM"
    prerequisites: "Required tools/access"
    tags:
      - kubernetes
      - monitoring
    installation: |
      Instructions for setup
    tools:
      - name: tool_name
        description: "What this tool does"
        command: |
          command to execute
        parameters:
          - name: param_name
            description: "Parameter description"
            required: true
        additionalInstructions: |
          How to interpret output

Variable Types

LLM-Inferred Variables (double braces)

command: |
  kubectl get pods -n {{ namespace }} -l app={{ app_name }}

The LLM will determine appropriate values based on context.

Environment Variables (single braces with $)

command: |
  curl -H "Authorization: Bearer ${API_TOKEN}" ${API_URL}/endpoint

Hidden from LLM, sourced from environment.

Example: Custom Grafana Toolset

toolsets:
  grafana-custom:
    description: "Query Grafana dashboards and panels"
    prerequisites: "Grafana API access"
    tags:
      - monitoring
      - grafana
    tools:
      - name: list_dashboards
        description: "List all Grafana dashboards"
        command: |
          curl -s -H "Authorization: Bearer ${GRAFANA_API_KEY}" \
            "${GRAFANA_URL}/api/search?type=dash-db" | jq '.[] | {uid, title}'

      - name: get_dashboard
        description: "Get a specific dashboard by UID"
        command: |
          curl -s -H "Authorization: Bearer ${GRAFANA_API_KEY}" \
            "${GRAFANA_URL}/api/dashboards/uid/{{ dashboard_uid }}" | jq '.dashboard'
        parameters:
          - name: dashboard_uid
            description: "Dashboard UID"
            required: true

      - name: query_panel
        description: "Query a specific panel's data"
        command: |
          curl -s -H "Authorization: Bearer ${GRAFANA_API_KEY}" \
            "${GRAFANA_URL}/api/ds/query" \
            -d '{"queries": [{"refId": "A", "datasourceId": \
            {{ datasource_id }}, "rawSql": "{{ query }}"}]}'
        parameters:
          - name: datasource_id
            description: "Datasource ID"
          - name: query
            description: "Query to execute"

Example: Custom Health Check Toolset

toolsets:
  service-health:
    description: "Check health of internal services"
    prerequisites: "Network access to services"
    tools:
      - name: check_endpoint_health
        description: "Check HTTP endpoint health"
        command: |
          curl -s -o /dev/null -w "%{http_code}" \
            http://{{ service }}.{{ namespace }}.svc.cluster.local:{{ port }}/health
        parameters:
          - name: service
            description: "Service name"
          - name: namespace
            description: "Kubernetes namespace"
          - name: port
            description: "Service port"
            default: "8080"
        additionalInstructions: |
          200 = healthy, 5xx = server error, 4xx = client error

      - name: check_database_connectivity
        description: "Test database connection"
        command: |
          kubectl run db-check --rm -i --restart=Never --image=postgres:15 \
            -- pg_isready -h {{ host }} -p {{ port }} -U {{ user }}
        parameters:
          - name: host
            description: "Database host"
          - name: port
            description: "Database port"
            default: "5432"
          - name: user
            description: "Database user"

Example: Kubernetes Diagnostics Toolset

toolsets:
  k8s-diagnostics:
    description: "Advanced Kubernetes diagnostic commands"
    tools:
      - name: check_node_pressure
        description: "Check for node resource pressure conditions"
        command: |
          kubectl get nodes -o json | jq '.items[] | {
            name: .metadata.name,
            conditions: [.status.conditions[] |
              select(.type | test("Pressure")) | {type, status}]
          }'

      - name: get_pod_resource_usage
        description: "Get actual resource usage for pods in namespace"
        command: |
          kubectl top pods -n {{ namespace }} --sort-by=memory
        parameters:
          - name: namespace
            description: "Target namespace"

      - name: check_pvc_status
        description: "Check PersistentVolumeClaim status"
        command: |
          kubectl get pvc -n {{ namespace }} -o json | jq '.items[] | {
            name: .metadata.name,
            status: .status.phase,
            capacity: .status.capacity.storage,
            storageClass: .spec.storageClassName
          }'
        parameters:
          - name: namespace
            description: "Target namespace"

      - name: get_events
        description: "Get recent events filtered by type"
        command: |
          kubectl get events -n {{ namespace }} \
            --sort-by='.lastTimestamp' \
            --field-selector type={{ event_type }} -o json | \
            jq '.items[-10:] | .[] |
              {time: .lastTimestamp, reason: .reason, message: .message}'
        parameters:
          - name: namespace
            description: "Target namespace"
          - name: event_type
            description: "Event type (Normal or Warning)"
            default: "Warning"

MCP Server Integration

HolmesGPT supports Model Context Protocol (MCP) servers for extended capabilities.

Configuration

# In Helm values.yaml
mcpServers:
  aws-mcp:
    command: "npx"
    args:
      - "-y"
      - "@aws-mcp/server"
    env:
      AWS_REGION: "us-east-1"

  github-mcp:
    command: "npx"
    args:
      - "-y"
      - "@github-mcp/server"
    env:
      GITHUB_TOKEN: "${GITHUB_TOKEN}"

Custom Docker Image for Additional Tools

When you need binaries not in the base image:

# Dockerfile
FROM us-central1-docker.pkg.dev/genuine-flight-317411/devel/holmes:latest

# Install custom tools
RUN apt-get update && apt-get install -y \
    postgresql-client \
    mysql-client \
    redis-tools

# Add custom scripts
COPY scripts/ /usr/local/bin/
RUN chmod +x /usr/local/bin/*.sh

# Add custom toolsets
COPY toolsets/ /etc/holmes/toolsets/

Build and use:

docker build -t my-holmes:latest .

# In Helm values
image:
  repository: my-registry/my-holmes
  tag: latest

Toolset Best Practices

1. Start Minimal: Enable only kubernetes/core and kubernetes/logs initially 2. Add Incrementally: Enable additional toolsets as needed 3. Security First: Disable internet and bash in production unless required 4. Use Environment Variables: Never hardcode secrets in toolset YAML 5. Test Custom Toolsets: Verify commands work before deploying 6. Document Prerequisites: Clearly state required permissions and access 7. Handle Large Outputs: Use llm_summarize transformer for verbose commands

HolmesGPT HTTP API Reference

REST API documentation for HolmesGPT server deployments.

Overview

HolmesGPT provides a REST API for programmatic access when deployed in Kubernetes. The API supports both synchronous requests and Server-Sent Events (SSE) for streaming responses.

Base URL

After Helm installation:

http://holmesgpt-holmes.<namespace>.svc.cluster.local

For local testing:

kubectl port-forward -n holmesgpt svc/holmesgpt-holmes 8080:80
# Base URL: http://localhost:8080

Endpoints

GET /api/model

List available AI models.

Request:

curl http://localhost:8080/api/model

Response:

{
  "models": ["sonnet", "opus", "gpt4", "gpt4o"]
}

POST /api/chat

General-purpose conversational interface.

Request:

curl -X POST http://localhost:8080/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "ask": "what pods are unhealthy in production namespace?",
    "model": "sonnet",
    "conversation_history": [],
    "include_tool_calls": true
  }'

Parameters:

Response:

{
  "analysis": "Based on my investigation...",
  "sections": {
    "alert_explanation": "...",
    "key_findings": "...",
    "root_causes": "...",
    "next_steps": "..."
  },
  "conversation_history": [...],
  "tool_calls": [...],
  "metadata": {
    "token_count": 1500,
    "model": "sonnet"
  }
}

POST /api/investigate

Automated incident investigation.

Request:

curl -X POST http://localhost:8080/api/investigate \
  -H "Content-Type: application/json" \
  -d '{
    "alert_name": "KubePodCrashLooping",
    "alert_labels": {
      "namespace": "production",
      "pod": "api-gateway-xxx"
    },
    "model": "sonnet"
  }'

Parameters:

Response:

{
  "analysis": "Investigation complete...",
  "sections": {
    "alert_explanation": "This alert fires when...",
    "key_findings": "1. Pod restarted 5 times...",
    "root_causes": "Memory limit exceeded...",
    "next_steps": "1. Increase memory limits..."
  },
  "tool_calls": [
    {
      "tool": "kubectl_get_pods",
      "parameters": {"namespace": "production"},
      "result": "...",
      "status": "success"
    }
  ]
}

POST /api/stream/investigate

Streaming investigation with SSE.

Request:

curl -X POST http://localhost:8080/api/stream/investigate \
  -H "Content-Type: application/json" \
  -H "Accept: text/event-stream" \
  -d '{
    "alert_name": "HighMemoryUsage",
    "model": "sonnet"
  }'

SSE Event Types:

Example SSE Stream:

event: start_tool_calling
data: {"tool": "kubectl_get_pods"}

event: tool_calling_result
data: {"tool": "kubectl_get_pods", "status": "success", "result": "..."}

event: ai_message
data: {"content": "I found several issues..."}

event: ai_answer_end
data: {"analysis": "...", "sections": {...}}

POST /api/issue_chat

Discuss specific issues with context.

Request:

curl -X POST http://localhost:8080/api/issue_chat \
  -H "Content-Type: application/json" \
  -d '{
    "ask": "what are the next steps?",
    "model": "sonnet",
    "issue_context": {
      "previous_analysis": "Memory limit issue identified...",
      "alert_name": "KubePodCrashLooping"
    },
    "conversation_history": [...]
  }'

POST /api/workload_health_check

Kubernetes workload health analysis.

Request:

curl -X POST http://localhost:8080/api/workload_health_check \
  -H "Content-Type: application/json" \
  -d '{
    "workload_name": "api-gateway",
    "workload_type": "Deployment",
    "namespace": "production",
    "model": "sonnet",
    "include_alert_history": true
  }'

Parameters:

POST /api/workload_health_chat

Conversational workload health discussion.

Request:

curl -X POST http://localhost:8080/api/workload_health_chat \
  -H "Content-Type: application/json" \
  -d '{
    "ask": "why is memory usage increasing?",
    "workload_name": "api-gateway",
    "namespace": "production",
    "model": "sonnet",
    "conversation_history": [...]
  }'

Response Structure

Standard Response

{
  "analysis": "Full text analysis...",
  "sections": {
    "alert_explanation": "Explanation of the alert/issue",
    "key_findings": "Important discoveries",
    "root_causes": "Identified root causes",
    "next_steps": "Recommended actions"
  },
  "conversation_history": [
    {"role": "user", "content": "..."},
    {"role": "assistant", "content": "..."}
  ],
  "tool_calls": [
    {
      "tool": "tool_name",
      "parameters": {"param": "value"},
      "result": "output",
      "status": "success"
    }
  ],
  "metadata": {
    "token_count": 1500,
    "context_window_used": 0.3,
    "truncated": false,
    "model": "sonnet"
  }
}

Error Response

{
  "error": "error_code",
  "message": "Human-readable error message",
  "details": {
    "field": "additional info"
  }
}

Error Codes

Authentication

By default, the API has no authentication. For production, add authentication via:

Kubernetes NetworkPolicy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: holmesgpt-api
  namespace: holmesgpt
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: holmes
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: allowed-namespace

Ingress with Authentication

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: holmesgpt
  annotations:
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: holmesgpt-auth
spec:
  rules:
    - host: holmesgpt.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: holmesgpt-holmes
                port:
                  number: 80

Rate Limiting

AI Provider Rate Limits

HolmesGPT API responses are subject to the rate limits of the underlying AI provider. When limits are exceeded, the API returns a rate_limited error:

{
  "error": "rate_limited",
  "message": "AI provider rate limit exceeded. Retry after 60 seconds.",
  "details": {
    "provider": "anthropic",
    "retry_after_seconds": 60
  }
}

Client-Side Rate Limiting

For production deployments, implement client-side rate limiting:

Python Example:

import time
from functools import wraps

def rate_limit(max_calls: int, period: int):
    """Decorator to rate limit API calls."""
    calls = []
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            now = time.time()
            # Remove old calls outside the period
            calls[:] = [c for c in calls if now - c < period]
            if len(calls) >= max_calls:
                sleep_time = period - (now - calls[0])
                time.sleep(sleep_time)
            calls.append(time.time())
            return func(*args, **kwargs)
        return wrapper
    return decorator

@rate_limit(max_calls=10, period=60)  # 10 calls per minute
def ask_holmes(question: str) -> dict:
    # API call implementation
    pass

Kubernetes-Level Rate Limiting

Use Ingress annotations for cluster-level rate limiting:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: holmesgpt
  annotations:
    nginx.ingress.kubernetes.io/limit-rps: "10"
    nginx.ingress.kubernetes.io/limit-connections: "5"
    nginx.ingress.kubernetes.io/limit-burst-multiplier: "2"

Recommended Limits by Use Case

Usage Examples

Python Client

import requests

HOLMES_URL = "http://localhost:8080"

def ask_holmes(question: str, model: str = "sonnet") -> dict:
    response = requests.post(
        f"{HOLMES_URL}/api/chat",
        json={
            "ask": question,
            "model": model,
            "include_tool_calls": True
        }
    )
    response.raise_for_status()
    return response.json()

def investigate_alert(alert_name: str, labels: dict) -> dict:
    response = requests.post(
        f"{HOLMES_URL}/api/investigate",
        json={
            "alert_name": alert_name,
            "alert_labels": labels,
            "model": "sonnet"
        }
    )
    response.raise_for_status()
    return response.json()

# Usage
result = ask_holmes("what pods are crashing?")
print(result["analysis"])

JavaScript/Node.js Client

const axios = require('axios');

const HOLMES_URL = 'http://localhost:8080';

async function askHolmes(question, model = 'sonnet') {
  const response = await axios.post(`${HOLMES_URL}/api/chat`, {
    ask: question,
    model: model,
    include_tool_calls: true
  });
  return response.data;
}

async function streamInvestigation(alertName) {
  const response = await fetch(`${HOLMES_URL}/api/stream/investigate`, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Accept': 'text/event-stream'
    },
    body: JSON.stringify({
      alert_name: alertName,
      model: 'sonnet'
    })
  });

  const reader = response.body.getReader();
  const decoder = new TextDecoder();

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    console.log(decoder.decode(value));
  }
}

// Usage
askHolmes('check cluster health').then(console.log);

Curl Examples

# Basic chat
curl -X POST http://localhost:8080/api/chat \
  -H "Content-Type: application/json" \
  -d '{"ask": "list unhealthy pods", "model": "sonnet"}'

# With conversation history
curl -X POST http://localhost:8080/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "ask": "tell me more about the first one",
    "model": "sonnet",
    "conversation_history": [
      {"role": "user", "content": "list unhealthy pods"},
      {"role": "assistant", "content": "Found 3 unhealthy pods..."}
    ]
  }'

# Workload health check
curl -X POST http://localhost:8080/api/workload_health_check \
  -H "Content-Type: application/json" \
  -d '{
    "workload_name": "nginx",
    "workload_type": "Deployment",
    "namespace": "default",
    "model": "sonnet"
  }'

HolmesGPT Installation Guide

Complete installation instructions for all deployment methods.

Prerequisites

• Kubernetes cluster (for Helm/in-cluster deployment)
• kubectl configured with cluster access
• API key from supported AI provider (Anthropic, OpenAI, Azure, etc.)
• Python 3.9+ (for pip/poetry installation)

Installation Methods

1. Homebrew (Mac/Linux) - Recommended for CLI

# Install
brew tap robusta-dev/homebrew-holmesgpt
brew install holmesgpt

# Verify installation
holmes ask --help

# Upgrade
brew upgrade holmesgpt

2. Pipx (Cross-platform)

# Install pipx first if needed
# macOS: brew install pipx
# Linux: python3 -m pip install --user pipx
# pipx ensurepath

# Install HolmesGPT
pipx install holmesgpt

# Verify
holmes ask --help

# Upgrade
pipx upgrade holmesgpt

3. Poetry (Development/Source)

# Clone repository
git clone https://github.com/robusta-dev/holmesgpt.git
cd holmesgpt

# Install with Poetry
poetry install --no-root

# Run via poetry
poetry run holmes ask --help

# Or activate shell
poetry shell
holmes ask --help

4. Docker

# Basic usage with OpenAI
docker run -it --net=host \
  -e OPENAI_API_KEY="your-api-key" \
  -v ~/.kube/config:/root/.kube/config \
  us-central1-docker.pkg.dev/genuine-flight-317411/devel/holmes \
  ask "what pods are unhealthy?"

# With Anthropic
docker run -it --net=host \
  -e ANTHROPIC_API_KEY="your-api-key" \
  -v ~/.kube/config:/root/.kube/config \
  us-central1-docker.pkg.dev/genuine-flight-317411/devel/holmes \
  ask "what pods are unhealthy?"

# Full mount for all credentials
docker run -it --net=host \
  -e ANTHROPIC_API_KEY="your-key" \
  -v ~/.holmes:/root/.holmes \
  -v ~/.aws:/root/.aws \
  -v ~/.config/gcloud:/root/.config/gcloud \
  -v ~/.kube/config:/root/.kube/config \
  us-central1-docker.pkg.dev/genuine-flight-317411/devel/holmes \
  ask "query here"

5. Kubernetes (Helm) - Production Deployment

# Add Helm repository
helm repo add robusta https://robusta-charts.storage.googleapis.com
helm repo update

# Create namespace
kubectl create namespace holmesgpt

# Create secret for API keys
kubectl create secret generic holmesgpt-secrets \
  --namespace holmesgpt \
  --from-literal=anthropic-api-key="your-anthropic-key"

# Create values.yaml (see below)

# Install
helm install holmesgpt robusta/holmes \
  --namespace holmesgpt \
  -f values.yaml

# Verify deployment
kubectl get pods -n holmesgpt
kubectl logs -n holmesgpt deployment/holmesgpt-holmes

# Upgrade
helm repo update
helm upgrade holmesgpt robusta/holmes \
  --namespace holmesgpt \
  -f values.yaml

# Uninstall
helm uninstall holmesgpt -n holmesgpt

Version Pinning (Production Recommended)

For production environments, pin to specific chart versions to ensure reproducible deployments and controlled upgrades.

# List available chart versions
helm search repo robusta/holmes --versions

# Install specific version
helm install holmesgpt robusta/holmes \
  --version 1.5.0 \
  --namespace holmesgpt \
  -f values.yaml

# Upgrade to specific version
helm upgrade holmesgpt robusta/holmes \
  --version 1.6.0 \
  --namespace holmesgpt \
  -f values.yaml

# Check current installed version
helm list -n holmesgpt

ArgoCD Application with Version Pinning:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: holmesgpt
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://robusta-charts.storage.googleapis.com
    chart: holmes
    targetRevision: 1.5.0  # Pin chart version
    helm:
      valueFiles:
        - values.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: holmesgpt
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Upgrade Strategy:

1. Review changelog for breaking changes 2. Test in non-production environment first 3. Update targetRevision in ArgoCD or --version in Helm 4. Monitor pod logs after upgrade

Helm values.yaml Examples

Minimal Configuration (OpenAI)

env:
  - name: OPENAI_API_KEY
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: openai-api-key

modelList:
  gpt4:
    api_key: "{{ env.OPENAI_API_KEY }}"
    model: openai/gpt-4.1
    temperature: 0

Anthropic Configuration (Recommended)

env:
  - name: ANTHROPIC_API_KEY
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: anthropic-api-key

modelList:
  sonnet:
    api_key: "{{ env.ANTHROPIC_API_KEY }}"
    model: anthropic/claude-sonnet-4-20250514
    temperature: 0
  opus:
    api_key: "{{ env.ANTHROPIC_API_KEY }}"
    model: anthropic/claude-opus-4-20250514
    temperature: 0

Azure OpenAI Configuration

env:
  - name: AZURE_API_KEY
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: azure-api-key
  - name: AZURE_API_BASE
    value: "https://your-resource.openai.azure.com"
  - name: AZURE_API_VERSION
    value: "2024-02-15-preview"

modelList:
  azure-gpt4:
    api_key: "{{ env.AZURE_API_KEY }}"
    model: azure/your-deployment-name
    api_base: "{{ env.AZURE_API_BASE }}"
    api_version: "{{ env.AZURE_API_VERSION }}"
    temperature: 0

AWS Bedrock Configuration

env:
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: aws-access-key-id
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: aws-secret-access-key
  - name: AWS_DEFAULT_REGION
    value: "us-east-1"

modelList:
  bedrock-claude:
    model: bedrock/anthropic.claude-3-sonnet-20240229-v1:0
    temperature: 0

Production Configuration

image:
  repository: robustadev/holmes
  tag: latest
  pullPolicy: IfNotPresent

replicaCount: 1

env:
  - name: ANTHROPIC_API_KEY
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: anthropic-api-key
  - name: PROMETHEUS_URL
    value: "http://prometheus-server.monitoring.svc.cluster.local"

modelList:
  sonnet:
    api_key: "{{ env.ANTHROPIC_API_KEY }}"
    model: anthropic/claude-sonnet-4-20250514
    temperature: 0

toolsets:
  kubernetes/core:
    enabled: true
  kubernetes/logs:
    enabled: true
  prometheus/metrics:
    enabled: true
  robusta:
    enabled: false

resources:
  requests:
    memory: "1024Mi"
    cpu: "100m"
  limits:
    memory: "2048Mi"

createServiceAccount: true

logLevel: INFO
enableTelemetry: false

# Node scheduling
nodeSelector: {}
tolerations: []
affinity: {}

# Pod disruption budget
podDisruptionBudget:
  enabled: true
  minAvailable: 1

Post-Installation Verification

CLI Verification

# Set API key
export ANTHROPIC_API_KEY="your-key"
# or
export OPENAI_API_KEY="your-key"

# Test basic query
holmes ask "list all namespaces"

# Test with specific cluster context
holmes ask "what pods are running in kube-system?"

Helm Verification

# Check pod status
kubectl get pods -n holmesgpt

# Check logs
kubectl logs -n holmesgpt deployment/holmesgpt-holmes

# Port forward for API access
kubectl port-forward -n holmesgpt svc/holmesgpt-holmes 8080:80

# Test API
curl -X POST http://localhost:8080/api/chat \
  -H "Content-Type: application/json" \
  -d '{"ask": "list pods in default namespace", "model": "sonnet"}'

Configuration File

Store common settings in ~/.holmes/config.yaml:

# Default model
model: sonnet

# Default toolsets
toolsets:
  - kubernetes/core
  - kubernetes/logs
  - prometheus/metrics

# Custom runbooks location
runbooks: ~/runbooks/

# Log level
log_level: INFO

Troubleshooting Installation

Common Issues

1. API Key Not Found

   # Verify environment variable
   echo $ANTHROPIC_API_KEY

   # Or set inline
   ANTHROPIC_API_KEY="key" holmes ask "test"

2. kubectl Access Issues

   # Verify kubeconfig
   kubectl get pods

   # Check current context
   kubectl config current-context

3. Helm Installation Fails

   # Check Helm repos
   helm repo list

   # Update repos
   helm repo update

   # Debug installation
   helm install holmesgpt robusta/holmes -f values.yaml --debug --dry-run

4. Permission Denied

   # Check RBAC
   kubectl auth can-i get pods --as=system:serviceaccount:holmesgpt:holmesgpt-holmes

HolmesGPT Integrations Reference

Complete guide for integrating HolmesGPT with alerting platforms, messaging services, and incident management tools.

Slack Integration

Sending Analysis to Slack

HolmesGPT can send investigation results directly to Slack channels.

CLI Usage:

# Send to a specific channel
holmes ask "why is payment-service crashing?" \
  --destination slack \
  --slack-token xoxb-your-bot-token \
  --slack-channel "#incidents"

# Send alert investigation to Slack
holmes investigate alertmanager \
  --alertmanager-url http://alertmanager:9093 \
  --destination slack \
  --slack-token $SLACK_BOT_TOKEN \
  --slack-channel "#alerts"

Creating a Slack Bot

1. Go to api.slack.com/apps and create a new app 2. Under "OAuth & Permissions", add these Bot Token Scopes:

• chat:write - Send messages
• chat:write.public - Send to channels without joining
• files:write - Upload files (for large outputs)

3. Install the app to your workspace 4. Copy the "Bot User OAuth Token" (starts with xoxb-)

Helm Configuration for Slack

env:
  - name: SLACK_BOT_TOKEN
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: slack-bot-token
  - name: SLACK_CHANNEL
    value: "#holmesgpt-alerts"

# In values.yaml destination configuration
destinations:
  slack:
    enabled: true
    default_channel: "#holmesgpt-alerts"

Slack Message Formatting

HolmesGPT formats messages with sections:

:mag: *HolmesGPT Investigation*

*Alert:* KubePodCrashLooping
*Namespace:* production

*Key Findings:*
• Pod restarted 5 times in the last hour
• OOMKilled events detected
• Memory usage exceeded limits

*Root Cause:*
Memory leak in application causing OOM kills

*Recommended Actions:*
1. Increase memory limits
2. Investigate memory leak
3. Enable memory profiling

PagerDuty Integration

Investigating PagerDuty Incidents

# Investigate open incidents
holmes investigate pagerduty \
  --pagerduty-api-key $PAGERDUTY_API_KEY

# Update incidents with analysis
holmes investigate pagerduty \
  --pagerduty-api-key $PAGERDUTY_API_KEY \
  --update

# Investigate specific incident
holmes investigate pagerduty \
  --pagerduty-api-key $PAGERDUTY_API_KEY \
  --incident-id P123ABC \
  --update

PagerDuty API Key Setup

1. Go to PagerDuty > Integrations > API Access Keys 2. Create a new API key with read/write access 3. Store securely in Kubernetes secret

kubectl create secret generic holmesgpt-secrets \
  --namespace holmesgpt \
  --from-literal=pagerduty-api-key="your-api-key"

Helm Configuration for PagerDuty

env:
  - name: PAGERDUTY_API_KEY
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: pagerduty-api-key

# Enable PagerDuty integration
integrations:
  pagerduty:
    enabled: true
    update_incidents: true
    add_notes: true

PagerDuty Integration Modes

OpsGenie Integration

Investigating OpsGenie Alerts

# Investigate open alerts
holmes investigate opsgenie \
  --opsgenie-api-key $OPSGENIE_API_KEY

# Update alerts with findings
holmes investigate opsgenie \
  --opsgenie-api-key $OPSGENIE_API_KEY \
  --update

# Filter by priority
holmes investigate opsgenie \
  --opsgenie-api-key $OPSGENIE_API_KEY \
  --priority P1,P2

OpsGenie API Key Setup

1. Go to OpsGenie > Settings > API key management 2. Create a new API key with appropriate permissions 3. Store in Kubernetes secret

env:
  - name: OPSGENIE_API_KEY
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: opsgenie-api-key

AlertManager Integration

Webhook Receiver Setup

Configure AlertManager to send alerts to HolmesGPT for automatic investigation.

AlertManager Configuration:

# alertmanager.yml
route:
  receiver: default
  routes:
    - match:
        severity: critical
      receiver: holmesgpt
      continue: true

receivers:
  - name: default
    # Your default receiver config

  - name: holmesgpt
    webhook_configs:
      - url: 'http://holmesgpt-holmes.holmesgpt.svc.cluster.local/api/investigate'
        send_resolved: false
        max_alerts: 10

Investigating AlertManager Alerts

# Basic investigation
holmes investigate alertmanager \
  --alertmanager-url http://alertmanager:9093

# With authentication
holmes investigate alertmanager \
  --alertmanager-url http://alertmanager:9093 \
  --alertmanager-username admin \
  --alertmanager-password $ALERTMANAGER_PASSWORD

# Filter by alert name
holmes investigate alertmanager \
  --alertmanager-url http://alertmanager:9093 \
  --alertname "KubePodCrashLooping"

# With custom runbook
holmes investigate alertmanager \
  --alertmanager-url http://alertmanager:9093 \
  -r ~/runbooks/kubernetes.yaml

Jira Integration

Investigating Jira Issues

# Investigate issues in a project
holmes investigate jira \
  --jira-url https://company.atlassian.net \
  --jira-username user@company.com \
  --jira-api-token $JIRA_TOKEN \
  --jira-project OPS

# Filter by JQL
holmes investigate jira \
  --jira-url https://company.atlassian.net \
  --jira-username user@company.com \
  --jira-api-token $JIRA_TOKEN \
  --jira-jql "project = OPS AND status = Open AND labels = incident"

Jira API Token Setup

1. Go to id.atlassian.com/manage-profile/security/api-tokens 2. Create a new API token 3. Store in secret

env:
  - name: JIRA_URL
    value: "https://company.atlassian.net"
  - name: JIRA_USERNAME
    value: "user@company.com"
  - name: JIRA_API_TOKEN
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: jira-api-token

GitHub Integration

Investigating GitHub Issues

# Investigate issues in a repository
holmes investigate github \
  --github-token $GITHUB_TOKEN \
  --github-repo owner/repo

# Filter by labels
holmes investigate github \
  --github-token $GITHUB_TOKEN \
  --github-repo owner/repo \
  --github-labels "bug,priority:high"

# Update issues with analysis
holmes investigate github \
  --github-token $GITHUB_TOKEN \
  --github-repo owner/repo \
  --update

GitHub Token Permissions

Required permissions for the GitHub token:

• repo - Full repository access (for private repos)
• public_repo - Public repository access only
• read:org - Read organization info (optional)

Webhook Configuration

Generic Webhook Output

Send HolmesGPT analysis to any webhook endpoint.

# Send to custom webhook
holmes ask "analyze cluster health" \
  --destination webhook \
  --webhook-url https://your-service.com/api/holmes \
  --webhook-headers "Authorization: Bearer $TOKEN"

Webhook Payload Structure

{
  "timestamp": "2024-01-15T10:30:00Z",
  "source": "holmesgpt",
  "type": "investigation",
  "data": {
    "query": "why is payment-service crashing?",
    "analysis": "Full analysis text...",
    "sections": {
      "alert_explanation": "...",
      "key_findings": "...",
      "root_causes": "...",
      "next_steps": "..."
    },
    "model": "sonnet",
    "token_count": 1500
  }
}

Webhook Helm Configuration

webhooks:
  - name: custom-webhook
    url: "https://your-service.com/api/alerts"
    headers:
      Authorization: "Bearer ${WEBHOOK_TOKEN}"
      Content-Type: "application/json"
    events:
      - investigation_complete
      - critical_alert

CI/CD Integration

GitHub Actions

name: Investigate Deployment Failure

on:
  workflow_run:
    workflows: ["Deploy"]
    types: [completed]

jobs:
  investigate:
    if: ${{ github.event.workflow_run.conclusion == 'failure' }}
    runs-on: ubuntu-latest
    steps:
      - name: Install HolmesGPT
        run: pip install holmesgpt

      - name: Configure kubectl
        uses: azure/k8s-set-context@v3
        with:
          kubeconfig: ${{ secrets.KUBECONFIG }}

      - name: Investigate Failure
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
        run: |
          holmes ask "analyze deployment failure for ${{ github.repository }}" \
            --destination slack \
            --slack-token $SLACK_BOT_TOKEN \
            --slack-channel "#deployments"

GitLab CI

investigate_failure:
  stage: post-deploy
  when: on_failure
  image: python:3.11
  script:
    - pip install holmesgpt
    - |
      holmes ask "why did deployment fail for $CI_PROJECT_NAME?" \
        --destination slack \
        --slack-token $SLACK_BOT_TOKEN \
        --slack-channel "#deployments"
  only:
    - main

Azure DevOps

- task: Bash@3
  displayName: 'Investigate Failure'
  condition: failed()
  inputs:
    targetType: 'inline'
    script: |
      pip install holmesgpt
      export ANTHROPIC_API_KEY=$(ANTHROPIC_API_KEY)
      holmes ask "analyze deployment failure for $(Build.Repository.Name)" \
        --destination slack \
        --slack-token $(SLACK_BOT_TOKEN) \
        --slack-channel "#azure-deployments"

Robusta Integration

HolmesGPT integrates seamlessly with Robusta for automated alert enrichment.

Robusta Playbook Configuration

# robusta_playbook.yaml
customPlaybooks:
  - triggers:
      - on_prometheus_alert:
          alert_name: KubePodCrashLooping
    actions:
      - holmes_investigate:
          model: sonnet
          add_to_timeline: true

Environment Variables for Robusta

env:
  - name: ROBUSTA_ACCOUNT_ID
    valueFrom:
      secretKeyRef:
        name: robusta-secrets
        key: account-id
  - name: ROBUSTA_SIGNING_KEY
    valueFrom:
      secretKeyRef:
        name: robusta-secrets
        key: signing-key

Best Practices

Secret Management

1. Never hardcode credentials - Use Kubernetes Secrets or external secret managers 2. Rotate tokens regularly - Set up token rotation policies 3. Use minimal permissions - Grant only required access levels 4. Audit access - Monitor and log integration usage

Rate Limiting Considerations

Error Handling

# Retry with exponential backoff
RETRY_COUNT=0
MAX_RETRIES=3

until holmes investigate pagerduty --pagerduty-api-key $KEY \
  || [ $RETRY_COUNT -eq $MAX_RETRIES ]; do
  RETRY_COUNT=$((RETRY_COUNT + 1))
  SLEEP_TIME=$((2 ** RETRY_COUNT))
  echo "Retry $RETRY_COUNT in $SLEEP_TIME seconds..."
  sleep $SLEEP_TIME
done

Multi-Integration Setup

# Complete multi-integration values.yaml
env:
  - name: ANTHROPIC_API_KEY
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: anthropic-api-key
  - name: SLACK_BOT_TOKEN
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: slack-bot-token
  - name: PAGERDUTY_API_KEY
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: pagerduty-api-key
  - name: GITHUB_TOKEN
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: github-token
  - name: ALERTMANAGER_URL
    value: "http://alertmanager.monitoring.svc.cluster.local:9093"

integrations:
  slack:
    enabled: true
    default_channel: "#holmesgpt"
  pagerduty:
    enabled: true
    update_incidents: true
  alertmanager:
    enabled: true
    webhook: true
  github:
    enabled: true

HolmesGPT Troubleshooting Guide

Common issues and solutions for HolmesGPT deployments.

Common Issues

1. API Key Not Found

Symptoms:

• "API key not found" error
• Authentication failures
• Empty responses

Solutions:

# Verify environment variable is set
echo $ANTHROPIC_API_KEY
echo $OPENAI_API_KEY

# Set inline for testing
ANTHROPIC_API_KEY="sk-ant-..." holmes ask "test"

# Check in Kubernetes
kubectl get secret holmesgpt-secrets -n holmesgpt -o yaml

# Verify secret is mounted
kubectl exec -n holmesgpt deployment/holmesgpt-holmes -- env | grep API_KEY

2. Kubernetes Access Issues

Symptoms:

• "pods is forbidden" error
• Empty resource lists
• RBAC permission denied

Solutions:

# Check service account permissions
kubectl auth can-i get pods \
  --as=system:serviceaccount:holmesgpt:holmesgpt-holmes

kubectl auth can-i list events \
  --as=system:serviceaccount:holmesgpt:holmesgpt-holmes

# Verify ClusterRole
kubectl get clusterrole holmesgpt-holmes -o yaml

# Verify ClusterRoleBinding
kubectl get clusterrolebinding holmesgpt-holmes -o yaml

# Check if service account exists
kubectl get sa -n holmesgpt

Fix RBAC:

# Add to Helm values
createServiceAccount: true
customClusterRoleRules:
  - apiGroups: [""]
    resources: ["pods", "pods/log", "services", "events", "nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets", "statefulsets", "daemonsets"]
    verbs: ["get", "list", "watch"]

3. Data Truncation

Symptoms:

• Incomplete analysis
• Missing important information
• "Data truncated" warnings

Solutions:

# Be more specific in queries
# Bad:
holmes ask "what's happening in the cluster?"

# Good:
holmes ask "what pods are crashing in production namespace?"

# Use time ranges
holmes ask "show errors from the last hour in payment-service"

# Target specific components
holmes ask "analyze payment-service deployment in production"

Configure output limits:

# In Helm values
env:
  - name: HOLMES_MAX_OUTPUT_LENGTH
    value: "50000"

4. Missing Data Access

Symptoms:

• "Cannot connect to Prometheus"
• Empty metrics
• Toolset connection failures

Solutions:

# Verify Prometheus connectivity
kubectl exec -n holmesgpt deployment/holmesgpt-holmes -- \
  curl -s http://prometheus-server.monitoring.svc.cluster.local/api/v1/query?query=up

# Check environment variables
kubectl exec -n holmesgpt deployment/holmesgpt-holmes -- env | grep PROMETHEUS

# Test from within cluster
kubectl run test-curl --rm -it --restart=Never --image=curlimages/curl -- \
  curl -s http://prometheus-server.monitoring.svc.cluster.local/api/v1/status/config

Fix connectivity:

# In Helm values
env:
  - name: PROMETHEUS_URL
    value: "http://prometheus-server.monitoring.svc.cluster.local"

5. Model Limitations

Symptoms:

• Poor analysis quality
• Incorrect conclusions
• Slow responses

Solutions:

# Use better model
holmes ask "complex issue" --model opus

# Check available models
cat ~/.holmes/config.yaml

# Use recommended models
# Best: Claude Sonnet 4.0/4.5, Claude Opus
# Good: GPT-4.1, GPT-4o
# Basic: GPT-3.5 (not recommended for complex issues)

6. Ineffective Prompts

Symptoms:

• Generic or unhelpful responses
• AI asks for more information
• Irrelevant suggestions

Solutions:

# Bad queries:
holmes ask "why is my pod not working?"
holmes ask "what's wrong?"
holmes ask "check everything"

# Good queries:
holmes ask "why is payment-service pod restarting in production namespace?"
holmes ask "analyze CrashLoopBackOff for api-gateway deployment"
holmes ask "investigate high memory usage in monitoring namespace pods"

Query improvement tips:

• Include namespace
• Specify deployment/pod name
• Mention the specific symptom
• Provide time context if relevant

7. Helm Installation Failures

Symptoms:

• helm install errors
• Missing resources
• Configuration not applied

Solutions:

# Debug installation
helm install holmesgpt robusta/holmes -f values.yaml --debug --dry-run

# Check values syntax
helm lint robusta/holmes -f values.yaml

# Verify repo is updated
helm repo update

# Check for existing resources
kubectl get all -n holmesgpt

# Clean up failed installation
helm uninstall holmesgpt -n holmesgpt
kubectl delete namespace holmesgpt

8. Pod Not Starting

Symptoms:

• Pod stuck in Pending/CrashLoopBackOff
• ImagePullBackOff errors
• OOMKilled

Solutions:

# Check pod status
kubectl describe pod -n holmesgpt -l app.kubernetes.io/name=holmes

# Check events
kubectl get events -n holmesgpt --sort-by='.lastTimestamp'

# Check logs
kubectl logs -n holmesgpt deployment/holmesgpt-holmes --previous

# Check resources
kubectl top pod -n holmesgpt

Common fixes:

# Increase resources
resources:
  requests:
    memory: "2048Mi"
    cpu: "200m"
  limits:
    memory: "4096Mi"

# Fix image pull
imagePullSecrets:
  - name: registry-credentials

9. Slow Responses

Symptoms:

• Queries take too long
• Timeouts
• Incomplete results

Solutions:

# Use faster model for simple queries
holmes ask "list pods" --model gpt4o

# Reduce scope
holmes ask "check payment-service only"

# Disable unnecessary toolsets

Optimize configuration:

# Disable unused toolsets
toolsets:
  internet:
    enabled: false
  confluence:
    enabled: false

# Use faster models for routine checks
modelList:
  fast:
    model: openai/gpt-4o-mini
    temperature: 0

10. Interactive Mode Issues

Symptoms:

• Commands not recognized
• Context lost
• Cannot execute /run commands

Solutions:

# Ensure interactive mode is started
holmes ask "question" --interactive

# Clear corrupted context
/clear

# Check available commands
/help

# If /run doesn't work, verify shell access
# Some deployments restrict shell execution

Diagnostic Commands

Check HolmesGPT Status

# Pod status
kubectl get pods -n holmesgpt -o wide

# Pod logs
kubectl logs -n holmesgpt deployment/holmesgpt-holmes -f

# Pod events
kubectl describe pod -n holmesgpt -l app.kubernetes.io/name=holmes

# Resource usage
kubectl top pod -n holmesgpt

Test Connectivity

# Test API endpoint (after port-forward)
kubectl port-forward -n holmesgpt svc/holmesgpt-holmes 8080:80 &
curl -X POST http://localhost:8080/api/chat \
  -H "Content-Type: application/json" \
  -d '{"ask": "test", "model": "sonnet"}'

# Test from within cluster
kubectl run test-holmes --rm -it --restart=Never --image=curlimages/curl -- \
  curl -X POST http://holmesgpt-holmes.holmesgpt.svc.cluster.local/api/chat \
  -H "Content-Type: application/json" \
  -d '{"ask": "test", "model": "sonnet"}'

Verify Configuration

# Check Helm values applied
helm get values holmesgpt -n holmesgpt

# Check configmaps
kubectl get configmap -n holmesgpt -o yaml

# Check secrets (without revealing values)
kubectl get secrets -n holmesgpt

# Check environment in pod
kubectl exec -n holmesgpt deployment/holmesgpt-holmes -- env | sort

Getting Help

1. Slack Community: Cloud Native Slack #holmesgpt channel 2. GitHub Issues: <https://github.com/robusta-dev/holmesgpt/issues> 3. Documentation: <https://holmesgpt.dev/> 4. DeepWiki AI Support: Built-in support assistant

Reporting Issues

Include in bug reports:

• HolmesGPT version
• Deployment method (CLI/Helm/Docker)
• AI provider and model
• Error messages and logs
• Steps to reproduce
• Expected vs actual behavior