Robusta Dev

RobustaDev

Comprehensive guide for Robusta - the SRE agent that transforms Kubernetes alerts into actionable insights using playbooks, AI investigation, and automated remediation.

Quick Reference

Workflow Routing

Installation Quick Start

Prerequisites

• Kubernetes cluster
• Helm 3.x installed
• kubectl configured

All-in-One Installation (Robusta + Prometheus)

# Generate configuration
pipx run robusta-cli gen-config --enable-prometheus-stack

# Or using Docker
curl -fsSL -o robusta https://docs.robusta.dev/master/_static/robusta
chmod +x robusta
./robusta gen-config --enable-prometheus-stack

# Install via Helm
helm repo add robusta https://robusta-charts.storage.googleapis.com
helm repo update
helm install robusta robusta/robusta \
  -f ./generated_values.yaml \
  --set clusterName=<YOUR_CLUSTER_NAME>

# Verify installation
kubectl get pods -A | grep robusta

Standalone Installation (Existing Prometheus)

pipx run robusta-cli gen-config
helm install robusta robusta/robusta -f ./generated_values.yaml

Playbook Structure

# Example playbook in generated_values.yaml
customPlaybooks:
  - triggers:
      - on_prometheus_alert:
          alert_name: KubePodCrashLooping
    actions:
      - logs_enricher: {}
      - pod_events_enricher: {}
    sinks:
      - slack

Trigger Types

Common Actions

Sink Configuration

Slack

sinksConfig:
  - slack_sink:
      name: main_slack
      slack_channel: alerts
      api_key: xoxb-your-token

Microsoft Teams

sinksConfig:
  - ms_teams_sink:
      name: teams_alerts
      webhook_url: https://outlook.office.com/webhook/...

PagerDuty

sinksConfig:
  - pagerduty_sink:
      name: pagerduty
      api_key: your-integration-key

Webhook (Generic)

sinksConfig:
  - webhook_sink:
      name: custom_webhook
      url: https://your-endpoint.com/alerts

Examples

Example 1: Install Robusta with Prometheus

User: "Install Robusta on my AKS cluster"
-> Generate config with gen-config --enable-prometheus-stack
-> Add Helm repo and install with cluster name
-> Verify pods are running

Example 2: Create crash loop enrichment playbook

User: "Add pod logs to CrashLoopBackOff alerts"
-> Create playbook with on_prometheus_alert trigger
-> Add logs_enricher and pod_events_enricher actions
-> Configure Slack sink for notifications

Example 3: Configure Slack notifications

User: "Send Robusta alerts to #k8s-alerts Slack channel"
-> Add slack_sink to sinksConfig
-> Set channel name and API key
-> Optionally filter by severity or namespace

Example 4: Debug missing alerts

User: "Robusta isn't sending alerts to Slack"
-> Check robusta-runner pod logs
-> Verify sink configuration in generated_values.yaml
-> Test with manual trigger: robusta playbooks trigger

Key Concepts

Alert Flow

Prometheus Alert -> AlertManager -> Robusta -> Playbook -> Actions -> Sinks

Playbook Components

1. Triggers - What events activate the playbook 2. Actions - What to do when triggered (enrich, remediate) 3. Sinks - Where to send the result

Smart Grouping

Robusta groups related alerts using Slack threads to reduce notification spam.

AI Investigation (HolmesGPT)

Optional AI-powered root cause analysis available with Robusta Pro or self-hosted HolmesGPT.

Reference Documentation

• references/Installation.md - Detailed installation guide
• references/Playbooks.md - Complete playbook reference
• references/Triggers.md - All trigger types
• references/Actions.md - Available actions
• references/Sinks.md - Sink configuration
• references/Troubleshooting.md - Common issues and fixes

External Resources

• Official Docs
• GitHub
• Robusta Platform (SaaS UI)

---

Gotchas

• `customPlaybooks` order matters — first match wins for the same trigger: A catch-all on_prometheus_alert without alert_name: filter at the top swallows every alert, and the specific-alert playbooks below never fire. Always put specific-alert playbooks first, generic last.
• `api_key` in `slack_sink` is the bot token (xoxb-), not the webhook URL: Pasting a webhook URL there silently fails — Robusta logs "auth failed" once and stops. Use slack_channel + api_key for bot mode, or switch to a different sink type for webhook delivery.
• `delete_pod` auto-remediation has no rate limit: A misconfigured CrashLoopBackOff playbook with delete_pod action against a flapping deployment will hot-loop pod deletions and exhaust pod IPs in the subnet. Always pair with on_prometheus_alert + a firing filter, not pod-state triggers.
• `gen-config --enable-prometheus-stack` installs a full kube-prometheus-stack: Running it on a cluster that already has Prometheus produces two Prometheus instances scraping each other. Use plain gen-config (no flag) when integrating with existing monitoring.
• `ms_teams_sink` webhook URLs were deprecated in 2024 — connectors retire: Existing webhook URLs keep working briefly then return 410. Migrate to Power Automate workflows or the new Teams sink format before the deprecation cliff hits.
• Smart Grouping uses Slack threads — breaks if the channel has threading disabled or webhook-only: Alerts post but the "group" collapses into individual messages, creating notification spam exactly when the feature is supposed to prevent it. Verify channel threading is enabled and you're using bot mode, not webhooks.

Actions Reference

Complete reference for Robusta playbook actions.

Log Actions

logs_enricher

Adds container logs to alert.

actions:
  - logs_enricher:
      container_name: main      # Optional: specific container
      previous_logs: true       # Include previous container logs
      warn_on_missing_label: true
      regex_replacer_patterns:  # Redact sensitive data
        - pattern: "password=.*"
          replacement: "password=REDACTED"

pod_events_enricher

Adds Kubernetes events for the pod.

actions:
  - pod_events_enricher:
      max_events: 10

Graph/Metrics Actions

pod_graph_enricher

Generates resource usage graphs for pod.

actions:
  - pod_graph_enricher:
      resource_type: Memory    # Memory or CPU
      display_limits: true     # Show limits line

cpu_graph_enricher

CPU-specific graph enricher.

actions:
  - cpu_graph_enricher:
      resource_type: CPU

node_cpu_enricher

Node-level CPU information.

actions:
  - node_cpu_enricher: {}

node_memory_enricher

Node-level memory information.

actions:
  - node_memory_enricher: {}

custom_graph_enricher

Custom PromQL query graph.

actions:
  - custom_graph_enricher:
      graph_title: "Request Rate"
      promql_query: "rate(http_requests_total{pod='$pod'}[5m])"
      graph_duration_minutes: 60

prometheus_enricher

Execute custom PromQL query.

actions:
  - prometheus_enricher:
      query: "sum(rate(http_requests_total[5m]))"

OOM Actions

pod_oom_killer_enricher

Information about OOM kill events.

actions:
  - pod_oom_killer_enricher: {}

oom_killer_graph_enricher

Memory graph with OOM events marked.

actions:
  - oom_killer_graph_enricher:
      delay_graph_s: 30  # Wait before generating

Status Actions

pod_status_enricher

Pod status and conditions.

actions:
  - pod_status_enricher: {}

deployment_status_enricher

Deployment status details.

actions:
  - deployment_status_enricher: {}

node_status_enricher

Node status and conditions.

actions:
  - node_status_enricher: {}

node_running_pods_enricher

List pods running on node.

actions:
  - node_running_pods_enricher:
      max_pods: 20

node_allocatable_resources_enricher

Node allocatable resources.

actions:
  - node_allocatable_resources_enricher: {}

cluster_status_enricher

Overall cluster status.

actions:
  - cluster_status_enricher: {}

related_pods_enricher

Find related pods (same deployment, etc.).

actions:
  - related_pods_enricher: {}

Remediation Actions

delete_pod

Delete the affected pod.

actions:
  - delete_pod:
      force: false          # Force delete
      grace_period: 30      # Seconds

node_bash_enricher

Run bash command on node.

actions:
  - node_bash_enricher:
      bash_command: "df -h"

pod_bash_enricher

Run command in pod container.

actions:
  - pod_bash_enricher:
      bash_command: "ls -la /app"

restart_loop_reporter

Report restart loop with details.

actions:
  - restart_loop_reporter:
      restart_reason: OOMKilled

Report Actions

event_report

Generate event report.

actions:
  - event_report:
      title: "Kubernetes Event"

create_finding

Create custom finding/alert.

actions:
  - create_finding:
      title: "Custom Alert"
      aggregation_key: "custom-alert"
      severity: HIGH
      description: "Something happened"

External Integration Actions

send_to_webhook

Send data to external webhook.

actions:
  - send_to_webhook:
      url: "https://my-api.com/alerts"
      headers:
        Authorization: "Bearer token"

jira_issue_reporter

Create Jira ticket.

actions:
  - jira_issue_reporter:
      project_key: "OPS"
      issue_type: "Bug"

Silencing Actions

silence_alert

Silence an alert temporarily.

actions:
  - silence_alert:
      duration_minutes: 60
      comment: "Known issue, investigating"

Customization

Adding Custom Actions

Custom actions can be defined via Python:

# In a custom playbook file
from robusta.api import *

@action
def my_custom_action(event: PodEvent):
    pod = event.get_pod()
    event.add_enrichment([
        MarkdownBlock(f"Pod {pod.metadata.name} info")
    ])

Load custom actions:

# In generated_values.yaml
playbookRepos:
  - url: https://github.com/my-org/my-playbooks
    key: my-actions

Action Parameters

All actions support these common parameters:

Installation Reference

Detailed installation reference for Robusta on Kubernetes.

Prerequisites

Helm Chart

# Add repository
helm repo add robusta https://robusta-charts.storage.googleapis.com
helm repo update

Configuration Generation

Using pipx (Recommended)

pipx run robusta-cli gen-config --enable-prometheus-stack

Using Docker

curl -fsSL -o robusta https://docs.robusta.dev/master/_static/robusta
chmod +x robusta
./robusta gen-config --enable-prometheus-stack

Configuration Options

Helm Values

Core Values

clusterName: my-cluster
globalConfig:
  signing_key: <generated>
  account_id: <generated>

sinksConfig:
  - slack_sink:
      name: main_slack
      slack_channel: alerts
      api_key: xoxb-...

runner:
  resources:
    requests:
      memory: 512Mi
      cpu: 250m
    limits:
      memory: 1Gi
      cpu: 1000m

Platform-Specific Values

GKE Autopilot

kube-prometheus-stack:
  coreDns:
    enabled: false
  kubeControllerManager:
    enabled: false
  kubeEtcd:
    enabled: false
  kubeScheduler:
    enabled: false
  kubeProxy:
    enabled: false

EKS

enablePlatformPlaybooks: true
# Requires EBS CSI driver installed

OpenShift

openshift:
  enabled: true
  createScc: true

Small Clusters

isSmallCluster: true
runner:
  resources:
    limits:
      memory: 256Mi
      cpu: 250m

Installation Commands

Standard Install

helm install robusta robusta/robusta \
  -f ./generated_values.yaml \
  --set clusterName=my-cluster \
  --namespace robusta \
  --create-namespace

Upgrade

helm repo update
helm upgrade robusta robusta/robusta \
  -f ./generated_values.yaml \
  --namespace robusta

Uninstall

helm uninstall robusta -n robusta
kubectl delete namespace robusta

Verification

# Check pods
kubectl get pods -n robusta

# Check runner logs
kubectl logs -n robusta deploy/robusta-runner --tail=50

# List loaded playbooks
kubectl exec -n robusta deploy/robusta-runner -- robusta playbooks list

# Send test notification
kubectl exec -n robusta deploy/robusta-runner -- robusta demo

Expected Pods

Playbooks Reference

Complete reference for Robusta playbook configuration.

Playbook Structure

customPlaybooks:
  - triggers:
      - <trigger_type>:
          <trigger_parameters>
    actions:
      - <action_name>:
          <action_parameters>
    sinks:
      - <sink_name>
    scope:
      include:
        - <filter>
      exclude:
        - <filter>

Built-in Playbooks

Robusta includes default playbooks for common alerts:

Playbook Priority

Playbooks are evaluated in order. First matching playbook wins. Use stop: true to prevent further processing:

customPlaybooks:
  - triggers:
      - on_prometheus_alert:
          alert_name: CriticalAlert
    actions:
      - logs_enricher: {}
    stop: true  # Don't process other playbooks

Scope Filtering

By Namespace

scope:
  include:
    - namespace: production
    - namespace: staging
  exclude:
    - namespace: kube-system

By Labels

scope:
  include:
    - labels:
        team: backend
        env: prod

By Alert Name (Regex)

scope:
  include:
    - name: KubePod.*
    - name: .*Memory.*

By Severity

triggers:
  - on_prometheus_alert:
      alert_name: ".*"
      alert_severity: critical

Common Playbook Patterns

Crash Loop Enrichment

customPlaybooks:
  - triggers:
      - on_prometheus_alert:
          alert_name: KubePodCrashLooping
    actions:
      - logs_enricher:
          previous_logs: true
      - pod_events_enricher: {}
      - pod_graph_enricher:
          resource_type: Memory

Resource Monitoring

customPlaybooks:
  - triggers:
      - on_prometheus_alert:
          alert_name: CPUThrottlingHigh
    actions:
      - cpu_graph_enricher:
          resource_type: CPU
      - node_cpu_enricher: {}
      - prometheus_enricher:
          query: "container_cpu_usage_seconds_total{pod='$pod'}"

Scheduled Reports

customPlaybooks:
  - triggers:
      - on_schedule:
          cron_schedule_repeat:
            cron_expression: "0 9 * * 1"  # Monday 9 AM
    actions:
      - cluster_status_enricher: {}
    sinks:
      - slack

Auto-Remediation

customPlaybooks:
  - triggers:
      - on_prometheus_alert:
          alert_name: KubePodCrashLooping
          alert_severity: critical
    actions:
      - logs_enricher: {}
      - delete_pod:
          force: false
          grace_period: 30

OOM Detection

customPlaybooks:
  - triggers:
      - on_pod_oom_killed: {}
    actions:
      - pod_oom_killer_enricher: {}
      - oom_killer_graph_enricher: {}
      - custom_graph_enricher:
          graph_title: "Memory Timeline"
          promql_query: "container_memory_working_set_bytes{pod='$pod'}"

Playbook Variables

Available variables in playbook actions:

Global Configuration

globalConfig:
  alertThrottleMinutes: 5      # Deduplicate alerts
  finding_processor_workers: 4  # Parallel processing
  timezone: "America/New_York"  # Timestamp timezone

Disabling Built-in Playbooks

builtinPlaybooks:
  - triggers:
      - on_prometheus_alert:
          alert_name: Watchdog
    actions: []  # Disable by setting empty actions

Sinks Reference

Complete reference for Robusta notification sinks.

Sink Configuration

All sinks are configured in generated_values.yaml:

sinksConfig:
  - <sink_type>:
      name: <unique_identifier>
      <sink_parameters>
      scope:
        include:
          - <filter>
        exclude:
          - <filter>

Communication Platforms

Slack

sinksConfig:
  - slack_sink:
      name: main_slack
      slack_channel: k8s-alerts
      api_key: xoxb-your-bot-token
      thread_messages: true       # Group in threads
      message_format: standard    # standard, minimal, detailed

Slack Bot Setup: 1. Create app at api.slack.com/apps 2. Add OAuth scopes: chat:write, files:write 3. Install to workspace 4. Invite bot to channel

Microsoft Teams

sinksConfig:
  - ms_teams_sink:
      name: teams_alerts
      webhook_url: https://outlook.office.com/webhook/...

Teams Webhook Setup: 1. Channel → ... → Connectors 2. Add "Incoming Webhook" 3. Copy webhook URL

Discord

sinksConfig:
  - discord_sink:
      name: discord_alerts
      url: https://discord.com/api/webhooks/...

Google Chat

sinksConfig:
  - google_chat_sink:
      name: gchat
      webhook_url: https://chat.googleapis.com/v1/spaces/...

Telegram

sinksConfig:
  - telegram_sink:
      name: telegram
      bot_token: "123456:ABC-DEF..."
      chat_id: "-100123456789"

Incident Management

PagerDuty

sinksConfig:
  - pagerduty_sink:
      name: pagerduty
      api_key: your-integration-key
      routing_key: your-routing-key  # Optional

OpsGenie

sinksConfig:
  - opsgenie_sink:
      name: opsgenie
      api_key: your-api-key
      teams:
        - platform-team
      priority: P2  # P1-P5

VictorOps (Splunk On-Call)

sinksConfig:
  - victorops_sink:
      name: victorops
      url: https://alert.victorops.com/integrations/generic/...
      routing_key: your-routing-key

Incident.io

sinksConfig:
  - incidentio_sink:
      name: incidentio
      api_key: your-api-key

Ticketing Systems

Jira

sinksConfig:
  - jira_sink:
      name: jira_tickets
      url: https://your-instance.atlassian.net
      username: your-email@company.com
      api_key: your-api-token
      project_name: OPS
      issue_type: Bug
      dedupe: true  # Prevent duplicates

ServiceNow

sinksConfig:
  - servicenow_sink:
      name: servicenow
      instance: your-instance
      username: admin
      password: your-password
      table: incident

Data/Analytics

Kafka

sinksConfig:
  - kafka_sink:
      name: kafka_alerts
      kafka_url: kafka-bootstrap:9092
      topic: robusta-alerts
      username: user           # Optional
      password: pass           # Optional
      security_protocol: SASL_SSL

DataDog

sinksConfig:
  - datadog_sink:
      name: datadog
      api_key: your-datadog-api-key

Elasticsearch

sinksConfig:
  - elasticsearch_sink:
      name: elasticsearch
      url: https://elasticsearch:9200
      index: robusta-alerts
      username: elastic
      password: your-password

Webhook (Generic)

sinksConfig:
  - webhook_sink:
      name: custom_webhook
      url: https://your-endpoint.com/alerts
      headers:
        Authorization: "Bearer token"
        Content-Type: "application/json"

Robusta Platform

Robusta UI (SaaS)

sinksConfig:
  - robusta_sink:
      name: robusta_ui
      token: <generated-during-setup>

Debugging

File Sink

sinksConfig:
  - file_sink:
      name: debug_file
      file_name: /tmp/robusta-alerts.log

SMTP/Email

sinksConfig:
  - mail_sink:
      name: email_alerts
      mailto: alerts@company.com
      smtp_host: smtp.company.com
      smtp_port: 587
      username: alerts@company.com
      password: your-password

Sink Filtering

By Namespace

sinksConfig:
  - slack_sink:
      name: prod_slack
      slack_channel: prod-alerts
      api_key: xoxb-...
      scope:
        include:
          - namespace: production

By Severity

sinksConfig:
  - pagerduty_sink:
      name: pagerduty_critical
      api_key: ...
      scope:
        include:
          - labels:
              severity: critical

By Alert Name

sinksConfig:
  - slack_sink:
      name: slack_pod_issues
      slack_channel: pod-alerts
      api_key: ...
      scope:
        include:
          - name: KubePod.*

Exclude Namespaces

sinksConfig:
  - slack_sink:
      name: main_slack
      slack_channel: alerts
      api_key: ...
      scope:
        exclude:
          - namespace: kube-system
          - namespace: monitoring

Multiple Sinks

Configure multiple sinks for different purposes:

sinksConfig:
  # All alerts to main channel
  - slack_sink:
      name: all_alerts
      slack_channel: k8s-alerts
      api_key: xoxb-...

  # Critical alerts to pager
  - pagerduty_sink:
      name: critical_pager
      api_key: ...
      scope:
        include:
          - labels:
              severity: critical

  # Production issues to dedicated channel
  - slack_sink:
      name: prod_alerts
      slack_channel: prod-critical
      api_key: xoxb-...
      scope:
        include:
          - namespace: production

  # Archive all to Elasticsearch
  - elasticsearch_sink:
      name: archive
      url: https://es:9200
      index: robusta-alerts

Testing Sinks

# Test all sinks
kubectl exec -n robusta deploy/robusta-runner -- robusta demo

# Check sink status in logs
kubectl logs -n robusta deploy/robusta-runner | grep -i sink

Triggers Reference

Complete reference for Robusta playbook triggers.

Prometheus/AlertManager Triggers

on_prometheus_alert

Fires when Prometheus AlertManager sends an alert.

triggers:
  - on_prometheus_alert:
      alert_name: KubePodCrashLooping  # Exact or regex
      alert_severity: critical          # Optional: info, warning, critical
      status: firing                    # Optional: firing, resolved

Parameters:

Kubernetes Event Triggers

on_kubernetes_warning_event

Fires on Kubernetes warning events.

triggers:
  - on_kubernetes_warning_event:
      include:
        - FailedScheduling
        - BackOff
        - Unhealthy
      exclude:
        - TooManyPods

on_kubernetes_any_event

Fires on any Kubernetes event (warning or normal).

triggers:
  - on_kubernetes_any_event:
      include:
        - Pulling
        - Pulled

Pod Lifecycle Triggers

on_pod_create

Fires when a pod is created.

triggers:
  - on_pod_create:
      name_prefix: api-
      namespace_prefix: production

on_pod_update

Fires when a pod is updated.

triggers:
  - on_pod_update:
      operation: Update

on_pod_delete

Fires when a pod is deleted.

triggers:
  - on_pod_delete: {}

on_pod_oom_killed

Fires when a container is OOM killed.

triggers:
  - on_pod_oom_killed:
      rate_limit: 1800  # Seconds between triggers

on_container_state_change

Fires on container state transitions.

triggers:
  - on_container_state_change:
      new_state: CrashLoopBackOff

Deployment Triggers

on_deployment_create

triggers:
  - on_deployment_create: {}

on_deployment_update

triggers:
  - on_deployment_update:
      operation: Update

on_deployment_delete

triggers:
  - on_deployment_delete: {}

on_deployment_replica_change

Fires when replica count changes.

triggers:
  - on_deployment_replica_change:
      increase: true
      decrease: true

Node Triggers

on_node_create

triggers:
  - on_node_create: {}

on_node_update

triggers:
  - on_node_update: {}

on_node_delete

triggers:
  - on_node_delete: {}

on_node_status_change

Fires when node status changes (Ready/NotReady).

triggers:
  - on_node_status_change:
      status: NotReady

Job Triggers

on_job_create

triggers:
  - on_job_create: {}

on_job_update

triggers:
  - on_job_update: {}

on_job_failure

Fires when a job fails.

triggers:
  - on_job_failure: {}

on_job_success

Fires when a job succeeds.

triggers:
  - on_job_success: {}

Schedule Triggers

on_schedule

Fires on a cron schedule.

triggers:
  - on_schedule:
      cron_schedule_repeat:
        cron_expression: "0 */6 * * *"  # Every 6 hours

Cron Format: minute hour day month weekday

Service/Ingress Triggers

on_service_create

triggers:
  - on_service_create: {}

on_ingress_create

triggers:
  - on_ingress_create: {}

HPA Triggers

on_horizontalpodautoscaler_update

triggers:
  - on_horizontalpodautoscaler_update: {}

Manual Triggers

on_manual_trigger

Can be triggered via API or CLI.

triggers:
  - on_manual_trigger:
      action_params:
        param1: value1

Trigger via CLI:

kubectl exec -n robusta deploy/robusta-runner -- \
  robusta playbooks trigger manual_trigger \
  --action-params '{"param1": "value1"}'

Trigger Filtering

All triggers support these common filters:

triggers:
  - on_pod_create:
      name_prefix: api-
      namespace_prefix: prod
      labels_selector: "app=backend,tier=api"

Troubleshooting Reference

Comprehensive troubleshooting guide for Robusta.

Quick Health Check

# Overall status
kubectl get pods -n robusta
kubectl get events -n robusta --sort-by='.lastTimestamp'

# Runner logs (main processor)
kubectl logs -n robusta deploy/robusta-runner --tail=100

# Forwarder logs (event collector)
kubectl logs -n robusta deploy/robusta-forwarder --tail=100

Common Issues

1. Pods Not Starting

Symptoms: Pending, CrashLoopBackOff, ImagePullBackOff

# Diagnose
kubectl describe pod -n robusta -l app=robusta-runner
kubectl get events -n robusta --sort-by='.lastTimestamp'

# Solutions

# Insufficient resources
helm upgrade robusta robusta/robusta \
  -f ./generated_values.yaml \
  --set isSmallCluster=true

# Image pull issues
kubectl get secret -n robusta | grep regcred

Resource Limits:

runner:
  resources:
    requests:
      memory: 256Mi
      cpu: 100m
    limits:
      memory: 512Mi
      cpu: 500m

2. Alerts Not Received

Symptoms: Prometheus alerts exist but Robusta doesn't process them

# Check AlertManager webhook config
kubectl get secret -n robusta alertmanager-robusta-kube-prometheus-alertmanager \
  -o jsonpath='{.data.alertmanager\.yaml}' | base64 -d

# Verify webhook receiver points to robusta-runner:5000

# Test AlertManager connectivity
kubectl exec -n robusta deploy/robusta-runner -- \
  curl -s http://alertmanager-operated:9093/api/v2/alerts

# Check for webhook delivery errors in AlertManager
kubectl logs -n robusta alertmanager-robusta-kube-prometheus-alertmanager-0

3. Slack Notifications Failing

Symptoms: Alerts processed but no Slack messages

# Check Slack configuration
kubectl logs -n robusta deploy/robusta-runner | grep -i slack

# Verify API key
kubectl get secret -n robusta robusta-runner-secret \
  -o jsonpath='{.data}' | base64 -d

# Test Slack API
kubectl exec -n robusta deploy/robusta-runner -- \
  curl -X POST https://slack.com/api/auth.test \
  -H "Authorization: Bearer xoxb-your-token"

Common Slack Issues:

4. Playbooks Not Loading

Symptoms: Custom playbooks not executing

# List loaded playbooks
kubectl exec -n robusta deploy/robusta-runner -- \
  robusta playbooks list

# Check for YAML errors
kubectl logs -n robusta deploy/robusta-runner | grep -i "playbook\|error\|yaml"

# Validate YAML locally
python3 -c "import yaml; yaml.safe_load(open('generated_values.yaml'))"

# Check ConfigMap
kubectl get configmap -n robusta robusta-runner-playbooks -o yaml

5. High Resource Usage

Symptoms: Runner pod consuming excessive CPU/memory

# Check usage
kubectl top pod -n robusta

# Check for alert storms
kubectl logs -n robusta deploy/robusta-runner | grep -c "Processing alert"

Solutions:

# Add rate limiting
globalConfig:
  alertThrottleMinutes: 5

# Reduce workers
runner:
  workers: 2

# Increase resources
runner:
  resources:
    limits:
      memory: 1Gi
      cpu: 1000m

6. Prometheus Stack Issues

Symptoms: Prometheus/Grafana pods failing

# Check Prometheus
kubectl get pods -n robusta -l app.kubernetes.io/name=prometheus
kubectl logs -n robusta prometheus-robusta-kube-prometheus-prometheus-0

# Check PVCs
kubectl get pvc -n robusta

# Check Grafana
kubectl logs -n robusta deploy/robusta-grafana

EKS Storage Issue:

# Ensure EBS CSI driver is installed
kubectl get csidrivers | grep ebs

7. Multi-Cluster Issues

Symptoms: Alerts from wrong cluster or missing cluster context

# Verify cluster name
helm get values robusta -n robusta | grep clusterName

# Update cluster name
helm upgrade robusta robusta/robusta \
  -f ./generated_values.yaml \
  --set clusterName=correct-cluster-name

8. Webhook Connectivity

Symptoms: External sinks not receiving data

# Test outbound connectivity
kubectl exec -n robusta deploy/robusta-runner -- \
  curl -I https://hooks.slack.com

# Check network policies
kubectl get networkpolicies -n robusta

# Test specific endpoints
kubectl exec -n robusta deploy/robusta-runner -- \
  curl -I https://api.pagerduty.com

Advanced Debugging

Enable Debug Logging

# In generated_values.yaml
runner:
  log_level: DEBUG

Access Runner Shell

kubectl exec -it -n robusta deploy/robusta-runner -- bash

# Inside container
robusta playbooks list
robusta sinks list
robusta version

Test Alert Processing

# Send test alert
kubectl exec -n robusta deploy/robusta-runner -- robusta demo

# Trigger specific playbook
kubectl exec -n robusta deploy/robusta-runner -- \
  robusta playbooks trigger prometheus_alert \
  --alert-name KubePodCrashLooping \
  --namespace default \
  --labels '{"severity":"warning","pod":"test-pod"}'

Inspect Internal State

# Check runner metrics
kubectl exec -n robusta deploy/robusta-runner -- \
  curl http://localhost:5000/metrics

# Check health endpoint
kubectl exec -n robusta deploy/robusta-runner -- \
  curl http://localhost:5000/health

Recovery Procedures

Reinstall Robusta

# Backup values
cp generated_values.yaml generated_values.yaml.bak

# Uninstall
helm uninstall robusta -n robusta

# Wait for cleanup
kubectl get pods -n robusta -w

# Reinstall
helm install robusta robusta/robusta \
  -f ./generated_values.yaml \
  --namespace robusta

Reset Configuration

# Regenerate config (keeps signing keys)
mv generated_values.yaml generated_values.yaml.old
pipx run robusta-cli gen-config --enable-prometheus-stack

# Merge old sink/playbook configs back
# Then reinstall

Force Pod Restart

kubectl rollout restart deployment -n robusta robusta-runner
kubectl rollout restart deployment -n robusta robusta-forwarder

Log Analysis

Common Log Patterns

Useful Log Commands

# Find errors
kubectl logs -n robusta deploy/robusta-runner | grep -i error

# Find specific alert processing
kubectl logs -n robusta deploy/robusta-runner | grep "KubePodCrashLooping"

# Watch logs in real-time
kubectl logs -n robusta deploy/robusta-runner -f

Getting Help

• Robusta Slack Community
• GitHub Issues
• Official Documentation

Configure Playbooks

Workflow for creating and managing Robusta playbooks.

Playbook Structure

customPlaybooks:
  - triggers:
      - <trigger_type>:
          <trigger_params>
    actions:
      - <action_name>:
          <action_params>
    sinks:
      - <sink_name>

Common Playbook Patterns

1. Enrich CrashLoopBackOff Alerts

customPlaybooks:
  - triggers:
      - on_prometheus_alert:
          alert_name: KubePodCrashLooping
    actions:
      - logs_enricher: {}
      - pod_events_enricher: {}
      - pod_graph_enricher:
          resource_type: Memory

2. Monitor High CPU Usage

customPlaybooks:
  - triggers:
      - on_prometheus_alert:
          alert_name: CPUThrottlingHigh
    actions:
      - cpu_graph_enricher:
          resource_type: CPU
      - node_cpu_enricher: {}

3. Node Not Ready Alerts

customPlaybooks:
  - triggers:
      - on_prometheus_alert:
          alert_name: KubeNodeNotReady
    actions:
      - node_status_enricher: {}
      - node_running_pods_enricher: {}
      - node_allocatable_resources_enricher: {}

4. Deployment Rollout Issues

customPlaybooks:
  - triggers:
      - on_prometheus_alert:
          alert_name: KubeDeploymentReplicasMismatch
    actions:
      - deployment_status_enricher: {}
      - related_pods_enricher: {}

5. Scheduled Health Check

customPlaybooks:
  - triggers:
      - on_schedule:
          cron_schedule_repeat:
            cron_expression: "0 */6 * * *"
    actions:
      - cluster_status_enricher: {}
    sinks:
      - slack

6. Auto-Remediation: Delete Stuck Pods

customPlaybooks:
  - triggers:
      - on_prometheus_alert:
          alert_name: KubePodCrashLooping
          alert_severity: critical
    actions:
      - logs_enricher: {}
      - delete_pod:
          force: false

7. OOMKilled Detection

customPlaybooks:
  - triggers:
      - on_pod_oom_killed: {}
    actions:
      - pod_oom_killer_enricher: {}
      - oom_killer_graph_enricher: {}

8. Custom Event Trigger

customPlaybooks:
  - triggers:
      - on_kubernetes_warning_event:
          include:
            - FailedScheduling
            - BackOff
    actions:
      - event_report: {}

Filtering Playbooks

By Namespace

customPlaybooks:
  - triggers:
      - on_prometheus_alert:
          alert_name: ".*"
    scope:
      include:
        - namespace: production
        - namespace: staging

By Label

customPlaybooks:
  - triggers:
      - on_prometheus_alert:
          alert_name: ".*"
    scope:
      include:
        - labels:
            team: backend

By Severity

customPlaybooks:
  - triggers:
      - on_prometheus_alert:
          alert_name: ".*"
          alert_severity: critical

Applying Changes

After modifying generated_values.yaml:

# Upgrade Helm release
helm upgrade robusta robusta/robusta \
  -f ./generated_values.yaml \
  --namespace robusta

# Verify playbooks loaded
kubectl logs -n robusta deploy/robusta-runner | grep -i playbook

# List active playbooks
kubectl exec -n robusta deploy/robusta-runner -- robusta playbooks list

Testing Playbooks

# Trigger a test alert manually
kubectl exec -n robusta deploy/robusta-runner -- \
  robusta playbooks trigger prometheus_alert \
  --alert-name TestAlert \
  --labels '{"severity":"warning"}'

Configure Sinks

Workflow for setting up notification destinations in Robusta.

Sink Configuration Location

All sinks are configured in generated_values.yaml under sinksConfig:

sinksConfig:
  - <sink_type>:
      name: <unique_name>
      <sink_parameters>

Communication Platforms

Slack

sinksConfig:
  - slack_sink:
      name: main_slack
      slack_channel: k8s-alerts
      api_key: xoxb-your-bot-token
      # Optional parameters
      # thread_messages: true          # Group alerts in threads
      # scope:                          # Filter alerts
      #   include:
      #     - namespace: production

Getting Slack Token: 1. Go to api.slack.com/apps 2. Create new app or use existing 3. Add chat:write and files:write scopes 4. Install to workspace 5. Copy Bot User OAuth Token

Microsoft Teams

sinksConfig:
  - ms_teams_sink:
      name: teams_alerts
      webhook_url: https://outlook.office.com/webhook/...

Getting Teams Webhook: 1. Go to Teams channel 2. Click ... > Connectors 3. Add "Incoming Webhook" 4. Copy webhook URL

Discord

sinksConfig:
  - discord_sink:
      name: discord_alerts
      url: https://discord.com/api/webhooks/...

Google Chat

sinksConfig:
  - google_chat_sink:
      name: gchat
      webhook_url: https://chat.googleapis.com/v1/spaces/...

Incident Management

PagerDuty

sinksConfig:
  - pagerduty_sink:
      name: pagerduty_critical
      api_key: your-integration-key
      # Optional
      # scope:
      #   include:
      #     - labels:
      #         severity: critical

OpsGenie

sinksConfig:
  - opsgenie_sink:
      name: opsgenie
      api_key: your-api-key
      # teams:
      #   - team_name

VictorOps (Splunk On-Call)

sinksConfig:
  - victorops_sink:
      name: victorops
      url: https://alert.victorops.com/integrations/generic/...
      routing_key: your-routing-key

Ticketing Systems

Jira

sinksConfig:
  - jira_sink:
      name: jira_tickets
      url: https://your-instance.atlassian.net
      username: your-email@company.com
      api_key: your-api-token
      project_name: OPS
      issue_type: Bug

ServiceNow

sinksConfig:
  - servicenow_sink:
      name: servicenow
      instance: your-instance
      username: admin
      password: your-password

Data & Integration

Kafka

sinksConfig:
  - kafka_sink:
      name: kafka_alerts
      kafka_url: kafka-bootstrap:9092
      topic: robusta-alerts

Webhook (Generic)

sinksConfig:
  - webhook_sink:
      name: custom_webhook
      url: https://your-endpoint.com/alerts
      # headers:
      #   Authorization: Bearer token

DataDog

sinksConfig:
  - datadog_sink:
      name: datadog
      api_key: your-datadog-api-key

File Sink (Debugging)

sinksConfig:
  - file_sink:
      name: debug_file
      file_name: /tmp/robusta-alerts.log

Robusta UI (SaaS)

sinksConfig:
  - robusta_sink:
      name: robusta_ui
      token: <generated-during-setup>

Filtering Alerts per Sink

By Namespace

sinksConfig:
  - slack_sink:
      name: prod_slack
      slack_channel: prod-alerts
      api_key: xoxb-...
      scope:
        include:
          - namespace: production
  - slack_sink:
      name: dev_slack
      slack_channel: dev-alerts
      api_key: xoxb-...
      scope:
        include:
          - namespace: development
          - namespace: staging

By Severity

sinksConfig:
  - pagerduty_sink:
      name: pagerduty_critical
      api_key: ...
      scope:
        include:
          - labels:
              severity: critical
  - slack_sink:
      name: slack_warnings
      slack_channel: warnings
      api_key: ...
      scope:
        include:
          - labels:
              severity: warning

By Alert Name

sinksConfig:
  - slack_sink:
      name: slack_pod_issues
      slack_channel: pod-alerts
      api_key: ...
      scope:
        include:
          - name: KubePod.*

Applying Changes

helm upgrade robusta robusta/robusta \
  -f ./generated_values.yaml \
  --namespace robusta

# Verify sinks loaded
kubectl logs -n robusta deploy/robusta-runner | grep -i sink

Testing Sinks

# Send test message to all sinks
kubectl exec -n robusta deploy/robusta-runner -- \
  robusta demo

Install Robusta

Step-by-step workflow for installing Robusta on Kubernetes clusters.

Prerequisites Check

# Verify Helm is installed
helm version

# Verify kubectl is configured
kubectl cluster-info

# Check cluster resources (for sizing)
kubectl get nodes

Installation Options

Option 1: All-in-One (Robusta + Prometheus Stack)

Best for new clusters without existing monitoring.

# Step 1: Generate configuration
pipx run robusta-cli gen-config --enable-prometheus-stack

# Or using Docker if pipx not available
curl -fsSL -o robusta https://docs.robusta.dev/master/_static/robusta
chmod +x robusta
./robusta gen-config --enable-prometheus-stack

# Step 2: Add Helm repository
helm repo add robusta https://robusta-charts.storage.googleapis.com
helm repo update

# Step 3: Install
helm install robusta robusta/robusta \
  -f ./generated_values.yaml \
  --set clusterName=my-cluster \
  --namespace robusta \
  --create-namespace

# Step 4: Verify
kubectl get pods -n robusta
kubectl logs -n robusta deploy/robusta-runner -f

Option 2: Standalone (Existing Prometheus)

For clusters with existing Prometheus/AlertManager.

# Generate config without Prometheus stack
pipx run robusta-cli gen-config

# Install
helm install robusta robusta/robusta \
  -f ./generated_values.yaml \
  --set clusterName=my-cluster

Platform-Specific Configurations

GKE Autopilot

helm install robusta robusta/robusta \
  -f ./generated_values.yaml \
  --set clusterName=my-gke-cluster \
  --set kube-prometheus-stack.coreDns.enabled=false \
  --set kube-prometheus-stack.kubeControllerManager.enabled=false \
  --set kube-prometheus-stack.kubeEtcd.enabled=false \
  --set kube-prometheus-stack.kubeScheduler.enabled=false \
  --set kube-prometheus-stack.kubeProxy.enabled=false

EKS

Ensure EBS CSI driver is installed:

# Install EBS CSI driver add-on via eksctl or AWS console
helm install robusta robusta/robusta \
  -f ./generated_values.yaml \
  --set clusterName=my-eks-cluster \
  --set enablePlatformPlaybooks=true

AKS

helm install robusta robusta/robusta \
  -f ./generated_values.yaml \
  --set clusterName=my-aks-cluster

Small/Test Clusters

For clusters with limited resources:

helm install robusta robusta/robusta \
  -f ./generated_values.yaml \
  --set isSmallCluster=true

OpenShift

helm install robusta robusta/robusta \
  -f ./generated_values.yaml \
  --set openshift.enabled=true \
  --set openshift.createScc=true

Post-Installation Verification

# Check all pods are running
kubectl get pods -n robusta

# Expected pods:
# - robusta-runner (main processor)
# - robusta-forwarder (event collector)
# - prometheus-* (if all-in-one install)

# Check logs for errors
kubectl logs -n robusta deploy/robusta-runner --tail=50

# Test connectivity to sinks
kubectl exec -n robusta deploy/robusta-runner -- robusta playbooks list

Upgrading

helm repo update
helm upgrade robusta robusta/robusta \
  -f ./generated_values.yaml \
  --namespace robusta

Uninstalling

helm uninstall robusta -n robusta
kubectl delete namespace robusta

Troubleshoot Robusta

Common issues and debugging steps for Robusta installations.

Quick Diagnostics

# Check pod status
kubectl get pods -n robusta

# Check runner logs
kubectl logs -n robusta deploy/robusta-runner --tail=100

# Check forwarder logs
kubectl logs -n robusta deploy/robusta-forwarder --tail=100

# Check events
kubectl get events -n robusta --sort-by='.lastTimestamp'

Common Issues

1. Pods Not Starting

Symptoms: Pods stuck in Pending, CrashLoopBackOff, or ImagePullBackOff

# Check pod events
kubectl describe pod -n robusta -l app=robusta-runner

# Check resource limits
kubectl get pods -n robusta -o yaml | grep -A5 resources

# For small clusters, reduce resources
helm upgrade robusta robusta/robusta \
  -f ./generated_values.yaml \
  --set isSmallCluster=true

2. Alerts Not Firing

Symptoms: Prometheus alerts exist but Robusta doesn't process them

# Verify AlertManager is sending to Robusta
kubectl get svc -n robusta

# Check AlertManager config
kubectl get secret -n robusta alertmanager-robusta-kube-prometheus-alertmanager \
  -o jsonpath='{.data.alertmanager\.yaml}' | base64 -d

# Verify webhook URL in AlertManager
# Should have receiver pointing to robusta-runner:5000

# Test AlertManager connection
kubectl exec -n robusta deploy/robusta-runner -- \
  curl -s http://alertmanager-operated:9093/api/v2/alerts

3. Slack Messages Not Sending

Symptoms: Alerts processed but no Slack notifications

# Check sink configuration in logs
kubectl logs -n robusta deploy/robusta-runner | grep -i slack

# Verify API key is set
kubectl get secret -n robusta robusta-runner-secret \
  -o jsonpath='{.data.SLACK_API_KEY}' | base64 -d

# Test Slack API directly
kubectl exec -n robusta deploy/robusta-runner -- \
  curl -X POST https://slack.com/api/auth.test \
  -H "Authorization: Bearer xoxb-your-token"

# Common issues:
# - Wrong channel name (don't include #)
# - Bot not invited to channel
# - Missing chat:write scope

4. Playbooks Not Loading

Symptoms: Custom playbooks defined but not executing

# List loaded playbooks
kubectl exec -n robusta deploy/robusta-runner -- \
  robusta playbooks list

# Check for YAML syntax errors
kubectl logs -n robusta deploy/robusta-runner | grep -i "playbook\|error\|yaml"

# Validate YAML syntax locally
python -c "import yaml; yaml.safe_load(open('generated_values.yaml'))"

5. High Memory/CPU Usage

Symptoms: Runner pod consuming excessive resources

# Check current usage
kubectl top pod -n robusta

# Check for alert storms
kubectl logs -n robusta deploy/robusta-runner | grep -c "Processing alert"

# Add rate limiting to playbooks
# In generated_values.yaml:
globalConfig:
  alertThrottleMinutes: 5

# Reduce concurrent processing
runner:
  resources:
    limits:
      memory: 512Mi
      cpu: 500m

6. Prometheus Stack Issues

Symptoms: Prometheus/Grafana pods failing (all-in-one install)

# Check Prometheus pods
kubectl get pods -n robusta -l app.kubernetes.io/name=prometheus

# Check PVC status (EKS/AKS may need storage class)
kubectl get pvc -n robusta

# For EKS without EBS CSI:
# Install EBS CSI driver first

# Check Grafana
kubectl logs -n robusta deploy/robusta-grafana

7. Multi-Cluster Issues

Symptoms: Alerts from wrong cluster or missing cluster context

# Verify cluster name is set
helm get values robusta -n robusta | grep clusterName

# Update cluster name
helm upgrade robusta robusta/robusta \
  -f ./generated_values.yaml \
  --set clusterName=correct-cluster-name

Advanced Debugging

Enable Debug Logging

# In generated_values.yaml
runner:
  log_level: DEBUG

Check Robusta Internals

# Access runner shell
kubectl exec -it -n robusta deploy/robusta-runner -- bash

# Inside container:
robusta playbooks list
robusta sinks list
robusta version

Test Alert Processing

# Send test alert
kubectl exec -n robusta deploy/robusta-runner -- \
  robusta demo

# Trigger specific playbook
kubectl exec -n robusta deploy/robusta-runner -- \
  robusta playbooks trigger prometheus_alert \
  --alert-name KubePodCrashLooping \
  --namespace default \
  --labels '{"severity":"warning","pod":"test-pod"}'

Network Connectivity

# Test sink connectivity from runner
kubectl exec -n robusta deploy/robusta-runner -- \
  curl -I https://slack.com

# Test AlertManager connectivity
kubectl exec -n robusta deploy/robusta-runner -- \
  curl http://alertmanager-operated:9093/-/healthy

Recovery Steps

Reinstall Robusta

# Uninstall
helm uninstall robusta -n robusta

# Wait for cleanup
kubectl get pods -n robusta -w

# Reinstall
helm install robusta robusta/robusta \
  -f ./generated_values.yaml \
  --namespace robusta

Reset Configuration

# Regenerate config
rm generated_values.yaml
pipx run robusta-cli gen-config --enable-prometheus-stack

# Reinstall with fresh config
helm upgrade --install robusta robusta/robusta \
  -f ./generated_values.yaml

Getting Help

• Robusta Slack Community
• GitHub Issues
• Official Docs