Az Aks Agent

Azure AKS Agent CLI Skill

Overview

The Agentic CLI for Azure Kubernetes Service (AKS) is an AI-powered troubleshooting and insights tool (currently in preview) that brings advanced diagnostics directly to your terminal. It allows you to ask natural language questions about your cluster's health, configuration, and issues without requiring deep Kubernetes expertise or knowledge of complex command syntax.

Primary Command: az aks agent

Quick Reference

Installation

# Prerequisites: Azure CLI version 2.76 or higher
az version

# Install the extension (takes 5-10 minutes)
az extension add --name aks-agent --debug

# Verify installation
az extension list
az aks agent --help

# Initialize LLM configuration (interactive wizard)
az aks agent-init

# Remove extension if needed
az extension remove --name aks-agent --debug

Basic Usage

# Get cluster credentials first
az aks get-credentials --resource-group <rg-name> --name <cluster-name>

# Start interactive troubleshooting
az aks agent -g <resource-group> -n <cluster-name>

# Ask a specific question
az aks agent -g <resource-group> -n <cluster-name> --query "What's wrong with my cluster?"

# Non-interactive mode (batch processing)
az aks agent -g <resource-group> -n <cluster-name> --no-interactive --query "Check pod health"

Workflow Decision Tree

What do you need to do?
├── Cluster Health Check?
│   └── Use: az aks agent --query "What's the health status of my cluster?"
├── Troubleshoot Pod Issues?
│   └── Use: az aks agent --query "Why are my pods failing?"
├── Networking Problems?
│   └── Use: az aks agent --query "Diagnose networking issues"
├── Storage Issues?
│   └── Use: az aks agent --query "Check storage configuration"
├── Security/RBAC Issues?
│   └── Use: az aks agent --query "Review RBAC configuration"
├── Node Pool Problems?
│   └── Use: az aks agent --query "Check node pool health"
└── Configuration Review?
    └── Use: az aks agent --query "Review cluster configuration"

Command Reference

Core Commands

Command Parameters

LLM Model Specifications

# Azure OpenAI
--model "azure/gpt-4o"
--model "azure/gpt-4o-mini"

# OpenAI
--model "gpt-4o"
--model "gpt-4o-mini"

# Anthropic
--model "anthropic/claude-sonnet-4"
--model "anthropic/claude-3-5-sonnet"

# Gemini
--model "gemini/gemini-pro"

Configuration

Environment Variables

# Azure OpenAI API Key
export AZURE_API_KEY="your-azure-openai-key"

# OpenAI API Key
export OPENAI_API_KEY="your-openai-key"

# Anthropic API Key
export ANTHROPIC_API_KEY="your-anthropic-key"

Config File Structure (~/.azure/aksAgent.config)

# Azure OpenAI Configuration
llm_provider: azure
azure_api_base: https://<your-endpoint>.openai.azure.com/
azure_api_version: 2025-04-01-preview
model: gpt-4o

# OR OpenAI Configuration
llm_provider: openai
model: gpt-4o

# OR Anthropic Configuration
llm_provider: anthropic
model: claude-sonnet-4

Azure OpenAI Requirements

• Deployment name: Must match model name
• Minimum TPM: 1,000,000+ (Tokens Per Minute)
• Minimum context size: 128,000+ tokens
• API Base Format: https://{endpoint}.openai.azure.com/ (NOT AI Foundry URI)

Common Use Cases

Cluster Health Analysis

# General health check
az aks agent -g myRG -n myCluster --query "What's the overall health of my cluster?"

# Node status
az aks agent -g myRG -n myCluster --query "Are all nodes healthy and ready?"

# Resource utilization
az aks agent -g myRG -n myCluster --query "Show me resource utilization across nodes"

Pod Troubleshooting

# Failed pods analysis
az aks agent -g myRG -n myCluster --query "Why are pods in CrashLoopBackOff?"

# Pending pods
az aks agent -g myRG -n myCluster --query "Why are some pods stuck in Pending state?"

# OOMKilled pods
az aks agent -g myRG -n myCluster --query "Investigate OOMKilled containers"

Networking Issues

# Network policy review
az aks agent -g myRG -n myCluster --query "Are there network policies blocking traffic?"

# DNS troubleshooting
az aks agent -g myRG -n myCluster --query "Diagnose DNS resolution issues"

# Service connectivity
az aks agent -g myRG -n myCluster --query "Why can't pods reach external services?"

Storage Troubleshooting

# PVC issues
az aks agent -g myRG -n myCluster --query "Why are PersistentVolumeClaims pending?"

# Storage class review
az aks agent -g myRG -n myCluster --query "Review storage class configuration"

Security Analysis

# RBAC review
az aks agent -g myRG -n myCluster --query "Are RBAC permissions configured correctly?"

# Security best practices
az aks agent -g myRG -n myCluster --query "What security improvements do you recommend?"

AKS Events Reference

Viewing Cluster Events

# Get cluster credentials first
az aks get-credentials --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER

# List all events
kubectl get events

# Filter by namespace
kubectl get events --namespace default

# Watch auto-repair events
kubectl get events --field-selector=source=aks-auto-repair --watch

# Detailed pod events
kubectl describe pod $POD_NAME

Event Types

Common Event Reasons

Event Fields

Best Practices

Effective Query Strategies

1. Start broad, then narrow

   # Start with general health
   "What's wrong with my cluster?"
   # Then focus on specific issues
   "Why are pods in namespace X failing?"

2. Provide context about symptoms

   "Pods are restarting frequently in the production namespace"
   "Services are experiencing intermittent timeouts"

3. Ask for specific recommendations

   "What changes do you recommend to improve cluster performance?"
   "How can I fix the networking issues you identified?"

4. Request historical analysis

   "What patterns do you see in recent pod failures?"
   "Have there been any unusual events in the last 24 hours?"

Security Considerations

• Ensure proper RBAC permissions are configured
• Use Azure AD integration for authentication
• Follow principle of least privilege
• Audit command usage through Azure activity logs
• Service account tokens for automation

Integration Tips

1. Combine with traditional monitoring: Use alongside Azure Monitor and Container Insights 2. Proactive monitoring: Run health checks regularly 3. Document findings: Save important diagnostic outputs 4. Enable Container Insights: For events beyond 1-hour retention

Troubleshooting the Agent

Installation Issues

# Verify Azure CLI version
az version

# Upgrade Azure CLI if needed
az upgrade

# Force reinstall extension
az extension remove --name aks-agent
az extension add --name aks-agent --debug

Authentication Issues

# Verify Azure login
az account show

# Re-authenticate
az login

# Check subscription
az account set --subscription <subscription-id>

LLM Connection Issues

# Reinitialize LLM configuration
az aks agent-init

# Check API key environment variable
echo $AZURE_API_KEY

# Test with explicit API key
az aks agent -g myRG -n myCluster --api-key "your-key"

Rate Limiting

• Symptom: Slow responses or errors
• Solution: Increase TPM quota in Azure OpenAI deployment
• Minimum recommended: 1,000,000 TPM

Important Notes

1. Preview Feature: This is currently in preview with limited warranty coverage 2. Not for Production Critical: Not recommended for production-critical decision making 3. Event Retention: Kubernetes events only persist for 1 hour by default 4. Context Window: Requires 128,000+ token context for optimal performance 5. Authentication: Always authenticate with az login before using

Resources

References

Core References

• references/cli-commands.md - Complete CLI command reference
• references/troubleshooting.md - Extended troubleshooting guide
• references/examples.md - Practical usage examples

Diagnostics & Monitoring

• references/diagnostics.md - AKS Diagnose and Solve Problems guide
• references/monitoring.md - Comprehensive AKS monitoring guide
• references/control-plane-metrics.md - Control plane metrics (API Server, etcd)

Troubleshooting Guides

• references/kubelet-logs.md - Kubelet logs access and analysis
• references/memory-saturation.md - Memory saturation identification and resolution
• references/node-auto-repair.md - Node auto-repair process and monitoring
• references/api-server-etcd.md - API server and etcd troubleshooting

External Documentation

• AKS Agent Overview
• AKS Agent Installation
• AKS Events
• AKS Diagnostics
• Monitor AKS
• Control Plane Metrics
• Kubelet Logs
• Memory Saturation
• Node Auto-Repair
• API Server/etcd Troubleshooting
• AKS Monitoring Reference
• AKS Agent Troubleshooting
• AKS Agent FAQ
• GitHub Repository
• Example Config

---

Gotchas

• Agent uses Azure-CLI session token — expired session silently fails to the API but surfaces as "agent not responding".
• Agent only sees Kubernetes API events, not custom controller events — your custom-resource problems are invisible.
• Top events report aggregates by reason+message — subtly different messages get separate buckets; frequency counts mislead.
• Cluster MSI vs Workload Identity: the agent uses cluster MSI; troubleshooting auth issues for a pod under workload identity requires re-auth from that pod's context.
• Preview features: AKS-managed Prometheus + AKS agent integration is GA in some regions, preview in others. Same CLI version, different behavior by region.

API Server and etcd Troubleshooting Guide

Common API Server Issues

Symptoms

Root Causes

1. Network rules blocking agent-to-API traffic 2. Custom webhook deadlock 3. Resource leakage (object accumulation) 4. AKS managed API server guard activation 5. Excessive LIST/PUT calls from clients 6. High etcd memory usage

etcd Database Management

Capacity Limits

Metric Names by Version

Diagnostic Commands

Check API Server Connectivity

kubectl aks config import \
  --subscription <subscriptionID> \
  --resource-group <resourceGroup> \
  --cluster-name <clusterName>

kubectl aks check-apiserver-connectivity --node <nodeName>

Monitor etcd Database Size

kubectl get --raw /metrics | grep -E "etcd_db_total_size_in_bytes|apiserver_storage_size_bytes|apiserver_storage_db_total_size_in_bytes"

Check API Server Metrics

kubectl get --raw /metrics | grep apiserver_

Identify Webhooks

kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations

Analyze FlowSchemas

kubectl get flowschemas
kubectl get prioritylevelconfigurations

Log Analytics Queries

Top API Users (Resource-Specific Mode)

AKSAudit
| where TimeGenerated between(now(-1h)..now())
| summarize count() by UserAgent
| top 10 by count_

Top API Users (Azure Diagnostics Mode)

AzureDiagnostics
| where Category == "kube-audit"
| extend event = parse_json(log_s)
| extend User = tostring(event.user.username)
| summarize count() by User
| top 10 by count_

P99 Latency by Operation

AKSAudit
| where TimeGenerated between(now(-1h)..now())
| extend HttpMethod = Verb
| extend Resource = tostring(ObjectRef.resource)
| extend latency = datetime_diff('millisecond', StageReceivedTime, RequestReceivedTime)
| summarize p99latency=percentile(latency, 99) by HttpMethod, Resource

Failed API Requests

AKSAudit
| where TimeGenerated > ago(1h)
| where ResponseStatus.code >= 400
| summarize count() by ResponseStatus.code, Verb, ObjectRef.resource
| order by count_ desc

Resolution Steps

Cause 1: Network Issues

Reconfigure network policies to allow unrestricted traffic between agent nodes and API server.

# Validate connectivity
kubectl aks check-apiserver-connectivity --node <nodeName>

Cause 2: Webhook Deadlock

# Remove problematic webhooks
kubectl delete validatingwebhookconfigurations <name>
kubectl delete mutatingwebhookconfigurations <name>

Cause 3: Resource Leakage

Implement Resource Quotas:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: object-quota
spec:
  hard:
    pods: "100"
    jobs.batch: "50"

Cleanup Completed Jobs:

# Delete completed jobs
kubectl delete jobs --field-selector status.successful=1

# Delete failed pods
kubectl delete pods --field-selector status.phase=Failed

# Bulk delete by label
kubectl delete pods -l app=old-app

Implement Auto-cleanup with TTL:

apiVersion: batch/v1
kind: Job
spec:
  ttlSecondsAfterFinished: 3600  # Auto-delete after 1 hour

Cause 4: API Server Guard Activation

# Identify guard
kubectl get flowschemas
kubectl get prioritylevelconfigurations

# Remove guard
kubectl delete flowschema aks-managed-apiserver-guard
kubectl delete prioritylevelconfiguration aks-managed-apiserver-guard

# Preserve custom configs
kubectl label prioritylevelconfiguration aks-managed-apiserver-guard \
  aks-managed-skip-update-operation=true

Cause 5: Excessive API Calls

Optimize Client Patterns:

• Use field selectors to limit LIST results
• Implement client-side caching
• Use informers instead of repeated LIST calls
• Batch operations where possible

Throttle Problematic Clients:

apiVersion: flowcontrol.apiserver.k8s.io/v1beta2
kind: FlowSchema
metadata:
  name: restrict-bad-client
spec:
  priorityLevelConfiguration:
    name: very-low-priority
  rules:
  - resourceRules:
    - resources: ["pods"]
      verbs: ["list"]
    subjects:
    - kind: ServiceAccount
      serviceAccount:
        name: bad-client-account
        namespace: default
---
apiVersion: flowcontrol.apiserver.k8s.io/v1beta2
kind: PriorityLevelConfiguration
metadata:
  name: very-low-priority
spec:
  type: Limited
  limited:
    assuredConcurrencyShares: 5
    limitResponse:
      type: Reject

Cause 6: High etcd Memory

• Follow Cause 3 & 5 solutions
• Move environment variables to ConfigMaps
• Split large Secrets/ConfigMaps
• Optimize resource specifications

Azure Portal Diagnostics

Access Path: AKS cluster → Diagnose and Solve Problems → Cluster and Control Plane Availability and Performance

Available Tools

Monitoring Best Practices

Proactive Monitoring

• Monitor etcd database size continuously
• Track API server latency metrics (P99)
• Set alerts on HTTP 429 response rates
• Monitor control plane component health

Alert Thresholds

Prevention Strategy

1. Implement resource quotas per namespace 2. Use TTL values for temporary objects 3. Implement client-side caching and informers 4. Set up field/label selector best practices 5. Regular cleanup of failed/completed jobs 6. Monitor excessive API users

Critical Warning

If API server becomes unresponsive due to severe etcd memory pressure, contact Azure support immediately rather than attempting troubleshooting steps.

Diagnostic Script

#!/bin/bash
# diagnose-control-plane.sh

echo "=== etcd Database Size ==="
kubectl get --raw /metrics 2>/dev/null | grep -E "etcd_db_total_size|apiserver_storage" | head -5

echo -e "\n=== API Server Metrics ==="
kubectl get --raw /metrics 2>/dev/null | grep -E "apiserver_request_total|apiserver_current_inflight" | head -10

echo -e "\n=== FlowSchemas ==="
kubectl get flowschemas

echo -e "\n=== Priority Level Configurations ==="
kubectl get prioritylevelconfigurations

echo -e "\n=== Validating Webhooks ==="
kubectl get validatingwebhookconfigurations

echo -e "\n=== Mutating Webhooks ==="
kubectl get mutatingwebhookconfigurations

echo -e "\n=== Object Counts ==="
echo "Pods: $(kubectl get pods -A --no-headers 2>/dev/null | wc -l)"
echo "Jobs: $(kubectl get jobs -A --no-headers 2>/dev/null | wc -l)"
echo "ConfigMaps: $(kubectl get configmaps -A --no-headers 2>/dev/null | wc -l)"
echo "Secrets: $(kubectl get secrets -A --no-headers 2>/dev/null | wc -l)"

Azure AKS Agent CLI Commands Reference

Installation Commands

Install Extension

# Standard installation
az extension add --name aks-agent

# Installation with debug output
az extension add --name aks-agent --debug

# Force upgrade existing installation
az extension add --name aks-agent --upgrade

Verify Installation

# List installed extensions
az extension list

# Show extension details
az extension show --name aks-agent

# Get help
az aks agent --help

Remove Extension

# Remove the extension
az extension remove --name aks-agent

# Remove with debug output
az extension remove --name aks-agent --debug

Configuration Commands

Initialize Configuration

# Interactive configuration wizard
az aks agent-init

# This wizard will:
# 1. Ask for LLM provider selection
# 2. Request API credentials
# 3. Save configuration to ~/.azure/aksAgent.config

Configuration File Location

# Default location
~/.azure/aksAgent.config

# Specify custom config file
az aks agent --config-file /path/to/custom/config.yaml

Core Agent Commands

Basic Usage

# Interactive mode (default)
az aks agent -g <resource-group> -n <cluster-name>

# With specific query
az aks agent -g <resource-group> -n <cluster-name> \
  --query "What's wrong with my cluster?"

# Non-interactive mode
az aks agent -g <resource-group> -n <cluster-name> \
  --no-interactive \
  --query "Check cluster health"

Complete Parameter Reference

az aks agent \
  --resource-group <resource-group>     # Required: Azure resource group
  --name <cluster-name>                 # Required: AKS cluster name
  --api-key <api-key>                   # Optional: LLM API key (overrides env/config)
  --config-file <path>                  # Optional: Custom config file path
  --max-steps <number>                  # Optional: Max investigation steps (default: 10)
  --model <model>                       # Optional: LLM model specification
  --no-interactive                      # Optional: Disable interactive mode
  --show-tool-output                    # Optional: Show tool execution output
  --refresh-toolsets                    # Optional: Refresh available toolsets

Model Specifications

Azure OpenAI Models

# GPT-4o (recommended)
az aks agent -g myRG -n myCluster --model "azure/gpt-4o"

# GPT-4o Mini
az aks agent -g myRG -n myCluster --model "azure/gpt-4o-mini"

OpenAI Models

# GPT-4o
az aks agent -g myRG -n myCluster --model "gpt-4o"

# GPT-4o Mini
az aks agent -g myRG -n myCluster --model "gpt-4o-mini"

Anthropic Models

# Claude Sonnet 4.0
az aks agent -g myRG -n myCluster --model "anthropic/claude-sonnet-4"

# Claude 3.5 Sonnet
az aks agent -g myRG -n myCluster --model "anthropic/claude-3-5-sonnet"

Gemini Models

# Gemini Pro
az aks agent -g myRG -n myCluster --model "gemini/gemini-pro"

Environment Variables

API Keys

# Azure OpenAI
export AZURE_API_KEY="your-azure-openai-api-key"

# OpenAI
export OPENAI_API_KEY="your-openai-api-key"

# Anthropic
export ANTHROPIC_API_KEY="your-anthropic-api-key"

# Gemini
export GEMINI_API_KEY="your-gemini-api-key"

Azure OpenAI Specific

# API Base URL (required for Azure OpenAI)
export AZURE_API_BASE="https://your-endpoint.openai.azure.com/"

# API Version
export AZURE_API_VERSION="2025-04-01-preview"

Related Azure CLI Commands

AKS Cluster Management

# Get cluster credentials (required before using agent)
az aks get-credentials \
  --resource-group <resource-group> \
  --name <cluster-name> \
  --overwrite-existing

# List clusters
az aks list --resource-group <resource-group> -o table

# Show cluster details
az aks show --resource-group <resource-group> --name <cluster-name>

# Check cluster health
az aks show \
  --resource-group <resource-group> \
  --name <cluster-name> \
  --query "agentPoolProfiles[].{name:name,count:count,vmSize:vmSize,status:provisioningState}"

Kubectl Commands for Events

# List all events
kubectl get events

# List events in specific namespace
kubectl get events -n <namespace>

# Watch events in real-time
kubectl get events --watch

# Watch auto-repair events
kubectl get events --field-selector=source=aks-auto-repair --watch

# Get events sorted by timestamp
kubectl get events --sort-by='.lastTimestamp'

# Get events for a specific pod
kubectl get events --field-selector involvedObject.name=<pod-name>

# Describe pod (includes events)
kubectl describe pod <pod-name> -n <namespace>

Azure Monitor Integration

# Enable Container Insights
az aks enable-addons \
  --resource-group <resource-group> \
  --name <cluster-name> \
  --addons monitoring \
  --workspace-resource-id <log-analytics-workspace-id>

# Check monitoring addon status
az aks show \
  --resource-group <resource-group> \
  --name <cluster-name> \
  --query "addonProfiles.omsagent"

Example Config Files

Azure OpenAI Configuration

# ~/.azure/aksAgent.config
llm_provider: azure
azure_api_base: https://my-openai.openai.azure.com/
azure_api_version: 2025-04-01-preview
azure_api_key: sk-xxxxxxxxxxxx
model: gpt-4o
max_steps: 10

OpenAI Configuration

# ~/.azure/aksAgent.config
llm_provider: openai
openai_api_key: sk-xxxxxxxxxxxx
model: gpt-4o
max_steps: 10

Anthropic Configuration

# ~/.azure/aksAgent.config
llm_provider: anthropic
anthropic_api_key: sk-ant-xxxxxxxxxxxx
model: claude-sonnet-4
max_steps: 10

Batch Processing Examples

Script for Multiple Clusters

#!/bin/bash
# check-all-clusters.sh

CLUSTERS=("cluster1" "cluster2" "cluster3")
RESOURCE_GROUP="my-resource-group"
QUERY="What's the overall health status?"

for cluster in "${CLUSTERS[@]}"; do
  echo "=== Checking $cluster ==="
  az aks agent \
    -g "$RESOURCE_GROUP" \
    -n "$cluster" \
    --no-interactive \
    --query "$QUERY"
  echo ""
done

CI/CD Pipeline Integration

# azure-pipelines.yml
steps:
  - task: AzureCLI@2
    inputs:
      azureSubscription: 'my-subscription'
      scriptType: 'bash'
      scriptLocation: 'inlineScript'
      inlineScript: |
        # Install extension
        az extension add --name aks-agent

        # Run health check
        az aks agent \
          -g $(RESOURCE_GROUP) \
          -n $(CLUSTER_NAME) \
          --no-interactive \
          --api-key $(AKS_AGENT_API_KEY) \
          --query "Are there any critical issues?"

Output Formatting

Show Tool Output

# Display what tools the agent is running
az aks agent -g myRG -n myCluster \
  --show-tool-output \
  --query "Check pod health"

Verbose Mode

# Enable debug output
az aks agent -g myRG -n myCluster --debug

# Combined with tool output
az aks agent -g myRG -n myCluster \
  --debug \
  --show-tool-output

Quick Reference Card

AKS Control Plane Metrics Reference

Overview

Control plane metrics provide visibility into the health and performance of AKS managed control plane components including API Server and etcd.

Prerequisites

• Managed Identity Authentication on AKS cluster
• Managed service for Prometheus in Azure Monitor
• Azure Private Link is NOT supported

Enable Control Plane Metrics

Step 1: Install Preview Extension

az extension add --name aks-preview
az extension update --name aks-preview

Step 2: Register Feature Flag

az feature register --namespace "Microsoft.ContainerService" \
  --name "AzureMonitorMetricsControlPlanePreview"

# Verify registration (takes a few minutes)
az feature show --namespace "Microsoft.ContainerService" \
  --name "AzureMonitorMetricsControlPlanePreview"

Step 3: Refresh Provider

az provider register --namespace "Microsoft.ContainerService"

Step 4: Enable Metrics on Cluster

az aks update --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP

Available Metrics

API Server Metrics

etcd Metrics

Default Configuration

# Default targets (ON by default)
controlplane-apiserver = true
controlplane-etcd = true

# Optional targets (OFF by default)
controlplane-cluster-autoscaler = false
controlplane-kube-scheduler = false
controlplane-kube-controller-manager = false
controlplane-node-auto-provisioning = false

Configuration Profiles

Option A: Minimal Ingestion (Default)

Only collects essential metrics for API server and etcd.

Option B: All Metrics from All Targets

# Download configmap
wget https://raw.githubusercontent.com/Azure/prometheus-collector/main/otelcollector/configmaps/ama-metrics-settings-configmap.yaml

# Edit: set minimalingestionprofile = false

# Apply
kubectl apply -f ama-metrics-settings-configmap.yaml -n kube-system

Option C: Specific Metrics from Specific Targets

# In configmap: set minimalingestionprofile = false
# Specify metrics with pipe-separated list

controlplane-apiserver = "apiserver_admission_webhook_admission_duration_seconds|apiserver_longrunning_requests"

# For histograms (include all three variants):
controlplane-apiserver = "apiserver_request_duration_seconds_bucket|apiserver_request_duration_seconds_sum|apiserver_request_duration_seconds_count"

Apply the configmap:

kubectl apply -f configmap-controlplane.yaml -n kube-system

Optional Targets

Cluster Autoscaler Metrics

controlplane-cluster-autoscaler = true

Available metrics:

• cluster_autoscaler_cluster_safe_to_autoscale
• cluster_autoscaler_scaled_up_nodes_total
• cluster_autoscaler_scaled_down_nodes_total
• cluster_autoscaler_unneeded_nodes_count
• cluster_autoscaler_unschedulable_pods_count

Kube Scheduler Metrics

controlplane-kube-scheduler = true

Available metrics:

• scheduler_pending_pods
• scheduler_unschedulable_pods
• scheduler_queue_incoming_pods_total
• scheduler_schedule_attempts_total

Kube Controller Manager Metrics

controlplane-kube-controller-manager = true

Available metrics:

• workqueue_depth
• rest_client_requests_total
• rest_client_request_duration_seconds

Node Auto-Provisioning (Karpenter)

controlplane-node-auto-provisioning = true

Available metrics:

• karpenter_pods_state
• karpenter_nodes_created_total
• karpenter_nodes_terminated_total

Querying Metrics

Azure Portal

1. Navigate to AKS cluster resource 2. Left menu → Monitor > Monitor Settings 3. Access linked Azure Monitor workspace 4. Under "Managed Prometheus" → Prometheus explorer

Pre-built Grafana Dashboards

• API Server: <https://grafana.com/grafana/dashboards/20331-kubernetes-api-server/>
• etcd: <https://grafana.com/grafana/dashboards/20330-kubernetes-etcd/>

CLI Commands Reference

Important Notes

1. Preview Feature: Not production-ready 2. Private Link: Not supported 3. Self-hosted Prometheus: Cannot scrape control plane metrics 4. Managed Prometheus Only: Only supported collection method 5. Configmap Namespace: Apply to kube-system namespace 6. Histogram Metrics: Always include _bucket, _sum, and _count suffixes 7. Data Latency: Several minutes after enabling 8. High-Cardinality Warning: apiserver_request_sli_duration_seconds_bucket not collected by default

Scrape Configuration

Default scrape interval: 30 seconds (configurable)

# Configmap fields to verify
default-targets-metrics-keep-list: <metrics>
minimal-ingestion-profile: true/false
default-scrape-settings-enabled: true/false

AKS Diagnostics Reference

Overview

AKS Diagnose and Solve Problems is an intelligent, self-diagnostic tool built into the Azure portal that helps identify and resolve cluster problems automatically with no extra configuration or billing cost.

Access Methods

Azure Portal (Primary)

1. Navigate to your AKS cluster in Azure Portal 2. From service menu, select Diagnose and solve problems 3. Select a troubleshooting category or use search 4. Review alerts and click View details 5. Follow remediation steps

Azure CLI - AKS Agent (AI-Powered)

# Describe cluster status
az aks agent "describe cluster <cluster-name> in resource group <rg-name>"

# Ask diagnostic questions
az aks agent "What's wrong with my cluster?"
az aks agent "How is my cluster [name] in resource group [rg]?"
az aks agent "Diagnose connectivity issues in my cluster"

AKS Periscope (Log Collection)

# Install prerequisite
az extension add --name aks-preview

# Collect diagnostic information
az aks kollect

# View options
az aks kollect -h

Periscope Collects:

• Container logs (kube-system namespace by default)
• Docker and Kubelet system service logs
• Network outbound connectivity checks
• Node IP Tables

Diagnostic Categories

1. Cluster and Control Plane Availability

• Service availability checks
• Throttling issue detection
• Control plane health status
• ETCD database health

2. Connectivity Issues

• Cluster DNS resolution errors
• Outbound communication routes
• Network connectivity validation
• Service-to-service connectivity

3. Best Practices

• VM resource provisioning recommendations
• Cluster upgrade planning
• Scaling operations guidance
• Subnet configuration validation

4. Overview (All Diagnostics)

• Runs all diagnostics across categories
• Displays comprehensive issue summary

Cluster Insight Test Categories

Available Metrics

Control Plane Metrics

Node Metrics

Pod Metrics

Cluster Autoscaler Metrics

Interpreting Results

Report Structure

Each diagnostic report provides:

1. Issue Summary - High-level problem overview 2. Error Details - Specific findings 3. Severity Level - Impact indication 4. Recommended Actions - Resolution steps 5. Documentation Links - Microsoft Learn references 6. Related Metrics - Performance data 7. Logging Data - Detailed logs

Metric Interpretation Guide

Diagnostic Settings CLI Commands

Create Diagnostic Setting

az monitor diagnostic-settings create \
  --resource /subscriptions/{sub-id}/resourceGroups/{rg}/providers/Microsoft.ContainerService/managedClusters/{cluster} \
  --name {diagnostic-setting-name} \
  --logs '[{
    "category": "kube-apiserver",
    "enabled": true
  }]' \
  --workspace /subscriptions/{sub-id}/resourceGroups/{rg}/providers/microsoft.operationalinsights/workspaces/{workspace}

Available Log Categories

# Control plane logs
kube-apiserver              # API server operations
kube-controller-manager     # Controller manager operations
kube-scheduler              # Scheduler operations
cloud-controller-manager    # Cloud controller operations

# Audit logs
kube-audit                  # All operations (including get/list)
kube-audit-admin            # Only modifying operations

# System logs
cluster-autoscaler          # Autoscaler operations
csi-azuredisk-controller    # Azure Disk CSI driver
csi-azurefile-controller    # Azure File CSI driver
karpenter-events            # Karpenter provisioner events

Log Analytics Queries

Query API Server Logs

AKSControlPlane
| where TimeGenerated > ago(1h)
| where log_s contains "error"

Query Audit Logs

AKSAudit
| where TimeGenerated > ago(24h)
| where ObjectRef_user contains "pod"

Query Warning Events

KubeEvents
| where TimeGenerated > ago(1h)
| where Type == "Warning"

Count Logs by Category

AzureDiagnostics
| where ResourceType == "MANAGEDCLUSTERS"
| summarize count() by Category

Best Practices

1. Start Broad: Use general queries like "What's wrong with my cluster?" 2. Be Descriptive: Provide context about observed symptoms 3. Review Carefully: Understand recommendations before implementing 4. Complement with Monitoring: Use alongside Azure Monitor 5. Collect Logs: Use AKS Periscope for detailed analysis 6. Cost Optimization: Disable kube-audit when not needed; use kube-audit-admin instead

Additional Tools

• AKS Periscope: Comprehensive log collection
• Resource Health: Historical health status in Azure Portal
• VS Code AKS Extension: Native IDE diagnostics support
• AKS Triage Practices Guide: Troubleshooting methodology

Azure AKS Agent Practical Examples

Setup Examples

Initial Setup for New Users

# Step 1: Verify Azure CLI version (must be 2.76+)
az version

# Step 2: Install the AKS Agent extension
az extension add --name aks-agent --debug

# Step 3: Login to Azure
az login

# Step 4: Set your subscription
az account set --subscription "My AKS Subscription"

# Step 5: Initialize LLM configuration
az aks agent-init

# Step 6: Get cluster credentials
az aks get-credentials \
  --resource-group cafehyna-rg \
  --name cafehyna-dev

# Step 7: Verify connection
kubectl get nodes

# Step 8: Start using the agent
az aks agent -g cafehyna-rg -n cafehyna-dev

Setup with Azure OpenAI

# Set environment variables
export AZURE_API_KEY="your-api-key"
export AZURE_API_BASE="https://my-openai.openai.azure.com/"
export AZURE_API_VERSION="2025-04-01-preview"

# Or create config file
cat > ~/.azure/aksAgent.config << 'EOF'
llm_provider: azure
azure_api_base: https://my-openai.openai.azure.com/
azure_api_version: 2025-04-01-preview
model: gpt-4o
max_steps: 10
EOF

# Test the setup
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "What's the cluster health status?"

Cluster Health Examples

Daily Health Check

# Comprehensive cluster health check
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Perform a comprehensive health check of my cluster including nodes, system pods, and resource utilization"

# Quick health summary
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Give me a quick health summary"

Node Health Analysis

# Check node status
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Are all nodes healthy and ready? Check for any NotReady nodes or resource pressure"

# Node pool analysis
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Analyze my node pools and their resource utilization"

# Spot node issues
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Are there any issues with spot instance nodes?"

Control Plane Health

# API server health
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Check the health of the Kubernetes API server and control plane components"

# etcd health
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "What's the status of etcd? Any latency or storage issues?"

Pod Troubleshooting Examples

CrashLoopBackOff Investigation

# General CrashLoopBackOff diagnosis
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "I have pods in CrashLoopBackOff state. Identify them and explain why they're crashing"

# Namespace-specific
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Why are pods crashing in the production namespace? Show logs and events"

Pending Pods Analysis

# Find pending pods
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Find all pending pods and explain why they can't be scheduled"

# Resource-related pending
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Are there pods pending due to insufficient CPU or memory?"

# Node selector issues
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Are there pods pending due to node selector or toleration mismatches?"

OOMKilled Containers

# Find OOMKilled containers
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Find containers that have been OOMKilled and analyze their memory patterns"

# Memory recommendations
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Which deployments need memory limit adjustments based on OOMKilled events?"

Image Pull Errors

# Image pull failures
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Find pods with ImagePullBackOff and identify the cause"

# Registry access issues
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Are there any container registry authentication issues?"

Networking Examples

Service Connectivity

# Service discovery issues
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Debug why Service X cannot reach Service Y in the cluster"

# LoadBalancer issues
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Why is my LoadBalancer service not getting an external IP?"

# Internal service DNS
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Verify DNS resolution is working for cluster services"

Network Policy Analysis

# Network policy audit
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "List all network policies and check if any are blocking expected traffic"

# Traffic flow analysis
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Is traffic from namespace A allowed to reach namespace B?"

Ingress Troubleshooting

# Ingress status
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Check the status of all Ingress resources and their backends"

# Certificate issues
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Are there any TLS certificate issues with my Ingresses?"

# Ingress controller health
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Is the NGINX ingress controller healthy?"

Storage Examples

PVC Troubleshooting

# Pending PVCs
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Find PersistentVolumeClaims stuck in Pending state and explain why"

# Storage class issues
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Are there any issues with Azure Disk or Azure File storage classes?"

# Volume attachment
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Check for pods stuck waiting for volume attachment"

CSI Driver Issues

# CSI driver status
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Check the health of CSI drivers in the cluster"

# Key Vault CSI
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Are there issues with Azure Key Vault CSI driver secret mounts?"

Security Examples

RBAC Analysis

# RBAC audit
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Review RBAC configuration for overly permissive roles"

# Service account analysis
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Are there service accounts with cluster-admin privileges that shouldn't have them?"

# Missing permissions
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Why is pod X getting Forbidden errors when accessing the API?"

Pod Security

# Security context issues
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Find pods running as root or with privileged containers"

# Pod Security Standards
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Are there pods violating Pod Security Standards?"

Secret Management

# Secret audit
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Are there any secrets that are not being used or might be exposed?"

# Secret mounting issues
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Check if secrets are properly mounted in pods"

Performance Examples

Resource Utilization

# Cluster resource usage
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Show me cluster-wide CPU and memory utilization with recommendations"

# High resource consumers
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Which pods are using the most CPU and memory?"

# Resource recommendations
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Based on actual usage, suggest resource request and limit adjustments"

Scaling Analysis

# HPA issues
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Why is my HorizontalPodAutoscaler not scaling?"

# Cluster autoscaler
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Is the cluster autoscaler working correctly? Are there pending pods that should trigger scale-up?"

Application-Specific Examples

Deployment Issues

# Deployment rollout status
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Check the rollout status of all deployments and identify any stuck rollouts"

# Failed deployments
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Why did the deployment of my-app fail?"

# Deployment recommendations
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "What improvements do you recommend for my deployment configurations?"

StatefulSet Issues

# StatefulSet health
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Check the health of StatefulSets and their ordered pod management"

# Database pods
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Are my database StatefulSet pods healthy with proper PVC attachments?"

Job/CronJob Issues

# Failed jobs
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Find failed Jobs and CronJobs and explain why they failed"

# CronJob schedule issues
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Are my CronJobs running on schedule? Any missed executions?"

Events Analysis Examples

Recent Cluster Events

# All recent events
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Show me all Warning events in the last hour"

# Critical events
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Are there any critical events that need immediate attention?"

Node Events

# Node events
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Show me node-related events including auto-repairs"

# Auto-repair analysis
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Have there been any node auto-repair events recently? What triggered them?"

Automation Examples

Health Check Script

#!/bin/bash
# daily-health-check.sh
# Run daily cluster health check and save report

RESOURCE_GROUP="cafehyna-rg"
CLUSTER_NAME="cafehyna-dev"
REPORT_DIR="./health-reports"
DATE=$(date +%Y-%m-%d)

mkdir -p "$REPORT_DIR"

echo "Running daily health check for $CLUSTER_NAME..."

az aks agent \
  -g "$RESOURCE_GROUP" \
  -n "$CLUSTER_NAME" \
  --no-interactive \
  --query "Perform a comprehensive health check including:
    1. Node status and resource pressure
    2. System pod health
    3. Warning events in the last 24 hours
    4. Resource utilization trends
    5. Any pods in error states
    6. Network connectivity status
    Provide a summary with severity levels and recommended actions." \
  > "$REPORT_DIR/health-$DATE.txt"

echo "Report saved to $REPORT_DIR/health-$DATE.txt"

Multi-Cluster Check

#!/bin/bash
# check-all-clusters.sh
# Check health across multiple clusters

RESOURCE_GROUP="cafehyna-rg"
CLUSTERS=("cafehyna-dev" "cafehyna-hub" "cafehyna-prd")

for cluster in "${CLUSTERS[@]}"; do
  echo ""
  echo "========================================"
  echo "Cluster: $cluster"
  echo "========================================"

  az aks get-credentials \
    --resource-group "$RESOURCE_GROUP" \
    --name "$cluster" \
    --overwrite-existing

  az aks agent \
    -g "$RESOURCE_GROUP" \
    -n "$cluster" \
    --no-interactive \
    --max-steps 5 \
    --query "Quick health status: any critical issues?"
done

Incident Response Template

#!/bin/bash
# incident-response.sh
# Quick incident response diagnostic

RESOURCE_GROUP="${1:-cafehyna-rg}"
CLUSTER_NAME="${2:-cafehyna-dev}"
NAMESPACE="${3:-default}"

echo "Starting incident response diagnostic..."
echo "Cluster: $CLUSTER_NAME"
echo "Namespace: $NAMESPACE"
echo ""

# Get cluster credentials
az aks get-credentials \
  --resource-group "$RESOURCE_GROUP" \
  --name "$CLUSTER_NAME" \
  --overwrite-existing

# Run comprehensive diagnostic
az aks agent \
  -g "$RESOURCE_GROUP" \
  -n "$CLUSTER_NAME" \
  --show-tool-output \
  --query "INCIDENT RESPONSE for namespace $NAMESPACE:
    1. List all pods in error states with their events
    2. Check for recent Warning events
    3. Verify node health
    4. Check resource constraints
    5. Review recent deployments or changes
    6. Network connectivity status
    7. Service endpoint health
    Provide root cause analysis and immediate remediation steps."

Interactive Session Examples

Starting Interactive Session

# Start interactive session
az aks agent -g cafehyna-rg -n cafehyna-dev

# In the interactive session, you can ask follow-up questions:
# > What's wrong with my cluster?
# > Can you provide more details about the networking issue you mentioned?
# > How do I fix the pending pods?
# > Show me the relevant kubectl commands

Guided Troubleshooting Flow

# Step 1: Start with broad question
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "What issues exist in my cluster?"

# Step 2: Dive deeper based on findings
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Tell me more about the pod failures in production namespace"

# Step 3: Get remediation steps
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "How do I fix the OOMKilled pods you identified?"

# Step 4: Verify the fix
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Verify that the memory limit changes I made are working"

Best Practice Query Examples

Proactive Queries

# Optimization recommendations
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "What optimizations do you recommend for my cluster?"

# Cost optimization
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Are there opportunities to reduce costs through resource right-sizing?"

# Security hardening
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "What security improvements do you recommend?"

Pre-Deployment Checks

# Readiness check
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Is the cluster ready for a new deployment? Check capacity and health"

# Namespace preparation
az aks agent -g cafehyna-rg -n cafehyna-dev \
  --query "Check if namespace 'new-app' has proper resource quotas and network policies"

AKS Kubelet Logs Guide

Overview

Kubelet logs provide critical information about node-level operations, pod lifecycle management, and container runtime interactions in AKS clusters.

Access Methods

Method A: kubectl Raw API (Quick - No SSH)

# Set node name
export NODE_NAME="aks-agentpool-xxxxxxx-0"

# Get kubelet logs via raw API
kubectl get --raw "/api/v1/nodes/$NODE_NAME/proxy/logs/messages" | grep kubelet

Advantages:

• Quickest method
• No SSH access required
• Useful for rapid diagnostics

Method B: SSH Connection (Detailed Access)

# 1. Establish SSH connection to node
# 2. Enter host environment
chroot /host

# 3. View kubelet logs
journalctl -u kubelet -o cat

Method C: Container Insights

Azure Monitor Container Insights provides:

• Syslog collection for kubelet logs
• Integration with Log Analytics
• Centralized log aggregation

Log Locations

Linux Nodes

Windows Nodes

Log Format

I0508 12:26:17.905042    8672 kubelet_node_status.go:497] Using Node Hostname from cloudprovider: "aks-agentpool-xxxxxxx-0"
I0508 12:26:28.920125    8672 server.go:796] GET /stats/summary: (10.370874ms) 200 [[Ruby] 10.244.0.x:52492]

Format Components

CLI Commands

Prerequisites

export RESOURCE_GROUP_NAME="<ResourceGroupName>"
export AKS_CLUSTER_NAME="<AKSClusterName>"
az aks get-credentials --resource-group $RESOURCE_GROUP_NAME --name $AKS_CLUSTER_NAME

Retrieve Kubelet Logs

# Via kubectl raw API
export NODE_NAME="aks-agentpool-xxxxxxx-0"
kubectl get --raw "/api/v1/nodes/$NODE_NAME/proxy/logs/messages" | grep kubelet

# Via az aks command invoke
az aks command invoke -g $RESOURCE_GROUP_NAME -n $AKS_CLUSTER_NAME \
  --command "chroot /host && journalctl -u kubelet -o cat"

Filter Logs

# Last 100 lines
kubectl get --raw "/api/v1/nodes/$NODE_NAME/proxy/logs/messages" | grep kubelet | tail -100

# Errors only
kubectl get --raw "/api/v1/nodes/$NODE_NAME/proxy/logs/messages" | grep -E "^E.*kubelet"

# Warnings and errors
kubectl get --raw "/api/v1/nodes/$NODE_NAME/proxy/logs/messages" | grep -E "^[EW].*kubelet"

# Time range (via journalctl on node)
journalctl -u kubelet --since "2024-01-01 00:00:00" --until "2024-01-01 12:00:00"

Common Issues in Kubelet Logs

Node Status Issues

Connectivity Issues

Resource Issues

Pod Lifecycle Issues

Troubleshooting Workflow

Step 1: Identify Issue Category

• Cluster creation failures
• Upgrade/scaling operations
• Node status problems
• Connectivity issues
• Performance degradation

Step 2: Collect Logs

# Quick collection
kubectl get --raw "/api/v1/nodes/$NODE_NAME/proxy/logs/messages" | grep kubelet > kubelet-logs.txt

# Detailed collection (SSH)
journalctl -u kubelet -o cat > kubelet-detailed.txt

Step 3: Analyze Patterns

# Search for errors
grep -E "^E" kubelet-logs.txt

# Search for specific issues
grep -i "certificate" kubelet-logs.txt
grep -i "connection refused" kubelet-logs.txt
grep -i "timeout" kubelet-logs.txt
grep -i "evicted" kubelet-logs.txt

Step 4: Correlate with Events

# Get node events
kubectl get events --field-selector involvedObject.name=$NODE_NAME

# Describe node
kubectl describe node $NODE_NAME

Key Diagnostic Patterns

Error Patterns to Search

Log Level Reference

Additional Tools

AKS Diagnose and Solve Problems

• Automated self-diagnostic in Azure Portal
• No additional configuration needed
• Covers connectivity, best practices, cluster health

AKS Periscope

• Detailed log collection tool
• Comprehensive node diagnostics
• Network connectivity checks

Data Collection Methods

• Real-time system insights capture
• TCP dump from nodes
• TCP packet capture from pods
• Container dumps (Windows)

Access Methods Comparison

Best Practices

1. Start with kubectl raw API for quick diagnostics 2. Use SSH for deep investigation when raw API is insufficient 3. Enable Container Insights for continuous log collection 4. Set up log-based alerts for critical patterns 5. Correlate with Kubernetes events for context 6. Document common patterns for your workloads

AKS Memory Saturation Troubleshooting

Overview

Memory saturation occurs when applications or processes need more memory than the container host can provide, or when the host exhausts available memory.

Identification Methods

Method 1: Container Insights (Azure Portal)

1. Navigate to Kubernetes services → Select cluster 2. Go to Monitoring → Insights 3. Select Nodes tab 4. Choose metric: Memory working set (computed from Allocatable) 5. Set percentiles to Max 6. Sort by Max % column

Method 2: kubectl top

# Node memory usage
kubectl top node

# Example output
NAME                                STATUS   CPU(cores)   MEMORY(bytes)   MEMORY%
aks-agentpool-12345678-vmss000000   Ready    250m         4500Mi          69%
aks-agentpool-12345678-vmss000001   Ready    180m         3000Mi          46%

Method 3: Inspektor Gadget (Advanced)

# Top 10 memory-consuming processes cluster-wide
kubectl gadget run top_process --sort -memoryRelative --max-entries 10

# By specific node
kubectl gadget run top_process --sort -memoryRelative --filter k8s.node==<node-name>

# By namespace
kubectl gadget run top_process --sort -memoryRelative --filter k8s.namespace==<namespace>

# By pod
kubectl gadget run top_process --sort -memoryRelative --filter k8s.podName==<pod-name>

Symptoms and Indicators

Symptom Progression

Memory Pressure → Pod Eviction → Node Not Ready → OOM Kill

Diagnostic Commands

Basic Diagnostics

# Node memory usage
kubectl top node

# Describe node (shows allocatable resources)
kubectl describe node <node-name>

# Node conditions
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="MemoryPressure")].status}{"\n"}{end}'

Detailed Pod Memory Analysis

# Pod memory usage on specific node
kubectl get pods --all-namespaces --output wide \
    | grep <node-name> \
    | awk '{print $1" "$2}' \
    | xargs -n2 kubectl top pods --namespace \
    | awk 'NR==1 || NR%2==0' \
    | sort -k3n \
    | column -t

Memory Events

# OOMKilled events
kubectl get events --all-namespaces --field-selector reason=OOMKilling

# Memory pressure events
kubectl get events --all-namespaces | grep -i memory

Resolution Steps

Step 1: Implement Memory Requests and Limits

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: myapp:latest
    resources:
      requests:
        memory: "180Mi"      # Close to actual usage
      limits:
        memory: "300Mi"      # Prevent overcommitting

Guidelines:

• Set requests close to actual usage
• Set limits slightly higher as buffer
• Monitor actual usage to tune values

Step 2: Enable Horizontal Pod Autoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 70

Step 3: Apply Pod Anti-Affinity

apiVersion: apps/v1
kind: Deployment
metadata:
  name: high-memory-app
spec:
  template:
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - high-memory-app
            topologyKey: kubernetes.io/hostname

Step 4: Scale Up VM SKUs

# Create new node pool with higher memory VMs
az aks nodepool add \
  --resource-group <rg> \
  --cluster-name <cluster> \
  --name highmempool \
  --node-count 3 \
  --node-vm-size Standard_E8s_v3

# Cordon existing nodes
kubectl cordon <old-node>

# Drain workloads
kubectl drain <old-node> --ignore-daemonsets --delete-emptydir-data

# Delete old node pool
az aks nodepool delete \
  --resource-group <rg> \
  --cluster-name <cluster> \
  --name oldpool

Step 5: Separate Workload Types

# Dedicate node pool for applications
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      nodeSelector:
        agentpool: userpool
      tolerations:
      - key: "workload"
        operator: "Equal"
        value: "user"
        effect: "NoSchedule"

Prevention Strategies

Resource Quota for Namespaces

apiVersion: v1
kind: ResourceQuota
metadata:
  name: mem-quota
  namespace: production
spec:
  hard:
    requests.memory: "10Gi"
    limits.memory: "20Gi"

Limit Range for Defaults

apiVersion: v1
kind: LimitRange
metadata:
  name: mem-limit-range
  namespace: production
spec:
  limits:
  - default:
      memory: "512Mi"
    defaultRequest:
      memory: "256Mi"
    type: Container

Monitoring Queries

Log Analytics - OOMKilled Containers

KubeEvents
| where Reason == "OOMKilling"
| project TimeGenerated, Namespace, Name, Message
| order by TimeGenerated desc

Memory Working Set by Node

InsightsMetrics
| where Name == "node_memory_working_set_percentage"
| summarize avg(Val) by Computer, bin(TimeGenerated, 5m)
| render timechart

Memory Saturation Alert Query

InsightsMetrics
| where Name == "node_memory_working_set_percentage"
| summarize MaxMemory = max(Val) by Computer
| where MaxMemory > 90

Critical Metrics to Monitor

Best Practices Summary

1. Set appropriate requests/limits for all containers 2. Monitor memory working set against allocatable (not total) 3. Use HPA to scale horizontally before vertical 4. Implement resource quotas per namespace 5. Use node pools to separate workload types 6. Set up alerts for memory pressure conditions 7. Regular capacity planning based on actual usage

AKS Monitoring Comprehensive Guide

Architecture Overview

AKS monitoring operates on a multi-layered architecture with five primary data sources:

┌─────────────────────────────────────────────────────────────┐
│                     Azure Monitor                            │
├────────────┬────────────┬────────────┬────────────┬─────────┤
│  Platform  │ Prometheus │  Activity  │  Resource  │Container│
│  Metrics   │  Metrics   │    Logs    │    Logs    │Insights │
│  (Free)    │ (Managed)  │ (Sub-level)│  (11 cat)  │  (App)  │
└────────────┴────────────┴────────────┴────────────┴─────────┘

Data Sources

Container Insights Setup

Prerequisites

• Log Analytics workspace (same subscription as AKS)
• Managed identity authentication (recommended)
• Azure Monitor Data Collection Rules (DCRs)

Enable via Azure CLI

az aks enable-addons \
  --resource-group <resource-group> \
  --name <cluster-name> \
  --addons monitoring \
  --workspace-resource-id <log-analytics-workspace-id>

Cost Optimization Groupings (via DCRs)

Resource Logs Configuration

Create Diagnostic Settings

az monitor diagnostic-settings create \
  --name AKS-Diagnostics \
  --resource /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.ContainerService/managedClusters/{cluster} \
  --logs '[
    {"category": "kube-audit", "enabled": true},
    {"category": "kube-audit-admin", "enabled": true},
    {"category": "kube-apiserver", "enabled": true},
    {"category": "kube-controller-manager", "enabled": true},
    {"category": "kube-scheduler", "enabled": true},
    {"category": "cluster-autoscaler", "enabled": true},
    {"category": "cloud-controller-manager", "enabled": true},
    {"category": "guard", "enabled": true},
    {"category": "csi-azuredisk-controller", "enabled": true},
    {"category": "csi-azurefile-controller", "enabled": true},
    {"category": "csi-snapshot-controller", "enabled": true}
  ]' \
  --workspace /subscriptions/{sub}/resourcegroups/{rg}/providers/microsoft.operationalinsights/workspaces/{workspace} \
  --export-to-resource-specific true

Log Categories

Collection Modes

Azure Diagnostics Mode

All data routes to AzureDiagnostics table, identified via Category column.

AzureDiagnostics
| where Category == "kube-apiserver"

Resource-Specific Mode (Recommended)

Data routes to dedicated tables:

• AKSAudit - All audit logs
• AKSAuditAdmin - Excludes get/list events
• AKSControlPlane - Control plane logs

AKSAudit
| where TimeGenerated > ago(1h)

AKSControlPlane
| where Category == "kube-apiserver"

Log Analytics Tables

Log Analytics Queries

Count Logs by Category

AzureDiagnostics
| where ResourceType == "MANAGEDCLUSTERS"
| summarize count() by Category

API Server Logs

AzureDiagnostics
| where Category == "kube-apiserver"

Detailed Audit Logs

let starttime = datetime("2023-02-23");
let endtime = datetime("2023-02-24");
AzureDiagnostics
| where TimeGenerated between(starttime..endtime)
| where Category == "kube-audit"
| extend event = parse_json(log_s)
| extend HttpMethod = tostring(event.verb)
| extend User = tostring(event.user.username)
| extend Apiserver = pod_s
| extend SourceIP = tostring(event.sourceIPs[0])
| project TimeGenerated, Category, HttpMethod, User, Apiserver, SourceIP, OperationName, event

Container Logs with Errors

ContainerLogV2
| where LogLevel == "error"
| project TimeGenerated, PodName, ContainerName, LogMessage

Pod Restart Analysis

KubePodInventory
| where PodRestartCount > 0
| summarize max(PodRestartCount) by PodName, Namespace
| order by max_PodRestartCount desc

Node Resource Usage

InsightsMetrics
| where Name == "node_cpu_usage_percentage" or Name == "node_memory_rss_percentage"
| summarize avg(Val) by Name, Computer, bin(TimeGenerated, 5m)
| render timechart

Alert Configuration

Recommended Alert Rules

Create Metric Alert

az monitor metrics alert create \
  --name "High-CPU-Alert" \
  --resource-group <rg> \
  --scopes <aks-resource-id> \
  --condition "avg node_cpu_usage_percentage > 80" \
  --window-size 5m \
  --evaluation-frequency 1m \
  --action <action-group-id>

Real-Time Monitoring

Live Data Features

• Live logs for pods, containers, workloads
• Live metrics for CPU, memory, network I/O
• Live events for Kubernetes resources
• Requires: Container Insights + direct Kubernetes API access

Enable Live Metrics

kubectl proxy &
# Access via Azure Portal Live Data feature

Network Metrics

Default Metrics (K8s 1.29+)

Disable Per-Node

kubectl label node <node-name> networking.azure.com/node-network-metrics=disabled

Integration Options

Best Practices

1. Use Resource-Specific Mode: Easier querying and Basic logs tier support 2. Enable ContainerLogV2: Better schema, cost savings with Basic tier 3. Implement Cost Groupings: Use DCRs to control ingestion 4. Set Up Alerts: Proactive issue detection 5. Use Prometheus for Metrics: Cloud-native compatibility 6. Retain Critical Logs: Configure appropriate retention periods 7. Disable Unnecessary Logs: kube-audit generates high volume; use kube-audit-admin

AKS Node Auto-Repair Reference

Overview

AKS continuously monitors worker node health and automatically initiates repairs when nodes become unhealthy. This is a managed service with no additional configuration required.

How It Works

Health Monitoring

• AKS monitors worker node health continuously
• Works with Azure VM platform
• Repairs initiated by aks-remediator service account

Detection Criteria

Trigger Threshold

• Node must be unhealthy for at least 5 minutes before repair initiates

Repair Process

Repair Sequence (Progressive)

Reboot → Reimage → Redeploy (Linux only)

Retry Logic

• Entire sequence retried up to 3 times
• Total completion window: Up to 1 hour

Timeline Example

Conditions That Block Auto-Repair

Shutdown Taints Present

# Auto-repair does NOT occur if these taints exist:
node.cloudprovider.kubernetes.io/shutdown
ToBeDeletedByClusterAutoscaler

Upgrade In Progress

# Auto-repair blocked with these annotations:
cluster-autoscaler.kubernetes.io/scale-down-disabled: "true"
kubernetes.azure.com/azure-cluster-autoscaler-scale-down-disabled-reason: "upgrade"

Other Blocking Conditions

• Network configuration errors preventing status reporting
• Node failed to initially register as healthy

Monitoring Auto-Repair Events

View Events

# Get all events
kubectl get events

# Watch auto-repair events
kubectl get events --field-selector source=aks-auto-repair --watch

# Events for specific node
kubectl get events --field-selector involvedObject.name=<node-name>

Event Retention

• Local retention: 1 hour
• Extended retention: Enable Container Insights (90+ days)

Event Types

Action Events

Error Events

Example Event Messages

# Start Event
Node auto-repair is initiating a reboot action due to NotReady status persisting for more than 5 minutes.

# End Event
Reboot action from node auto-repair is completed.

# Error Event
Node auto-repair reboot action failed due to an operation failure. See error details: [Error code]

Manual Health Checks

# View node status
kubectl get nodes

# Describe specific node
kubectl describe node <node-name>

# Check node conditions
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}'

# View node events
kubectl get events --field-selector involvedObject.name=<node-name>

Diagnostic Script

#!/bin/bash
# check-node-health.sh

echo "=== Node Status ==="
kubectl get nodes

echo -e "\n=== NotReady Nodes ==="
kubectl get nodes | grep NotReady

echo -e "\n=== Node Conditions ==="
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\t"}{.status.conditions[?(@.type=="Ready")].reason}{"\n"}{end}'

echo -e "\n=== Auto-Repair Events ==="
kubectl get events --field-selector source=aks-auto-repair --sort-by='.lastTimestamp'

echo -e "\n=== Recent Warning Events ==="
kubectl get events --field-selector type=Warning --sort-by='.lastTimestamp' | head -20

Configuration

No Direct Configuration Available

Node auto-repair is a managed service with:

• Enabled by default
• Fixed detection and retry logic
• Cannot be disabled
• Cannot be customized

Recommendations

1. Enable Container Insights for extended event retention 2. Monitor Events Regularly for anomalies 3. Manual Intervention if issues persist after auto-repair 4. Review Troubleshooting Docs for persistent failures

Integration with Other Features

Cluster Autoscaler

• Auto-repair coordinates with cluster autoscaler
• Respects scale-down disabled annotations
• Handles autoscaler shutdown taints

Node Pools

• Auto-repair applies to all node pools
• Works with both system and user pools
• Handles spot instance nodes

Upgrade Operations

• Auto-repair paused during upgrades
• Resumes after upgrade completion
• Respects upgrade annotations

Troubleshooting Persistent Issues

Node Remains Unhealthy After Auto-Repair

1. Check kubelet logs 2. Review node events 3. Verify network connectivity 4. Check for resource exhaustion 5. Contact Azure support if needed

Commands for Investigation

# Get kubelet logs
kubectl get --raw "/api/v1/nodes/<node>/proxy/logs/messages" | grep kubelet

# Check network connectivity
kubectl run nettest --image=busybox --rm -it --restart=Never -- wget -qO- kubernetes.default.svc.cluster.local

# Review node resources
kubectl describe node <node> | grep -A 10 "Allocated resources"

Best Practices

1. Enable Container Insights for long-term event monitoring 2. Set up alerts for repeated auto-repair events 3. Review events weekly for patterns 4. Document recurring issues for trend analysis 5. Plan capacity to prevent resource-related failures 6. Use multiple node pools for workload isolation

Azure AKS Agent Troubleshooting Guide

Installation Issues

Extension Installation Fails

Symptom: az extension add --name aks-agent fails with an error.

Diagnosis:

# Check Azure CLI version
az version

# Version must be 2.76 or higher

Solution:

# Upgrade Azure CLI
az upgrade

# Or install specific version
# On macOS
brew upgrade azure-cli

# On Ubuntu/Debian
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

# Retry installation
az extension add --name aks-agent --debug

Extension Already Installed Error

Symptom: Error indicates extension is already installed.

Solution:

# Remove existing extension
az extension remove --name aks-agent

# Reinstall
az extension add --name aks-agent

Authentication Issues

Not Logged In to Azure

Symptom: Please run 'az login' to setup account.

Solution:

# Interactive login
az login

# Or with device code (for remote sessions)
az login --use-device-code

# Or with service principal
az login --service-principal \
  -u <app-id> \
  -p <password> \
  --tenant <tenant-id>

Wrong Subscription

Symptom: Cluster not found but exists in a different subscription.

Diagnosis:

# Check current subscription
az account show

# List all subscriptions
az account list -o table

Solution:

# Set correct subscription
az account set --subscription <subscription-id-or-name>

# Verify
az account show

Cluster Credentials Not Found

Symptom: Unable to connect to the cluster.

Solution:

# Get cluster credentials
az aks get-credentials \
  --resource-group <resource-group> \
  --name <cluster-name> \
  --overwrite-existing

# Verify connection
kubectl get nodes

LLM Configuration Issues

API Key Not Set

Symptom: Error: API key not found

Diagnosis:

# Check environment variable
echo $AZURE_API_KEY    # For Azure OpenAI
echo $OPENAI_API_KEY   # For OpenAI

Solution:

# Set environment variable
export AZURE_API_KEY="your-api-key"

# Or run configuration wizard
az aks agent-init

# Or provide key directly
az aks agent -g myRG -n myCluster --api-key "your-key"

Invalid API Base URL

Symptom: Connection errors when using Azure OpenAI.

Cause: Using AI Foundry URI instead of direct OpenAI endpoint.

Solution:

# Correct format (direct OpenAI endpoint)
export AZURE_API_BASE="https://your-resource.openai.azure.com/"

# Incorrect format (AI Foundry - will not work)
# export AZURE_API_BASE="https://your-aiservices.azure.com/"

Invalid API Version

Symptom: API version not supported error.

Solution:

# Use a supported API version
export AZURE_API_VERSION="2025-04-01-preview"

# Or update in config file
# ~/.azure/aksAgent.config
# azure_api_version: 2025-04-01-preview

Model Not Found

Symptom: Model deployment not found.

Cause: Deployment name doesn't match model name.

Solution:

# Verify deployment exists in Azure OpenAI Studio
# Deployment name should match model name (e.g., "gpt-4o")

# List deployments
az cognitiveservices account deployment list \
  --name <openai-resource-name> \
  --resource-group <resource-group> \
  -o table

Performance Issues

Slow Response Times

Symptom: Agent takes a long time to respond.

Causes:

1. Insufficient TPM (Tokens Per Minute) quota 2. High cluster complexity 3. Network latency

Solutions:

# Check current TPM in Azure Portal or:
az cognitiveservices account deployment show \
  --name <openai-resource-name> \
  --resource-group <resource-group> \
  --deployment-name <deployment-name>

# Increase TPM quota (minimum recommended: 1,000,000)
# Done via Azure Portal: Azure OpenAI > Resource > Deployments > Edit

# Reduce max steps for faster (but less thorough) analysis
az aks agent -g myRG -n myCluster --max-steps 5

Rate Limiting Errors

Symptom: 429 Too Many Requests errors.

Solution:

# 1. Increase TPM quota (Azure Portal)

# 2. Reduce request frequency

# 3. Use a different model with higher quota
az aks agent -g myRG -n myCluster --model "azure/gpt-4o-mini"

Context Window Exceeded

Symptom: Errors about context length or token limits.

Cause: Cluster data exceeds model's context window.

Solution:

# Use a model with larger context window
# gpt-4o supports 128,000+ tokens

# Reduce max steps to limit data collection
az aks agent -g myRG -n myCluster --max-steps 5

# Focus query on specific namespace/resource
az aks agent -g myRG -n myCluster \
  --query "Check pods in the production namespace only"

Cluster Access Issues

Insufficient RBAC Permissions

Symptom: Forbidden errors when querying cluster resources.

Diagnosis:

# Check current user's permissions
kubectl auth can-i --list

# Check specific permission
kubectl auth can-i get pods -n kube-system

Solution:

# Request cluster-admin role (or appropriate permissions)
# Contact cluster administrator

# Or use admin credentials
az aks get-credentials \
  --resource-group <resource-group> \
  --name <cluster-name> \
  --admin

Network Connectivity Issues

Symptom: Cannot connect to API server.

Diagnosis:

# Test kubectl connectivity
kubectl cluster-info

# Check network
nc -zv <api-server-fqdn> 443

Solution:

# If using private cluster, ensure VPN/ExpressRoute connection

# If behind proxy
export HTTPS_PROXY=http://proxy:8080
export NO_PROXY=<api-server-fqdn>

# Refresh credentials
az aks get-credentials \
  --resource-group <resource-group> \
  --name <cluster-name> \
  --overwrite-existing

Common AKS Events Issues

Events Not Showing

Symptom: kubectl get events returns no results.

Cause: Events expire after 1 hour by default.

Solution:

# Enable Container Insights for long-term storage
az aks enable-addons \
  --resource-group <resource-group> \
  --name <cluster-name> \
  --addons monitoring \
  --workspace-resource-id <workspace-id>

# Query historical events via Azure Monitor

Missing Auto-Repair Events

Symptom: No events from aks-auto-repair source.

Diagnosis:

# Check if auto-repair is enabled
az aks show \
  --resource-group <resource-group> \
  --name <cluster-name> \
  --query "agentPoolProfiles[].enableAutoScaling"

Solution:

# Enable auto-repair (enabled by default in newer clusters)
# Events are automatically emitted when repairs occur

# Watch for events
kubectl get events --field-selector=source=aks-auto-repair --watch

Diagnostic Commands

Full System Check

#!/bin/bash
# diagnostics.sh - Run full AKS Agent diagnostics

echo "=== Azure CLI Version ==="
az version

echo -e "\n=== Logged In Account ==="
az account show

echo -e "\n=== Extension Status ==="
az extension list | grep aks-agent

echo -e "\n=== Environment Variables ==="
echo "AZURE_API_KEY: ${AZURE_API_KEY:+SET (hidden)}"
echo "OPENAI_API_KEY: ${OPENAI_API_KEY:+SET (hidden)}"
echo "AZURE_API_BASE: $AZURE_API_BASE"

echo -e "\n=== Kubectl Context ==="
kubectl config current-context

echo -e "\n=== Cluster Connection ==="
kubectl cluster-info

echo -e "\n=== Node Status ==="
kubectl get nodes

echo -e "\n=== Config File ==="
cat ~/.azure/aksAgent.config 2>/dev/null || echo "No config file found"

Test LLM Connection

# Simple query to test LLM connectivity
az aks agent -g myRG -n myCluster \
  --query "Say hello" \
  --max-steps 1 \
  --debug

Verbose Debugging

# Enable all debug output
az aks agent -g myRG -n myCluster \
  --debug \
  --show-tool-output \
  --query "Check cluster health"

Error Message Reference

Getting Help

Official Resources

• AKS Agent Troubleshooting Guide
• AKS Agent FAQ
• GitHub Issues

Support Channels

# Check AKS support policies
az aks show \
  --resource-group <resource-group> \
  --name <cluster-name> \
  --query "sku.tier"

# Note: AKS Agent is in preview and may have limited support