Airunway Aks Setup

AI Runway AKS Setup

This skill walks users from a bare Kubernetes cluster to a running AI model deployment. Follow each step in sequence unless the user provides skip-to-step N to resume from a specific phase.

Cost awareness: GPU node pools incur significant compute charges (A100-80GB can cost $3–5+/hr). Confirm the user understands cost implications before provisioning GPU resources.

Prerequisites

This skill assumes an AKS cluster already exists. If the user does not have a cluster, hand off to the azure-kubernetes skill first to provision one (with a GPU node pool unless CPU-only inference is acceptable), then return here.

Quick Reference

When to Use This Skill

Use this skill when the user wants to:

• Set up AI Runway on an existing AKS cluster from scratch
• Install the AI Runway controller and CRDs
• Assess GPU hardware compatibility for model deployment
• Choose and install an inference provider (KAITO, Dynamo, KubeRay)
• Deploy their first AI model to AKS via AI Runway
• Resume a partially-complete AI Runway setup from a specific step

MCP Tools

This skill uses no MCP tools. All cluster operations are performed directly via kubectl and make.

Rules

1. Execute steps in sequence — load the reference for each step as you reach it 2. Report cluster state at each step: ✓ healthy, ✗ missing/failed 3. Ask for user confirmation before any install or deployment action 4. If a step is already complete, report status and skip to the next step 5. If the user provides skip-to-step N, start at step N; assume prior steps are complete

Steps

Error Handling

For full error handling and rollback procedures, see troubleshooting.md.

GPU Compatibility Reference

This reference is used by the airunway-aks-setup skill during Step 3 — GPU Assessment to match detected GPU hardware to known compatibility profiles and surface constraints before model deployment.

GPU Profiles

Attention Backends

Compatibility Warnings

Surface these warnings when the following GPUs are detected:

T4

Warning: T4 GPUs do not support bfloat16. You must configure --dtype float16 in serving arguments. Failure to do so causes errors or silent dtype casting.

V100

Warning: V100 GPUs do not support bfloat16 and have limited flash attention support. Use xformers backend and --dtype float16.

Model Sizing & Recommendations

For VRAM sizing estimates and starter model recommendations, see model-sizing.md.

Model Sizing & Starter Recommendations

VRAM Sizing Guide

Estimate how much VRAM a model requires. Values are weights only — add 20–30% overhead for KV cache and activations during inference.

Example: 4× A100 80 GB = 320 GB total. Llama-3.1-70B at bfloat16 ≈ 168 GB with overhead — fits across 4 GPUs with tensor parallelism.

Starter Model Recommendations

Gated Models

These models are gated on HuggingFace and require an access token:

• meta-llama/Llama-3.1-8B-Instruct
• meta-llama/Llama-3.1-70B-Instruct

Non-gated alternatives (no token required):

• microsoft/Phi-3-mini-4k-instruct (MIT license)
• microsoft/Phi-3-medium-128k-instruct (MIT license)
• google/gemma-3-1b-it-qat-q8_0-gguf (Gemma license)

PowerShell Command Variants

This reference provides PowerShell equivalents for commands in steps 1, 4, 5, and 6 that use Bash-specific syntax.

Prerequisite Check (Step 1)

@('kubectl', 'make', 'curl') | ForEach-Object {
  if (Get-Command $_ -ErrorAction SilentlyContinue) {
    Write-Host "✓ $_ found"
  } else {
    Write-Host "✗ $_ NOT FOUND — install before continuing"
  }
}

GPU Detection (Step 1)

# Extract GPU count and model per node (requires NVIDIA device plugin labels)
# PowerShell does not need the bash single-quote escaping — pass the jsonpath directly
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.nvidia\.com/gpu}{"\t"}{.metadata.labels.nvidia\.com/gpu\.product}{"\n"}{end}'

Note: If the output shows empty GPU fields, use the fallback:

kubectl describe nodes | Select-String -Pattern "nvidia" -Context 0,2

Provider Check (Step 4)

# Check if providers are already registered (errors indicate CRD not yet installed — expected on a fresh cluster)
kubectl get inferenceproviderconfigs --all-namespaces

Provider Discovery (Step 4)

# List available providers
Get-ChildItem providers/

# Check default image for a provider
Get-Content providers/<provider>/Makefile | Select-String -Pattern 'IMG\s*\?='

HuggingFace Token Secret (Step 5)

$token = Read-Host -Prompt "HuggingFace token" -AsSecureString
$bstr = [System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($token)
try {
  [System.Runtime.InteropServices.Marshal]::PtrToStringAuto($bstr) |
    Set-Content -NoNewline -Encoding UTF8 -Path "$env:TEMP\hf-token.txt"
} finally {
  [System.Runtime.InteropServices.Marshal]::ZeroFreeBSTR($bstr)
}

kubectl create secret generic hf-token `
  --from-file=token="$env:TEMP\hf-token.txt" `
  -n <namespace> `
  --dry-run=client -o yaml | kubectl apply -f -

Remove-Item -Force "$env:TEMP\hf-token.txt"

ModelDeployment CR (Step 5)

For gated models (Llama etc.):

$manifest = @"
apiVersion: airunway.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: <model-name>
  namespace: <namespace>
spec:
  model:
    id: <model-id>
    huggingFaceTokenSecretRef:
      name: hf-token
      key: token
  provider:
    name: <provider-name>
"@
$manifest | kubectl apply -f -

For non-gated models (Phi-3, Gemma etc.):

$manifest = @"
apiVersion: airunway.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: <model-name>
  namespace: <namespace>
spec:
  model:
    id: <model-id>
  provider:
    name: <provider-name>
"@
$manifest | kubectl apply -f -

Smoke Test (Step 6)

$endpoint = kubectl get modeldeployment <model-name> -n <namespace> -o jsonpath='{.status.endpoint}'

if ([string]::IsNullOrEmpty($endpoint)) {
  Write-Host "Endpoint not yet available — model may still be starting"
} else {
  $body = @{
    model    = "<model-id>"
    messages = @(@{ role = "user"; content = "Hello!" })
  } | ConvertTo-Json -Depth 3

  Invoke-RestMethod -Method Post -Uri "$endpoint/v1/chat/completions" `
    -ContentType "application/json" -Body $body
}

Step 1 — Cluster Verification

Goal: Confirm prerequisites are met, validate the cluster connection, and inventory available nodes and GPUs.

Prerequisites

Verify required CLI tools are available before proceeding:

# Check all required tools are installed
for tool in kubectl make curl; do
  command -v "$tool" >/dev/null 2>&1 && echo "✓ $tool found" || echo "✗ $tool NOT FOUND — install before continuing"
done

See powershell-notes.md for the PowerShell equivalent.

If any tool is missing, STOP and tell the user which tools to install before continuing.

Cluster Connection

# Confirm active context
kubectl config current-context

# Inventory all nodes
kubectl get nodes -o wide

GPU Detection

# Extract GPU count and model per node (requires NVIDIA device plugin labels)
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable['\''nvidia.com/gpu'\'']}{"\t"}{.metadata.labels['\''nvidia.com/gpu.product'\'']}{"\n"}{end}'

Note: The above command relies on nvidia.com/gpu.product node labels, which are set by the NVIDIA device plugin or GPU operator. If the output shows empty GPU fields, try the fallback:

>

```bash

# Fallback: check node descriptions for GPU capacity

kubectl describe nodes | grep -A 5 "Allocatable:" | grep -i nvidia

```

>

If neither approach shows GPUs but you know the nodes have GPU hardware, the NVIDIA device plugin may not be installed yet. Guide the user to install it before proceeding.

Report to user:

• Cluster context name
• Total node count and GPU node count
• Per GPU type: model, count, VRAM per card, total cluster VRAM

Decision logic:

• No kubeconfig context → STOP. Tell user to configure kubeconfig (e.g., az aks get-credentials) and retry.
• No GPU nodes detected → Note "CPU-only cluster" and proceed; CPU-only inference is available via KAITO + llama.cpp.

Step 2 — Controller Status & Installation

Goal: Ensure the AI Runway controller and CRDs are installed and healthy.

# Check CRD presence
kubectl get crd modeldeployments.airunway.ai

# Check controller pod health
kubectl get pods -n airunway-system -l control-plane=controller-manager

If already installed and healthy: Report version and pod status, skip to Step 3.

If not installed: Ask user to confirm, then from the repository root run:

make controller-install   # Install CRDs
make controller-deploy    # Deploy controller manager

Note: make targets must be run from the AI Runway repository root.

Verify rollout:

kubectl rollout status deployment/airunway-controller-manager -n airunway-system --timeout=120s

Error handling:

• CrashLoopBackOff → kubectl logs -n airunway-system -l control-plane=controller-manager --previous
• Rollout timeout → kubectl describe deployment airunway-controller-manager -n airunway-system
• No rule to make target → Navigate to repo root and retry

Step 3 — GPU Assessment

Goal: Match detected hardware to known profiles and surface compatibility constraints.

Load and consult gpu-profiles.md.

For each GPU type detected in Step 1: 1. Look up the model in gpu-profiles.md 2. Report: VRAM per card, total cluster VRAM, supported dtypes, recommended attention backend 3. Surface compatibility warnings:

CPU-only path: Note CPU-only inference is available via KAITO + llama.cpp. Recommend google/gemma-3-1b-it-qat-q8_0-gguf.

Use the Model Sizing Guide in model-sizing.md to calculate maximum model size for the cluster.

Step 4 — Provider Recommendation & Installation

Goal: Select and install the right inference provider for the detected hardware.

# Check if providers are already registered
kubectl get inferenceproviderconfigs --all-namespaces 2>/dev/null || kubectl get inferenceproviderconfigs

See powershell-notes.md for the PowerShell equivalent.

If providers already registered: Show name and status, skip installation.

Provider recommendation logic:

Default to KAITO unless the user has a specific reason to choose otherwise. KAITO is the simplest to set up and handles most use cases. Dynamo is for teams that need to independently scale the prefill and decode stages of inference for high throughput. KubeRay is for teams already invested in the Ray ecosystem.

Present recommendation with reasoning. Ask user to confirm before installing.

Installation — from the repository root:

First, check the provider's Makefile or README for the default image:

# List available providers and their default images
ls providers/
cat providers/<provider>/Makefile | grep -E 'IMG\s*\?='

See powershell-notes.md for the PowerShell equivalent.

Then deploy:

cd providers/<provider>
make deploy IMG=<image>

Tip: If the Makefile defines a default IMG, you can omit the IMG= argument and just run make deploy.

Verify registration:

kubectl get inferenceproviderconfigs <provider> -o yaml

Check status.ready: true. If not ready within 2 minutes, inspect provider pod logs with kubectl logs <pod-name> and optionally use kubectl describe pod <pod-name> to review events and pod status.

Step 5 — First Model Deployment

Goal: Deploy a starter model that fits the cluster's GPU capacity and reach Ready status.

Model recommendations (based on cluster VRAM from Step 3 — see gpu-profiles.md for full sizing guide):

Present recommendation. Ask user to confirm or choose their own model.

Naming Convention

• `model-name`: A DNS-safe Kubernetes resource name derived from the model ID (e.g., meta-llama/Llama-3.1-8B-Instruct → llama-3-1-8b-instruct)
• `model-id`: The full HuggingFace model identifier (e.g., meta-llama/Llama-3.1-8B-Instruct)

Gated Models (HuggingFace Token)

Llama-family models are gated on HuggingFace and require an access token. Read the token interactively to avoid exposing it in shell history:

# Read token without echo, store in a securely permissioned temp file
hf_token_file="$(mktemp)"
trap 'rm -f "$hf_token_file"' EXIT

read -r -s -p "Enter HuggingFace token: " hf_token
printf '\n'
printf '%s' "$hf_token" > "$hf_token_file"
chmod 600 "$hf_token_file"
unset hf_token

kubectl create secret generic hf-token \
  --from-file=token="$hf_token_file" \
  -n <namespace> \
  --dry-run=client -o yaml | kubectl apply -f -

See powershell-notes.md for the PowerShell equivalent.

Apply the ModelDeployment CR

Use the appropriate manifest based on whether the model is gated.

For gated models (Llama etc. — requires HuggingFace token):

kubectl apply -f - <<EOF
apiVersion: airunway.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: <model-name>
  namespace: <namespace>
spec:
  model:
    id: <model-id>
    huggingFaceTokenSecretRef:
      name: hf-token
      key: token
  provider:
    name: <provider-name>
EOF

For non-gated models (Phi-3, Gemma etc. — no token required):

kubectl apply -f - <<EOF
apiVersion: airunway.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: <model-name>
  namespace: <namespace>
spec:
  model:
    id: <model-id>
  provider:
    name: <provider-name>
EOF

See powershell-notes.md for the PowerShell equivalent.

Monitor Deployment

kubectl get modeldeployment <model-name> -n <namespace> -w

Wait until STATUS = Ready. If not ready within 10 minutes:

• kubectl describe modeldeployment <model-name> -n <namespace> — check events
• kubectl get pods -l modeldeployment=<model-name> -n <namespace> — check pod status

Note: Large models (13B+) may take significantly longer than 10 minutes because weights must be downloaded first. For 70B models, expect 20–40+ minutes depending on network speed. Check pod logs to confirm download progress rather than assuming failure.

Step 6 — Summary & Smoke Test

Goal: Confirm everything is working and guide the user on next steps.

Report:

• Cluster context and node inventory (Step 1)
• Controller version and namespace (Step 2)
• GPU types and constraints in effect (Step 3)
• Provider installed and registration status (Step 4)
• Model deployed and endpoint URL (Step 5)

Smoke Test

Retrieve the endpoint and test it (only after STATUS = Ready):

Replace <namespace> with the namespace used during deployment (default: default).

ENDPOINT=$(kubectl get modeldeployment <model-name> -n <namespace> -o jsonpath='{.status.endpoint}')

if [ -z "$ENDPOINT" ]; then
  echo "Endpoint not yet available — model may still be starting"
else
  curl -X POST "${ENDPOINT}/v1/chat/completions" \
    -H "Content-Type: application/json" \
    -d '{"model": "<model-id>", "messages": [{"role": "user", "content": "Hello!"}]}'
fi

If the endpoint is a cluster-internal URL (e.g., http://svc-name.namespace:port), set up port-forwarding first:

>

```bash

# Discover the service port (vLLM: 8000, llama.cpp: 8080)

SERVICE_PORT=$(kubectl get svc <service-name> -n <namespace> -o jsonpath='{.spec.ports[0].port}')

>

kubectl port-forward svc/<service-name> 8080:${SERVICE_PORT} -n <namespace> &

# Then use http://localhost:8080 as the endpoint

```

See powershell-notes.md for the PowerShell equivalent.

Expected output: A JSON response with a choices array containing the model's reply.

Suggest next steps:

• Open the AI Runway Web UI to browse and deploy additional models
• Configure an ingress gateway for external access
• Review controller/config/samples/ for advanced ModelDeployment options

Troubleshooting & Rollback

Error Handling

Rollback

If a step fails and you need to undo a partial setup, work backwards through the steps: