K8s Service Mesh

Kubernetes Service Mesh (Istio)

Traffic management, security, and observability using kubectl-mcp-server's Istio/Kiali tools.

When to Apply

Use this skill when:

• User mentions: "Istio", "service mesh", "mTLS", "VirtualService", "traffic shifting"
• Operations: traffic management, canary deployments, security policies
• Keywords: "sidecar", "proxy", "traffic split", "mutual TLS"

Priority Rules

Quick Reference

Quick Status Check

Detect Istio Installation

istio_detect_tool()

Check Proxy Status

istio_proxy_status_tool()
istio_sidecar_status_tool(namespace)

Analyze Configuration

istio_analyze_tool(namespace)

Traffic Management

VirtualServices

List and inspect:

istio_virtualservices_list_tool(namespace)
istio_virtualservice_get_tool(name, namespace)

See TRAFFIC-SHIFTING.md for canary and blue-green patterns.

DestinationRules

istio_destinationrules_list_tool(namespace)

Gateways

istio_gateways_list_tool(namespace)

Traffic Shifting Patterns

Canary Release (Weight-Based)

VirtualService for 90/10 split:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: my-service
spec:
  hosts:
  - my-service
  http:
  - route:
    - destination:
        host: my-service
        subset: stable
      weight: 90
    - destination:
        host: my-service
        subset: canary
      weight: 10

Apply and verify:

kubectl_apply(vs_yaml, namespace)
istio_virtualservice_get_tool("my-service", namespace)

Header-Based Routing

Route beta users:

http:
- match:
  - headers:
      x-user-type:
        exact: beta
  route:
  - destination:
      host: my-service
      subset: canary
- route:
  - destination:
      host: my-service
      subset: stable

Security (mTLS)

See MTLS.md for detailed mTLS configuration.

PeerAuthentication (mTLS Mode)

istio_peerauthentications_list_tool(namespace)

AuthorizationPolicy

istio_authorizationpolicies_list_tool(namespace)

Observability

Proxy Metrics

istio_proxy_status_tool()

Hubble (Cilium Integration)

If using Cilium with Istio:

hubble_flows_query_tool(namespace)
cilium_endpoints_list_tool(namespace)

Troubleshooting

Sidecar Not Injected

istio_sidecar_status_tool(namespace)

Traffic Not Routing

istio_analyze_tool(namespace)
istio_virtualservice_get_tool(name, namespace)
istio_destinationrules_list_tool(namespace)
istio_proxy_status_tool()

mTLS Failures

istio_peerauthentications_list_tool(namespace)

Common Issues

Multi-Cluster Service Mesh

Istio multi-cluster setup:

istio_proxy_status_tool(context="primary")
istio_virtualservices_list_tool(namespace, context="primary")

istio_proxy_status_tool(context="remote")

Prerequisites

• Istio: Required for all Istio tools

  istioctl install --set profile=demo

Related Skills

• k8s-deploy - Deployment with traffic shifting
• k8s-security - Authorization policies

Istio mTLS Configuration

Mutual TLS setup and troubleshooting.

mTLS Modes

PeerAuthentication

Namespace-Wide STRICT

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: my-namespace
spec:
  mtls:
    mode: STRICT

Workload-Specific

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: my-app-mtls
  namespace: my-namespace
spec:
  selector:
    matchLabels:
      app: my-app
  mtls:
    mode: STRICT

Port-Level Override

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: my-app-mtls
spec:
  selector:
    matchLabels:
      app: my-app
  mtls:
    mode: STRICT
  portLevelMtls:
    8080:
      mode: PERMISSIVE  # Allow plaintext on this port

DestinationRule for mTLS

Client-side mTLS configuration:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: my-app
spec:
  host: my-app.my-namespace.svc.cluster.local
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL  # Use Istio certs

TLS Modes in DestinationRule

Mesh-Wide Policy

Enable STRICT mTLS for entire mesh:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system  # Mesh-wide
spec:
  mtls:
    mode: STRICT

MCP Commands

# List PeerAuthentications
istio_peerauthentications_list_tool(namespace)

# List DestinationRules
istio_destinationrules_list_tool(namespace)

# Check proxy status
istio_proxy_status_tool()

# Analyze for issues
istio_analyze_tool(namespace)

Migration to STRICT mTLS

Step 1: Inventory Non-Mesh Services

istio_sidecar_status_tool(namespace)
# Identify services without sidecars

Step 2: Enable PERMISSIVE (Default)

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: my-namespace
spec:
  mtls:
    mode: PERMISSIVE

Step 3: Verify All Traffic is mTLS

Monitor in Kiali or Prometheus that all traffic is encrypted.

Step 4: Switch to STRICT

spec:
  mtls:
    mode: STRICT

Troubleshooting

Connection Refused

Cause: Client not using mTLS, server requires STRICT.

Check:

istio_peerauthentications_list_tool(namespace)
istio_sidecar_status_tool(namespace)  # Client has sidecar?

Resolution:

• Ensure client has sidecar
• Or set PERMISSIVE mode

Certificate Errors

Check:

istio_proxy_status_tool()
# Look for certificate sync issues

Common Causes:

• Certificate expired
• istiod not running
• Time sync issues

Mixed mTLS Modes

Symptoms: Intermittent failures between services.

Check:

istio_peerauthentications_list_tool(namespace)
istio_destinationrules_list_tool(namespace)

Resolution:

• Ensure consistent modes
• PeerAuthentication matches DestinationRule

Best Practices

1. Start with PERMISSIVE

• Enables gradual migration
• No disruption to existing traffic

2. Namespace-Level Policies

• Easier to manage than per-workload
• Override only when necessary

3. Audit Regularly

   for ns in namespaces:
       istio_peerauthentications_list_tool(ns)

4. Monitor Certificate Expiry

• Istio auto-rotates certs
• But check istiod health

5. Test in Staging First

• STRICT mode can break connectivity
• Verify all services have sidecars

Istio Traffic Shifting Patterns

Advanced traffic management with VirtualServices.

Weight-Based Routing (Canary)

Gradual Rollout

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
  - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 90
    - destination:
        host: reviews
        subset: v2
      weight: 10

Progressive Weights

MCP Commands

# Apply weight change
apply_manifest(virtualservice_yaml, namespace)

# Verify
istio_virtualservice_get_tool(name, namespace)

Header-Based Routing

Beta Users

http:
- match:
  - headers:
      x-user-type:
        exact: beta
  route:
  - destination:
      host: reviews
      subset: v2
- route:
  - destination:
      host: reviews
      subset: v1

Internal Testing

http:
- match:
  - headers:
      x-internal:
        exact: "true"
  route:
  - destination:
      host: reviews
      subset: canary

Cookie-Based Routing

http:
- match:
  - headers:
      cookie:
        regex: "^(.*?;)?(user=beta)(;.*)?$"
  route:
  - destination:
      host: reviews
      subset: v2

Source-Based Routing

By Service Identity

http:
- match:
  - sourceLabels:
      app: productpage
      version: v2
  route:
  - destination:
      host: reviews
      subset: v2

Mirroring (Shadow Traffic)

Test new version with production traffic:

http:
- route:
  - destination:
      host: reviews
      subset: v1
  mirror:
    host: reviews
    subset: v2
  mirrorPercentage:
    value: 100.0

Fault Injection

Delay

http:
- fault:
    delay:
      percentage:
        value: 10
      fixedDelay: 5s
  route:
  - destination:
      host: reviews

Abort

http:
- fault:
    abort:
      percentage:
        value: 10
      httpStatus: 500
  route:
  - destination:
      host: reviews

Timeout and Retry

http:
- timeout: 10s
  retries:
    attempts: 3
    perTryTimeout: 3s
    retryOn: 5xx,reset
  route:
  - destination:
      host: reviews

DestinationRule for Subsets

Required companion to VirtualService:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: reviews
spec:
  host: reviews
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
  - name: canary
    labels:
      version: canary

Traffic Shifting Workflow

# 1. Ensure DestinationRule exists
istio_destinationrules_list_tool(namespace)

# 2. Check current VirtualService
istio_virtualservice_get_tool(name, namespace)

# 3. Apply new weights
apply_manifest(updated_vs, namespace)

# 4. Verify change
istio_virtualservice_get_tool(name, namespace)

# 5. Monitor (with Kiali or Prometheus)
istio_analyze_tool(namespace)

Rollback

Quick rollback to stable version:

http:
- route:
  - destination:
      host: reviews
      subset: v1
    weight: 100

Apply immediately:

apply_manifest(rollback_vs, namespace)