Kubernetes Skill

KubeShark: Failure-Mode Workflow for Kubernetes

Run this workflow top to bottom.

1) Capture execution context

Record before writing manifests:

• cluster version (e.g. 1.30, 1.31) and distribution (EKS, GKE, AKS, k3s, vanilla)
• target namespace and environment criticality (dev/staging/prod)
• workload type (Deployment, StatefulSet, Job, CronJob, DaemonSet)
• deployment method (raw YAML, Helm, Kustomize, operator-managed)
• policy enforcement (Pod Security Admission level, Kyverno, OPA/Gatekeeper)
• cloud provider and CNI (affects networking, storage classes, load balancers)
• platform controllers/add-ons (GitOps, observability, ingress, service mesh, autoscaling)

If unknown, state assumptions explicitly.

2) Diagnose likely failure mode(s)

Select one or more based on user intent and risk:

• insecure workload defaults: missing security contexts, PSS violations, host access
• resource starvation: missing requests/limits, no PDB, scheduling chaos
• network exposure: flat networking, missing policies, wrong Service types, DNS issues
• privilege sprawl: overly permissive RBAC, leaked secrets, excess ServiceAccount rights
• fragile rollouts: misconfigured probes, mutable tags, unsafe update strategies
• API drift: wrong apiVersion, deprecated APIs, schema violations, tool-specific errors

3) Load only the relevant reference file(s)

Primary failure-mode references:

• references/insecure-workload-defaults.md
• references/resource-starvation.md
• references/network-exposure.md
• references/privilege-sprawl.md
• references/fragile-rollouts.md
• references/api-drift.md

Supplemental references (only when needed):

• references/deployment-patterns.md
• references/stateful-patterns.md
• references/job-patterns.md
• references/daemonset-operator-patterns.md
• references/security-hardening.md
• references/observability.md
• references/multi-tenancy.md
• references/storage-and-state.md
• references/helm-patterns.md
• references/kustomize-patterns.md
• references/validation-and-policy.md
• references/examples-good.md
• references/examples-bad.md
• references/do-dont-patterns.md

Conditional Reference Retrieval (CRR) references (load only when the signal is detected):

• references/conditional/eks-patterns.md for EKS, AWS, IRSA, EKS Pod Identity, AWS Load Balancer Controller, EBS/EFS CSI, Karpenter
• references/conditional/gke-patterns.md for GKE, Autopilot, Workload Identity Federation for GKE, Dataplane V2, GCE Ingress, Config Sync
• references/conditional/aks-patterns.md for AKS, Microsoft Entra Workload ID, Azure CNI, AGIC, Azure Disk/File/Blob CSI
• references/conditional/openshift-patterns.md for OpenShift, OKD, ROSA, ARO, Routes, SCCs, OLM, oc
• references/conditional/gitops-controllers.md for Argo CD, ApplicationSet, Flux, GitOps reconciliation, sync waves
• references/conditional/observability-stacks.md for Prometheus Operator, ServiceMonitor, PodMonitor, OpenTelemetry, Loki, Grafana

Do not load multiple CRR files unless the task spans multiple detected platforms/tools.

4) Propose fix path with explicit risk controls

For each fix, include:

• why this addresses the failure mode
• what could still go wrong at deploy time or runtime
• guardrails (validation commands, policy checks, rollback path)

5) Generate implementation artifacts

When applicable, output:

• Kubernetes manifests (YAML with security contexts, resource limits, labels)
• Helm values/templates or Kustomize overlays
• NetworkPolicies, RBAC resources, PodDisruptionBudgets
• Policy rules (Kyverno/OPA) and admission controls

6) Validate before finalize

Always provide validation steps tailored to deployment method and risk tier:

• kubectl apply --dry-run=server or kubectl diff
• kubeconform for schema validation against target cluster version
• cross-resource consistency check (label/selector/port alignment)
• policy scan (PSS profile check, Kyverno/OPA audit)

Never recommend direct production apply without reviewed diff and approval.

7) Output contract

Return:

• assumptions and cluster version floor
• selected failure mode(s)
• chosen remediation and tradeoffs
• validation/test plan
• rollback/recovery notes (rollout undo, revision history, data safety)

Summary

• What changed?
• Why is this needed?
• Which failure mode(s) does this address?

Failure-Mode Coverage

Select all that apply:

• [ ] insecure-workload-defaults
• [ ] resource-starvation
• [ ] network-exposure
• [ ] privilege-sprawl
• [ ] fragile-rollouts
• [ ] api-drift
• [ ] Not applicable

Quality Impact

• Hallucination or error pattern reduced:
• Expected quality gain:
• Token-cost impact (higher/lower/neutral):

Validation Performed

• [ ] Ran markdown/link checks
• [ ] Checked SKILL frontmatter and file structure
• [ ] Verified all referenced files exist
• [ ] (If content change) sanity-checked YAML examples for correctness

Safety Checklist

• [ ] No secrets or credentials added
• [ ] No contradictory guidance across files
• [ ] Cluster version and API version statements are internally consistent
• [ ] Examples are newly written and not copied from external repos

Changelog

v1.0.0

Initial release of Kubernetes Skill (KubeShark).

• 6 primary failure modes: insecure workload defaults, resource starvation, network exposure, privilege sprawl, fragile rollouts, API drift
• 7-step failure-mode-first diagnostic workflow
• 20 granular reference files covering failure modes, workload patterns, cross-cutting concerns, tooling, and examples
• Production-ready YAML, Helm, and Kustomize examples
• Good/bad/do-dont pattern banks with LLM mistake checklists
• HonKit documentation site
• GitHub Actions CI validation and docs deployment
• Conventional commits and semantic versioning

Code of Conduct

Our Standards

We are committed to providing a welcoming and respectful environment for everyone, regardless of experience level.

Expected behavior:

• Be respectful and constructive in discussions and code reviews
• Accept constructive criticism gracefully
• Focus on what is best for the project and community
• Show empathy towards other contributors

Unacceptable behavior:

• Harassment, insults, or derogatory comments
• Trolling or deliberately inflammatory remarks
• Publishing others' private information without consent
• Any conduct that would be considered inappropriate in a professional setting

Scope

This code of conduct applies to all project spaces: issues, pull requests, discussions, and any other communication channels associated with this repository.

Attribution

This Code of Conduct is adapted from the Contributor Covenant, version 2.1.

Contributing

Thanks for contributing to Kubernetes Skill (KubeShark).

Goal of this repo

Improve Kubernetes manifest quality while staying lean on token usage.

Every change should answer:

• which failure mode does this prevent?
• what measurable quality gain does it provide?
• is the token cost justified?

Development flow

1. create a branch 2. make focused changes 3. run local checks 4. open PR using .github/PULL_REQUEST_TEMPLATE.md

Local checks

# quick sanity checks
rg -n "FIXME|placeholder-text" README.md SKILL.md references/*.md
python - <<'PY'
from pathlib import Path
assert Path('SKILL.md').exists()
assert Path('README.md').exists()
for p in [
  'references/insecure-workload-defaults.md',
  'references/resource-starvation.md',
  'references/network-exposure.md',
  'references/privilege-sprawl.md',
  'references/fragile-rollouts.md',
  'references/api-drift.md',
]:
  assert Path(p).exists(), f'missing {p}'
print('basic structure OK')
PY

Content rules

• Keep examples original and clearly distinct.
• Prefer failure-mode framing over generic "best-practice dump" text.
• Avoid cloud-provider-specific deep dives unless they directly reduce a known LLM failure mode.
• Keep claims precise; avoid vague "always" language when tradeoffs exist.
• Default to the Pod Security Standards restricted profile in all examples.

Required for PR approval

• clear mapping to one or more failure modes
• no contradictory guidance across references
• updated links/indexes if files were moved/renamed
• validation workflow passing (.github/workflows/validate.yml)

Security

• never commit credentials, tokens, or secret values
• do not paste real cluster state or kubeconfig data
• do not include real IP addresses, hostnames, or cloud account identifiers

Community

Open an issue with:

• observed hallucination/failure pattern
• minimal reproducible prompt/context
• expected behavior

Token Efficiency

How KubeShark minimizes context window consumption while maximizing manifest generation quality.

The Problem

Context window space is a finite resource. Every token spent on skill content is a token unavailable for the user's actual manifests, conversation history, and tool results. A monolithic skill file that dumps thousands of lines of Kubernetes guidance wastes context on information irrelevant to the current task. This is not just inefficient -- it degrades output quality by forcing the model to process noise alongside signal.

KubeShark's Approach

KubeShark is designed around three principles:

Lean Activation

The core SKILL.md is approximately 85 lines (~650 tokens). It contains no YAML examples, no inline manifests, no tutorial material. It is purely procedural: a 7-step workflow the model follows. This means the skill activates with minimal context cost regardless of the task.

Granular References

Depth lives in 20 separate reference files organized by concern:

• 6 failure mode files -- insecure workload defaults, resource starvation, network exposure, privilege sprawl, fragile rollouts, API drift
• 4 workload pattern files -- Deployments, StatefulSets, Jobs/CronJobs, DaemonSets and operators
• 4 cross-cutting concern files -- security hardening, observability, multi-tenancy, storage and state
• 3 tooling files -- Helm patterns, Kustomize patterns, validation and policy
• 3 pattern bank files -- good examples, bad examples, do/don't checklist

The model loads only the 1-2 files relevant to the diagnosed failure mode. A query about probe configuration never loads the RBAC guidance. A query about Helm chart structure never loads the NetworkPolicy patterns.

Selective Loading

Step 3 of the workflow explicitly instructs the model to load only the relevant references. This is not a suggestion -- it is a structural constraint built into the diagnostic flow.

Content Inclusion Rules

Content enters KubeShark only when at least one condition is met:

• It materially lowers the probability of insecure, unreliable, or invalid manifest generation
• It prevents common deploy-time or runtime surprises (probe cascades, selector mismatches, OOMKills)
• It encodes operational guardrails that general model knowledge cannot reliably infer

Content is excluded when:

• It is generic Kubernetes knowledge with low failure impact
• It is cloud-provider-specific deep configuration that belongs in project docs
• It duplicates an existing rule without adding a new decision signal

What Models Need Help With

LLMs have strong general Kubernetes knowledge but consistently fail on specific operational details:

• Security contexts -- models frequently omit them entirely, producing root-running containers
• Cross-resource consistency -- label/selector/port alignment across Deployment, Service, Ingress, HPA, PDB
• API version currency -- models generate removed APIs from training data (e.g., extensions/v1beta1)
• Provider-specific constraints -- storage class capabilities, CNI behavior, load balancer semantics
• Probe design -- liveness probes that check external dependencies, causing cascading failures

Models generally do not need help with basic YAML syntax, resource kind selection, or standard field names. KubeShark avoids restating what models already know reliably.

Core Principle

High signal density. Every line in every reference file must earn its token cost by reducing the probability of a specific, named failure mode.

Multi-Tenancy

Running multiple teams, environments, or customers in a single Kubernetes cluster requires defense-in-depth isolation. Namespaces are the primary boundary, but a namespace without quotas, network policies, RBAC scoping, and Pod Security Admission is an open door. This guide covers the five layers of namespace isolation and when to use separate clusters instead.

Namespace as the Isolation Unit

Every Kubernetes isolation mechanism is scoped to namespaces: RBAC, NetworkPolicy, ResourceQuota, LimitRange, and Pod Security Admission. A well-configured tenant namespace enforces all five simultaneously. An unconfigured namespace provides none of them.

Layer 1: ResourceQuota

Every tenant namespace must have a ResourceQuota. Without it, a single tenant can consume all cluster CPU, memory, and storage, starving other tenants.

ResourceQuota sets aggregate caps on the namespace: total CPU requests, memory limits, pod count, PVC count, and service types. When a ResourceQuota exists, every pod in the namespace must specify resource requests and limits or admission is rejected. This enforces resource discipline across all workloads.

Key settings for shared clusters:

• services.nodeports: "0" prevents tenants from claiming node ports that conflict across namespaces.
• services.loadbalancers limits the number of cloud load balancers a tenant can provision.
• persistentvolumeclaims caps storage consumption.

Layer 2: LimitRange

LimitRange complements ResourceQuota by setting per-container defaults and bounds. Without LimitRange, a pod that omits resource specifications is rejected by the quota (since the quota requires explicit resources). LimitRange provides sensible defaults so that "lazy" deployments still get resource boundaries.

LimitRange also sets min/max bounds per container, preventing a single container from requesting disproportionate resources (e.g., 32Gi memory in a namespace with a 40Gi quota).

Layer 3: NetworkPolicy

By default, pods in different namespaces can communicate freely. Default-deny NetworkPolicy is the minimum viable network isolation for multi-tenancy.

A complete namespace network baseline consists of three policies: 1. Default deny all -- blocks all ingress and egress for every pod in the namespace. 2. Allow DNS -- permits egress to kube-system on port 53 so service discovery works. 3. Allow intra-namespace -- permits pods within the same namespace to communicate.

Additional policies are added as needed for cross-namespace communication (e.g., allowing the ingress controller namespace to reach application pods).

The AND/OR semantics of NetworkPolicy rules are critical for multi-tenancy: a namespaceSelector and podSelector in the same from entry are AND-ed (both must match). Separate from entries are OR-ed. Getting this wrong can either block legitimate traffic or open traffic to the entire cluster.

Layer 4: RBAC Scoping

Use namespace-scoped Role and RoleBinding for tenant access. ClusterRole and ClusterRoleBinding grant access across all namespaces and should be reserved for platform administrators.

Tenant RBAC should follow least privilege:

• Developers: get, list, watch, create, update, patch, delete on workload resources (Deployments, Services, ConfigMaps, Jobs). Read-only on Secrets.
• CI/CD pipelines: create, update, patch on Deployments and ConfigMaps. No access to Secrets (use external secret management).
• Monitoring: get, list, watch on pods, events, and metrics endpoints.

Never use wildcards (verbs: ["*"], resources: ["*"]) in tenant roles. See the Privilege Sprawl deep dive for details.

Layer 5: Pod Security Admission

Every tenant namespace must have PSA labels enforcing at minimum the baseline profile, and preferably restricted:

labels:
  pod-security.kubernetes.io/enforce: restricted
  pod-security.kubernetes.io/audit: restricted
  pod-security.kubernetes.io/warn: restricted

Set all three modes (enforce, audit, warn). enforce blocks non-compliant pods. audit logs violations. warn shows warnings to users during kubectl apply. Using all three provides defense-in-depth and visibility into violations that audit mode catches but enforce mode has not yet been enabled for.

Naming Conventions

Consistent namespace naming enables policy automation and cost attribution:

• Environment-based: prod-payments, staging-orders.
• Team-based: platform-monitoring, alpha-api.
• Tenant-based (SaaS): acme-prod, acme-staging.

Pick one pattern and enforce it with admission webhooks. Inconsistent naming makes RBAC, cost allocation, and policy application error-prone.

What Namespaces Do Not Isolate

Namespaces are a soft boundary. They do not provide:

• Node-level isolation. Pods from different namespaces share the same node kernel. A container escape or noisy neighbor affects all tenants on that node. Use taints/tolerations and dedicated node pools for hard isolation.
• Cluster-scoped resources. ClusterRoles, CRDs, PersistentVolumes, and Nodes are visible cluster-wide.
• Network without NetworkPolicy. Namespaces without NetworkPolicy allow all traffic by default.
• Container runtime isolation. A kernel exploit reaches the host regardless of namespace. Use sandboxed runtimes (gVisor, Kata Containers) for untrusted workloads.

When to Use Separate Clusters

Rule of thumb: use namespaces for internal teams in the same trust domain. Use separate clusters when tenants are external customers, have different compliance requirements, or when the blast radius of a cluster-level failure is unacceptable.

Further Reading

• Namespaces
• KubeShark Privilege Sprawl
• KubeShark Network Exposure

Storage and State

Misconfigured storage is the only Kubernetes failure mode that can cause irreversible data loss. Unlike compute issues (which resolve by restarting pods) or network issues (which resolve by fixing policies), a deleted PersistentVolume with reclaimPolicy: Delete destroys the underlying disk permanently. Every storage decision must account for data durability.

The PV/PVC Model

Kubernetes abstracts storage through three resources:

• PersistentVolume (PV): Represents a piece of provisioned storage -- a cloud disk, an NFS share, or a local SSD. PVs are cluster-scoped, not namespaced.
• PersistentVolumeClaim (PVC): A namespaced request for storage. Specifies size, access mode, and StorageClass. The control plane binds the PVC to a PV that satisfies its requirements.
• StorageClass: Defines how PVs are dynamically provisioned. Specifies the CSI driver, parameters (disk type, encryption), reclaim policy, and binding mode.

Dynamic provisioning is the default workflow: a PVC references a StorageClass, the CSI driver provisions a volume, and the control plane creates a PV and binds it to the PVC automatically.

StorageClass: Critical Fields

Two StorageClass defaults are dangerous for production data:

`reclaimPolicy: Delete` (the default) destroys the underlying volume when the PVC is deleted. A single kubectl delete pvc command permanently deletes the data. Production StorageClasses must use Retain, which preserves the volume for manual recovery.

`volumeBindingMode: Immediate` (the default) provisions the volume before a pod is scheduled. This can place the volume in a different availability zone than the pod, causing the pod to stay Pending indefinitely. WaitForFirstConsumer provisions the volume in the same zone as the pod.

Always set allowVolumeExpansion: true so PVCs can be resized without recreation. PVCs can be expanded but never shrunk.

Access Modes

The most common mistake: requesting ReadWriteMany with a block storage provisioner. Block storage is physically attached to a single node and cannot support RWX. The PVC stays in Pending state with no clear error message. Use a file storage solution (EFS, Filestore, Azure Files) for shared access.

For databases, prefer ReadWriteOncePod over ReadWriteOnce. RWO allows multiple pods on the same node to mount the volume, which can cause data corruption. RWOP restricts access to exactly one pod.

Dynamic Provisioning and CSI Drivers

Each cloud provider and storage platform has a CSI driver:

All major CSI drivers support snapshots, volume expansion, and encryption. Always enable encryption (parameters.encrypted: "true") for production StorageClasses.

VolumeSnapshot for Backup and Restore

VolumeSnapshots provide point-in-time copies of PVCs. They are the primary mechanism for data protection before destructive operations:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: db-snapshot-2025-04-12
spec:
  volumeSnapshotClassName: csi-snapclass
  source:
    persistentVolumeClaimName: data-postgres-0

To restore, create a new PVC with dataSource referencing the snapshot. The CSI driver provisions a new volume from the snapshot data.

Critical rules for snapshots:

• Always snapshot before PVC deletion, StorageClass migration, or major upgrades.
• Snapshots may be crash-consistent, not application-consistent. For databases, run logical backups (pg_dump, mysqldump) alongside snapshots.
• Test restore procedures regularly. A backup never restored is not a backup.

Ephemeral Storage: emptyDir

emptyDir volumes are tied to the pod lifecycle -- deleted when the pod is removed. Use them for scratch space, caches, and temporary files required by readOnlyRootFilesystem: true.

Always set sizeLimit on emptyDir volumes. Without it, a runaway process can fill the node's disk and trigger eviction of every pod on that node. For in-memory emptyDirs (medium: Memory), the size counts against the container's memory limit.

StatefulSet volumeClaimTemplates

StatefulSets create one PVC per replica automatically. PVCs created by volumeClaimTemplates are intentionally not deleted when the StatefulSet is deleted or scaled down -- this protects data. To reclaim storage, delete the PVCs manually after verifying the data is no longer needed.

The persistentVolumeClaimRetentionPolicy field (1.27+) can configure automatic PVC deletion on scale-down or StatefulSet deletion, but use it with extreme caution in production.

fsGroup and Permissions

When running containers as non-root with readOnlyRootFilesystem: true, mounted PVCs may not be writable because the volume's filesystem ownership does not match the container's user. Set fsGroup in the pod security context to ensure the mounted volume is writable by the pod's group:

securityContext:
  runAsUser: 10000
  runAsGroup: 10000
  fsGroup: 10000

Without fsGroup, the pod mounts the volume but cannot write to it, causing application errors that appear to be permission issues inside the container.

Further Reading

• Persistent Volumes
• Storage Classes
• KubeShark Resource Starvation

Workload Patterns

Kubernetes provides five workload resource types, each designed for a specific execution model. Choosing the wrong type forces workarounds that break update semantics, storage management, and scaling behavior. This guide provides a decision framework for selecting the right workload type.

Decision Matrix

Deployment

Use when: Pods are interchangeable and need no stable identity or persistent local storage. Web servers, REST/gRPC APIs, microservices, frontend proxies, stateless queue workers.

Key considerations:

• Always set replicas >= 2 for production with a PodDisruptionBudget.
• Use topologySpreadConstraints to distribute across zones and nodes.
• Pair with HPA for elastic scaling. Set scaleDown.stabilizationWindowSeconds to prevent flapping.
• Never put app.kubernetes.io/version in selector.matchLabels -- selectors are immutable and this breaks upgrades.

Common mistake: Using a Deployment with a RWO PersistentVolumeClaim and replicas > 1. Only one pod can mount a RWO volume at a time. The second replica stays Pending. Use a StatefulSet with per-pod volumes or switch to RWX storage.

StatefulSet

Use when: Pods need stable network identity (predictable DNS per pod), stable per-pod storage (PVC follows the pod across reschedules), or ordered deployment. Databases (PostgreSQL, MySQL), message brokers (Kafka, RabbitMQ), consensus systems (etcd, ZooKeeper).

Key considerations:

• Requires a headless Service (clusterIP: None) for per-pod DNS: <pod>.<service>.<ns>.svc.cluster.local.
• volumeClaimTemplates create one PVC per pod. PVCs are never auto-deleted on scale-down to protect data.
• podManagementPolicy: OrderedReady (default) creates pods sequentially. Use Parallel when pods initialize independently.
• Set terminationGracePeriodSeconds to 60-120 seconds for databases. The default 30 seconds is insufficient for clean shutdown.

Common mistake: Using a StatefulSet when a Deployment with a single PVC or an external database would suffice. If you only need storage (not per-pod identity), a Deployment is simpler. StatefulSets add operational complexity for ordered rollouts, scale-down behavior, and PVC lifecycle management.

DaemonSet

Use when: Exactly one pod must run on every qualifying node. Log collectors (Fluent Bit, Vector), monitoring agents (node-exporter, Datadog), network plugins (Cilium), CSI node drivers, security agents (Falco).

Key considerations:

• DaemonSets have no replicas field. The scheduler places one pod per qualifying node automatically.
• Resources are multiplied across every node. 100m CPU x 200 nodes = 20 CPU cores cluster-wide. Be conservative with requests.
• Use nodeSelector or nodeAffinity to target specific node pools. Add tolerations for tainted nodes (control-plane, GPU).
• Use a custom PriorityClass (not system-node-critical) for application-level agents.

Common mistake: Specifying a replicas field. DaemonSets do not support it -- the API rejects the manifest.

Job

Use when: Work runs to completion and then stops. Database migrations, data exports, ETL pipelines, one-time scripts, ML training runs.

Key considerations:

• restartPolicy must be Never or OnFailure. The default Always is rejected by the API for Jobs.
• Always set activeDeadlineSeconds to prevent runaway jobs.
• Always set ttlSecondsAfterFinished to auto-clean completed Jobs and their pods.
• Jobs may retry on failure. Every Job must be idempotent -- assume at-least-once execution.
• Use podFailurePolicy (1.26+) to distinguish retryable from fatal errors.

Common mistake: Using restartPolicy: Always, which is the default for pods but invalid for Jobs. LLMs frequently omit restartPolicy in Job specs, relying on the default that the API rejects.

CronJob

Use when: Work runs on a recurring schedule. Report generation, cache warming, log rotation, periodic health checks, certificate renewal.

Key considerations:

• Set concurrencyPolicy: Forbid by default. Overlapping runs cause resource exhaustion and data corruption.
• Set startingDeadlineSeconds to skip runs that are too late (prevents burst of overdue jobs after controller downtime).
• Set timeZone explicitly. Without it, the schedule uses the controller's clock (typically UTC).
• CronJobs have three label levels (CronJob, jobTemplate, pod template). All three need consistent labels.

Common mistake: Leaving concurrencyPolicy at the default Allow, which permits overlapping runs. A CronJob that takes 10 minutes, scheduled every 5 minutes, will accumulate concurrent instances until the cluster runs out of resources.

Anti-Patterns

• StatefulSet for stateless workloads. Adds unnecessary complexity. Use a Deployment.
• Deployment for one-shot tasks. The pod restarts forever after completion. Use a Job.
• DaemonSet when only some nodes need the workload. Use nodeSelector to target the correct subset, not a blanket DaemonSet with no selector.
• CronJob for long-running daemons. If the workload should run continuously, use a Deployment with HPA.

Further Reading

• Workloads
• KubeShark Good Patterns
• KubeShark Bad Patterns

Changelog

All notable changes to the Kubernetes Skill (KubeShark) are documented here. This project uses Semantic Versioning.

For the repository-level changelog, see CHANGELOG.md.

---

v1.0.0

Initial release of KubeShark.

Failure Modes

• 6 primary failure modes: insecure workload defaults, resource starvation, network exposure, privilege sprawl, fragile rollouts, API drift
• 7-step failure-mode-first diagnostic workflow (diagnose before generate)

Reference Files

• 20 granular reference files covering failure modes, workload patterns, cross-cutting concerns, tooling, and examples
• LLM mistake checklists in every reference file that covers a risk domain

Pattern Banks

• 8 production-ready good examples with annotated YAML
• 8 common anti-pattern bad examples with explanations
• Do/Don't checklist spanning 9 categories

Tooling

• Helm chart pattern guidance with template conventions
• Kustomize overlay and patch patterns
• Validation and policy enforcement (kubeconform, Kyverno, OPA/Gatekeeper, Polaris)

Infrastructure

• HonKit documentation site
• GitHub Actions CI validation and docs deployment
• Conventional commits and semantic versioning

Contributing

Thanks for contributing to Kubernetes Skill (KubeShark). This is a condensed guide. For the full version, see CONTRIBUTING.md.

Core Principle

Every change must map to a failure mode. Before submitting, answer three questions:

1. Which failure mode does this prevent? 2. What measurable quality gain does it provide? 3. Is the token cost justified?

Development Flow

1. Branch -- create a feature or fix branch from main 2. Change -- make focused changes; keep PRs small and single-purpose 3. Check -- run local checks (see below) 4. PR -- open a pull request using the PR template

Local Checks

# Verify no placeholder text remains
rg -n "FIXME|placeholder-text" README.md SKILL.md references/*.md

# Verify required files exist
python - <<'PY'
from pathlib import Path
assert Path('SKILL.md').exists()
assert Path('README.md').exists()
for p in [
  'references/insecure-workload-defaults.md',
  'references/resource-starvation.md',
  'references/network-exposure.md',
  'references/privilege-sprawl.md',
  'references/fragile-rollouts.md',
  'references/api-drift.md',
]:
  assert Path(p).exists(), f'missing {p}'
print('basic structure OK')
PY

Content Rules

• Keep examples original and clearly distinct
• Prefer failure-mode framing over generic best-practice text
• Avoid cloud-provider-specific deep dives unless they directly reduce a known LLM failure mode
• Keep claims precise; avoid vague "always" language when tradeoffs exist
• Default to the PSS restricted profile in all examples

Required for PR Approval

• Clear mapping to one or more failure modes
• No contradictory guidance across references
• Updated links and indexes if files were moved or renamed
• Validation workflow passing (.github/workflows/validate.yml)

Security

• Never commit credentials, tokens, or secret values
• Do not paste real cluster state or kubeconfig data
• Do not include real IP addresses, hostnames, or cloud account identifiers

Reporting Issues

Open an issue with: the observed hallucination or failure pattern, a minimal reproducible prompt/context, and the expected behavior.

Failure Modes

KubeShark organizes Kubernetes risks into six named failure modes. Every piece of guidance in the skill maps to at least one of these. Content that does not reduce the probability of any failure mode is excluded.

These are not arbitrary categories. They represent the six most common ways LLM-generated Kubernetes manifests cause real damage in production.

---

1. Insecure Workload Defaults

Containers running with overly permissive security settings because no explicit security context was provided.

Symptoms:

• Containers running as root (UID 0)
• Pods admitted without any securityContext
• Linux capabilities not dropped (CAP_NET_RAW, CAP_SYS_ADMIN still present)
• hostPath volumes mounted into workload pods
• Privileged containers that can escape to the node
• PodSecurity admission rejecting manifests at deploy time

Common causes:

• Upstream example manifests and Helm chart defaults rarely include security contexts
• LLMs train on those permissive examples and reproduce them verbatim
• securityContext has both pod-level and container-level fields; omitting either leaves gaps
• Confusion between PSS levels (privileged, baseline, restricted)

Risk pattern: A Deployment without a security context deploys successfully, runs as root, and becomes a container escape vector when a CVE is exploited. The cluster accepts it without complaint.

---

2. Resource Starvation

Workloads deployed without proper resource requests and limits, leading to scheduling failures, evictions, and cascading outages.

Symptoms:

• OOMKilled containers exceeding memory limits
• Pods stuck in Pending because the scheduler cannot find a node
• Node pressure evictions killing BestEffort pods
• CPU throttling causing invisible latency spikes
• Noisy neighbors starving co-located pods
• HPA flapping between replica counts

Common causes:

• Missing requests and limits entirely (BestEffort QoS, first to be evicted)
• Arbitrary round numbers (cpu: 1, memory: 1Gi) without profiling
• No PodDisruptionBudget -- voluntary disruptions take down all replicas
• CPU limits set too close to requests, causing constant CFS throttling
• No LimitRange to catch misconfigured pods at admission

Risk pattern: A pod without resource requests gets scheduled on an overcommitted node. Under load, the kubelet evicts it. The replacement pod lands on another overcommitted node. The cycle continues until the workload is effectively unavailable.

---

3. Network Exposure

Cluster networking left in the default open state, exposing all pods to all other pods and potentially to the internet.

Symptoms:

• All pods can reach all pods (Kubernetes default)
• Unexpected external exposure via NodePort or LoadBalancer Services
• DNS resolution failures from wrong Service names or missing namespace qualifiers
• Silent routing to nothing when Service selectors do not match pod labels
• Lateral movement after compromise because no NetworkPolicy exists
• Ingress 404s or 502s from path/backend mismatches

Common causes:

• Kubernetes has no network segmentation by default -- every pod can reach every other pod
• LLMs generate NodePort and LoadBalancer Services when ClusterIP is sufficient
• Service selectors silently fail when labels do not match (zero errors, zero traffic)
• No policy means allow-all, not deny-all
• Egress policies are forgotten -- ingress-only policies still allow unrestricted outbound

Risk pattern: A compromised pod in one namespace freely connects to the database in another namespace. No NetworkPolicy exists, so every service in the cluster is reachable. The blast radius of a single vulnerability is the entire cluster.

---

4. Privilege Sprawl

RBAC permissions, ServiceAccount tokens, and secret access granted far beyond what workloads actually require.

Symptoms:

• ClusterRoleBinding with cluster-admin attached to a workload ServiceAccount
• Rules containing verbs: ["*"] or resources: ["*"]
• Pods running with the default ServiceAccount (shared identity across the namespace)
• automountServiceAccountToken: true on pods that never call the Kubernetes API
• Secrets injected as environment variables (visible in kubectl describe pod and crash dumps)

Common causes:

• Copy-pasting cluster-admin bindings from quickstart guides
• Using wildcards to "get it working" and never scoping down
• Not creating dedicated ServiceAccounts per workload
• Misunderstanding that Kubernetes Secrets are base64-encoded, not encrypted
• Injecting secrets via env instead of volume mounts or external operators

Risk pattern: A web application pod runs with the default ServiceAccount, which has a ClusterRoleBinding to cluster-admin left over from initial setup. An SSRF vulnerability in the application allows an attacker to read the mounted token and take full control of the cluster.

---

5. Fragile Rollouts

Deployments that break during updates due to misconfigured probes, mutable image tags, or missing graceful shutdown handling.

Symptoms:

• Cascading restarts across all pods (liveness probe checks an external dependency)
• Dropped connections and 502s during deploys (readiness probe passes too early)
• All replicas unavailable simultaneously (maxUnavailable too high)
• Version drift across pods (:latest tag with cached layers)
• Pods killed before finishing in-flight requests (no preStop hook)
• Slow-starting apps killed in restart loops (no startup probe)

Common causes:

• Misunderstanding the difference between liveness and readiness probes
• Checking external dependencies (databases, APIs) in liveness probes
• Using :latest tags, which are mutable and nondeterministic
• Not setting terminationGracePeriodSeconds or preStop hooks
• maxUnavailable and maxSurge left at defaults without considering replica count

Risk pattern: A Deployment with a liveness probe that checks database connectivity deploys successfully. The database has a brief network blip. Every pod fails its liveness check simultaneously. Kubernetes restarts all pods at once, causing a full outage that outlasts the original database blip.

---

6. API Drift

Manifests using wrong, deprecated, or removed API versions that fail silently or break on cluster upgrades.

Symptoms:

• no matches for kind "Ingress" in version "extensions/v1beta1" (removed API)
• Warning: policy/v1beta1 PodDisruptionBudget is deprecated (deprecated, not yet removed)
• Fields silently ignored after upgrade (existed in beta, removed in stable)
• Helm templates render valid YAML but kubectl apply fails
• kubeconform reports schema violations

Common causes:

• LLM training data contains outdated manifests from blog posts and Stack Overflow
• Copy-paste from tutorials written for the Kubernetes 1.18-1.21 era
• Helm charts pinned to old API versions without Capabilities checks
• Not running schema validation against the target cluster version
• Confusing "deprecated" (still works, prints warning) with "removed" (hard failure)

Risk pattern: An LLM generates a manifest with apiVersion: extensions/v1beta1 for an Ingress resource. This was removed in Kubernetes 1.22. The manifest looks correct, passes YAML linting, but fails on any modern cluster. The correct version is networking.k8s.io/v1.

---

How Failure Modes Are Used

Failure modes drive the entire KubeShark workflow:

1. Step 2 (Diagnose) selects the relevant failure modes based on the task. 2. Step 3 (Load references) pulls the reference files that correspond to the diagnosed failure modes. 3. Step 4 (Propose) structures recommendations around preventing the specific risks identified. 4. Step 7 (Output contract) lists which failure modes were addressed, making the response auditable.

Most tasks involve multiple failure modes. A Deployment creation task typically triggers insecure workload defaults, resource starvation, and fragile rollouts at minimum. The workflow ensures none are overlooked.

Philosophy

This page describes the design rationale behind KubeShark. For the full treatment, see PHILOSOPHY.md in the repository root.

---

Failure-Mode-First vs. Reference Manuals

The core insight: telling an LLM what good Kubernetes looks like is less effective than telling it how to think about Kubernetes problems.

A static reference manual gives the model information but no diagnostic process. There is no risk assessment step, no structured output, and no way to verify that the right concerns were addressed. The model reads the reference and generates whatever it thinks fits.

KubeShark takes the opposite approach. The core SKILL.md is an operational workflow, not a knowledge dump. It forces a diagnostic sequence: capture context, identify failure modes, load only relevant references, propose fixes with risk controls, validate, and deliver a structured output contract. The model diagnoses before it generates.

---

Why Kubernetes Needs This More Than Terraform

Terraform fails explicitly. A misconfiguration surfaces at terraform plan or terraform apply with a clear error message. Kubernetes is different in three critical ways:

Silent failures are common. A Service with the wrong selector deploys successfully but routes to nothing. A NetworkPolicy with a mistyped label silently does nothing. A probe pointing to the wrong port passes creation but fails at runtime. The cluster accepts the manifest without complaint -- failures surface only when traffic arrives.

Runtime is continuous. Terraform is plan-and-apply. Kubernetes is a continuous reconciliation loop. A misconfigured liveness probe does not just fail once -- it restarts the pod every 30 seconds forever. A missing PodDisruptionBudget does not just affect one deploy -- it allows every future rolling update to take down all replicas simultaneously.

The blast radius is multi-dimensional. Terraform operates at infrastructure provisioning time. Kubernetes operates across provisioning, deployment, runtime, networking, scheduling, and security simultaneously. An LLM must reason about all these dimensions for every resource it generates.

These properties make a diagnostic workflow essential. Without one, the LLM produces syntactically valid but operationally dangerous manifests -- and the cluster silently accepts them.

---

Token Efficiency as Design Constraint

Context window space is a finite resource. Every token spent on skill content is a token unavailable for the user's actual manifests, conversation history, and tool results.

KubeShark is designed for minimal activation cost:

• SKILL.md is ~85 lines (~650 tokens). It contains no YAML examples, no inline manifests, and no tutorial material. It is purely procedural.
• 20 granular reference files. The model loads only the 1-2 files relevant to the diagnosed failure mode per query.
• No duplication. A query about probe configuration never loads the RBAC guidance. A query about Helm chart structure never loads the NetworkPolicy patterns.

A single large reference file would force the model to process thousands of irrelevant tokens. Twenty small files let it load precisely what it needs.

---

LLM-Aware Guardrails

Every reference file that covers a risk domain includes an LLM mistake checklist -- a list of specific errors that language models make when generating Kubernetes configurations:

• Omitting securityContext entirely, producing manifests that run as root
• Setting liveness probes that check external dependencies, causing cascading restarts
• Using apiVersion: extensions/v1beta1 for Ingress (removed in 1.22)
• Generating RBAC with wildcard verbs and resources on ClusterRoleBindings
• Omitting resource requests and limits, or using arbitrary round numbers
• Using :latest image tags without imagePullPolicy override
• Creating Services with selectors that do not match any pod labels

These checklists exist because the model needs to know what it gets wrong, not just what is correct. A reference that only shows the right pattern still allows the model to hallucinate the wrong one. A reference that explicitly names the hallucination pattern reduces it.

---

Output Contracts for Auditability

Every KubeShark response ends with a structured output contract: assumptions, failure modes addressed, remediation choices and tradeoffs, validation plan, and rollback notes.

This is a deliberate design choice. Kubernetes manifests applied to a cluster have real operational consequences. The output contract makes every response auditable -- a reviewer can check whether the model's assumptions matched reality, whether the right risks were identified, and whether the rollback path is viable, all before applying anything.

Without an output contract, the user receives a manifest and must independently assess whether it is safe. The contract shifts that burden: the model states what it assumed and what it did not account for.

---

Default Security Posture

KubeShark defaults to the Pod Security Standards restricted profile. Every generated workload includes:

• runAsNonRoot: true
• allowPrivilegeEscalation: false
• readOnlyRootFilesystem: true
• capabilities: { drop: ["ALL"] }
• seccompProfile: { type: RuntimeDefault }

The restricted profile prevents the largest class of container escape vulnerabilities. Deviations are allowed only when the user explicitly requests them, and the deviation is documented in the output contract with justification.

This is a secure-by-default posture. It is easier to relax security with documented justification than to retroactively harden manifests that were generated permissively.

Workflow

KubeShark operates through a 7-step workflow defined in SKILL.md. The workflow runs top to bottom on every Kubernetes task. This page explains what each step does and why it exists.

---

Step 1: Capture Execution Context

Before writing any YAML, KubeShark records the environment it is operating in. This prevents the most common LLM failure: generating manifests that assume a generic cluster and ignore the user's actual setup.

Context captured:

When any dimension is unknown, KubeShark states the assumption explicitly rather than guessing silently. These assumptions appear in the output contract (Step 7) so the user can verify them.

---

Step 2: Diagnose Failure Modes

This is the step that distinguishes KubeShark from a reference manual. Before generating anything, the workflow identifies which of the six failure modes are relevant to the task.

The six failure modes:

1. Insecure workload defaults -- missing security contexts, PSS violations, host access, excessive capabilities 2. Resource starvation -- missing requests/limits, no QoS strategy, absent PodDisruptionBudgets, scheduling chaos 3. Network exposure -- flat networking, missing NetworkPolicies, wrong Service types, DNS misconfigurations 4. Privilege sprawl -- overly permissive RBAC, leaked secrets, unscoped ServiceAccount tokens 5. Fragile rollouts -- misconfigured probes, mutable image tags, unsafe update strategies, missing graceful shutdown 6. API drift -- wrong apiVersion, deprecated APIs, schema violations, tool-specific structural errors

Most tasks trigger multiple failure modes. A "create a Deployment with an Ingress" request involves at least insecure workload defaults, network exposure, and fragile rollouts. The diagnosis step ensures none of these are overlooked.

See Failure Modes for a detailed breakdown of each.

---

Step 3: Load Targeted References

KubeShark includes 20 reference files, but only 1-2 are loaded per query. This is a deliberate token efficiency decision: loading all references would burn thousands of tokens on irrelevant guidance.

Reference selection logic:

• A probe configuration question loads fragile-rollouts.md -- it never touches privilege-sprawl.md or network-exposure.md.
• A Helm chart task loads helm-patterns.md and the failure-mode reference for the workload being charted.
• A security review loads insecure-workload-defaults.md and security-hardening.md.

Reference categories:

Each reference file is self-contained. No file depends on another being loaded simultaneously.

---

Step 4: Propose Fix Path

For every recommendation, KubeShark provides three things:

1. Why this addresses the failure mode -- the causal link between the fix and the diagnosed risk. 2. What could still go wrong -- runtime behavior, edge cases, and deployment-time risks that remain even after the fix. 3. Guardrails -- validation commands, policy checks, and rollback paths that protect against the remaining risks.

This structure prevents a common LLM pattern: recommending a fix without acknowledging its limitations. A liveness probe fix that does not mention the risk of checking external dependencies is incomplete. A NetworkPolicy recommendation that does not mention egress is incomplete.

---

Step 5: Generate Artifacts

When the task calls for implementation, KubeShark produces the appropriate artifacts:

• Kubernetes manifests -- YAML with security contexts, resource limits, proper labels, and annotations
• Helm values and templates -- chart structure following Helm best practices
• Kustomize overlays -- base/overlay structure with proper patch formats
• NetworkPolicies -- default-deny with explicit allow rules
• RBAC resources -- least-privilege Roles and RoleBindings with dedicated ServiceAccounts
• PodDisruptionBudgets -- tuned to workload replica count and availability requirements
• Policy rules -- Kyverno ClusterPolicies or OPA/Gatekeeper ConstraintTemplates

All generated manifests default to the Pod Security Standards restricted profile: runAsNonRoot: true, allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, drop: ["ALL"] capabilities, and RuntimeDefault seccomp profile.

---

Step 6: Validate

KubeShark never recommends applying directly to production without validation. Every response includes validation steps matched to the deployment method and risk level:

• `kubectl apply --dry-run=server` or `kubectl diff` -- catches API-level errors without making changes
• `kubeconform` -- schema validation against the target cluster version to catch API drift
• Cross-resource consistency checks -- verifies that labels, selectors, ports, and names align across Deployments, Services, Ingress, PDBs, HPAs, and NetworkPolicies
• Policy scan -- PSS profile compliance check, Kyverno audit, or OPA/Gatekeeper dry-run

Cross-resource consistency is especially important because Kubernetes silently accepts mismatched selectors. A Service with a selector that matches no pods deploys without error -- the failure only surfaces when traffic arrives.

---

Step 7: Output Contract

Every KubeShark response ends with a structured output contract containing five sections:

The output contract makes every response auditable. A reviewer can check whether the assumptions match reality, whether the right failure modes were identified, and whether the rollback path is viable -- all before applying anything to the cluster.

Bad Patterns -- Common LLM Anti-Patterns

These eight anti-patterns represent manifests that LLMs frequently generate. Each one compiles and appears valid but has serious issues in production. The danger of these patterns is that Kubernetes accepts them without error -- the failure only surfaces at runtime or under load.

For full annotated YAML with detailed explanations of what is wrong in each case, see references/examples-bad.md.

1. Deployment Running as Root with No Security Context

No securityContext at pod or container level -- container runs as root by default. Missing runAsNonRoot, allowPrivilegeEscalation: false, readOnlyRootFilesystem, capabilities drop, seccomp profile. Also lacks resource requests, probes, and standard labels. Uses :latest tag.

Failure modes: Insecure workload defaults, resource starvation, fragile rollouts.

2. Service with Selector That Matches No Pods

Service selector includes version: v1 but pods have version: v2. Kubernetes does not warn about selector mismatches -- the Service silently has zero endpoints. A frequent LLM mistake when updating version labels on the Deployment without updating the Service.

3. ClusterRoleBinding with cluster-admin for a Single-Namespace App

Binds a single-namespace application ServiceAccount to cluster-admin, granting unrestricted access to the entire cluster. If the service account token is compromised, the attacker owns every namespace, every resource, every verb.

Failure mode: Privilege sprawl.

4. Liveness Probe Checking External Database

Liveness probe depends on pg_isready against an external database. If the database is briefly unavailable, Kubernetes kills all API pods, causing cascading failure: database blip leads to thundering herd reconnects and further overload.

5. Deployment with :latest Tag and No imagePullPolicy

Uses the mutable :latest tag. Different nodes may pull different versions, causing inconsistent behavior across replicas. Rollbacks are impossible because every revision points to the same tag.

6. Ingress Using Removed API Version

Uses extensions/v1beta1 (removed in Kubernetes 1.22) with the deprecated kubernetes.io/ingress.class annotation, old backend syntax (serviceName/servicePort), and missing pathType. LLMs frequently generate this because training data contains many examples of the old API.

Failure mode: API drift.

7. Secret Data in a ConfigMap

Stores database passwords, API keys, and AWS credentials in a ConfigMap instead of a Secret. ConfigMaps are stored unencrypted in etcd and appear in plain text in kubectl describe, logs, and version control.

8. PVC with ReadWriteMany on an Unsupported Provider

Requests ReadWriteMany access mode with a gp3 (AWS EBS) storage class. EBS volumes only support ReadWriteOnce. The PVC will be stuck in Pending state with no clear error. LLMs frequently pair RWX with block storage classes because they do not track provider-specific storage capabilities.

---

Each anti-pattern maps to one or more of KubeShark's six failure modes. The reference file includes the exact broken YAML so you can study the specific mistakes and understand why Kubernetes does not catch them at admission time.

Do/Don't Quick Reference Checklist

A terse, actionable checklist of Kubernetes best practices organized by category. Each line is a standalone rule. The default security posture is the PSS restricted profile.

For the full checklist with every rule, see references/do-dont-patterns.md.

Categories Covered

The checklist spans eight categories that map directly to KubeShark's failure modes:

How to Use

Use this checklist as a final review pass before applying any manifest to a cluster. Each DO/DON'T rule is self-contained -- you can check them individually without reading the surrounding context. The checklist is designed for both human review and LLM self-verification during manifest generation.

Relationship to Failure Modes

The categories map directly to KubeShark's six named failure modes:

• Security Contexts, RBAC -- insecure workload defaults, privilege sprawl
• Resource Management -- resource starvation
• Networking -- network exposure
• Probes and Rollouts, Image Management -- fragile rollouts
• Storage, Configuration, Namespaces -- cross-cutting concerns that affect multiple failure modes

Every rule in the checklist exists because it prevents a specific, observed failure pattern. No generic advice is included unless it maps to a real failure mode.

Good Patterns -- Production-Ready Examples

These eight patterns demonstrate production-ready Kubernetes manifests that follow the PSS restricted profile, include proper labels, and set explicit resource constraints. Each pattern is annotated with key points explaining why specific choices were made.

For the full annotated YAML of every pattern below, see references/examples-good.md.

1. Minimal Production Deployment

A complete Deployment with full security context (pod-level and container-level), resource bounds, liveness and readiness probes, topology spread constraints, and standard app.kubernetes.io/* labels. Demonstrates the readOnlyRootFilesystem pattern with an emptyDir /tmp mount.

Key takeaway: Both pod-level and container-level securityContext are required. Topology spread prevents all replicas landing on one node.

2. Default-Deny NetworkPolicy

A two-resource pattern: a blanket deny-all policy (empty podSelector) followed by a targeted allow policy. Demonstrates allowing specific ingress from an ingress controller, scoped egress to a database, and mandatory DNS egress to kube-dns.

Key takeaway: Always allow DNS egress (UDP/TCP 53 to kube-dns) or name resolution breaks silently.

3. Scoped RBAC for CI Deployer

Namespace-scoped Role and RoleBinding for a CI pipeline ServiceAccount. Only grants the specific verbs and resources needed for deployment -- no delete, no cluster-admin, no ClusterRoleBinding.

4. CronJob with Lifecycle Controls

A CronJob with concurrencyPolicy: Forbid, startingDeadlineSeconds, activeDeadlineSeconds, ttlSecondsAfterFinished, history limits, and proper security context. Demonstrates safe scheduled job configuration that prevents overlapping runs and auto-cleans completed pods.

Key takeaway: activeDeadlineSeconds kills jobs that hang; ttlSecondsAfterFinished auto-cleans completed pods.

5. Ingress with TLS and Path-Based Routing

Uses the current networking.k8s.io/v1 API with ingressClassName (not the deprecated annotation), TLS configuration, and path-based routing with explicit pathType. More specific paths listed first.

6. HPA with Scale-Down Stabilization

An HPA using autoscaling/v2 with separate scale-up and scale-down behaviors. Scale-down is conservative (300s stabilization window, 25% per minute limit) while scale-up is aggressive. Targets both CPU and memory utilization.

7. Namespace with Quota, LimitRange, and PSA Labels

A complete namespace setup: PSA labels enforcing the restricted profile, a ResourceQuota capping total resource consumption, and a LimitRange providing defaults and bounds for containers that omit resource specs.

8. ExternalSecret for Vault Integration

Namespace-scoped SecretStore with Vault backend using Kubernetes auth, and an ExternalSecret that syncs credentials with a refresh interval. Demonstrates deletionPolicy: Retain to prevent accidental secret loss.

Key takeaway: Use namespace-scoped SecretStore (not ClusterSecretStore) unless multiple namespaces genuinely share the same Vault path.

---

Each of these patterns addresses one or more of KubeShark's six named failure modes. Use them as starting points and adapt to your cluster's specific requirements.

FM6: API Drift

Kubernetes follows a strict API deprecation lifecycle: beta APIs are introduced, stable APIs replace them, and beta APIs are eventually removed. LLMs hallucinate removed API versions more than any other type of Kubernetes error because their training data contains years of blog posts, tutorials, and Stack Overflow answers written for older cluster versions.

The Deprecation Lifecycle

Every API migration follows the same pattern:

1. Beta API introduced -- a new resource or feature enters as v1beta1 under an API group. 2. Stable API introduced -- the resource graduates to v1. The beta version is deprecated in the same release or shortly after. 3. Beta API removed -- typically 2-3 minor versions after deprecation, per the Kubernetes deprecation policy. From this point, the API server rejects manifests using the old version with a hard error.

"Deprecated" means the API still works but prints a warning. "Removed" means it fails. LLMs do not distinguish between these states.

Major Migrations LLMs Get Wrong

Ingress: extensions/v1beta1 to networking.k8s.io/v1

Removed in Kubernetes 1.22. This is the most frequently hallucinated API version because Ingress existed as a beta for years (1.1 through 1.21) and generated enormous amounts of training data.

The structural changes in v1 are not just a version swap:

• spec.backend renamed to spec.defaultBackend.
• serviceName and servicePort (flat fields) replaced by service.name and service.port.number (nested).
• pathType is required on every path -- it was optional in beta.
• ingressClassName replaces the kubernetes.io/ingress.class annotation.

An LLM that generates extensions/v1beta1 will also use the old field structure, compounding the error.

PodDisruptionBudget: policy/v1beta1 to policy/v1

Removed in Kubernetes 1.25. The v1 API makes spec.selector immutable after creation and adds spec.unhealthyPodEvictionPolicy. LLMs frequently generate policy/v1beta1 because PDB examples in training data predate 1.25.

HorizontalPodAutoscaler: autoscaling/v2beta1 and v2beta2 to autoscaling/v2

v2beta1 removed in 1.25, v2beta2 removed in 1.26. The key structural change: targetAverageUtilization (a top-level field in beta) moves to target.averageUtilization (nested under target in v2). LLMs mix beta and stable field structures unpredictably.

Other Removed APIs

Schema Validation

There are two levels of manifest validity, and LLM-generated manifests can fail at either:

• Structural validity: Does the YAML conform to the schema for this API version? Caught by kubeconform or --dry-run=server. Wrong field names, wrong nesting, unknown fields.
• Semantic validity: Does the manifest make sense in context? Does the referenced Service exist? Is the port correct? Caught only at apply time or with policy tools.

kubeconform validates manifests against the OpenAPI schema for a specific Kubernetes version. Always pin the version to match your target cluster:

kubeconform -kubernetes-version 1.30.0 -strict manifests/

The -strict flag rejects unknown fields, which catches the common case where an LLM generates fields from one API version in a manifest tagged with a different version.

Helm-Specific Drift

Helm templates can produce syntactically valid YAML that uses the wrong API version. The template renders without error, but kubectl apply fails on the cluster. Use Capabilities.APIVersions to branch on cluster version:

{% raw %}

{{- if .Capabilities.APIVersions.Has "networking.k8s.io/v1" }}
apiVersion: networking.k8s.io/v1
{{- else }}
apiVersion: networking.k8s.io/v1beta1
{{- end }}

{% endraw %}

Another common Helm drift error: broken Go template expressions that fail silently. {% raw %}{{ .Values.replicas }}{% endraw %} evaluates to empty (not an error) if replicas is not defined in values.yaml. Always use defaults: {% raw %}{{ .Values.replicas | default 3 }}{% endraw %}.

Kustomize-Specific Drift

Kustomize strategic merge patches specify a target with group, version, and kind. If the API group in the patch does not match the resource, the patch silently fails to apply -- no error, no warning, just unpatched output.

What LLMs Get Wrong

1. `extensions/v1beta1` for Ingress. Removed since 1.22, but still the most common LLM-generated Ingress API version. 2. Beta HPA API versions. Mixing autoscaling/v2beta1 field structures with autoscaling/v2 API version, or vice versa. 3. Flat Ingress backend fields. Using serviceName/servicePort instead of the nested service.name/service.port.number structure. 4. Missing `pathType` on Ingress paths. Required in networking.k8s.io/v1 but optional in beta. LLMs trained on beta examples omit it. 5. `batch/v1beta1` for CronJob. Removed since 1.25, but CronJob tutorials from the beta era are abundant in training data. 6. No schema validation in the workflow. LLMs generate manifests without suggesting validation, so errors are discovered only at deploy time.

Prevention

The most effective defense against API drift is automated validation in the CI pipeline:

1. `kubeconform` with -strict and -kubernetes-version matching the target cluster. 2. `pluto` scans manifests, Helm charts, and running clusters for deprecated and removed APIs. 3. `--dry-run=server` validates against the live API server schema, catching CRD and admission webhook issues that offline tools miss.

Run all three: kubeconform in CI, pluto as a pre-commit check, and dry-run=server in the deployment pipeline.

Further Reading

• Kubernetes Deprecation Policy
• API Migration Guide
• KubeShark Validation and Policy Guide

FM5: Fragile Rollouts

A bad rollout is worse than no rollout. Misconfigured probes, mutable image tags, and missing graceful shutdown logic turn routine deployments into outages. Fragile rollouts are the failure mode most likely to cause user-facing downtime because they activate during the exact moment the system is changing.

The Three Probe Types

Kubernetes provides three probes, each with a distinct purpose. Confusing them is the leading cause of cascading failures:

• Liveness probe: "Is the process alive?" If it fails, the kubelet kills and restarts the container. This probe must check only the process itself -- never external dependencies.
• Readiness probe: "Can this pod serve traffic?" If it fails, the pod is removed from Service endpoints. This is where dependency checks belong -- if the database is down, the pod should stop receiving requests but should not be killed.
• Startup probe: "Has initialization finished?" Disables liveness and readiness checks until it succeeds. Required for applications with slow startup (JVM warmup, ML model loading, large cache priming).

Cascading Failure From Liveness Probes

The single most dangerous rollout misconfiguration is a liveness probe that checks an external dependency. When the database goes down:

1. The liveness probe fails on all pods simultaneously. 2. The kubelet restarts all pods. 3. Pods restart, database is still down, liveness fails again. 4. The entire service enters CrashLoopBackOff with exponential backoff. 5. When the database recovers, the service takes minutes to recover because of the backoff timer.

If the liveness probe only checked "is the main thread responsive?", the pods would have stayed up and resumed serving immediately when the database returned. The readiness probe would have removed them from traffic in the meantime.

The :latest Tag Trap

Using :latest as an image tag introduces three problems:

1. Nondeterminism: Different nodes may cache different image layers. After a rollout, some pods run version A and others run version B, depending on which nodes had cached layers. 2. Impossible rollbacks: kubectl rollout undo re-deploys the same :latest tag, which may now point to a newer (broken) image. 3. Silent drift: No change is detected by the Deployment controller because the tag has not changed, even though the image content has.

With imagePullPolicy: IfNotPresent (the default for non-:latest tags), nodes use cached images. With :latest, the default policy is Always, but some environments override this, creating inconsistent behavior.

The fix: always use immutable tags -- semantic versions (v2.4.1), git SHAs, or digests (@sha256:...).

Rolling Update Strategy

The strategy.rollingUpdate fields control how many pods are replaced simultaneously:

• `maxSurge`: How many extra pods above the desired count during the update. Higher values speed up rollouts but consume more resources.
• `maxUnavailable`: How many pods can be unavailable during the update. Set to 0 for zero-downtime deployments (requires maxSurge >= 1).
• `minReadySeconds`: How long a new pod must be Ready before it counts as Available. Catches pods that start successfully but crash shortly after (e.g., failing to connect to a dependency after initialization).

For critical services, use maxSurge: 1, maxUnavailable: 0. This ensures capacity never drops below the desired count during a rollout.

Graceful Shutdown

When Kubernetes terminates a pod, two things happen in parallel: 1. The pod is removed from Service endpoints (asynchronous). 2. The container receives SIGTERM.

Because endpoint removal is asynchronous, the pod may still receive traffic for several seconds after SIGTERM. Without a preStop hook, the application begins shutting down while requests are still arriving, causing dropped connections and 502 errors.

The fix is a preStop sleep of 3-5 seconds to allow endpoint propagation before the application begins its shutdown sequence:

lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 5"]

Set terminationGracePeriodSeconds to a value that exceeds the preStop sleep plus the application's drain time. The default of 30 seconds is often insufficient for applications with long-lived connections.

Init Containers for Dependency Waiting

Dependencies should be waited on in init containers, not liveness probes. An init container blocks pod startup until the dependency is available, then exits. This keeps the probe system focused on runtime health, not startup prerequisites:

initContainers:
  - name: wait-for-db
    image: busybox:1.36
    command: ["sh", "-c", "until nc -z postgres 5432; do sleep 2; done"]

What LLMs Get Wrong

1. Liveness probe checking database connectivity. The number one cause of cascading outages. Liveness should check only process health. 2. Same endpoint for liveness and readiness. These probes have different purposes and should hit different endpoints (/healthz for liveness, /ready for readiness). 3. No startup probe for slow applications. JVM apps, Python ML services, and applications loading large datasets need 60-120+ seconds to start. Without a startup probe, the liveness probe kills them during initialization. 4. `failureThreshold: 1` on liveness. A single blip (GC pause, network hiccup) kills the pod. Use at least 3. 5. `:latest` tag with no registry prefix. image: myapp:latest with no registry means the kubelet looks in the default registry, which varies by runtime configuration. 6. Missing `preStop` hook. Traffic arrives after SIGTERM, causing dropped connections. 7. `maxUnavailable` too high. With 3 replicas and maxUnavailable: 2, only 1 pod serves traffic during rollout -- a single failure causes a complete outage.

Real-World Impact

• Cloudflare outage (2019): A misconfigured health check caused a cascading restart of edge proxies across multiple data centers, resulting in a global 30-minute outage.
• GitLab incident (2021): A canary deployment with no readiness probe sent traffic to pods still loading their configuration, causing elevated error rates for 45 minutes.
• Shopify Black Friday (2020): Aggressive liveness probes combined with database latency caused pod restarts during peak traffic, requiring manual intervention to stabilize.

Rollout fragility is entirely preventable. Every field -- probes, strategy, shutdown hooks, image tags -- has a correct configuration that eliminates the corresponding failure mode.

Further Reading

• Configure Liveness, Readiness, and Startup Probes
• KubeShark Good Patterns
• KubeShark Bad Patterns

FM1: Insecure Workload Defaults

Kubernetes does not ship with secure defaults. Pods created without explicit security contexts run as root, retain all Linux capabilities, and have writable root filesystems. This is the single most impactful failure mode for LLM-generated manifests because training data overwhelmingly consists of insecure examples.

Why This Matters

OWASP Kubernetes Top Ten ranks "Insecure Workload Configurations" as K01 -- the number one risk. A compromised container running as root with full capabilities can escape to the host node, access the cloud metadata service, pivot to other workloads, and exfiltrate secrets. Every missing security control compounds the blast radius.

Security Context: Pod-Level vs Container-Level

Kubernetes splits security settings across two scopes, and both must be configured:

• Pod-level (spec.securityContext): applies to all containers including init containers. This is where runAsNonRoot, runAsUser, runAsGroup, fsGroup, and seccompProfile belong.
• Container-level (spec.containers[].securityContext): per-container overrides. This is where allowPrivilegeEscalation, readOnlyRootFilesystem, and capabilities belong.

Omitting either level leaves gaps. A pod-level runAsNonRoot: true without container-level capabilities.drop: [ALL] still retains dangerous capabilities like CAP_NET_RAW (used for ARP spoofing and network sniffing within the cluster).

Pod Security Standards

Kubernetes enforces security through Pod Security Admission (PSA), which evaluates pods against three profiles:

PSA is enforced via namespace labels. A namespace without these labels has no enforcement -- pods run with whatever the manifest specifies, including fully privileged.

Capabilities and Privilege Escalation

Linux capabilities grant fine-grained privileges. The default Docker/containerd capability set includes CAP_NET_RAW, CAP_SETUID, CAP_SETGID, and others that attackers exploit for container escapes. The hardened baseline is:

securityContext:
  capabilities:
    drop:
      - ALL

If a workload genuinely needs a specific capability (e.g., NET_BIND_SERVICE to bind port 443), add only that one capability back. Never leave the default set in place.

The allowPrivilegeEscalation: false field is equally critical. Without it, a process inside the container can gain more privileges than its parent process through setuid binaries or other escalation vectors. This field must be set at the container level -- setting it at the pod level has no effect.

Host Namespace Access

Setting hostNetwork, hostPID, or hostIPC to true breaks the container isolation boundary entirely. hostNetwork exposes the pod to the node's network stack and bypasses all NetworkPolicy enforcement. hostPID lets the container see and signal every process on the node. These fields must be false (the default) for all application workloads.

AppArmor and Seccomp

Seccomp restricts which system calls a container can make. The RuntimeDefault profile blocks dangerous syscalls like ptrace and mount while allowing normal application behavior. Under PSS restricted, seccompProfile.type: RuntimeDefault is mandatory at the pod level.

AppArmor provides mandatory access control on top of seccomp. As of Kubernetes 1.30, AppArmor has graduated to a first-class field (securityContext.appArmorProfile), replacing the older annotation-based approach (container.apparmor.security.beta.kubernetes.io/<name>). For clusters running 1.30+, use the native field. For older clusters, use the annotation. LLMs frequently mix these two approaches in the same manifest.

Custom seccomp profiles (type: Localhost) can further restrict syscall access beyond RuntimeDefault, but require the profile to be available on every node. Use RuntimeDefault as the starting point unless specific workload requirements demand a custom profile.

What LLMs Get Wrong

LLMs reproduce patterns from their training data, which is dominated by quickstart guides and blog posts without security hardening. The most frequent errors:

1. Omitting security context entirely. The most common mistake. The generated manifest has no securityContext at either level. 2. Setting `runAsNonRoot` but not `runAsUser`. The kubelet checks the image metadata at runtime -- if the image specifies USER root, the pod fails to start with a confusing error. 3. Dropping capabilities partially. Dropping SYS_ADMIN but not all capabilities still leaves NET_RAW, SETUID, and others. 4. Forgetting init containers. Security context on main containers but not init containers leaves a privilege escalation window during pod startup. 5. Confusing pod-level and container-level fields. Putting allowPrivilegeEscalation at the pod level (where it is ignored) instead of the container level. 6. Missing `readOnlyRootFilesystem`. Without it, an attacker can write binaries into the container filesystem. Combine with emptyDir mounts for /tmp and any other write paths.

Real-World Impact

• Tesla cryptojacking (2018): Kubernetes dashboard exposed without authentication, pods deployed with no security context, cryptominers ran as root on GPU nodes.
• Shopify bug bounty (2020): A container escape via CAP_SYS_ADMIN in a pod that did not drop capabilities, granting access to the underlying node.
• Capital One breach (2019): While not Kubernetes-specific, the pattern is identical -- overly permissive workload identity plus missing runtime restrictions enabled lateral movement from a single SSRF to full S3 access.

The common thread: every breach was amplified by workloads running with more privileges than they needed. Secure defaults are not optional -- they are the primary defense against turning a single vulnerability into a cluster-wide compromise.

Further Reading

• OWASP Kubernetes Top Ten - K01
• Pod Security Standards
• KubeShark Security Hardening Guide

FM3: Network Exposure

Kubernetes networking is flat by default. Every pod can reach every other pod on any port, across all namespaces. There is no firewall, no segmentation, no access control until you explicitly create NetworkPolicy resources. This default-open posture means a single compromised container can reach databases, internal APIs, and cloud metadata endpoints without restriction.

The Default-Open Problem

Unlike traditional networks where firewalls deny traffic by default, Kubernetes starts with full connectivity. Installing a CNI plugin that supports NetworkPolicy (Calico, Cilium, Antrea) is necessary but not sufficient -- the plugin only enforces policies that exist. A namespace with zero NetworkPolicy objects allows all traffic regardless of the CNI plugin.

The correct baseline is a default-deny policy in every namespace, followed by explicit allow rules for required communication paths.

Service Types and Exposure Risk

LLMs frequently generate LoadBalancer or NodePort Services when ClusterIP is sufficient. Always default to ClusterIP and expose externally only through an Ingress controller with TLS termination.

The Silent Selector Mismatch

The most frustrating Kubernetes networking bug produces no error, no warning, and no log entry. When a Service selector does not match any pod labels, the Service gets zero Endpoints. Traffic sent to the Service simply vanishes -- connections time out or receive connection refused errors.

This happens because:

• The pod label says app: api-server but the Service selector says app: api (typo).
• The selector includes a version label that changes on deploy (e.g., version: v2 in the selector, but the new pods have version: v3).
• Labels are case-sensitive: App: api-server does not match app: api-server.

Always verify with kubectl get endpoints <service-name> after any Service or Deployment change.

NetworkPolicy AND/OR Logic

The most common NetworkPolicy mistake is confusing AND and OR semantics in from/to rules:

• Same list item = AND: A namespaceSelector and podSelector in the same from entry must both match.
• Separate list items = OR: Two separate from entries are unioned -- traffic matching either rule is allowed.

Getting this wrong can either block legitimate traffic or open traffic to the entire cluster. A single misplaced hyphen in YAML changes the behavior completely.

Egress Policies and DNS

A default-deny egress policy blocks all outbound traffic including DNS resolution. If you forget to allow DNS (port 53 UDP and TCP to kube-system), every service lookup fails and the application appears to have network connectivity issues when it actually has a policy misconfiguration.

Always include a DNS egress rule when writing egress policies:

egress:
  - to:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: kube-system
    ports:
      - protocol: UDP
        port: 53
      - protocol: TCP
        port: 53

DNS Performance and ndots

Kubernetes defaults to ndots: 5, meaning any hostname with fewer than 5 dots triggers search domain expansion. For a call to api.stripe.com (2 dots), the resolver first tries api.stripe.com.production.svc.cluster.local, then api.stripe.com.svc.cluster.local, then api.stripe.com.cluster.local, and finally the actual address. This multiplies DNS queries by 4-5x for every external call.

Fix with dnsConfig.options: [{name: ndots, value: "2"}] or append a trailing dot to external hostnames (api.stripe.com.).

Lateral Movement After Compromise

Without NetworkPolicy, an attacker who compromises a single pod can: 1. Scan the entire cluster network to discover services. 2. Access databases directly (bypassing application-level auth). 3. Reach the cloud metadata endpoint (169.254.169.254) to steal IAM credentials. 4. Pivot to other namespaces to access higher-privilege workloads. 5. Exfiltrate data to external endpoints without restriction.

NetworkPolicy is the primary control against lateral movement. It reduces the blast radius of any single compromise from "the entire cluster" to "the pods this workload is explicitly allowed to reach."

What LLMs Get Wrong

1. No NetworkPolicy at all. The most common error. The generated manifests include Deployments and Services but no network segmentation. 2. Ingress-only policies. Writing a policy with only ingress rules still allows unrestricted egress. Always specify both policyTypes: [Ingress, Egress]. 3. Forgetting DNS egress. Blocking all egress without a DNS exception breaks all service discovery. 4. NodePort as default. Generating type: NodePort when ClusterIP would suffice, exposing the service on every node. 5. Missing `ingressClassName`. Omitting it in an Ingress resource relies on a default IngressClass that may not exist, causing silent 404s. 6. Wrong port mapping. Confusing the Service port, targetPort, and Ingress backend port.number. The Ingress backend references the Service port, not the container port. 7. `hostNetwork: true` without justification. Bypasses all NetworkPolicy enforcement entirely.

Real-World Impact

Lateral movement is the primary attack vector in Kubernetes breaches. The 2022 Sysdig threat report found that 87% of container images contained a high or critical vulnerability, and the average time from initial compromise to lateral movement was under 10 minutes in clusters without NetworkPolicy.

Network segmentation is not optional security hardening -- it is the minimum viable defense for any multi-service deployment.

Further Reading

• Network Policies
• KubeShark Security Hardening Guide
• KubeShark Do/Don't Checklist

FM4: Privilege Sprawl

Privilege sprawl occurs when workloads accumulate more Kubernetes API access than they need. It compounds the impact of every other failure mode -- a compromised container with cluster-admin permissions turns a single vulnerability into a full cluster takeover. RBAC misconfigurations are silent, hard to audit, and rarely reviewed after initial setup.

RBAC Fundamentals

Kubernetes RBAC has four resource types:

• Role: grants permissions within a single namespace.
• ClusterRole: grants permissions cluster-wide or across all namespaces.
• RoleBinding: binds a Role (or ClusterRole) to subjects within one namespace.
• ClusterRoleBinding: binds a ClusterRole to subjects across the entire cluster.

The principle of least privilege means using namespace-scoped Roles unless the workload genuinely needs cluster-wide access. Most application workloads need zero Kubernetes API access at all.

Wildcard Permissions

Rules containing verbs: ["*"], resources: ["*"], or apiGroups: ["*"] grant unrestricted access. A single wildcard rule can negate every other security control in the cluster. Wildcards appear frequently in quickstart guides and Helm chart defaults because they "just work" -- but they grant far more access than any workload needs.

Always enumerate specific verbs (get, list, watch, create, update, patch, delete), specific resources (pods, configmaps, deployments), and specific API groups ("", apps, batch). Use resourceNames to restrict access to specific named resources when possible.

The Default ServiceAccount Problem

Every namespace has a default ServiceAccount. Every pod that does not specify serviceAccountName uses it. Every pod that uses it shares the same identity. This means:

• A single RoleBinding granting permissions to the default SA affects every pod in the namespace.
• If any pod in the namespace is compromised, the attacker inherits whatever permissions the default SA has.
• RBAC audit trails cannot distinguish between workloads using the same SA.

The fix: create a dedicated ServiceAccount for every workload. Set automountServiceAccountToken: false on both the ServiceAccount and the Pod spec for workloads that never call the Kubernetes API (which is most of them).

automountServiceAccountToken

By default, Kubernetes mounts a service account token into every pod at /var/run/secrets/kubernetes.io/serviceaccount/token. This token grants whatever permissions the SA has. For workloads that never call the Kubernetes API (web servers, batch processors, data pipelines), this token is pure attack surface.

Setting automountServiceAccountToken: false on the pod spec removes the token mount entirely. For workloads that do need API access, use projected token volumes with explicit audience and expiration instead of the legacy static token.

Secrets Are Not Encrypted

The most dangerous misconception about Kubernetes Secrets is that they are secure. They are not:

• Base64 is not encryption. kubectl get secret -o yaml shows the value. echo <value> | base64 -d decodes it. Any user or ServiceAccount with get secrets RBAC in the namespace can read every secret.
• etcd stores secrets in plaintext by default. Without explicit EncryptionConfiguration, secrets are stored unencrypted in the cluster's backing store.
• Environment variable injection exposes secrets. Secrets injected via env.valueFrom.secretKeyRef are visible in kubectl describe pod, process listings (/proc/<pid>/environ), and crash dumps.

The hardened approach: 1. Mount secrets as files via volumeMounts, not environment variables. 2. Enable etcd encryption at rest as a baseline. 3. Use external secret management (External Secrets Operator with AWS Secrets Manager, GCP Secret Manager, or HashiCorp Vault) for production secrets. 4. Use Sealed Secrets for secrets that must be stored in Git.

Token Projection for API Access

When a workload genuinely needs to call the Kubernetes API, use bound service account token volumes instead of the default mount:

volumes:
  - name: kube-api-token
    projected:
      sources:
        - serviceAccountToken:
            audience: "https://kubernetes.default.svc"
            expirationSeconds: 3600
            path: token

Projected tokens are short-lived and audience-scoped, limiting the damage if the token is leaked.

What LLMs Get Wrong

1. Binding `cluster-admin` to workload ServiceAccounts. The most dangerous mistake. Appears in quickstart-style outputs when the LLM does not know the specific permissions needed. 2. Using wildcards for convenience. verbs: ["*"] and resources: ["*"] appear frequently because they avoid enumeration. 3. Omitting `serviceAccountName`. The pod silently uses the default SA, sharing identity with every other pod in the namespace. 4. Leaving `automountServiceAccountToken` at default. The token is mounted even when the workload never calls the API. 5. Injecting secrets as environment variables. Using env.valueFrom.secretKeyRef instead of volume mounts. 6. Hardcoding secret values in manifests. Plaintext passwords in env.value fields, committed to version control. 7. Treating base64 as encryption. Generating a Secret resource and assuming the data is protected.

Real-World Impact

• Shopify Kubernetes bug bounty: An attacker gained access to a pod with excessive RBAC permissions, then used kubectl from inside the pod to read secrets from other namespaces.
• Kubernetes CVE-2018-1002105: A privilege escalation vulnerability in the API server. Clusters where workloads already had broad RBAC permissions experienced full compromise; clusters with least-privilege RBAC contained the blast radius.
• Uber breach (2022): While not Kubernetes-specific, the pattern -- hardcoded credentials in source code -- is identical to the secrets-in-env antipattern that LLMs reproduce.

Privilege sprawl is cumulative and invisible until exploitation. Every unnecessary permission is an expansion of the attack surface that persists indefinitely unless explicitly revoked.

Further Reading

• RBAC Authorization
• KubeShark Security Hardening Guide
• KubeShark Do/Don't Checklist

FM2: Resource Starvation

Every container in a Kubernetes cluster shares finite CPU, memory, and disk. Without explicit resource requests and limits, workloads compete unpredictably -- a single runaway process can starve an entire node. Resource starvation is the most common cause of production instability in Kubernetes and the hardest to diagnose after the fact.

QoS Classes and Eviction Order

Kubernetes assigns a Quality of Service class to every pod based on how its resource fields are configured. This class determines eviction priority when a node runs out of resources:

A pod with no resources block at all is BestEffort. Under node memory pressure, the kubelet kills BestEffort pods first, then Burstable pods exceeding their requests, and Guaranteed pods only as a last resort. Running BestEffort in production is never acceptable.

CPU Throttling: The Invisible Latency Killer

CPU is a compressible resource -- when a container hits its CPU limit, the kernel's Completely Fair Scheduler (CFS) throttles it rather than killing it. This causes latency spikes that are invisible in standard metrics. A container with a 250m CPU limit that needs 300m for a request will pause mid-execution, adding unpredictable delays.

Current best practice for most application workloads: set CPU requests but omit CPU limits. This allows bursting to available capacity without CFS throttling. Set CPU limits only when running in multi-tenant clusters that require hard fairness guarantees, or when Guaranteed QoS is specifically needed.

Memory: Always Set a Limit

Memory is incompressible. When a container exceeds its memory limit, the kernel OOM-kills the process immediately. There is no throttling, no warning -- the process is terminated and the container restarts. Set memory limits 25-50% above observed p99 usage to absorb garbage collection spikes and temporary allocations.

LimitRange and ResourceQuota

Namespace-level guardrails catch workloads that slip through without resource specifications:

• LimitRange sets default requests/limits for containers that omit them, and enforces min/max bounds per container. Without a LimitRange, a single container can request all available resources on a node.
• ResourceQuota caps aggregate resource consumption per namespace: total CPU, memory, pod count, PVC count. When a ResourceQuota exists, every pod must specify resources or admission is rejected.

Both should be present in every production namespace. LimitRange provides sensible defaults; ResourceQuota prevents a single namespace from starving the rest of the cluster.

OOMKill Cascades

A particularly dangerous pattern occurs when OOMKills cascade. Pod A exceeds its memory limit and is killed. Its traffic shifts to pods B and C, which now handle more load, consume more memory, and also get OOMKilled. Within seconds, the entire service is in CrashLoopBackOff. This is especially common with JVM workloads where heap sizing does not account for off-heap memory, native threads, and container overhead.

PodDisruptionBudgets

Without a PDB, voluntary disruptions (node upgrades, autoscaler scale-downs, kubectl drain) can terminate all replicas simultaneously. A PDB with maxUnavailable: 1 ensures at least N-1 replicas remain running during planned disruptions. Critical rules:

• Never set minAvailable equal to replicas -- it blocks all voluntary disruptions including cluster upgrades.
• The PDB selector must exactly match the pod labels. A mismatched selector silently protects nothing.
• PDBs only protect against voluntary disruptions. Node crashes and OOMKills bypass PDB constraints.

HPA Pitfalls

The Horizontal Pod Autoscaler scales replicas based on metrics, but misconfiguration causes more problems than it solves:

• Target utilization too high (90%): No headroom for traffic spikes. By the time new pods start, the existing pods are overwhelmed.
• Target utilization too low (30%): Wasteful. The cluster runs 3x the needed capacity.
• No scale-down stabilization: HPA scales down aggressively by default. A brief traffic dip removes pods, then the next spike overwhelms the reduced fleet. Set scaleDown.stabilizationWindowSeconds: 300.
• HPA `minReplicas` below PDB `minAvailable`: HPA scales down to a count that violates the disruption budget, causing node drains to block indefinitely.

Topology Spread and Anti-Affinity

Three replicas on the same node provide zero high availability. A single node failure takes all of them down. Use topologySpreadConstraints to distribute pods across zones and nodes:

• Zone-level spread with whenUnsatisfiable: DoNotSchedule prevents all replicas from landing in one availability zone.
• Node-level spread with whenUnsatisfiable: ScheduleAnyway provides a soft preference that does not block scheduling in small clusters.

What LLMs Get Wrong

1. Omitting resources entirely. The most common error. The pod becomes BestEffort and is evicted first under any pressure. 2. Round-number guessing. cpu: 1 and memory: 1Gi without profiling. Requests should reflect measured steady-state usage, not arbitrary values. 3. Setting CPU limits by default. CFS throttling causes latency spikes. Omit CPU limits unless multi-tenancy or Guaranteed QoS requires them. 4. Memory limit equal to request. Zero headroom means any spike triggers OOMKill. Allow 25-50% margin. 5. Forgetting PDB. Multiple replicas without a PDB is false redundancy -- a node drain kills them all. 6. Topology spread missing. Three replicas with no spread constraints may all schedule to the same node.

Real Incidents

• Datadog outage (2023): A cascading OOMKill across monitoring agents caused loss of observability during a separate infrastructure incident, delaying diagnosis by hours.
• GitHub rate limiting regression: CPU throttling on API servers caused p99 latency to spike from 50ms to 2s. Removing CPU limits restored performance immediately.
• Zalando postmortem: A missing PDB allowed a cluster upgrade to drain all pods of a critical payment service simultaneously, causing a 15-minute outage.

Resource management is not optimization -- it is correctness. A manifest without resource configuration is incomplete.

Further Reading

• Managing Resources for Containers
• Pod Quality of Service Classes
• KubeShark Good Patterns

Installation

KubeShark can be installed in three ways depending on your environment: direct clone (recommended), marketplace install, or per-project setup for Codex.

---

Option 1: Direct Clone (Recommended)

Clone the repository into your Claude Code skills directory. Claude Code auto-discovers skills in ~/.claude/skills/ -- no restart or configuration needed.

macOS / Linux

git clone https://github.com/LukasNiessen/kubernetes-skill.git ~/.claude/skills/kubernetes-skill

Windows (PowerShell)

git clone https://github.com/LukasNiessen/kubernetes-skill.git "$env:USERPROFILE\.claude\skills\kubernetes-skill"

Windows (Command Prompt)

git clone https://github.com/LukasNiessen/kubernetes-skill.git "%USERPROFILE%\.claude\skills\kubernetes-skill"

After cloning, the skill is active immediately. Claude Code reads SKILL.md on the next Kubernetes-related prompt.

---

Option 2: Marketplace Install

Claude Code includes a built-in plugin system with marketplace support. This avoids manual cloning.

Add the marketplace source and install:

/plugin marketplace add LukasNiessen/kubernetes-skill
/plugin install kubernetes-skill

Or use the interactive manager:

1. Run /plugin in Claude Code. 2. Switch to the Discover tab. 3. Find KubeShark and install.

The marketplace reads .claude-plugin/marketplace.json in the repository to register KubeShark as an installable plugin.

---

Option 3: Codex Per-Project Setup

Codex has no global skill system. Setup is per-project: clone the skill into your repository and reference it from AGENTS.md.

Step 1 -- Clone into your project:

git clone https://github.com/LukasNiessen/kubernetes-skill.git .kubernetes-skill

Step 2 -- Reference in AGENTS.md:

Create or edit AGENTS.md in your repository root and add:

## Kubernetes

When working with Kubernetes manifests, Helm charts, or Kustomize overlays,
follow the workflow in `.kubernetes-skill/SKILL.md`.
Load references from `.kubernetes-skill/references/` as needed.

Codex will follow the workflow whenever it encounters Kubernetes tasks in the project.

---

Updating

KubeShark is a plain Git repository. Pull the latest changes to update:

macOS / Linux:

cd ~/.claude/skills/kubernetes-skill && git pull

Windows (PowerShell):

cd "$env:USERPROFILE\.claude\skills\kubernetes-skill"; git pull

Codex projects:

cd .kubernetes-skill && git pull

---

Uninstalling

Remove the cloned directory to uninstall.

macOS / Linux:

rm -rf ~/.claude/skills/kubernetes-skill

Windows (PowerShell):

Remove-Item -Recurse -Force "$env:USERPROFILE\.claude\skills\kubernetes-skill"

Codex projects:

rm -rf .kubernetes-skill

Also remove the corresponding section from AGENTS.md if you added one.

Marketplace installs:

/plugin uninstall kubernetes-skill

---

Verifying Installation

Confirm the skill is installed correctly by checking that SKILL.md exists:

macOS / Linux:

ls ~/.claude/skills/kubernetes-skill/SKILL.md

Windows (PowerShell):

Test-Path "$env:USERPROFILE\.claude\skills\kubernetes-skill\SKILL.md"

If the file exists, KubeShark is ready. You can also verify by asking Claude Code a Kubernetes question -- the response should follow the 7-step workflow and include an output contract with assumptions, failure modes, and rollback notes.

Quick Start

Get KubeShark running in under two minutes.

---

1. Install

git clone https://github.com/LukasNiessen/kubernetes-skill.git ~/.claude/skills/kubernetes-skill

See the Installation guide for Windows commands and alternative install methods.

---

2. Use It

Explicit invocation

Prefix your prompt with /kubernetes-skill to invoke the skill directly:

/kubernetes-skill Create a production-ready Deployment for a Node.js API with autoscaling

/kubernetes-skill Review my StatefulSet for security and reliability issues

Automatic activation

KubeShark activates automatically when Claude Code detects a Kubernetes-related task. No prefix needed:

Create a Helm chart for a PostgreSQL StatefulSet with backup CronJobs

Review my deployment.yaml for security issues

Both invocation methods produce the same structured output.

---

3. What to Expect

Every KubeShark response follows a 7-step workflow:

The output contract at the end is the key differentiator. It makes every response auditable -- you can verify assumptions and check the rollback path before applying anything to your cluster.

---

4. Example Tasks

KubeShark handles a wide range of Kubernetes work. Here are common task types to try:

Deployment creation

/kubernetes-skill Create a production Deployment for a Python Flask API with 3 replicas, resource limits, and an Ingress

Security review

/kubernetes-skill Review this Deployment for security issues and harden it with proper security contexts, NetworkPolicies, and RBAC

Helm chart generation

/kubernetes-skill Create a Helm chart for a Redis cluster with configurable replicas and persistent storage

Kustomize overlay

/kubernetes-skill Build a Kustomize overlay structure with base, staging, and production variants for my microservice

RBAC setup

/kubernetes-skill Create least-privilege RBAC for a monitoring service that needs read access to pods and metrics across all namespaces

Troubleshooting

/kubernetes-skill My pods are stuck in CrashLoopBackOff with OOMKilled status. Here is my manifest -- diagnose and fix it.

Probe configuration

/kubernetes-skill Add proper liveness, readiness, and startup probes for a Java Spring Boot app that takes 90 seconds to start

CI pipeline validation

/kubernetes-skill Create a CI pipeline step that validates all manifests with kubeconform and checks for policy violations with Kyverno

Helm Chart Best Practices

Helm is the standard package manager for Kubernetes. KubeShark follows these conventions when generating or reviewing Helm charts. For full YAML examples and the LLM mistake checklist, see references/helm-patterns.md.

Chart.yaml

Every chart must declare apiVersion: v2 (mandatory for Helm 3), a SemVer version that bumps on every chart change, and an independent appVersion tracking the application release. The type field should be application or library. Include a concise description field.

Key rules:

• version follows SemVer and must change on every chart modification -- Helm repositories serve stale versions from cache if the version is not bumped
• appVersion tracks the application release independently of the chart version
• Declare sub-chart dependencies in Chart.yaml under dependencies, not in a separate requirements.yaml

values.yaml Structure

Group values by resource type. Provide secure defaults that match the PSS restricted profile out of the box:

• image -- repository, tag (defaults to appVersion), pullPolicy
• securityContext -- runAsNonRoot, allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities.drop: ["ALL"]
• resources -- explicit requests and memory limits
• probes -- liveness and readiness paths, ports, and initial delays
• ingress -- disabled by default with enabled: false
• serviceAccount -- create: true, blank name, empty annotations

Document every section with # -- comments so helm-docs can auto-generate documentation.

Template Helpers (_helpers.tpl)

Define reusable named templates for fullname, labels, selectorLabels, and serviceAccountName. All templates should:

• Truncate names to 63 characters (Kubernetes DNS label limit)
• Support nameOverride and fullnameOverride values
• Use include (not template) so output can be piped to nindent

Template Conventions

• Use {% raw %}{{- ... -}}{% endraw %} whitespace trimming to prevent blank lines in rendered output.
• Always pipe string values through {% raw %}{{ .Values.foo | quote }}{% endraw %}.
• Use {% raw %}{{ toYaml .Values.resources | nindent N }}{% endraw %} for nested objects -- never render at column 0.
• Wrap optional resources in {% raw %}{{- if .Values.ingress.enabled }}{% endraw %} conditionals.
• Use {% raw %}{{ required "message" .Values.key }}{% endraw %} for values that must be supplied by the user.

Dependency Management

Declare sub-charts in Chart.yaml under dependencies. Run helm dependency update to generate Chart.lock. Use condition or tags to make sub-charts optional. Commit both Chart.yaml and Chart.lock to version control.

Security Defaults

Charts should ship with secure defaults out of the box. Users who need to relax security (e.g., for a CNI plugin that requires host networking) can override values explicitly, but the default path should produce a PSS-restricted-compliant workload.

Key defaults to include in every chart's values.yaml:

• Pod-level securityContext with runAsNonRoot: true and seccompProfile: RuntimeDefault
• Container-level securityContext with allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, and capabilities.drop: ["ALL"]
• automountServiceAccountToken: false unless the workload calls the Kubernetes API

Testing Pipeline

Run these checks in order during development and CI:

1. `helm lint ./chart` -- catch syntax and structural errors 2. `helm template release-name ./chart -f values-prod.yaml` -- render manifests locally 3. `kubeconform -kubernetes-version X.Y.0 -strict` -- validate rendered output against target cluster schemas 4. `helm test release-name` -- run in-cluster test pods post-install

Integrate these steps into your CI pipeline so every chart change is validated before merge. The schema validation step (kubeconform) is especially important because helm lint does not validate against the Kubernetes API schema.

Common LLM Mistakes

The most frequent Helm-specific errors LLMs produce include: missing {% raw %}{{-{% endraw %} whitespace control, omitting | nindent N on toYaml calls, forgetting to quote string values, hardcoding labels instead of using include helpers, not providing defaults for image tags, and not bumping the chart version. See the full checklist in the reference file.

Kustomize Patterns

Kustomize provides template-free customization of Kubernetes manifests using overlays and patches. KubeShark follows these conventions when generating or reviewing Kustomize configurations. For full YAML examples and the LLM mistake checklist, see references/kustomize-patterns.md.

Base/Overlay Structure

Organize manifests in a standard directory layout:

• base/ -- contains the core kustomization.yaml, Deployment, Service, and Namespace manifests shared across all environments
• overlays/dev/, overlays/staging/, overlays/production/ -- environment-specific customizations that reference the base
• components/ -- reusable cross-cutting features (e.g., monitoring, network policies) that any overlay can include

Every kustomization.yaml must declare apiVersion: kustomize.config.k8s.io/v1beta1, kind: Kustomization, and a resources list. Use resources (not the deprecated bases field) for base references.

Patches

Strategic Merge Patch -- merge into an existing resource structure. Best for adding or overriding specific fields like replica count or resource limits. The patch must include metadata.name to match the target resource.

JSON Patch -- add, remove, or replace at a specific path. Required for array element manipulation. Use /- to append to arrays, explicit indices to target known positions. Applied via inline patch blocks with a target selector.

Generators

configMapGenerator and secretGenerator create ConfigMaps and Secrets with an automatic content hash appended to the name. This hash-based naming triggers rolling updates when configuration changes -- a significant advantage over manually managed ConfigMaps.

When overriding a base generator in an overlay, use behavior: merge to extend existing values rather than creating a duplicate resource.

Components

Components use apiVersion: kustomize.config.k8s.io/v1alpha1 and kind: Component. They package reusable features (ServiceMonitor resources, Prometheus scrape annotations, sidecar injections) that any overlay can opt into via the components field.

Common Transformers

Kustomize provides several built-in transformers for cross-cutting modifications:

• `namePrefix` / `nameSuffix` -- add prefixes or suffixes to all resource names
• `commonLabels` -- add labels to all resources and their selectors (use with caution on mutable resources; see LLM mistakes)
• `commonAnnotations` -- add annotations to all resources
• `namespace` -- set the namespace on all resources in the kustomization

Image Transformer

Override image references without patching the Deployment directly:

• Use newTag for tag overrides during development
• Use digest for immutable production references
• The image transformer matches on the name field in container image references, so the name must match exactly

When to Use Kustomize vs Helm

Production Overlay Pattern

A typical production overlay references the base, sets the namespace, applies production labels, patches resource limits, overrides ConfigMap values with behavior: merge, pins image tags, and adds production-only resources like HPAs. See the reference file for a complete example.

Common LLM Mistakes

The most frequent Kustomize-specific errors LLMs produce include: using the deprecated bases field, omitting metadata.name in strategic merge patches, applying commonLabels to resources with immutable selectors, forgetting content hashes in resource references, wrong array indices in JSON patches, and using the wrong apiVersion for components. See the full checklist in the reference file.

Kubernetes Skill for Claude Code — KubeShark

KubeShark is a failure-mode-first Kubernetes skill for Claude Code and Codex. It prevents common LLM hallucinations in Kubernetes manifest generation by diagnosing risks before writing YAML.

Why use it

• Prevents hallucinations -- 6 named failure modes with targeted reference files
• Token-efficient -- ~650 token activation cost, granular references loaded on demand
• Production-ready defaults -- Pod Security Standards restricted profile, proper resource management, cross-resource validation
• 20 reference files -- covering security, networking, RBAC, probes, storage, Helm, Kustomize, and more

Key features

• Failure-mode-first diagnostic workflow (diagnose before generate)
• Output contracts with assumptions, tradeoffs, and rollback notes
• LLM mistake checklists in every reference file
• Cross-resource consistency validation (label/selector/port alignment)
• Helm and Kustomize pattern guidance
• Policy engine integration (Kyverno, OPA/Gatekeeper)

Quick install

git clone https://github.com/LukasNiessen/kubernetes-skill.git ~/.claude/skills/kubernetes-skill

License

MIT -- see LICENSE.