Kubernetes Operator

Kubernetes Operator

Build operators that reconcile correctly. Most operator bugs are not Kubernetes bugs — they are reconcile-loop bugs: missing finalizers, blocking calls, no requeue on transient errors, status drift, RBAC over-grants. This skill catches them deterministically before they reach a cluster.

When to use

• Building a new Kubernetes Operator (controller for a CRD)
• Reviewing an existing operator for capability-level gaps
• Auditing a CRD spec for status/conditions/finalizer correctness
• Choosing a framework (controller-runtime / kubebuilder / operator-sdk / metacontroller / KOPF)
• Designing the API surface of a Custom Resource
• Hardening RBAC, leader election, or webhook validation

When NOT to use

• Plain Helm chart packaging → use helm-chart-builder
• Standard kubectl operations / blue-green deploys → use senior-devops
• General k8s security posture → use cloud-security
• "I want to run a workload" — that's a Deployment / Job, not an operator

Core principle: an operator is a reconcile loop, not a script

observe(actual) → desired = read(spec) → diff(actual, desired) → act → update(status)
                                                                          ↓
                                                                   requeue / done

Operators that fail are the ones that: 1. Treat reconcile as imperative (do this, then this, then this) instead of declarative (make actual=desired, idempotently) 2. Don't requeue transient failures 3. Don't use finalizers, leaving orphan resources 4. Mutate spec instead of status 5. Don't use the status subresource (status updates trigger spec reconciles → loop) 6. Block in reconcile (long HTTP calls, locks) 7. Forget leader election → split-brain on multi-replica deploys

The 3 tools below catch each of these.

Quick start

SKILL=engineering/kubernetes-operator/skills/kubernetes-operator

# Validate a CRD design
python "$SKILL/scripts/crd_validator.py" --crd config/crd/myapp.yaml

# Lint a Go reconcile function
python "$SKILL/scripts/reconcile_lint.py" --controller controllers/myapp_controller.go

# Score against OperatorHub Capability Levels (1-5)
python "$SKILL/scripts/operator_capability_audit.py" --operator-dir .

The 3 Python tools

All stdlib-only. Run with --help.

crd_validator.py

Validates a CRD YAML against operator-pattern best practices.

python scripts/crd_validator.py --crd config/crd/myapp.yaml
python scripts/crd_validator.py --crd config/crd/ --format json

Checks:

• spec.versions[*].subresources.status is set (status subresource)
• spec.scope is Namespaced (not Cluster) unless explicitly justified
• Singular and listKind defined
• spec.versions[*].schema.openAPIV3Schema has type definitions (no x-kubernetes-preserve-unknown-fields: true at top level)
• A version is marked served: true AND storage: true
• Conditions array is in the schema (allows metav1.Conditions)
• Printer columns include Age and Status/Phase

reconcile_lint.py

Lints a Go controller reconcile function for anti-patterns.

python scripts/reconcile_lint.py --controller controllers/myapp_controller.go

Checks (regex-based heuristics):

• Returns are (ctrl.Result, error) shape
• Errors trigger a non-zero requeue (return ctrl.Result{Requeue: true}, err)
• client.Update() on the spec object is flagged (controllers should update only status)
• time.Sleep inside reconcile is flagged (use RequeueAfter)
• HTTP calls without context cancellation are flagged
• Missing defer after a finalizer add
• No IsConditionTrue / SetCondition calls when conditions present in CRD
• Reconcile function exceeds 80 lines (extract subroutines)

operator_capability_audit.py

Scores an operator against OperatorHub's 5 Capability Levels.

python scripts/operator_capability_audit.py --operator-dir .

Levels:

• L1 — Basic Install: CRD defined, controller deploys it
• L2 — Seamless Upgrades: PDBs, conversion webhooks, version skew strategy
• L3 — Full Lifecycle: backups, restores, failure recovery
• L4 — Deep Insights: metrics endpoint, Prometheus rules, alerts
• L5 — Auto Pilot: auto-scaling, auto-tuning, anomaly detection

Reports current level + concrete next steps to advance one level.

Tooling landscape

Pick a framework based on language and complexity. See references/tooling_landscape.md.

Decision rules:

• New operator + Go shop → kubebuilder
• New operator + Python shop → KOPF
• New operator + can't pick a language → metacontroller
• OpenShift target → operator-sdk

CRD design principles

See references/crd_design.md for full detail. Quick rules:

1. status is the source of truth for the controller's view of the world. Spec is what the user wants; status is what the controller observed. 2. Use the status subresource. Without it, status updates re-trigger reconcile (loop). 3. Use Conditions. Ready, Reconciling, Degraded. Each carries a reason and message. 4. Add finalizers. Without finalizers, deletion races the controller and orphans external resources. 5. Version your CRD from day 1. v1alpha1 → v1beta1 → v1. Plan a conversion webhook. 6. Validate via OpenAPI v3 schema. Don't rely on the controller for validation that should fail at admission. 7. Use `additionalPrinterColumns` for `kubectl get`. Show Age, Phase, Ready at minimum. 8. Namespace your CRDs unless they manage cluster-scoped resources.

Reconcile loop principles

See references/reconcile_loop.md for full detail. Quick rules:

1. Idempotent. Reconciling the same state twice → same result, zero side effects. 2. Read once, decide, act. Don't observe the world repeatedly during reconcile. 3. Update status, not spec. Spec belongs to the user. 4. Return errors that requeue. Use ctrl.Result{RequeueAfter: ...} for known transient cases. 5. Never block. No time.Sleep. No long HTTP calls without context. 6. Use the cache. Read via the controller's cached client; only escape the cache for a specific reason. 7. Leader-elect when running >1 replica. Otherwise enable single-replica mode. 8. Set OwnerReferences. Cascading deletion is the operator pattern's free gift.

Workflows

Workflow 1: Bootstrap a new operator (Go + kubebuilder)

1. Pick a Group/Version/Kind: e.g., apps.example.com/v1alpha1, kind=MyApp
2. kubebuilder init --domain example.com --repo github.com/org/myapp-operator
3. kubebuilder create api --group apps --version v1alpha1 --kind MyApp
4. Run crd_validator.py on config/crd/bases/apps.example.com_myapps.yaml
   → Fix every WARN before writing controller code
5. Implement the reconcile function (Karpathy principle 2: simplest correct version first)
6. Run reconcile_lint.py on controllers/myapp_controller.go
7. Run operator_capability_audit.py --operator-dir . — confirm L1
8. Test in a kind cluster: kubectl apply -f config/samples/
9. Add status conditions; aim for L2 in the same PR

Workflow 2: Audit an existing operator

1. Run operator_capability_audit.py --operator-dir <path>
2. Run crd_validator.py --crd config/crd/
3. Run reconcile_lint.py --controller controllers/
4. Triage findings:
   - FAIL → block release; fix before next deploy
   - WARN → file an issue; fix in next 30 days
5. Document current capability level in README; commit
6. Plan one capability level advancement per quarter

Workflow 3: Choose a framework

1. Identify primary language constraint (team skill)
2. Identify deployment target (vanilla k8s vs OpenShift)
3. Identify operator complexity (single CRD vs multi-CRD vs cluster-wide)
4. Cross-reference with references/tooling_landscape.md
5. Build a 1-week proof-of-concept before committing

References

• references/operator_pattern.md — what an operator IS, when to use vs alternatives
• references/crd_design.md — CRD design principles, versioning, conversion webhooks
• references/reconcile_loop.md — reconcile patterns, error handling, idempotency
• references/tooling_landscape.md — framework comparison + decision tree

Slash command

/operator-audit — Run all 3 tools on an operator repo and produce a markdown report.

Asset templates

• assets/crd_template.yaml — CRD with status subresource, conditions, finalizer hint, printer columns
• assets/reconcile_skeleton.go — Go controller reconcile function with idempotency, conditions, finalizers, requeue patterns

Anti-patterns

• *`time.Sleep(30 time.Second) inside reconcile** — block other reconciles. Use RequeueAfter`.
• `r.Client.Update(ctx, obj)` to set status — use r.Status().Update(ctx, obj) instead.
• No leader election + 2+ replicas — split-brain.
• No finalizer — external resources orphan on deletion.
• CRD without status subresource — status updates trigger spec reconciles (infinite loop).
• Reconcile function > 200 lines — extract reconcileXxx subroutines per condition.
• `x-kubernetes-preserve-unknown-fields: true` on spec root — defeats validation.
• Imperative reconcile — "if creating, do A; if updating, do B; if deleting, do C". Wrong shape. Reconcile = make actual=desired, regardless of how we got here.

Verifiable success

A team using this skill should achieve:

• 100% of new CRDs pass crd_validator.py before merge
• All reconcile functions pass reconcile_lint.py strict mode
• Operators reach OperatorHub Capability Level 3 (Full Lifecycle) before public release
• Mean time to fix a reconcile bug: <1 day (no infinite loops in production)

CRD design

Custom Resource Definitions (CRDs) define the API surface of your operator. A bad CRD design locks you into hard-to-evolve schemas, forces wrapper APIs, and creates user-facing UX problems via kubectl.

Anatomy of a production CRD

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: myapps.apps.example.com         # plural.group
spec:
  group: apps.example.com
  names:
    kind: MyApp                         # PascalCase
    plural: myapps                      # lowercase
    singular: myapp                     # lowercase
    listKind: MyAppList                 # KindList
    shortNames: [ma]                    # optional
  scope: Namespaced                     # or Cluster (justify)
  versions:
    - name: v1alpha1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [version]
              properties:
                version:
                  type: string
                  pattern: '^[0-9]+\.[0-9]+\.[0-9]+$'
                replicas:
                  type: integer
                  minimum: 1
                  maximum: 100
                  default: 3
            status:
              type: object
              properties:
                phase:
                  type: string
                  enum: [Pending, Running, Failed]
                conditions:
                  type: array
                  items:
                    type: object
                    required: [type, status, lastTransitionTime]
                    properties:
                      type:    { type: string }
                      status:  { type: string, enum: ["True", "False", "Unknown"] }
                      reason:  { type: string }
                      message: { type: string }
                      lastTransitionTime: { type: string, format: date-time }
                observedGeneration: { type: integer }
      subresources:
        status: {}                      # CRITICAL — see below
        scale:                          # if scaling is meaningful
          specReplicasPath: .spec.replicas
          statusReplicasPath: .status.readyReplicas
      additionalPrinterColumns:
        - name: Phase
          type: string
          jsonPath: .status.phase
        - name: Ready
          type: string
          jsonPath: .status.conditions[?(@.type=="Ready")].status
        - name: Age
          type: date
          jsonPath: .metadata.creationTimestamp

Required structural elements

1. Status subresource — subresources.status: {}

Without it:

• r.Status().Update(ctx, obj) doesn't work — falls back to r.Update
• Status updates re-trigger spec reconcile → loop
• RBAC can't be split between spec writers and status writers

Always declare it.

2. Conditions array

Use the standard metav1.Condition shape. Required fields: type, status, lastTransitionTime. Recommended: reason, message, observedGeneration.

Conventional condition types:

• Ready — overall readiness
• Reconciling — controller is actively working
• Degraded — operating but with reduced capability
• Progressing — change in progress (mostly for Deployments-style flows)

Use meta.SetStatusCondition() from k8s.io/apimachinery/pkg/api/meta — don't write to the slice directly.

3. observedGeneration

Track which spec generation the controller has acted on:

status.ObservedGeneration = obj.Generation

Lets users tell whether status reflects the latest spec or a previous one.

4. Printer columns

kubectl get myapp UX is determined by additionalPrinterColumns. Always include:

• Phase or Ready (status)
• Age (so users know when it was created)

Optionally: replicas, version, key spec field.

5. Validation in the schema, not the controller

Express constraints declaratively:

Reserve controller validation for cross-field rules and external dependencies (e.g., "this name is taken in our DB").

6. Avoid x-kubernetes-preserve-unknown-fields: true

It disables structural validation. Sometimes needed (e.g., raw kubectl apply patches), but never at the spec root. Use it sparingly on a single sub-tree.

Versioning strategy

CRDs evolve. Plan from day 1:

Conversion webhook required when:

• Multiple versions are served simultaneously
• A field's shape changed between versions

For simple field renames, x-kubernetes-conversion-strategy: None works.

Scope: Namespaced vs Cluster

Default to Namespaced. Cluster-scoped CRDs:

• Can't be RBAC-restricted by namespace
• Can't have OwnerReferences from namespaced parents
• Are appropriate only for cluster-wide resources (StorageClass-like things)

If your operator manages namespace-bound things (apps, databases, queues), use Namespaced.

Naming

• Group: <domain>.<reverse-domain> — e.g., apps.example.com. Don't use generic groups (com, io).
• Kind: PascalCase, singular, descriptive — MyApp, Database, Cache. Avoid MyAppResource (the Resource suffix is implicit).
• Plural: lowercase, plural — myapps, databases, caches.
• Short name: 2-3 letters; check for conflicts with built-in resources.

Validation tooling

• kubectl apply --dry-run=server — validates against your CRD
• kubectl explain <kind>.<field> — shows what your schema documents
• crd_validator.py — this skill's tool, structural rules

Documentation in the schema

Use the description field on every property. kubectl explain reads it:

properties:
  replicas:
    type: integer
    minimum: 1
    description: |
      Number of replicas to run. Production deployments should use ≥3.
      Increases above 100 require quota approval.

Anti-patterns

• Top-level `x-kubernetes-preserve-unknown-fields: true` — defeats validation
• No `scope:` declared — defaults to namespaced but make intent explicit
• No printer columns — kubectl get shows only NAME AGE
• Conditions written by hand (not via SetStatusCondition) — easy to lose lastTransitionTime
• Status fields that duplicate spec — keep them separate
• Using `metadata.annotations` to encode operator state — use status fields
• Single huge CRD with 50+ fields — split into multiple CRDs (e.g., MyApp + MyAppBackup + MyAppRestore)

The operator pattern

An operator is a controller that reconciles a Custom Resource (CR) toward its declared spec. It encodes operational knowledge — installation, upgrades, backups, failover — that would otherwise live in tribal knowledge or runbooks.

When you need an operator

Build an operator when:

• The application has nontrivial lifecycle operations (backup, restore, version upgrade, failover) that go beyond a simple Deployment
• The application has statefulness or topology that Helm/Deployment can't express (leader election, peer discovery, rolling state migration)
• Multiple teams need to provision instances of the application via a Kubernetes API, not a custom UI
• The application's operational discipline is documented in runbooks but unevenly applied

Don't build an operator when:

• A Helm chart is enough (most stateless apps fit here)
• A CronJob can run the operational task on a schedule
• The custom logic is a one-time migration (use a Job)
• Three engineers can manage it via Deployment + ConfigMap

Operator pattern shape

┌────────────────────────────────────────────────────────┐
│  apiVersion: apps.example.com/v1alpha1                 │
│  kind: MyApp                       ← Custom Resource   │
│  spec:                                                 │
│    replicas: 3                     ← user's intent     │
│    version: 1.4.2                                      │
│  status:                                               │
│    conditions:                     ← controller's view │
│      - type: Ready                                     │
│        status: "True"                                  │
│    phase: Running                                      │
└────────────────────────────────────────────────────────┘
                       ↑
                       │ owns
                       │
┌────────────────────────────────────────────────────────┐
│  controller.Reconcile(ctx, req) ⟶ ctrl.Result, error   │
│   1. read CR (the spec) from the cache                 │
│   2. read actual state (Pods, Services, ConfigMaps)    │
│   3. diff actual against desired                       │
│   4. act idempotently to converge                      │
│   5. update status with observed state                 │
│   6. return RequeueAfter or done                       │
└────────────────────────────────────────────────────────┘

Reconcile runs whenever:

• The CR changes
• A child resource changes
• A periodic resync fires (default 10h, configurable)
• An explicit requeue from a previous run

Spec vs status — the cardinal split

Violating the split is the #1 cause of operator bugs:

• Mutating spec from the controller → user changes get overwritten
• Updating status without the subresource → status update triggers spec reconcile → loop

Reconcile must be idempotent

Reconcile is called repeatedly for the same state. The function must:

• Produce the same outcome regardless of call count
• Use Create-or-Update patterns (controllerutil.CreateOrUpdate)
• Compare current state to desired before writing
• Never assume "this is the first time we've seen this resource"

Idempotence test: if reconcile is called 100 times in a row with the same spec and no external change, the system must converge after the first call and do nothing on the next 99.

OwnerReferences and cascading deletion

Every child resource the operator creates must have its OwnerReferences set to the parent CR. Then:

• Deleting the CR deletes children automatically
• The garbage collector handles orphan cleanup
• The operator doesn't need explicit teardown logic for owned resources

External resources (cloud DBs, S3 buckets, DNS records) don't have OwnerReferences. Use finalizers to clean them up.

Finalizers

A finalizer blocks deletion until the controller has cleaned up external state.

1. User: kubectl delete myapp foo
2. API server: sets metadata.deletionTimestamp; does NOT delete
3. Controller: sees deletionTimestamp; does cleanup; removes finalizer
4. API server: deletion now proceeds

Without a finalizer, external resources orphan. With one, the controller has a guaranteed hook to run cleanup before the CR disappears.

Conditions

The standard pattern for status reporting:

status:
  conditions:
    - type: Ready          # type values are operator-defined
      status: "True"        # True | False | Unknown
      reason: "AllReady"    # PascalCase, programmatic
      message: "All replicas ready"  # human-readable
      lastTransitionTime: "2026-05-08T12:00:00Z"
    - type: Reconciling
      status: "False"
      reason: "Idle"
      lastTransitionTime: "2026-05-08T12:00:00Z"

Use meta/v1.Conditions and meta/v1.SetStatusCondition from kubebuilder/controller-runtime — don't roll your own.

Webhooks

Two types:

• ValidatingWebhook — reject invalid CRs at admission (better than failing in reconcile)
• MutatingWebhook — fill in defaults / inject sidecars (use sparingly; surprising side effects)

Run webhooks in the same controller binary or a sidecar; cert-manager rotates the certs.

Anti-patterns

• Imperative reconcile: "if event = create, do X; if event = update, do Y". Wrong shape. Reconcile = make actual=desired regardless of how we got here.
• No status subresource: status updates re-trigger reconcile.
• Status mutation in many places: centralize in a setStatus helper.
• Reconcile depending on event order: events can be missed; reconcile must converge from any starting state.
• Long reconcile (>2 min): blocks the work queue; split work via RequeueAfter.

Decision flow: when an operator is the right answer

Need: I want to manage <X> in Kubernetes.

Is <X> a stateless web app?           → Deployment + Service. Done.
Is <X> a stateless web app with config?  → Deployment + ConfigMap.
Need version upgrade automation?       → Helm. Done.
Need stateful behaviour (leader, peers)? → StatefulSet.
Need application-aware operations
  (backup, version migration, repair)?  → Operator.
Need to expose <X> as a k8s resource
  to other teams?                       → Operator.

When in doubt: start with Helm. Move to an operator only when Helm can't express the operational logic.

The reconcile loop

Reconcile is the heart of an operator. Most operator bugs are reconcile-loop bugs. The patterns below are deterministic — copy them.

Skeleton — Reconcile(ctx, req)

func (r *MyAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := log.FromContext(ctx)

    // 1. Fetch the CR
    var cr appsv1alpha1.MyApp
    if err := r.Get(ctx, req.NamespacedName, &cr); err != nil {
        if apierrors.IsNotFound(err) {
            return ctrl.Result{}, nil  // CR is gone; nothing to do
        }
        return ctrl.Result{}, err  // transient error → requeue
    }

    // 2. Handle deletion via finalizer
    if !cr.DeletionTimestamp.IsZero() {
        return r.reconcileDelete(ctx, &cr)
    }
    if !controllerutil.ContainsFinalizer(&cr, finalizerName) {
        controllerutil.AddFinalizer(&cr, finalizerName)
        return ctrl.Result{}, r.Update(ctx, &cr)
    }

    // 3. Mark Reconciling
    meta.SetStatusCondition(&cr.Status.Conditions, metav1.Condition{
        Type: "Reconciling", Status: metav1.ConditionTrue,
        Reason: "InProgress", Message: "Converging to desired state",
        ObservedGeneration: cr.Generation,
    })

    // 4. Do the work, idempotently
    res, err := r.reconcileNormal(ctx, &cr)

    // 5. Update status (always — even on error)
    if statusErr := r.Status().Update(ctx, &cr); statusErr != nil {
        log.Error(statusErr, "failed to update status")
        return res, errors.Join(err, statusErr)
    }

    return res, err
}

The 5-step shape

1. Fetch the CR. Handle NotFound cleanly — the CR may have been deleted between event and reconcile. 2. Handle deletion. If DeletionTimestamp is set, run cleanup, remove finalizer, return. 3. Set Reconciling condition. Mark that the controller is working. 4. Do work idempotently. Use CreateOrUpdate, compare desired-vs-actual, only act on differences. 5. Update status. Even on error — partial progress is signal.

Idempotence patterns

Pattern: CreateOrUpdate

deployment := &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: cr.Name, Namespace: cr.Namespace}}
op, err := controllerutil.CreateOrUpdate(ctx, r.Client, deployment, func() error {
    deployment.Spec.Replicas = &cr.Spec.Replicas
    deployment.Spec.Template.Spec.Containers = buildContainers(&cr.Spec)
    return controllerutil.SetControllerReference(&cr, deployment, r.Scheme)
})
if err != nil { return ctrl.Result{}, err }
log.Info("deployment", "operation", op)  // "created", "updated", or "unchanged"

This pattern is idempotent by construction.

Pattern: SetControllerReference

Always set the OwnerReference so cascading deletion works:

controllerutil.SetControllerReference(&cr, child, r.Scheme)

Pattern: Finalizer for external resources

const finalizerName = "myapp.apps.example.com/finalizer"

func (r *MyAppReconciler) reconcileDelete(ctx context.Context, cr *appsv1alpha1.MyApp) (ctrl.Result, error) {
    if !controllerutil.ContainsFinalizer(cr, finalizerName) {
        return ctrl.Result{}, nil
    }
    if err := r.deleteExternalResources(ctx, cr); err != nil {
        return ctrl.Result{RequeueAfter: 30 * time.Second}, err
    }
    controllerutil.RemoveFinalizer(cr, finalizerName)
    return ctrl.Result{}, r.Update(ctx, cr)
}

Error handling and requeue

Don't use `time.Sleep` inside reconcile. It blocks the work queue, starving other reconciles. Use RequeueAfter.

Status update patterns

// Set a condition
meta.SetStatusCondition(&cr.Status.Conditions, metav1.Condition{
    Type: "Ready", Status: metav1.ConditionTrue,
    Reason: "AllReady", Message: "all components healthy",
    ObservedGeneration: cr.Generation,
})

// Track observed generation
cr.Status.ObservedGeneration = cr.Generation

// Update status — uses /status subresource
if err := r.Status().Update(ctx, &cr); err != nil { ... }

Never call r.Update(ctx, &cr) to update status. It uses the spec subresource, which the user owns.

Read once, decide, act

Don't observe the world repeatedly during reconcile. The cache is read-only and consistent within a single reconcile pass:

// Good: read once, decide, act
var pods corev1.PodList
r.List(ctx, &pods, client.InNamespace(cr.Namespace), client.MatchingLabels{"app": cr.Name})
desired := computeDesired(&cr, &pods)
applyDesired(ctx, r.Client, desired)

// Bad: observe-act-observe-act
for _, container := range cr.Spec.Containers {
    pod := r.Get(...)               // re-reading the cache
    if needsRestart(pod) {
        r.Delete(...)
        pod = r.Get(...)             // again
        ...
    }
}

Predicates — filter events you don't care about

func (r *MyAppReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&appsv1alpha1.MyApp{}).
        Owns(&appsv1.Deployment{}).
        WithEventFilter(predicate.GenerationChangedPredicate{}).  // ignore status-only updates
        Complete(r)
}

GenerationChangedPredicate skips reconciles when only status changed — important to avoid loops.

Leader election

Always enable leader election when running >1 controller replica:

mgr, _ := manager.New(cfg, manager.Options{
    LeaderElection:   true,
    LeaderElectionID: "myapp-operator-leader",
})

Without it: split-brain. Two controllers both think they own the resource and fight.

Performance — bounded reconcile time

A reconcile pass should complete in <30s for typical work, <2min for heavy work. Longer = the work queue starves other reconciles.

If work takes longer:

• Break into phases; emit RequeueAfter between them
• Move long-running work to a separate process (Job)
• Cache expensive computations on cr.Status

Logging conventions

log := log.FromContext(ctx).WithValues("phase", "create-deployment")
log.Info("creating deployment", "name", cr.Name)
log.Error(err, "failed to create deployment")

• Use log.FromContext(ctx) — picks up controller-runtime's contextual logger
• Use Info for normal flow, Error for retryable failures
• Add structured fields, not formatted strings

Anti-patterns checklist

• time.Sleep inside reconcile → starves queue; use RequeueAfter
• os.Exit / log.Fatal → kills the controller; return an error
• panic → same; return an error
• r.Update to set status → use r.Status().Update
• r.Update of the CR while the user could be editing it → use r.Status().Update or use Patch
• Reading the same resource multiple times in one reconcile → read once
• Reconcile body > 80 lines → extract reconcileXxx subroutines per phase
• HTTP calls without ctx → can't cancel during shutdown
• No requeue path for transient errors → silent failures
• Missing OwnerReferences on children → cascading deletion broken

Tooling landscape

Five mainstream operator frameworks. Pick by language, complexity, and target environment.

At-a-glance

Decision tree

Primary language?
├── Go ──┬── Need scaffolding + opinionated path → kubebuilder
│        ├── Targeting OpenShift / OLM            → operator-sdk (Go)
│        └── Library-only, full control            → controller-runtime
├── Python ─────────────────────────────────────────→ KOPF
├── Java   ─────────────────────────────────────────→ java-operator-sdk
└── Other (Node, Ruby, Rust)
        └── webhook-based, polyglot                  → metacontroller

controller-runtime (Go library)

What it is: The Go library that everyone else builds on. Provides Manager, Reconciler, cache, client, predicates, leader election.

Use when:

• You need fine-grained control over the manager and event sources
• You're building reusable operator components
• Your team has Go experience and prefers libraries to scaffolders

Skip when:

• You want bootstrap-by-CLI (use kubebuilder)
• You don't speak Go

Example:

mgr, _ := ctrl.NewManager(cfg, ctrl.Options{Scheme: scheme})
ctrl.NewControllerManagedBy(mgr).
    For(&apps.MyApp{}).
    Complete(&MyAppReconciler{Client: mgr.GetClient()})
mgr.Start(ctx)

kubebuilder (Go scaffolder)

What it is: The standard scaffolding tool. Wraps controller-runtime with project layout, code generation, and the kubebuilder CLI.

Use when:

• New Go operator
• You want predictable project structure
• You'll publish the operator publicly

Workflow:

kubebuilder init --domain example.com --repo github.com/org/myapp-operator
kubebuilder create api --group apps --version v1alpha1 --kind MyApp
make manifests
make generate
make run

Strengths: Excellent docs, mature, used by everyone from cert-manager to Crossplane.

Weaknesses: Some teams find the layout opinionated; sometimes hard to escape from.

operator-sdk (Red Hat / OpenShift)

What it is: Wraps kubebuilder for Go and adds Helm-based and Ansible-based operators (no Go required).

Use when:

• Targeting OpenShift / OLM (Operator Lifecycle Manager)
• Building a Helm-based operator from an existing chart
• Building an Ansible-based operator from existing playbooks

Helm-based operator:

operator-sdk init --plugins=helm --domain example.com --group apps --version v1 --kind MyApp
operator-sdk create api --group apps --version v1 --kind MyApp --helm-chart=./mychart

The operator's reconcile becomes helm upgrade --install. Fast on-ramp; less power.

Ansible-based operator: Similar, but reconcile invokes a playbook. Useful for ops teams already deep in Ansible.

Skip when:

• Vanilla k8s target (kubebuilder is more direct)
• You want a Go operator without OpenShift coupling

metacontroller (webhook-based, language-agnostic)

What it is: Runs in-cluster, watches your CRDs, and POSTs webhook calls to your endpoints with desired-state computations. You implement the logic in any language behind an HTTP endpoint.

Use when:

• Polyglot team (Python, Node, Ruby, etc.)
• Want to avoid Go and Java
• Operator logic is genuinely simple (compute children from parent)

Example sync hook:

# Python webhook returns desired children given parent + observed
def sync(request):
    parent = request['parent']
    return {
        'status': {'phase': 'Ready'},
        'children': [{'apiVersion': 'apps/v1', 'kind': 'Deployment', ...}],
    }

Strengths: No Go required; fast iteration in any language.

Weaknesses: Lower ecosystem activity; not great for complex multi-CRD operators; webhook-based latency.

KOPF (Python)

What it is: A Python framework for building operators. Async-first, decorator-based, no scaffolding step.

Use when:

• Python shop
• Operator logic is moderate complexity
• Want fast iteration without recompilation

Example:

import kopf

@kopf.on.create('apps.example.com', 'v1alpha1', 'myapps')
async def create_fn(spec, name, namespace, logger, **_):
    logger.info(f"creating MyApp {name}")
    # ... create children
    return {'phase': 'Ready'}

@kopf.on.delete('apps.example.com', 'v1alpha1', 'myapps')
async def delete_fn(spec, name, namespace, **_):
    # cleanup external resources
    pass

Strengths:

• Async/await native (good for many concurrent reconciles)
• No code generation
• Good for ML/data teams already in Python

Weaknesses:

• Smaller ecosystem than Go
• Some features lag controller-runtime (e.g., complex caching)
• Python startup cost in the controller pod

java-operator-sdk

What it is: Java framework, Quarkus integration, modeled after controller-runtime.

Use when: JVM shop with strong Spring/Quarkus skills.

Skip when: You don't already have a JVM ops setup.

Comparison: complexity vs control

control ↑
        │  controller-runtime  (full control, library)
        │       │
        │  kubebuilder         (scaffolded controller-runtime)
        │       │
        │  operator-sdk Go     (kubebuilder + OLM)
        │       │
        │  KOPF                (Python decorators)
        │       │
        │  java-operator-sdk   (JVM)
        │       │
        │  operator-sdk Ansible (playbooks)
        │       │
        │  operator-sdk Helm   (chart-based)
        │       │
        │  metacontroller      (webhook hooks)
        ↓
complexity ↓

Higher control = more code, more flexibility. Lower complexity = faster start, less power.

Cross-cutting concerns

Regardless of framework:

• Webhooks for validation — reject bad CRs at admission
• cert-manager — rotate webhook certs automatically
• Prometheus — /metrics endpoint via controller-runtime's built-in metrics
• OLM (Operator Lifecycle Manager) — for OperatorHub publishing
• OperatorHub Capability Levels — see operator_capability_audit.py

Migration paths

Selection checklist

Before committing:

• [ ] Identify primary language constraint
• [ ] Target environment (vanilla k8s vs OpenShift/OLM)
• [ ] Operator complexity: 1 CRD vs many
• [ ] Need webhooks?
• [ ] Need OLM publishing?
• [ ] Build a 1-week proof-of-concept; verify reconcile latency, status update flow, and dev-loop ergonomics