Eks Platform Engineering

EKS Platform Engineering

Guidance for designing and building an Internal Developer Platform (IDP) on Amazon EKS. This skill is opinionated: it teaches one proven, integrated golden-path stack end to end rather than cataloguing every option. The reference architecture and tool choices below are the recommended default; deviate only with a reason.

This is the "how do I build a platform that other teams self-serve from" skill. For "how do I run a single EKS cluster well" (compute, networking, security, cost, upgrades), use eks-best-practices instead.

When to Use This Skill

Activate when the user wants to:

• Build or design an Internal Developer Platform / developer self-service on EKS
• Stand up or wire together a developer portal (Backstage), GitOps (ArgoCD/Argo Workflows), progressive delivery (Argo Rollouts), or promotion (Kargo)
• Define golden paths — standardized, paved ways for app teams to ship
• Abstract AWS resources behind Kubernetes (ACK, kro) or applications behind OAM/KubeVela
• Enable self-service environment or resource provisioning
• Measure platform success (DORA metrics)
• Use GenAI (Amazon Q Developer) to author platform templates/manifests
• Extend the platform to AI/ML or data-engineering workloads

Don't use this skill for:

• Single-cluster EKS architecture, sizing, cost, or upgrade decisions with no self-service/platform angle → eks-best-practices
• Standalone Terraform/OpenTofu module authoring → terraform-skill
• Discovering what's already running on a cluster → eks-recon
• Generic Kubernetes concepts (Claude knows these)

What Is an Internal Developer Platform

CNCF defines a platform as "an integrated collection of capabilities defined and presented according to the needs of the platform's users." An IDP packages tools, services, and automation so application, ML, and data teams self-serve — provisioning environments, resources, and deployments without tickets — while a platform team owns the paved paths and guardrails.

Why it matters: speed (faster delivery), control (safe, consistent operations), cost (economies of scale), and continuous improvement (shared golden paths). Gartner projects 80% of large software orgs will have platform engineering teams by 2026.

The core principle — separation of concerns:

• Platform team defines how: portal templates, OAM component/trait definitions, ACK controllers, CI/CD scaffolds, promotion stages, guardrails.
• App/ML/data teams choose what: pick a template, fill parameters, push code. They never touch raw Kubernetes or AWS APIs.

See references/idp-architecture.md for the full reference architecture and the IDP value/challenge framework.

The Opinionated Platform Stack

This is the recommended, integration-tested stack. Each layer has one default tool.

Maturity note on `kro`: kro (which bundles many resources into one simple custom resource) is newer and less battle-tested than the rest of this stack. It works well, but confirm its current maturity fits your risk tolerance before you standardize on it. If you want a more established tool for the same job — composing AWS and Kubernetes resources behind one API — Crossplane is the proven alternative.

Cluster topology — hub-and-spoke (default): a hub cluster runs the platform control plane (ArgoCD, Backstage, GitLab, Keycloak, Kargo); spoke clusters (dev, prod) run workloads. EKS Auto Mode is the default cluster type so the platform team isn't managing nodes. Infrastructure metadata flows to clusters via the GitOps Bridge pattern. Details: references/idp-architecture.md.

The Golden Paths

A golden path is a paved, opinionated route from intent to running software, with guardrails baked in. The platform ships these as Backstage templates. The four core paths:

1. Provision an environment — Backstage template → kro composes VPC + EKS Auto Mode cluster + add-ons + ArgoCD registration. Minutes, not days. → references/infrastructure-abstraction.md 2. Provision an AWS resource — Backstage template → ACK manifest committed to Git → ArgoCD syncs → ACK provisions the AWS resource. → references/infrastructure-abstraction.md 3. Onboard an application — Backstage template → kro CICDPipeline → Argo Workflows CI + ArgoCD CD. Developer contract: provide a Dockerfile + an OAM Application manifest; the platform does the rest. → references/developer-portal-backstage.md, references/gitops-delivery.md 4. Ship safely to production — Argo Rollouts canary with gates in dev; Kargo promotes the validated artifact to prod (auto in dev, manual approval for prod). → references/progressive-delivery.md

The developer contract (what app teams provide vs. what the platform handles) is the design heart of every golden path — see references/golden-paths.md.

Guardrails — making the paved path the safe path

Self-service is only safe if the guardrails are built into the golden path, so they apply automatically and a developer cannot accidentally skip them:

• Policy-as-code — an admission policy engine (Kyverno or OPA Gatekeeper) automatically rejects unsafe workloads at deploy time. Examples: no privileged containers, images only from approved registries, required labels present.
• Supply-chain security — the platform's CI signs each image and produces an SBOM (a bill of materials of what is inside the image), and the cluster only admits signed images. This guarantees every running container actually came from your pipeline, not from somewhere untrusted.
• Least-privilege IAM, Pod Security Admission, and ingress conventions — encoded once in the OAM traits, so every app inherits them by default instead of each team getting them right by hand.

The platform team owns these controls; app teams get them for free. For the cluster-level depth behind each one, use eks-best-practices (its security.md and security-supply-chain.md references).

Application Modeling — OAM / KubeVela

Developers describe apps with the OAM Application CRD, not raw Deployments/Services/Ingresses:

• Component = a runnable unit (e.g. appmod-service, dynamodb-table).
• Trait = an operational add-on attached to a component (e.g. path-based-ingress, component-iam-policy).
• Platform engineers author component/trait definitions (in CUE); developers just reference them by type and set parameters, ordered with dependsOn.

This is the abstraction that lets a developer request "an app + a DynamoDB table + an IAM-scoped service account + ingress" in one declarative file. Full model, CUE authoring, and the appmod-service example: references/application-model-oam.md.

Progressive Delivery and Promotion

The platform's appmod-service component defaults to an Argo Rollouts canary: 20% → 40% → 60% → 80% → 100%, with quality gates that auto-rollback on failure:

• Functional gate (at 20%) — smoke/correctness test.
• Performance gate (at 80%) — load/latency test (e.g. Artillery).
• Metrics gate — developer-defined Prometheus queries (e.g. avg response time > 3s → fail).

Kargo orchestrates multi-stage promotion: a Warehouse watches ECR for new images; the dev stage auto-promotes; the prod stage requires manual approval and promotes the exact same artifact validated in dev. All promotions are Git commits (auditable, reversible). Strategies (canary/blue-green/A-B), gate config, and the Kargo Warehouse/Stage/Freight model: references/progressive-delivery.md.

Measuring Platform Success

A platform you can't measure is a platform you can't justify. Track the four DORA metrics — deployment frequency, lead time for changes, change failure rate, recovery time — with Apache DevLake ingesting signals from Argo Workflows/Rollouts and GitLab, visualized in Grafana. Measurement is zero-overhead: it's wired in when a team onboards via Backstage.

Pair delivery metrics with cost visibility (showback): attribute spend per team/tenant and surface it in the portal so each team sees what its workloads cost. DORA tells you how fast you ship; showback tells you what it costs to run. Cluster-level cost levers (Spot, Graviton, right-sizing, Karpenter consolidation) live in eks-best-practices (cost-optimization.md).

Framework and dashboards: references/measuring-success.md.

GenAI-Assisted Platform Engineering

Amazon Q Developer accelerates both tracks: code generation (app developers) and platform generation (platform engineers — OAM components, Backstage templates, deployment manifests). The reliable pattern is reference example + target schema + prompt → generated artifact → human review. Always human-in-the-loop; expect hallucinations. Prompt patterns, the /dev feature agent, and "platform-meta as RAG context" → references/genai-platform-engineering.md.

Identity and Multi-Tenancy

Keycloak provides SSO across all tools; EKS Pod Identity (preferred) / IRSA give workloads credential-free AWS access; per-team namespaces, RBAC, and one-repo-per-component keep tenants isolated. Details: references/identity-and-tenancy.md.

Platform for AI/ML and Data Engineering

The same golden-path model extends to ML and data teams via Backstage templates:

• Model development — JupyterHub (multi-user notebooks, Keycloak SSO).
• Model serving — Ray Serve (Backstage template → Git → ArgoCD → Ray cluster → inference endpoint).
• Data engineering — Spark Operator (Backstage template → Argo Workflows → SparkApplication CRD).
• Key pattern — a DaemonSet pre-pulls large ML images to nodes to kill cold-start latency.

Full ML/data golden paths: references/aiml-data-platform.md.

How to Use the References

This skill uses progressive disclosure — the essentials are above; load a reference only when the task needs that depth:

Sources

• Platform Engineering on EKS workshop (PEEKS) and its companion appmod-blueprints
• Internal Developer Platform · CNCF Platforms White Paper
• Backstage · Argo Project (CD, Workflows, Rollouts) · Kargo
• AWS Controllers for Kubernetes (ACK) · kro · KubeVela / OAM
• Amazon EKS Auto Mode · EKS Pod Identity · Amazon Q Developer · Apache DevLake (DORA)

Platform for AI/ML and Data Engineering

The same golden-path model — Backstage template → Git → ArgoCD → controllers — extends to ML and data teams. The platform handles the undifferentiated heavy lifting (infra, GitOps, SSO, observability) so ML/data engineers focus on models and pipelines.

Why platform engineering for AI/ML

ML delivery has hard problems (data lineage, reproducibility, large images, GPU scheduling, drift) and an end-to-end lifecycle (data prep → model dev → serving → monitoring) that benefits from MLOps collaboration between data scientists, ML engineers, and ops. A platform standardizes that lifecycle: productivity, scalability, consistency, faster time-to-market, resource efficiency, security, and compliance — by default.

OSS-first on Kubernetes. The platform favors Kubernetes-native OSS so ML/data workloads run on the same substrate as everything else: Ray, JupyterHub, Spark-on-K8s (and, in the broader landscape, Kubeflow, MLflow, Airflow).

The image pre-pull pattern (key platform decision)

AI/ML container images are large (Jupyter 4GB+, framework images 3GB+), so cold starts are painful. The platform runs a DaemonSet that pre-pulls these images to nodes in the background, so notebooks/serving pods start near-instantly. This is an opinionated platform-level solution to a problem every ML team would otherwise hit independently.

Golden path: model development — JupyterHub

JupyterHub provides multi-user, self-service notebooks on Kubernetes. Components: Hub (auth/management), Proxy (routing), and per-user notebook servers.

• Integrated with Keycloak SSO — same identity as the rest of the platform; users just log in.
• Platform manages compute provisioning, packages, and notebook persistence; ML engineers get a ready environment (e.g. build a scikit-learn classifier with no infra setup).
• Extension points: real datasets, GPU instance types, save models to S3, connect MLflow.

Golden path: model serving — Ray Serve

Ray on Kubernetes (Ray cluster + Ray K8s Operator + Ray Serve) for scalable, low-latency model serving.

Backstage "Ray Service" template → Git repo (manifests) → ArgoCD → Ray Serve resources → inference endpoint

• ML engineer supplies: name, namespace, worker replicas, model, max tokens. Platform supplies: provisioning, GitOps pipeline, Ray Dashboard observability, the endpoint.
• Auto-scaling, multi-framework, zero-downtime updates, operator-managed — the model is replaceable with the team's own.
• Test the endpoint with a simple curl POST to /generate.

Golden path: data engineering — Spark Operator

Spark-on-Kubernetes via the Spark Operator, jobs expressed as SparkApplication CRDs and orchestrated by Argo Workflows.

Backstage "Spark job" template → Git repo (manifests) → ArgoCD → Argo Workflows → Spark Operator → SparkApplication → driver/executor pods

• Data engineer supplies: app name + the mainApplicationFile (the PySpark script). Platform supplies: Spark infra, workflow orchestration, and observability (Backstage Spark tab + Argo Workflows UI).
• kubectl get sparkapplications.sparkoperator.k8s.io -A to inspect runs.

The throughline

ML serving, data jobs, and apps all follow the identical loop — a Backstage template generates a Git repo, ArgoCD deploys it, and a controller (Ray/Spark Operator, or KubeVela for apps) reconciles it. One platform, one delivery model, three audiences. That uniformity is the payoff: the platform team maintains one set of paved paths; ML and data teams self-serve exactly like app teams.

Application Model — OAM / KubeVela

The abstraction that lets developers describe an application — and its infrastructure dependencies — in one declarative file, without writing Deployments, Services, Ingresses, IAM, or ACK manifests directly.

OAM concepts

• Component — a runnable unit of work (a web service, a worker, or even a resource like a DynamoDB table).
• Trait — an operational feature attached to a component (ingress, autoscaling, an IAM policy, monitoring).
• Application — the developer-facing CRD that composes components + their traits.

KubeVela implements OAM on Kubernetes. Developers write kind: Application; KubeVela renders it into the underlying Kubernetes (and, via custom components, AWS) resources.

Separation of concerns

• Platform engineers author component and trait definitions (in CUE) — type-safe, composable templates. They own the "how."
• Developers reference a component/trait by type and set its parameters. They own the "what." They never see the rendered Deployment/Service.

A minimal app:

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: first-app
spec:
  components:
    - name: express-server
      type: webservice          # platform-defined component type
      properties:
        image: oamdev/hello-world
        ports: [{ port: 80, containerPort: 8000, expose: true }]

KubeVela renders this into a Deployment + Service: kubectl get deployment,service -l app.oam.dev/name=first-app.

The platform's app components (the appmod-service pattern)

The PEEKS platform ships richer, opinionated components so a developer can request an app and its dependencies in one ordered manifest using dependsOn:

spec:
  components:
    - name: dynamodb-table          # 1. the AWS resource (via ACK under the hood)
      type: dynamodb-table
      properties: { tableName: app-table, partitionKeyName: pk, sortKeyName: sk, region: us-west-2 }
      traits:
        - type: component-iam-policy
          properties: { service: dynamodb }
    - name: app-sa                  # 2. a service account scoped to that table (Pod Identity)
      type: dp-service-account
      properties: { componentNamesForAccess: [dynamodb-table], clusterName: peeks-spoke-dev, clusterRegion: us-west-2 }
      dependsOn: [dynamodb-table]
    - name: backend                 # 3. the app itself (canary via Argo Rollouts by default)
      type: appmod-service
      properties: { image: "<ecr-uri>:<tag>", port: 80, targetPort: 8080, replicas: 2, serviceAccount: app-sa }
      dependsOn: [app-sa]
      traits:
        - type: path-based-ingress
          properties: { rewritePath: true, http: { /app: 80 } }

What this single file produced: a DynamoDB table (ACK), a scoped IAM role + Pod Identity association, a canary-enabled Deployment + Service, and a path-based ingress — in dependency order. That is the power of the model: infrastructure + app + networking + security as one developer-authored unit.

Common appmod-service parameters: image, replicas, port/targetPort, serviceAccount, resources, env. Common traits: path-based-ingress, component-iam-policy. Plus the progressive-delivery gates (functionalGate/performanceGate/metrics) from progressive-delivery.md.

CUE and extensibility

Component/trait definitions are written in CUE (e.g. the webservice definition lives in KubeVela's templates). CUE is chosen over raw YAML for type-safety and composition. Platform engineers add new component/trait types without changing the core platform; developers pick them up immediately by type. GenAI (Amazon Q) is commonly used to author new component definitions from an existing one + a CRD schema (see genai-platform-engineering.md).

Why OAM here

• Developers ship faster — one file, no Kubernetes plumbing.
• Standards enforced centrally — every app gets the platform's ingress/IAM/rollout conventions by construction.
• GitOps-native — Applications are YAML in Git, reconciled by ArgoCD, rendered by KubeVela; multi-cluster from a single definition.

Developer Portal — Backstage

Backstage is the platform's front door: a self-service catalog plus software templates (the "scaffolder"). Developers interact with the platform here and rarely touch raw Kubernetes or AWS.

Why Backstage

• Open-source, CNCF, plugin-based — extensible without forking.
• Software Templates encode golden paths with parameters and guardrails.
• Service Catalog gives a single pane of glass (components, APIs, docs, owners).
• Integrates with Keycloak SSO so it shares identity with every other tool.

Anatomy of a software template

A Backstage template is a folder:

my-template/
├── template.yaml          # definition: parameters + steps (actions)
└── skeleton/              # scaffolded files, rendered with parameter values
    ├── catalog-info.yaml  # registers the new component in the catalog
    └── manifests/         # e.g. an OAM Application or ACK manifest

• `template.yaml` declares the form parameters the developer fills, then the ordered steps/actions: fetch skeleton → publish to Git → register in ArgoCD/catalog → trigger provisioning.
• `skeleton/` is the payload: the manifests and catalog-info.yaml that get committed to a new Git repo, rendered with the developer's parameter values.

The self-service flow (every template follows this shape)

Developer fills Backstage form
   → template creates a new GitLab repo with rendered skeleton (manifests)
   → registers the repo/app with ArgoCD (and the catalog)
   → ArgoCD syncs the manifests to the target cluster
   → controllers reconcile (KubeVela renders the app; ACK provisions AWS resources)

This is the same loop whether the artifact is an AWS resource (ACK manifest), an environment (kro composition), a CI/CD pipeline (kro CICDPipeline), or an ML serving endpoint (Ray Serve manifest). One repo per provisioned component is the default — it keeps GitOps reconciliation and ownership clean.

Opinionated template conventions

• Guardrails via fixed parameters / configuration tiers. Pre-define profiles (e.g. an S3 "standard" config) so developers pick a vetted option rather than free-typing settings.
• Namespace + region as parameters with sensible defaults (e.g. us-west-2, team namespace).
• Templates reference cluster-side definitions. A template emits an OAM Application that references a type (e.g. ddb-table) whose component definition the platform team owns — the template is just the friendly form over it.
• Register via `catalog-info.yaml` so the new component appears in the catalog with its owner, links (source repo, ArgoCD app, dashboards), and docs.

Catalog as the operational hub

Once registered, a component's catalog page links its source repo, its ArgoCD application (sync/health), its dashboards, and (for ML/data) execution details (Spark runs, Argo Workflow pipelines). This gives developers troubleshooting visibility without cluster access — directly answering the "fixing problems" adoption challenge.

Generating templates with GenAI

Amazon Q Developer can scaffold a whole new template from an existing one (e.g. "use the S3 template to make a DynamoDB template"). Pattern: point Q at a reference template + the target resource's OAM/CRD schema, generate, then diff against a validated version before registering. See genai-platform-engineering.md.

GenAI-Assisted Platform Engineering

Amazon Q Developer accelerates both sides of the platform: developers generating application code, and platform engineers generating the platform's own artifacts (OAM components, Backstage templates, deployment manifests). Always human-in-the-loop.

Two tracks

• Code generation (app developers) — snippets, functions, whole features in unfamiliar languages.
• Platform generation (platform engineers) — OAM component definitions, Backstage templates, deployment/IaC manifests, runbooks, automation.

The goal is to accelerate adoption of platform practices, not replace judgment.

The reliable generation pattern

reference example  +  target schema/CRD  +  a precise prompt  →  generated artifact  →  human review (diff)  →  use

Examples from the platform:

• Generate an OAM component — "use the S3 OAM component definition as a template and create a ddb-table component from this DynamoDB ACK CRD; only required properties except billingMode: PAY_PER_REQUEST; 4 params; namespace vela-system." Deploy to vela-system; developers then self-serve via type: ddb-table.
• Generate a Backstage template — "use the S3 Backstage template's folder structure and stages to create a DynamoDB template; reference this OAM definition." Produces template.yaml + skeleton/ (catalog-info + manifests).
• Generate a deployment manifest — "create an OAM Application using these KubeVela templates, in strict dependsOn order: DynamoDB table → service account → app → path-based ingress; default/required params; reference src/ for context."

Prompt engineering essentials

• Be specific: state the goal, the inputs, the constraints, and the expected output shape.
• Provide context: point at the reference file, the schema, and existing code (src/, platform-meta/examples/).
• Decompose complex tasks; iterate — GenAI is non-deterministic, so the same prompt may vary.
• Give acceptance criteria (error handling, tests, docs, conventions).

platform-meta as ad-hoc RAG; Q Developer Customizations as the future

The platform keeps a platform-meta folder of component definitions and examples that serves as context (RAG) for Q when generating manifests — pointing Q at it makes generations follow platform conventions. The forward path is Q Developer Customizations: the platform team trains the model on its component knowledge, so developers get platform-aware generations without manually supplying context.

The /dev feature agent

Q Developer's /dev agent does multi-step feature development (plan → discover → generate → edit → verify → loop) and returns a multi-file diff. Prompt it with the routes/behavior, the persistence layer, requirements (error handling, tests, docs), and references to existing services so it follows established patterns.

Human-in-the-loop is non-negotiable

Expect hallucinations (non-existent methods) and missed integration steps (e.g. a new route not wired into main). Always diff generated artifacts against a validated version before registering/deploying. GenAI produces the skeleton; the engineer validates and finishes. Then it ships through the same golden path as hand-written code — git push → CI/CD → canary.

GitOps Delivery — ArgoCD + Argo Workflows

The delivery backbone. Git is the single source of truth; ArgoCD reconciles cluster state to match; Argo Workflows runs the build pipelines that feed it.

Division of labor

• Argo Workflows (+ Argo Events) = CI. Webhook-triggered, container-native pipelines that build images, run tests, push to ECR, and update deployment manifests in Git.
• ArgoCD = CD. Watches Git, detects manifest changes, syncs them to the target cluster, reports sync/health, and detects drift.

Keeping CI and CD separate (rather than one monolithic pipeline) means a failed build never half-deploys, and the deployed state is always exactly what's in Git.

ArgoCD patterns

• App-of-apps / ApplicationSets — a bootstrap Application points at a Git path of child Applications; ApplicationSets template one app per cluster/environment from a generator (e.g. the cluster-registration secrets from the GitOps Bridge). This is how 60+ platform add-ons stay in sync across hub + spokes.
• Cluster registration — a new spoke registers with the hub's ArgoCD; ApplicationSets then deploy the full add-on set to it automatically.
• One ArgoCD Application per deployment target — e.g. app-dev-cd watches deployment/dev/, app-prod-cd watches deployment/prod/. Promotion = changing what's in the prod path (done by Kargo, see progressive-delivery.md).
• SSO — ArgoCD logs in via Keycloak; no separate credentials.

The application CI/CD pipeline (golden path 3)

The platform provisions an app's whole pipeline from one Backstage template that emits a kro CICDPipeline (see infrastructure-abstraction.md). That single CRD reconciles into:

• Argo Workflow templates (build + push to ECR + update manifest)
• Argo Events sensor/trigger (fires the workflow on git push)
• ECR repository
• ArgoCD applications for dev and prod (<app>-dev-cd, <app>-prod-cd)
• build caches

Two workflows typically run: a one-time *-setup-workflow (cache warming / provisioning) and the recurring *-initial-build-workflow (build → push → update the image: field in deployment/dev/application.yaml).

The developer contract for delivery

The developer provides only: 1. A `Dockerfile` in the repo (the platform expects nothing else for build — multi-stage builds, static linking, non-root, minimal base images are the recommended convention). 2. An OAM `Application` manifest in deployment/dev/ describing what to run (see application-model-oam.md).

Then: git push → Argo Workflows builds and pushes the image and bumps the manifest → ArgoCD syncs → KubeVela renders the app + its DynamoDB/IAM/ingress. "Deploying" is just pushing to Git.

Verifying

• kro pipeline object: kubectl get cicdpipelines <name> -n <team-ns> → expect STATE=ACTIVE, SYNCED=True.
• ArgoCD app: check Synced/Healthy (UI or argocd app get <app>); force with argocd app sync <app>.
• Workflow run: Argo Workflows UI shows build progress; kubectl get pods -n <team-ns> --watch shows the rollout.
• Allow ~3–5 minutes for ArgoCD to detect a Git change on default sync intervals.

Golden Paths

A golden path is a paved, opinionated route from intent to running software, with guardrails baked in. It is the central deliverable of a platform team — not a document, but a working, self-service flow shipped as a Backstage template.

The design heart — the developer contract

Every golden path is a contract: a small, stable set of things the developer provides, and everything else the platform handles.

Keep the left column tiny and stable. The narrower the developer contract, the lower the cognitive load and the higher adoption.

The four core golden paths

1. Provision an environment — template → kro composes VPC + EKS Auto Mode + add-ons + ArgoCD registration. 2. Provision an AWS resource — template → ACK manifest → Git → ArgoCD → resource exists. 3. Onboard an application — template → kro CICDPipeline → Argo Workflows CI + ArgoCD CD; developer provides Dockerfile + OAM manifest. 4. Ship safely to prod — Argo Rollouts canary + gates in dev; Kargo promotes the validated artifact to prod.

All four share the same Backstage → Git → ArgoCD → controllers loop (see developer-portal-backstage.md).

Design principles

• Make the golden path the easiest path. Adoption is the #1 IDP challenge; if the paved road is also the path of least resistance, teams take it.
• Guardrails, not gates everywhere. Encode standards (PSA levels, IAM scoping, ingress conventions, configuration tiers) into the templates and OAM definitions so compliance is the default, not a review step.
• Provide escape hatches. Opinionated ≠ locked-in. A team with a genuine edge case can drop to a lower abstraction; just make the common case trivial.
• One repo per provisioned component. Clean GitOps reconciliation and clear ownership.
• Self-service onboarding. New apps, environments, resources, and even DORA monitoring start from a template — no tickets.
• Iterate from real usage. Watch what teams actually do; if every app independently reinvents the same manifest or script, fold it into a component/template.

Onboarding a new team (typical sequence)

1. Team picks the "app CI/CD pipeline" template in Backstage, names the app and target cluster. 2. kro CICDPipeline reconciles; ArgoCD dev/prod apps appear; ECR repo created. 3. Team adds a Dockerfile + an OAM Application to their repo and pushes. 4. CI builds, CD deploys to dev with canary + gates; DORA tracking is live automatically. 5. When dev is validated, Kargo promotes to prod (manual approval).

Anti-patterns

• A golden path that requires Kubernetes/AWS knowledge to use — the abstraction has leaked.
• A wide, unstable developer contract — every new requirement becomes a new field the developer must understand.
• Golden paths with no measurement — you can't tell if they're helping (see measuring-success.md).
• Treating the platform as a project, not a product — no iteration, no adoption.

Identity and Multi-Tenancy

How the platform gives one identity across all tools, credential-free AWS access to workloads, and isolation between teams sharing the platform.

Keycloak — unified SSO

Keycloak is the identity backbone. Every platform tool — Backstage, ArgoCD, Argo Workflows, Kargo, Grafana, GitLab, JupyterHub, Ray Dashboard — federates authentication through Keycloak via OIDC/SAML.

• One login, access everything. A user authenticates once; the session propagates across the toolchain ("Log in via Keycloak").
• Decoupled identity. Identity management is separated from the portal and the CI/CD tools, so auth policy changes in one place.
• Why it matters for adoption. Per-tool credentials are friction; SSO removes it and centralizes access control.

Workload identity — EKS Pod Identity / IRSA

Workloads (and platform controllers like ACK) get AWS permissions without stored credentials:

In the OAM model, the dp-service-account component creates a service account bound to a scoped IAM role and wires the Pod Identity association — so a developer requesting "access to my DynamoDB table" gets a least-privilege role automatically, never an access key. Controllers (ACK) likewise use IRSA/Pod Identity, so no AWS keys live in the cluster.

Multi-tenancy

The platform is shared; tenants are isolated by construction:

• Per-team namespaces (e.g. team-rust, team-java) with RBAC.
• One repo per component/app, so GitOps boundaries match ownership.
• Scoped IAM — each workload's role grants only the resources it declared (via component-iam-policy + dp-service-account).
• Spoke clusters separate dev and prod entirely (own VPC + cluster); the hub never runs tenant workloads.
• Configuration tiers / PSA / policy — templates and OAM definitions enforce pod-security and resource standards so tenants can't drift below the baseline.

Putting it together

A tenant team logs into Backstage (Keycloak), orders a pipeline and resources via templates (landing in their namespace + repos), and their workloads run with scoped Pod Identity roles on a spoke cluster — all without the platform team issuing a single credential or running a single manual provisioning step.

IDP Architecture

The reference architecture for an Internal Developer Platform on EKS, and the framework for reasoning about IDP value and adoption.

What an IDP is (and isn't)

An IDP is not a single product — it's an integrated collection of capabilities presented to platform users (developers, ML engineers, data engineers) so they can self-serve. CNCF: "an integrated collection of capabilities defined and presented according to the needs of the platform's users."

Analogy: a train-station platform. The platform adds value only when people actively use it. An IDP centralizes tools, services, and automation so app teams focus on their work, not infrastructure.

The four goals of a platform team

1. Help developers work independently (self-service). 2. Reduce cognitive load. 3. Create reusable best practices ("golden paths"). 4. Automate common tasks (clusters, CI/CD, resource provisioning).

Value

Challenges (and the counter-move)

Reference architecture (opinionated)

                         ┌──────────────────────────────────────────┐
   Developer ──▶ Backstage (portal, templates) ── Keycloak SSO ──────┤
                         │                                            │
                         ▼  git commit                                │
                      GitLab  ──webhook──▶ Argo Workflows (CI) ──▶ ECR │
                         │                                            │
                         ▼                                            │
                      ArgoCD (GitOps CD) ───────────────────────────▶ ┘
                         │  reconciles
        ┌────────────────┼─────────────────────────────────────┐
        ▼                ▼                                       ▼
   HUB cluster      SPOKE: dev                              SPOKE: prod
   (control plane)  (EKS Auto Mode)                         (EKS Auto Mode)
   ArgoCD/Backstage  Argo Rollouts (canary+gates)           Argo Rollouts
   GitLab/Keycloak   ACK · kro · KubeVela                   ACK · kro · KubeVela
   Kargo · DevLake   External Secrets                       External Secrets
        │  Kargo promotes validated artifact: dev ─────────────▶ prod
        └─────────────────────────────────────────────────────────┘

Hub-and-spoke topology

• Hub cluster — runs the platform control plane: ArgoCD, Backstage, GitLab, Keycloak, Kargo, DevLake, Grafana. It is the management plane, not a workload plane.
• Spoke clusters — dev and prod, run application/ML/data workloads. Each spoke registers with the hub's ArgoCD and receives platform add-ons automatically.
• Why hub-and-spoke: centralizes control-plane operations, isolates workloads, and lets new spokes self-register and inherit the full add-on set via GitOps.
• EKS Auto Mode is the default cluster type for both hub and spokes so the platform team does not manage nodes, scaling, or patching.

The GitOps Bridge pattern

The "GitOps Bridge" passes infrastructure metadata (cluster name, region, IAM role ARNs, VPC IDs, OIDC provider, add-on config) from the IaC/provisioning layer into the cluster as Kubernetes secrets/labels that ArgoCD ApplicationSets read. This decouples "how the cluster was built" from "what gets deployed onto it" — the same GitOps definitions target any registered cluster because the bridge supplies the per-cluster specifics.

Practically: 1. Provisioning (CDK/Terraform/kro) creates the cluster + a standardized cluster-registration secret containing metadata. 2. ArgoCD ApplicationSets template add-ons/apps using values from that secret. 3. New clusters become deployment targets the moment their registration secret exists.

Component map (what each tool owns)

See the matching references for each layer's depth.

Infrastructure Abstraction — ACK and kro

How the platform turns Kubernetes into the universal control plane for AWS resources (ACK) and composes many resources into one self-service unit (kro).

ACK — AWS Controllers for Kubernetes

ACK represents AWS resources (S3, DynamoDB, IAM, EC2, ECR, EKS, …) as Kubernetes Custom Resources. A controller reconciles each CRD to the actual AWS resource.

Why ACK in this platform:

• Kubernetes-native — no separate IaC tool or context switch; same kubectl/RBAC/GitOps model as everything else.
• GitOps-friendly — an ACK manifest in Git is reconciled by ArgoCD and provisioned by the controller, with drift detection for free.
• Credential-free — controllers use IRSA/Pod Identity; no AWS keys in the cluster.

The provisioning golden path:

Backstage template → ACK manifest committed to Git → ArgoCD syncs → ACK controller provisions the AWS resource

Example: a developer fills an "S3 bucket" form (name, env, region, namespace, config tier) → a kind: Bucket ACK manifest lands in a new repo → ArgoCD applies it → the bucket exists. Verify in ArgoCD (synced/healthy) + the AWS console.

Conventions: namespace-scoped resources for multi-tenancy; configuration tiers (platform-defined profiles) so developers choose vetted settings rather than free-typing.

kro — Kubernetes Resource Orchestrator

kro composes multiple resources into a single custom API. The platform team defines a ResourceGraphDefinition; developers instantiate one CRD and get the whole graph.

Why kro: it raises the abstraction from "many manifests" to "one intent." Two platform-defining examples:

• `CICDPipeline` — one CRD reconciles into Argo Workflow templates + Argo Events sensor/trigger + ECR repo + dev/prod ArgoCD applications + caches. This is what the app-onboarding Backstage template emits (see gitops-delivery.md).

  kubectl get cicdpipelines <app>-cicd-pipeline -n <team-ns>   # STATE=ACTIVE, SYNCED=True

• Environment provisioning — a kro composition creates VPC + EKS Auto Mode cluster + add-ons + ArgoCD registration as one unit (golden path 1). Self-service environments in minutes, not days; each gets its own VPC + cluster for isolation.

ACK vs kro — they layer, not compete

• ACK = the building blocks (individual AWS resources as CRDs).
• kro = the composition (bundle ACK CRDs + native K8s objects into one developer-facing API).
• A kro ResourceGraphDefinition commonly contains ACK resources. The Backstage template is the friendly form over the kro CRD; kro fans out to ACK + native resources; ArgoCD keeps it all reconciled.

Why not raw Terraform/CloudFormation here

The platform deliberately keeps provisioning in the Kubernetes/GitOps plane so there's one reconciliation model, one RBAC model, and one source of truth. Terraform/CDK still bootstrap the platform itself (the hub, initial clusters); ACK/kro handle day-to-day self-service on top. (For standalone Terraform module work, that's the terraform-skill, not this skill.)

Measuring Platform Success

A platform you can't measure is a platform you can't improve or justify. The default metrics framework is DORA, collected automatically by Apache DevLake and visualized in Grafana.

Why measure

Industry signal: a large share of platform teams don't measure success at all, yet leading adopters report markedly faster time to market. Without measurement you can't identify bottlenecks, make data-driven decisions, validate investment, or demonstrate ROI. DORA gives a vocabulary both engineers and business leaders understand.

The four DORA metrics

Balance matters: velocity (frequency, lead time) and stability (failure rate, recovery). Healthy platforms move lead times from weeks to hours and failure rates from ~20% to under 5% while keeping recovery fast.

Apache DevLake — the measurement engine

• Purpose-built for DORA, with standardized, industry-aligned calculations.
• Integrates across the toolchain (Git, CI/CD, deployment, issue tracking).
• Automated collection (no manual data gathering), historical trending, and cross-team comparison.

Platform integration (how the data flows)

GitLab webhooks (commits, PRs, issues)  ┐
Argo Workflows (deployment events)       ├─▶ Apache DevLake ─▶ Grafana DORA dashboards
Argo Rollouts (deploy success/failure)  ┘        (calculates + stores)

• Argo Workflows processes deployment/measurement events.
• Argo Rollouts signals whether a deploy succeeded.
• DevLake computes and stores the four metrics.
• Grafana renders DORA Overview + per-metric detail dashboards.

Zero-overhead, self-service measurement

The key design choice: measurement is wired in when a team onboards via Backstage. Creating the CI/CD pipeline also deploys the DORA tracking workflows and the GitLab webhook event managers. Teams get measured simply by using the platform — no extra tooling or process. This is what makes DORA sustainable rather than a one-off audit.

Using the metrics as a platform team

• Identify bottlenecks (e.g. long lead time → CI is slow; high failure rate → weak gates).
• Make data-driven roadmap decisions and validate that a platform change actually helped.
• Demonstrate ROI to leadership in a shared language.
• Compare/benchmark teams to spread the best golden paths.

Progressive Delivery and Promotion

How the platform rolls out new versions safely (Argo Rollouts) and promotes them across environments (Kargo).

Strategies

The platform's appmod-service OAM component wires Argo Rollouts canary automatically — developers get progressive delivery without writing Rollout specs.

The default canary

new version → 20% traffic   [functional gate]
            → 40% (pause)
            → 60% (pause)
            → 80%           [performance gate / metrics gate]
            → 100%          old version retired

Any gate failure (or an image-pull failure, etc.) triggers automatic rollback to the last stable version.

Quality gates (Argo Rollouts Analyses)

Three gate types, all developer-configured, platform-executed:

• Functional gate — a smoke/correctness check at ~20% traffic. Example: verify the served page color matches expected.

  functionalGate: { pause: "20s", image: "httpd:alpine", extraArgs: "red" }   # "red" = expected

• Performance gate — a load/latency check at ~80% traffic (e.g. Artillery image), pass/fail on a threshold.

  performanceGate: { pause: "10s", image: "httpd:alpine", extraArgs: "160" }  # 160 = max avg ms

• Metrics gate — developer-defined Prometheus queries, the most powerful gate. Each criterion: a function (sum|avg|max|min|count) over a metric, a comparison, a threshold, and whether breaching means success or failure.

  metrics:
    pause: "2s"
    evaluationCriteria:
      - interval: "1s"
        count: 1
        function: "avg"
        successOrFailCondition: "fail"        # breaching this fails the rollout
        metric: "rocket_http_requests_duration_seconds_sum"
        comparisonType: ">"
        threshold: 3                          # avg response time > 3s → rollback

Principle — developers own "healthy." The platform supplies the mechanism (canary + analysis); each team defines the metrics and thresholds that define health for their app. Gates are config in the OAM manifest, not platform code.

Multi-stage promotion — Kargo

Kargo orchestrates dev→prod promotion GitOps-natively. Resources:

• Project — Kargo namespace for the app (<app>-kargo).
• Warehouse — watches ECR for new images (produces "Freight").
• Stages — dev (auto-promote) and prod (manual approval).
• PromotionTask — how to update the app's manifests for a stage.

Flow:

Argo Workflows builds image → Warehouse detects it
   → dev stage AUTO-promotes: commits image into deployment/dev/application.yaml → ArgoCD deploys dev
   → human clicks "Promote" in Kargo UI for prod
   → prod stage commits the SAME image into deployment/prod/application.yaml → ArgoCD deploys prod

Key properties:

• Same-artifact promotion — prod runs the exact image that passed dev; no rebuild, no drift.
• Auto dev / manual prod — speed in lower environments, an approval gate for production.
• GitOps-native — every promotion is a Git commit: auditable, reversible.

Verify: kubectl get warehouse,stages -n <app>-kargo -o wide; confirm the prod manifest's image: updated after promotion.

Watching a rollout

kubectl argo rollouts get rollout <name> -n <ns> -w     # live progression + gate status
kubectl argo rollouts retry rollout <name> -n <ns>      # retry after a fix