Platform Engineering

Platform Engineering

Purpose

Build Internal Developer Platforms (IDPs) that provide self-service infrastructure, reduce cognitive load, and accelerate developer productivity through golden paths and platform-as-product thinking.

Platform engineering represents the evolution beyond traditional DevOps, focusing on creating product-quality internal platforms that treat developers as customers. The discipline addresses the developer productivity crisis where engineers spend 30-40% of time on infrastructure and tooling instead of features.

When to Use This Skill

Trigger this skill when:

• Building or improving an internal developer platform
• Designing a developer portal (Backstage, Port, or commercial IDP)
• Implementing golden paths and software templates
• Establishing or restructuring a platform engineering team
• Measuring and improving developer experience (DevEx)
• Integrating IDP with infrastructure, CI/CD, observability, or security tools
• Driving platform adoption across an engineering organization
• Assessing platform maturity and identifying capability gaps

Core Concepts

Platform as Product

Treat internal platforms with the same rigor as customer-facing products:

Product Management Approach:

• Define platform vision, strategy, and roadmap
• Identify developer "customers" and their pain points
• Measure success via adoption metrics, satisfaction surveys, and business impact
• Iterate based on feedback loops and usage analytics
• Balance new capabilities with platform reliability and support

Key Differences from Traditional DevOps:

• DevOps focuses on delivery pipelines; platform engineering builds comprehensive developer experiences
• Platform teams operate as product teams (product managers, UX designers, engineers)
• Success measured by developer productivity and satisfaction, not just infrastructure metrics
• Self-service is the primary interface, not ticket queues

Internal Developer Platform (IDP) Architecture

Three-Layer Architecture:

1. Developer Portal (Frontend)

• Service catalog: Inventory of services with ownership, dependencies, health status
• Software templates: Project scaffolding with best practices baked in
• Documentation hub: Centralized, searchable, version-controlled docs
• Self-service workflows: Environment provisioning, deployments, access requests

2. Platform Orchestration (Backend)

• Infrastructure provisioning: Multi-cloud resource management
• Environment management: Dev, staging, production lifecycle
• Deployment automation: GitOps-based continuous delivery
• Configuration management: Separation of app and infrastructure concerns

3. Integration Layer (Glue)

• CI/CD integration: Pipeline visibility and triggering
• Observability: Metrics, logs, traces surfaced in portal
• Security: Vulnerability scanning, policy enforcement, secrets management
• FinOps: Cost visibility, budgets, optimization recommendations

For detailed architecture patterns and component breakdowns, see references/idp-architecture.md.

Golden Paths and Scaffolding

Golden Path Principle: Provide opinionated templates that handle 80% of use cases while allowing escape hatches for the remaining 20%.

Template Components:

• Repository structure and boilerplate code
• Infrastructure as code (Kubernetes manifests, Terraform)
• CI/CD pipeline configurations
• Observability instrumentation (metrics, logging, tracing)
• Security configurations (RBAC, network policies, secrets)
• Documentation templates (README, runbooks, architecture diagrams)

Constraint Mechanisms:

• Policy-as-code enforcement (OPA, Kyverno) for security and compliance
• Resource limits and quotas to prevent over-provisioning
• Required health checks and observability instrumentation
• Approved base images and dependency scanning

For template design patterns and examples, see references/golden-paths.md.

Developer Experience (DevEx) Optimization

Cognitive Load Reduction:

• Abstract infrastructure complexity without hiding necessary details
• Provide sensible defaults with clear override mechanisms
• Use progressive disclosure (simple for common cases, advanced options available)
• Consolidate tooling (single developer portal vs. 15+ separate tools)

Key Metrics:

DORA Metrics:

• Deployment frequency (how often code reaches production)
• Lead time for changes (commit to production duration)
• Mean time to recovery (MTTR for incidents)
• Change failure rate (percentage of deployments causing incidents)

SPACE Framework:

• Satisfaction: Developer happiness via surveys and NPS
• Performance: Throughput and efficiency of work completed
• Activity: Code commits, PRs, deployments (context, not raw counts)
• Communication: Collaboration quality, discoverability
• Efficiency: Minimize interruptions, reduce toil

Platform-Specific Metrics:

• Platform adoption rate (percentage of teams using platform)
• Self-service rate (actions completed without platform team tickets)
• Onboarding time (new developer to first production deployment)
• Template usage (which golden paths are adopted)
• Support ticket volume and resolution time

Platform Maturity Assessment

Assess current platform capabilities using a 5-level maturity model:

Level 0: Ad-Hoc - Manual provisioning, no standardization Level 1: Basic Automation - Some IaC and CI/CD, limited self-service Level 2: Paved Paths - Golden path templates, early portal, limited coverage Level 3: Self-Service Platform - Comprehensive portal, 80%+ self-service Level 4: Product-Driven Platform - Data-driven, product team structure, FinOps integration Level 5: AI-Augmented Platform - AI-assisted troubleshooting, predictive optimization

For detailed assessment framework, gap analysis, and improvement roadmap, see references/maturity-model.md.

Decision Frameworks

Build vs. Buy IDP

Choose Open Source (Backstage) when:

• Large enterprise (1000+ engineers)
• Dedicated platform team available (5-10 engineers)
• Deep customization required
• Open-source ecosystem preferred
• Long-term investment (3+ year horizon)

Choose Commercial IDP (Port, Humanitec, Cortex) when:

• Mid-size organization (100-1000 engineers)
• Faster time-to-value needed (3-6 months vs. 6-12 months)
• Prefer managed solution with vendor support
• Limited platform engineering resources (<5 engineers)
• Standard use cases (web apps, microservices, CI/CD)

Choose Hybrid Approach when:

• Large organization needing both flexibility and speed
• Complex infrastructure requiring orchestration backend
• Want best-in-class portal + orchestration components
• Willing to integrate multiple systems (e.g., Backstage + Humanitec)

For complete decision tree, selection criteria, and ROI calculations, see references/decision-frameworks.md.

Golden Path Design: Flexibility vs. Standardization

Spectrum of Control:

High Standardization (Regulated Industries):

• Limited technology choices, mandatory templates
• Policy enforcement via admission controllers (OPA, Kyverno)
• Escape hatches require approval process

Balanced Approach (Recommended for Most):

• Recommended golden paths (easy, well-documented, supported)
• Alternatives allowed with documentation
• Soft enforcement (defaults + education, not hard blocks)
• Clear ownership for deviations ("deviate and own")

High Flexibility (Innovative Organizations):

• Golden paths as suggestions (not requirements)
• Minimal policy enforcement (only critical security)
• "Build it, run it" ownership model

For detailed guidance on choosing the right balance and enforcement strategies, see references/decision-frameworks.md.

Platform Team Structure

Centralized Model:

• Single platform team (5-20 engineers) serving entire organization
• Best for: Small to mid-size orgs (100-500 engineers)

Federated Model:

• Central team (5-10 engineers) + embedded engineers (1-2 per business unit)
• Best for: Large orgs (500-2000+ engineers), multiple business units

Hub-and-Spoke Model:

• Central "hub" team (3-5 engineers) + "spoke" teams contributing plugins
• Best for: Organizations with strong open-source culture

For team sizing, roles, responsibilities, and governance models, see references/decision-frameworks.md.

Tool Recommendations

Developer Portals

Backstage (Open Source, CNCF)

• Trust Score: 78.7/100, 8,876 code snippets
• Software catalog, scaffolder, TechDocs, plugin ecosystem
• Recommended for: Enterprises with platform teams

Port (Commercial)

• Managed platform, modern UI/UX, faster time-to-value
• Recommended for: Mid-size orgs (100-1000 engineers)

Cortex (Commercial SaaS)

• Enterprise IDP, compliance focus, engineering standards enforcement
• Recommended for: Regulated industries

Platform Orchestration

Crossplane (Open Source, CNCF)

• Trust Score: 67.4/100, universal control plane for multi-cloud
• Kubernetes-native declarative infrastructure
• Recommended for: Multi-cloud abstractions

Humanitec (Commercial)

• Platform Orchestrator backend, environment and deployment management
• Recommended for: Complex infrastructure, complements portals

Terraform Cloud (Commercial)

• Mature IaC orchestration, workspace management
• Recommended for: Terraform-heavy organizations

GitOps Continuous Delivery

Argo CD (Open Source, CNCF) - RECOMMENDED

• Trust Score: 91.8/100 (HIGHEST)
• Declarative GitOps for Kubernetes, multi-cluster management
• Industry-leading documentation and community

Flux (Open Source, CNCF)

• Toolkit approach, Kubernetes-native
• Good for: GitOps-native operations

For detailed tool comparisons, integration patterns, and selection criteria, see references/tool-recommendations.md.

Implementation Guides

Bootstrapping a Platform

Foundation Phase (Months 1-3): 1. Define platform vision and form platform team (3-5 members) 2. Interview developers to identify pain points 3. Set up developer portal (Backstage or commercial) 4. Create initial service catalog and first golden path template

Pilot Phase (Months 4-6): 1. Select 2-3 pilot teams for white-glove onboarding 2. Rapid iteration based on feedback 3. Expand to 3-5 golden path templates 4. Integrate key tools (CI/CD, monitoring, secrets)

Expansion Phase (Months 7-12): 1. Scale to 20-50% of engineering teams 2. Build self-service documentation and training 3. Establish platform SLOs and on-call rotation 4. Internal evangelization (demos, champions program)

Maturity Phase (Year 2+): 1. 80%+ adoption across organization 2. Platform team operates as product team 3. Continuous improvement via metrics and feedback 4. AI-assisted capabilities, policy-as-code expansion

For detailed implementation steps and bootstrapping code, see references/implementation-backstage.md.

Creating Golden Path Templates

Template Design Process: 1. Identify most common use case (web app, API, data pipeline) 2. Define opinionated choices (language, framework, deployment pattern) 3. Create repository structure and infrastructure manifests 4. Configure CI/CD pipeline with security scanning 5. Instrument observability and document usage 6. Test with pilot team before broad rollout

Template Categories:

• Full-stack web application (backend API + frontend + database)
• Data pipeline (ETL/ELT with orchestration)
• Machine learning service (model serving, monitoring)
• Event-driven microservice (message broker integration)
• Scheduled job (cron jobs, batch processing)

For template examples, scaffolding code, and customization patterns, see references/golden-paths.md and examples/ directory.

Driving Platform Adoption

Evangelization Strategies:

• Showcase pilot team successes (internal blog posts, demos)
• Lunch-and-learns on platform capabilities
• Internal champions program (power users helping peers)
• Office hours and Slack/Teams support channels

Incentive Alignment:

• Make platform easier than alternatives (golden paths are "paved roads")
• Integrate with workflows developers already use
• Provide immediate value (faster onboarding, better visibility)
• Celebrate early adopters, showcase their successes

For adoption metrics, tracking dashboards, and success patterns, see references/maturity-model.md.

Quick Reference

Platform Engineering Checklist

Strategy and Vision:

• [ ] Platform vision and charter documented
• [ ] Platform team formed with clear roles
• [ ] Developer pain points identified via interviews
• [ ] Success metrics defined (DORA, SPACE, adoption)

IDP Foundation:

• [ ] Developer portal deployed (Backstage, Port, or commercial)
• [ ] Service catalog established (ownership, dependencies, health)
• [ ] First golden path template created and validated
• [ ] Documentation hub accessible to all engineers

Self-Service Capabilities:

• [ ] Environment provisioning (dev, staging, production)
• [ ] Deployment automation (GitOps with Argo CD or Flux)
• [ ] CI/CD integration visible in portal
• [ ] Observability dashboards per-service

Security and Compliance:

• [ ] Policy-as-code enforcement (OPA, Kyverno)
• [ ] Secrets management integrated (Vault, cloud providers)
• [ ] Vulnerability scanning in pipelines
• [ ] RBAC and access controls configured

Operations and Support:

• [ ] Platform SLOs defined and monitored
• [ ] Support channels established (Slack, office hours)
• [ ] Incident response playbooks documented
• [ ] Feedback loops and usage analytics in place

Common Pitfalls

Building Too Much Upfront:

• Start small (1 golden path, pilot team) and iterate
• Avoid "boil the ocean" syndrome

Ignoring Developer Feedback:

• Establish continuous feedback loops, not just quarterly surveys

Over-Standardization:

• Provide clear escape hatches for advanced use cases

Under-Measuring Success:

• Track DORA metrics, satisfaction surveys, self-service rates

Treating Platform as IT Project:

• Platform engineering is product development, not infrastructure provisioning
• Requires product managers, UX designers, customer focus

Integration with Other Skills

Related Skills:

• kubernetes-operations: Cluster operations, namespace management, RBAC, network policies
• infrastructure-as-code: Terraform, Pulumi for infrastructure provisioning integrated with platform
• gitops-workflows: GitOps principles, Argo CD / Flux implementation patterns
• building-ci-pipelines: CI/CD pipeline design integrated into platform templates
• security-hardening: Security best practices enforced through golden paths
• secret-management: Secrets management integrated into platform (Vault, cloud providers)
• observability: Monitoring, logging, tracing integrated into developer portal

Cross-Skill Workflows:

Platform Bootstrapping: 1. Use infrastructure-as-code to provision platform infrastructure 2. Use kubernetes-operations to configure clusters 3. Deploy developer portal (Backstage) on platform infrastructure 4. Integrate gitops-workflows (Argo CD) for continuous delivery 5. Add observability integrations (Prometheus, Grafana plugins)

Golden Path Creation: 1. Design template based on common use case 2. Use building-ci-pipelines patterns for CI/CD configuration 3. Apply security-hardening best practices (SAST, container scanning) 4. Integrate secret-management (Vault, encrypted configs) 5. Add observability instrumentation (metrics, logging, tracing)

Example Use Cases

Use Case 1: E-Commerce Platform Team

Context: 300-engineer e-commerce company, microservices architecture, manual provisioning causing bottlenecks.

Approach: Deploy Backstage, create 3 golden paths, integrate Argo CD, pilot with 3 teams, expand to 20 teams over 6 months.

Results: Onboarding time 2 days → 2 hours, deployment frequency 2x/week → 10x/day, developer NPS +35.

Use Case 2: Financial Services Platform

Context: 1500-engineer bank, strict compliance, legacy infrastructure, fragmented tooling.

Approach: Adopt Port (commercial), high standardization golden paths, OPA Gatekeeper, federated model, Terraform Cloud.

Results: Compliance audit prep 3 weeks → 3 days, infrastructure drift incidents 90% reduction, per-service cost attribution.

Use Case 3: Startup Platform

Context: 50-engineer startup, rapid growth, need fast developer onboarding.

Approach: Lightweight Backstage (2 engineers), 2 golden paths, GitHub Actions, PaaS infrastructure (Fly.io), documentation focus.

Results: New engineer to production 1 day (vs. 2 weeks), 100% self-service, 2 engineers supporting 50 developers.

For code examples and template structures, see examples/ directory.

Platform Engineering Decision Frameworks

Table of Contents

1. Build vs. Buy IDP Decision Matrix 2. Golden Path Design: Flexibility vs. Standardization 3. Platform Adoption Strategy 4. Platform Team Structure

Build vs. Buy IDP Decision Matrix

When to Build Custom Platform

Indicators:

• ✅ Unique requirements not met by commercial/open-source solutions
• ✅ Large engineering organization (1000+ engineers) with dedicated platform team (10+ engineers)
• ✅ Strong internal expertise in platform technologies (Kubernetes, IaC, CI/CD)
• ✅ Need for deep customization and control
• ✅ Open-source ecosystem preference (avoid vendor lock-in)
• ✅ Long-term investment (3+ year horizon)
• ✅ Existing infrastructure complex (multi-cloud, hybrid, legacy systems)

Risks:

• ⚠️ High upfront development cost (6-12 months to MVP)
• ⚠️ Ongoing maintenance burden (dedicated team required)
• ⚠️ Risk of building features that already exist in commercial products
• ⚠️ Slower time-to-value compared to buying

Recommended Approach:

• Start with Backstage (open source) as foundation
• Build custom plugins for unique integrations
• Contribute plugins back to community
• Examples: Spotify, American Airlines, Netflix

ROI Calculation:

• Platform team cost: 10 engineers × $200K = $2M/year
• Upfront development: 6-12 months to MVP
• Ongoing: Platform features, bug fixes, upgrades
• Break-even: When (developer productivity gains × engineer count) > platform team cost
• Typical: Organizations with 500+ engineers see positive ROI in 12-18 months

When to Buy Commercial IDP

Indicators:

• ✅ Mid-size organization (100-1000 engineers) with limited platform resources (<5 platform engineers)
• ✅ Need fast time-to-value (3-6 months to production)
• ✅ Prefer managed infrastructure and support
• ✅ Standard use cases (web apps, microservices, CI/CD)
• ✅ Budget available for tooling (vs. headcount)
• ✅ Want to focus engineering on business value, not platform

Risks:

• ⚠️ Licensing costs (can be significant at scale)
• ⚠️ Potential vendor lock-in (migration difficult)
• ⚠️ Limited customization compared to open source
• ⚠️ Reliance on vendor roadmap for new features

Recommended Approach:

• Evaluate Port, Humanitec, Cortex, or Harness
• Start with pilot team (20-50 developers)
• Measure ROI (developer time saved, deployment frequency increase)
• Scale if metrics show clear value

ROI Calculation:

• Commercial IDP cost: $50-200 per developer/month
• For 500 engineers: $30K-120K/month = $360K-1.44M/year
• Small platform team: 2-3 engineers × $200K = $400K-600K/year
• Total cost: $760K-2M/year (vs. $2M+ for build)
• Faster time-to-value: 3-6 months vs. 6-12 months
• Lower risk: Proven product, vendor support

When to Adopt Hybrid Approach

Indicators:

• ✅ Large organization needing both flexibility and speed
• ✅ Complex infrastructure requiring orchestration backend
• ✅ Want open-source portal with commercial orchestration (or vice versa)
• ✅ Willing to integrate multiple systems

Recommended Approach:

• Portal: Backstage (open source, customizable)
• Orchestration: Humanitec or Crossplane (infrastructure abstraction)
• GitOps: Argo CD (open source, proven)
• Observability: Commercial (Datadog, New Relic) or open source (Prometheus + Grafana)
• Example: Many enterprises in 2025 using Backstage + Humanitec combination

Integration Considerations:

• Ensure portals can integrate with orchestration backends (APIs, webhooks)
• Backstage + Humanitec: Humanitec plugin for Backstage available
• Backstage + Crossplane: Kubernetes plugin shows Crossplane resources
• Unified authentication (SSO across all platform components)

Decision Tree

Start: Need Internal Developer Platform?
│
├─ Organization Size?
│  ├─ <100 engineers → Use cloud provider tools (AWS Proton, GCP, Azure) or simple CI/CD (GitHub Actions + documentation)
│  ├─ 100-500 engineers → Buy commercial IDP (Port, Humanitec) OR adopt Backstage with minimal customization
│  ├─ 500-1000 engineers → Buy commercial IDP OR invest in Backstage with dedicated team (2-3 engineers)
│  └─ 1000+ engineers → Build on Backstage with large platform team (5-10 engineers) OR hybrid (Backstage + commercial orchestration)
│
├─ Platform Engineering Expertise?
│  ├─ Limited (<2 experienced engineers) → Buy commercial IDP
│  ├─ Moderate (2-5 engineers) → Backstage with community plugins
│  └─ Strong (5+ engineers) → Custom Backstage build
│
├─ Time to Value?
│  ├─ Urgent (<3 months) → Buy commercial IDP
│  ├─ Moderate (3-6 months) → Backstage with minimal customization
│  └─ Long-term (6-12 months) → Custom platform build
│
├─ Customization Needs?
│  ├─ Standard use cases → Commercial IDP
│  ├─ Some unique requirements → Backstage with custom plugins
│  └─ Highly unique → Full custom build on Backstage
│
└─ Budget?
   ├─ Limited tooling budget → Backstage (open source)
   ├─ Moderate budget → Commercial IDP or hybrid
   └─ Flexible budget → Best-in-class tools across stack (hybrid)

Selection Criteria Comparison

Golden Path Design: Flexibility vs. Standardization

Principle: The 80/20 Rule

Golden paths should handle 80% of use cases; remaining 20% get escape hatches.

Why This Matters:

• Too rigid: Developers route around platform, create shadow IT
• Too flexible: No productivity gains, duplicate effort across teams
• Balanced: High adoption (easy path), innovation enabled (escape hatches)

Spectrum of Control

High Standardization (Low Flexibility)

When to Use:

• Regulated industries (finance, healthcare, government)
• Security/compliance critical (SOC2, HIPAA, PCI-DSS)
• High risk of misconfigurations leading to outages or vulnerabilities
• Junior engineering teams needing guardrails

Characteristics:

• Limited technology choices (approved list only)
• Mandatory templates (cannot opt-out)
• Policy enforcement via admission controllers (OPA, Kyverno)
• Escape hatches require approval process (platform team review, architecture board)

Examples:

• Must use approved base images (security scanning passed)
• Cannot disable network policies (micro-segmentation enforced)
• Required resource limits enforced (prevent noisy neighbor)
• Mandatory observability instrumentation (metrics, logs, traces)

Implementation:

• OPA Gatekeeper policies: Reject deployments without required labels
• Kyverno policies: Mutate deployments to add security context
• Approval workflows: Production access requires manager + platform team approval
• Template versioning: All services must use template v2.x or higher

Risks:

• Stifles innovation (cannot experiment with new technologies)
• Frustration from advanced engineers ("getting in the way")
• Slower adoption of new best practices (approval process bottleneck)
• Can drive engineers to workarounds or shadow IT

Mitigation:

• Clear documentation on why constraints exist
• Fast-track approval process for trusted teams
• Regular review of constraints (remove outdated ones)
• Exemption process for justified cases

Balanced Approach (Recommended for Most)

When to Use:

• Most organizations (standard risk tolerance)
• Mix of junior and senior engineers
• Want to encourage best practices without blocking innovation

Characteristics:

• Recommended golden paths (easy, well-documented, supported)
• Alternatives allowed with documentation (justify deviations)
• Soft enforcement (defaults + education, not hard blocks)
• Easy escalation for exceptions (self-service + platform team review if needed)

Examples:

• Recommended: Use approved frameworks (Node.js, Python, Go), but custom frameworks allowed with runbook
• Default: Standard observability (Prometheus + Grafana), but custom metrics/logging allowed
• Encouraged: Use templates for new projects, but manual setup possible
• Documented: "If you deviate, you own support for that component"

Implementation:

• Templates are default in portal, but "advanced setup" option available
• Policy engines in audit mode (warn, not block)
• Documentation clearly marks "recommended" vs. "experimental"
• Platform team office hours for discussing deviations

Benefits:

• High adoption (golden paths are easiest option)
• Innovation enabled (advanced users can experiment)
• Clear expectations (deviations documented and owned)
• Balanced autonomy and standardization

Success Metrics:

• 80%+ of services use golden paths
• Advanced users happy (can deviate when needed)
• Platform team not overwhelmed with support for custom setups

High Flexibility (Low Standardization)

When to Use:

• Highly innovative organizations (startups, research labs)
• Senior engineering teams (know best practices)
• Experimentation culture (fail fast, learn quickly)

Characteristics:

• Golden paths as suggestions (not requirements)
• Minimal policy enforcement (only critical security/compliance)
• "You build it, you run it" ownership model
• Platform provides tools, not mandates

Examples:

• Choose any language, framework, database (platform-agnostic)
• Custom infrastructure configurations allowed
• Observability and security guidance provided, not enforced
• Engineers own full stack for their services

Implementation:

• Portal provides templates as starting points
• No policy engines (except security minimums)
• Documentation is guidance, not rules
• Platform team available for consultation, not enforcement

Risks:

• Inconsistency across teams (harder to support)
• Security/compliance gaps (if not careful)
• Duplicate effort (teams solving same problems differently)
• High cognitive load (engineers must know everything)

Mitigation:

• Regular architecture reviews (share patterns)
• Security scanning enforced (even for custom setups)
• Platform team publishes "patterns we've seen work"
• Shared libraries for common functions

Decision Matrix

Recommendations by Use Case

1. Financial Services Platform:

• Approach: High Standardization
• Rationale: Regulatory compliance (SOC2, PCI-DSS), audit requirements
• Implementation:
• Mandatory templates with strict policies
• Escape hatches require architecture review board approval
• Policy-as-code enforced (OPA Gatekeeper)
• All deployments logged and auditable
• Example Policy: "All services must use approved base images from internal registry, pass security scanning, and enforce encryption at rest and in transit"

2. E-Commerce Platform:

• Approach: Balanced
• Rationale: Mix of standard web apps and innovative features, junior to senior engineers
• Implementation:
• Golden paths for web apps, APIs, data pipelines
• Custom solutions allowed with documentation and ownership
• Platform team provides guidance, not blockers
• Soft enforcement (warnings, not blocks)
• Example Policy: "Recommended to use Node.js microservice template, but custom setups allowed if team documents architecture and owns support"

3. Early-Stage Startup:

• Approach: High Flexibility
• Rationale: Speed and innovation critical, senior engineers, limited platform resources
• Implementation:
• Minimal platform (CI/CD + monitoring basics)
• Engineers own full stack
• Golden paths emerge over time from repeated patterns
• Platform team publishes guidance, not rules
• Example Policy: "No required templates. Platform team provides example repos and is available for consultation."

Platform Adoption Strategy

Phase 1: Foundation (Months 1-3)

Goals:

• Establish platform vision and charter
• Form platform team and assign roles
• Build executive sponsorship
• Choose IDP technology stack

Activities:

Discovery:

• Interview developers (15-20 individual interviews)
• Pain points: What slows you down?
• Workflows: How do you currently deploy?
• Tool preferences: What tools do you wish you had?
• Audit existing infrastructure and tooling
• Inventory: Cloud resources, CI/CD pipelines, monitoring tools
• Gaps: What's missing? What's duplicated?
• Identify quick wins (high-impact, low-effort improvements)
• Example: Centralized logging if currently scattered

Strategy:

• Define platform vision and principles
• Vision: "Enable developers to focus on business logic, not infrastructure"
• Principles: Self-service, secure by default, gradual adoption
• Establish success metrics
• Baseline DORA metrics (current deployment frequency, lead time)
• Developer satisfaction (baseline survey)
• Platform KPIs (adoption rate, self-service rate)
• Secure budget and headcount for platform team
• Business case: ROI from developer productivity gains
• Headcount: 3-5 initial platform engineers

Foundation:

• Set up developer portal (Backstage, Port, or commercial)
• Deploy to cloud infrastructure
• Configure authentication (SSO)
• Basic homepage and documentation
• Create initial service catalog
• Manually document top 20 services
• Ownership, repository links, runbook links
• Build first golden path template
• Identify most common use case (e.g., Node.js API)
• Scaffold: Repository structure, CI/CD, deployment config
• Test with pilot service

Success Criteria:

• Platform vision documented and communicated to engineering org
• Platform team formed (3-5 initial members with clear roles)
• Developer portal accessible to all engineers
• 1 golden path template available and tested

Timeline: 3 months (Month 1: Discovery, Month 2: Strategy + Team, Month 3: Foundation build)

Phase 2: Pilot (Months 4-6)

Goals:

• Validate platform with pilot teams
• Prove ROI (faster onboarding, reduced incidents, higher satisfaction)
• Iterate based on feedback

Activities:

Pilot Selection:

• Choose 2-3 teams (15-30 total developers)
• Criteria: Friendly (willing to give feedback), representative of broader org, mix of greenfield and brownfield
• Greenfield: New projects (test full workflow from scratch)
• Brownfield: Existing services (test migration path)
• Committed to giving feedback and iterating

Enablement:

• White-glove onboarding for pilot teams
• Kickoff meeting: Platform overview, goals, feedback process
• Dedicated Slack channel for pilot teams
• Platform engineer embedded with each pilot team (first 2 weeks)
• Weekly office hours (platform team available for questions)
• Document feedback and pain points
• Feedback log: Issues, feature requests, confusion points
• Prioritize by impact and frequency

Iteration:

• Rapid fixes for blockers (prioritize pilot team needs)
• Goal: Fix critical issues within 1-2 days
• Example: If CI/CD integration broken, fix immediately
• Expand golden paths (2-3 additional templates)
• Based on pilot team needs (e.g., Python service, React frontend)
• Integrate key tools (CI/CD, monitoring, secrets)
• Backstage plugins: GitHub Actions, Prometheus, Vault
• Single pane of glass for developers

Success Criteria:

• 3 pilot teams using platform successfully
• 50% reduction in onboarding time for new services (pilot teams)
• Measure: Time from code scaffold to production deployment
• Positive feedback from pilot teams (satisfaction survey: >7/10)
• 3-5 golden path templates available

Timeline: 3 months (Month 4: Pilot selection + onboarding, Months 5-6: Iteration + expansion)

Phase 3: Expansion (Months 7-12)

Goals:

• Scale platform to broader organization (20-50% of teams)
• Build internal champions and evangelists
• Establish platform operations and support model

Activities:

Evangelization:

• Internal blog posts and demos (showcase pilot team successes)
• Case studies: "Team X reduced deployment time from 2 hours to 10 minutes"
• Metrics: DORA improvements, developer satisfaction
• Lunch-and-learns (platform capabilities, new features)
• Monthly sessions, different topics (golden paths, self-service, troubleshooting)
• Internal champions program (power users who help others)
• Recruit 1-2 champions per team
• Champions get early access to features, direct line to platform team
• Recognize champions (internal awards, blog posts)

Self-Service:

• Comprehensive documentation (getting started, recipes, troubleshooting)
• Getting started: 15-minute quick start guide
• Recipes: Common tasks (deploy service, add database, configure monitoring)
• Troubleshooting: FAQ, common errors
• Video tutorials and walkthroughs
• Screen recordings: Creating service from template, deploying to production
• Slack/Teams channel for community support
• Platform team monitors, but community can answer questions
• Knowledge base built from Q&A

Operations:

• Platform SLOs and monitoring (uptime, latency, error rate)
• SLO: 99.5% uptime for developer portal
• SLO: <5s latency for API calls
• SLO: <1% error rate for deployments
• On-call rotation for platform team
• 24/7 coverage for production incidents
• Escalation path: On-call → platform lead → director
• Incident response playbooks
• Portal down: Steps to diagnose and recover
• Deployment failures: Troubleshooting guide

Expansion:

• Onboard 10-20 additional teams
• Self-service onboarding (documentation + videos)
• Platform team office hours for questions
• Expand service catalog (100+ services documented)
• Automated discovery where possible (scan repositories)
• Add advanced features (ephemeral environments, FinOps integration)
• Ephemeral environments: Per-PR preview deployments
• FinOps: Cost visibility per service/team

Success Criteria:

• 20-50% of engineering teams using platform
• Self-service adoption (80% of onboarding without platform team involvement)
• Platform SLOs met (99.5% uptime)
• Measurable DORA metric improvements (5x deployment frequency, 50% reduction in lead time)

Timeline: 6 months (Months 7-9: Evangelization + self-service, Months 10-12: Expansion + operations)

Phase 4: Maturity (Year 2+)

Goals:

• Platform as standard way of working (80%+ adoption)
• Continuous improvement based on metrics and feedback
• Platform team as product team (roadmap, OKRs, customer focus)

Activities:

Product Management:

• Hire product manager for platform team
• Quarterly roadmap planning (prioritized by impact)
• Feature requests from teams
• Platform team initiatives
• Prioritization: Impact × Effort matrix
• Developer NPS surveys (measure satisfaction quarterly)
• Track trends, identify problem areas
• Usage analytics (which features used, which neglected)
• Instrument portal with analytics (page views, feature usage)
• Identify underutilized features (consider deprecation or improvement)

Advanced Capabilities:

• AI-assisted features (chatbots for common questions, recommendation engines)
• FinOps optimization (automated right-sizing, cost anomaly detection)
• Policy-as-code expansion (comprehensive guardrails for security, compliance)

Ecosystem:

• Contribute back to open-source community (Backstage plugins)
• Partner integrations (third-party tools)
• Internal marketplace (teams can publish reusable components)

Success Criteria:

• 80%+ of engineering teams using platform
• Top-quartile DORA metrics (compared to industry benchmarks)
• High developer NPS (>50)
• Platform team operates as product team (clear roadmap, customer focus)

Timeline: Ongoing (Year 2+)

Adoption Metrics to Track

Platform Team Structure

Model 1: Centralized Platform Team

Structure:

• Single platform team (5-20 engineers) serving entire organization
• Team owns: Developer portal, infrastructure orchestration, CI/CD, observability integration
• Team provides: Golden paths, support, strategic roadmap

When to Use:

• Small to mid-size organizations (100-500 engineers)
• Consistent technology stack across organization
• Limited platform engineering expertise (need to concentrate talent)

Roles and Responsibilities:

Platform Lead (1):

• Overall platform strategy and vision
• Roadmap prioritization
• Stakeholder management (engineering leadership, security, finance)

Platform Product Manager (0-1):

• Developer customer research
• Feature prioritization based on impact
• Success metrics and analytics

Platform Engineers (3-10):

• Build and maintain developer portal (Backstage plugins, integrations)
• Create and update golden path templates
• Integrate infrastructure orchestration (Crossplane, Terraform)
• Developer support (office hours, troubleshooting)

SRE / On-Call (1-2):

• Platform operations and monitoring
• Incident response
• SLO tracking and performance optimization

DevEx Specialist (0-1):

• Developer experience research
• Documentation and training
• Usability testing of platform features

Pros:

• ✅ Concentrated expertise (deep specialization)
• ✅ Consistent platform experience across organization
• ✅ Efficient resource utilization (no duplication)
• ✅ Clear ownership and accountability

Cons:

• ❌ Risk of bottleneck (all requests go through one team)
• ❌ May not understand all domain-specific needs
• ❌ Can become ivory tower (disconnected from app teams)
• ❌ Limited scalability (one team supporting 100s of developers)

Best Practices:

• Embed platform engineers with app teams temporarily (3-6 months rotations)
• Learn domain-specific needs
• Build relationships, trust
• Self-service focus (minimize ticket-based work)
• Goal: 90%+ of actions self-service
• Regular office hours and demos (stay connected to developers)
• Weekly office hours (drop-in for questions)
• Monthly demos (new features, best practices)

Scaling Limits:

• Works well up to 500 engineers
• Beyond 500, consider federated or hub-and-spoke model

Model 2: Federated Platform Model

Structure:

• Central platform team (5-10 engineers) sets standards and provides core capabilities
• Embedded platform engineers (1-2 per business unit/product area) customize for domains
• Shared responsibility: Central team (core platform), embedded engineers (domain-specific integrations)

When to Use:

• Large organizations (500-2000+ engineers)
• Multiple business units or product lines with different needs
• Some platform expertise distributed across organization

Roles and Responsibilities:

Central Platform Team (5-10 engineers):

• Platform strategy and vision
• Core developer portal (Backstage core, authentication, service catalog)
• Shared golden path templates (common use cases)
• Platform standards and governance
• Tool evaluations and vendor management

Embedded Platform Engineers (1-2 per business unit):

• Domain-specific golden paths (e.g., ML platform templates)
• Custom integrations (business unit-specific tools)
• Local developer support and evangelization
• Feedback conduit to central team

Reporting Structure:

• Embedded engineers report to: Business unit lead (dotted line to central platform lead)
• Or: Central platform lead (dotted line to business unit lead)
• Key: Clear alignment on priorities and responsibilities

Pros:

• ✅ Scales well (distributed support across domains)
• ✅ Domain expertise (embedded engineers understand specific needs)
• ✅ Balance of consistency (central standards) and flexibility (domain customization)
• ✅ Reduces bottlenecks (embedded engineers handle domain-specific work)

Cons:

• ❌ Coordination overhead (central + embedded teams must align)
• ❌ Risk of fragmentation (domains diverge if not careful)
• ❌ Requires more total headcount (embedded engineers across domains)

Best Practices:

• Clear charter: Central team owns platform core, embedded teams own integrations
• Written charter: What's centralized vs. domain-specific
• Decision log: When decisions made, rationale
• Regular sync meetings (central + all embedded engineers)
• Bi-weekly: Share progress, discuss challenges
• Quarterly: Strategic planning, roadmap alignment
• Shared knowledge base (central team documents patterns, embedded teams contribute)
• Confluence/Notion: Platform architecture, best practices, recipes
• Rotation program (embedded engineers rotate through central team 3-6 months)
• Cross-pollination of ideas
• Career development

Scaling Limits:

• Works well from 500 to 5000+ engineers
• Number of embedded engineers scales with business units (not total headcount)

Model 3: Hub-and-Spoke Model

Structure:

• Central "hub" team (3-5 platform engineers) provides foundational platform
• "Spoke" teams (app/product teams) contribute plugins, templates, integrations
• Community-driven: Platform team curates, app teams contribute

When to Use:

• Organizations with strong open-source culture
• High engineering maturity (senior engineers across teams)
• Want to distribute platform ownership (avoid bottleneck)

Roles and Responsibilities:

Hub Team (3-5 engineers):

• Core developer portal infrastructure
• Platform governance (quality standards for contributions)
• Curate community contributions (review, merge, deprecate)
• Strategic roadmap (platform direction)
• Tooling and infrastructure for contribution (e.g., plugin development kit)

Spoke Teams (distributed):

• Contribute plugins for tools they use (e.g., Datadog plugin)
• Build domain-specific templates (e.g., ML service template)
• Share reusable components (libraries, scripts)
• Dog-food platform, provide feedback

Contribution Process:

• RFC (Request for Comments) for major features
• Pull requests to platform repository
• Code review by hub team
• Documentation required for all contributions
• Maintenance responsibility: Contributors own their plugins

Pros:

• ✅ Distributes ownership (not reliant on one team)
• ✅ Scales without headcount (community contributions)
• ✅ Innovation from diverse perspectives
• ✅ High engagement (teams invested in platform success)

Cons:

• ❌ Quality control challenges (varied contribution quality)
• ❌ Maintenance burden (who maintains old plugins?)
• ❌ Requires strong governance (avoid chaos)
• ❌ Not suitable for low-maturity orgs (needs senior engineers)

Best Practices:

• Clear contribution guidelines
• Code quality standards
• Documentation requirements
• Testing requirements
• Deprecation policy (sunset unused/unmaintained plugins)
• Recognition for contributors (internal awards, blog posts, conference talks)
• Hub team capacity reserved for governance (not building all features)

Scaling Limits:

• Works well for mature organizations (500+ engineers, strong open-source culture)
• Requires active community management

Team Sizing Guidelines

Choosing the Right Model

Decision Factors:

1. Organization Size:

• <500 engineers → Centralized
• 500-2000 → Federated
• 2000+ → Federated or Hub-and-Spoke

2. Engineering Culture:

• Traditional (centralized IT) → Centralized
• Matrixed (business units) → Federated
• Open-source culture → Hub-and-Spoke

3. Platform Engineering Expertise:

• Concentrated (few experts) → Centralized
• Distributed (expertise in business units) → Federated
• High maturity (senior engineers across org) → Hub-and-Spoke

4. Customization Needs:

• Standardized (one tech stack) → Centralized
• Varied (different needs per business unit) → Federated
• Experimental (lots of innovation) → Hub-and-Spoke

5. Budget:

• Limited headcount → Centralized or Hub-and-Spoke
• Can support embedded engineers → Federated

Golden Path Templates

Table of Contents

1. Template Design Principles 2. Full-Stack Web Application Template 3. Data Pipeline Template 4. Machine Learning Service Template 5. Event-Driven Microservice Template 6. Scheduled Job Template

Template Design Principles

The 80/20 Rule

Golden paths should cover 80% of use cases while providing clear escape hatches for the remaining 20%.

Key Principles:

• Opinionated but flexible: Make strong recommendations, allow overrides
• Security by default: Bake in security best practices (RBAC, network policies, scanning)
• Observable from day one: Include metrics, logging, tracing instrumentation
• Production-ready: Include all necessary configs (health checks, resource limits, monitoring)
• Self-documenting: Clear README, inline comments, architecture diagrams
• Version-controlled: Templates evolve, provide migration paths

Template Anatomy

Every golden path template should include:

1. Application Code: Framework setup, boilerplate, example endpoints 2. Infrastructure as Code: Kubernetes manifests, Terraform modules 3. CI/CD Pipeline: Build, test, deploy automation 4. Observability: Metrics exporters, structured logging, tracing setup 5. Security: RBAC, network policies, secret references, scanning config 6. Documentation: README, runbooks, architecture diagrams

Constraint Mechanisms

Required (Hard Constraints):

• Health check endpoints (liveness, readiness)
• Resource limits (CPU, memory)
• Security scanning (SAST, container scanning)
• Logging to stdout/stderr (structured JSON)

Recommended (Soft Constraints):

• Approved frameworks (Node.js, Python, Go preferred)
• Standard observability stack (Prometheus, Grafana)
• Naming conventions (kebab-case for services)

Optional (Escape Hatches):

• Custom middleware/libraries
• Alternative databases (must be from approved list)
• Advanced Kubernetes features (with platform team consultation)

Full-Stack Web Application Template

Use Case

Web application with backend API, database, and frontend (SPA or SSR).

Technology Stack

Backend:

• Node.js + Express (or FastAPI for Python, Gin for Go)
• PostgreSQL database

Frontend:

• React (or Vue, Angular)
• Vite build tool

Infrastructure:

• Kubernetes deployment
• Ingress for routing
• Horizontal Pod Autoscaler

Directory Structure

my-web-app/
├── catalog-info.yaml          # Backstage catalog metadata
├── README.md
├── backend/
│   ├── src/
│   │   ├── index.ts           # Entry point
│   │   ├── app.ts             # Express app
│   │   ├── routes/            # API routes
│   │   ├── models/            # Database models
│   │   └── middleware/        # Custom middleware
│   ├── package.json
│   ├── tsconfig.json
│   └── Dockerfile
├── frontend/
│   ├── src/
│   │   ├── App.tsx
│   │   ├── components/
│   │   └── pages/
│   ├── package.json
│   ├── vite.config.ts
│   └── Dockerfile
├── k8s/
│   ├── namespace.yaml
│   ├── backend-deployment.yaml
│   ├── backend-service.yaml
│   ├── frontend-deployment.yaml
│   ├── frontend-service.yaml
│   ├── postgres.yaml          # Dev environment only
│   ├── ingress.yaml
│   └── hpa.yaml
├── .github/
│   └── workflows/
│       ├── backend-ci.yaml
│       └── frontend-ci.yaml
└── docs/
    ├── index.md               # TechDocs
    └── architecture.md

Backend Example (Node.js + Express)

src/index.ts:

import app from './app';
import { logger } from './utils/logger';

const PORT = process.env.PORT || 3000;

app.listen(PORT, () => {
  logger.info(`Server started on port ${PORT}`);
});

src/app.ts:

import express from 'express';
import helmet from 'helmet';
import cors from 'cors';
import promBundle from 'express-prom-bundle';
import { healthRouter } from './routes/health';
import { apiRouter } from './routes/api';
import { errorHandler } from './middleware/error';
import { requestLogger } from './middleware/logging';

const app = express();

// Security
app.use(helmet());
app.use(cors());

// Metrics (Prometheus)
const metricsMiddleware = promBundle({
  includeMethod: true,
  includePath: true,
  includeStatusCode: true,
  includeUp: true,
  customLabels: { service: 'my-web-app' },
  promClient: { collectDefaultMetrics: {} }
});
app.use(metricsMiddleware);

// Logging
app.use(requestLogger);

// Body parsing
app.use(express.json());
app.use(express.urlencoded({ extended: true }));

// Routes
app.use('/health', healthRouter);
app.use('/api', apiRouter);

// Error handling
app.use(errorHandler);

export default app;

Kubernetes Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-web-app-backend
  labels:
    app: my-web-app
    component: backend
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-web-app
      component: backend
  template:
    metadata:
      labels:
        app: my-web-app
        component: backend
    spec:
      containers:
      - name: backend
        image: my-registry/my-web-app-backend:latest
        ports:
        - containerPort: 3000
          name: http
        - containerPort: 9090
          name: metrics
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: database-credentials
              key: url
        livenessProbe:
          httpGet:
            path: /health/liveness
            port: http
          initialDelaySeconds: 10
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /health/readiness
            port: http
          initialDelaySeconds: 5
          periodSeconds: 5
        resources:
          requests:
            memory: "256Mi"
            cpu: "100m"
          limits:
            memory: "512Mi"
            cpu: "500m"

Data Pipeline Template

Use Case

ETL/ELT data processing pipeline with workflow orchestration.

Technology Stack

• Workflow orchestrator: Apache Airflow (or Prefect, Dagster)
• Data connectors: S3, BigQuery, Snowflake, databases
• Data quality: Great Expectations or dbt tests
• Scheduling: Cron or orchestrator built-in

Directory Structure

my-data-pipeline/
├── catalog-info.yaml
├── README.md
├── dags/
│   └── my_pipeline.py         # Airflow DAG
├── tasks/
│   ├── extract.py
│   ├── transform.py
│   └── load.py
├── tests/
│   └── test_pipeline.py
├── k8s/
│   └── cronjob.yaml           # Alternative to Airflow
├── requirements.txt
└── Dockerfile

Airflow DAG Example

dags/my_pipeline.py:

from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
from datetime import datetime, timedelta
import logging

default_args = {
    'owner': 'data-team',
    'depends_on_past': False,
    'email_on_failure': True,
    'email_on_retry': False,
    'retries': 2,
    'retry_delay': timedelta(minutes=5),
}

dag = DAG(
    'my_data_pipeline',
    default_args=default_args,
    description='Extract, transform, load data pipeline',
    schedule_interval='0 2 * * *',  # Daily at 2 AM
    start_date=datetime(2025, 1, 1),
    catchup=False,
    tags=['etl', 'production'],
)

def extract_data(**context):
    """Extract data from source"""
    logging.info("Extracting data from S3")
    s3_hook = S3Hook(aws_conn_id='aws_default')
    data = s3_hook.read_key('source-data/input.csv', bucket_name='my-bucket')
    # Process and return
    context['ti'].xcom_push(key='raw_data', value=data)

def transform_data(**context):
    """Transform data"""
    logging.info("Transforming data")
    raw_data = context['ti'].xcom_pull(key='raw_data')
    # Transformation logic
    transformed_data = raw_data  # Placeholder
    context['ti'].xcom_push(key='transformed_data', value=transformed_data)

def load_data(**context):
    """Load data to destination"""
    logging.info("Loading data to BigQuery")
    transformed_data = context['ti'].xcom_pull(key='transformed_data')
    # Load to BigQuery
    # ...

extract_task = PythonOperator(
    task_id='extract',
    python_callable=extract_data,
    dag=dag,
)

transform_task = PythonOperator(
    task_id='transform',
    python_callable=transform_data,
    dag=dag,
)

load_task = PythonOperator(
    task_id='load',
    python_callable=load_data,
    dag=dag,
)

extract_task >> transform_task >> load_task

Machine Learning Service Template

Use Case

ML model serving for inference (REST API).

Technology Stack

• Model serving: FastAPI or TorchServe
• Model registry: MLflow or Weights & Biases
• Inference: GPU support (optional)
• Monitoring: Drift detection, performance metrics

Directory Structure

my-ml-service/
├── catalog-info.yaml
├── README.md
├── src/
│   ├── main.py                # FastAPI app
│   ├── model_loader.py
│   ├── inference.py
│   └── monitoring.py
├── models/
│   └── model.pkl              # Serialized model
├── k8s/
│   ├── deployment.yaml        # With GPU resources
│   ├── service.yaml
│   └── hpa.yaml
├── requirements.txt
└── Dockerfile

FastAPI Inference Example

src/main.py:

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from prometheus_client import Counter, Histogram, make_asgi_app
import numpy as np
import logging

from .model_loader import load_model
from .inference import predict
from .monitoring import track_prediction

app = FastAPI(title="ML Inference Service")

# Metrics
prediction_counter = Counter('predictions_total', 'Total predictions')
prediction_latency = Histogram('prediction_latency_seconds', 'Prediction latency')

# Load model at startup
model = load_model('models/model.pkl')

class PredictionRequest(BaseModel):
    features: list[float]

class PredictionResponse(BaseModel):
    prediction: float
    confidence: float

@app.get("/health/liveness")
async def liveness():
    return {"status": "alive"}

@app.get("/health/readiness")
async def readiness():
    if model is None:
        raise HTTPException(status_code=503, detail="Model not loaded")
    return {"status": "ready"}

@app.post("/predict", response_model=PredictionResponse)
async def predict_endpoint(request: PredictionRequest):
    with prediction_latency.time():
        try:
            result = predict(model, np.array(request.features))
            prediction_counter.inc()
            track_prediction(request.features, result)
            return PredictionResponse(
                prediction=result['value'],
                confidence=result['confidence']
            )
        except Exception as e:
            logging.error(f"Prediction error: {e}")
            raise HTTPException(status_code=500, detail=str(e))

# Metrics endpoint for Prometheus
metrics_app = make_asgi_app()
app.mount("/metrics", metrics_app)

Event-Driven Microservice Template

Use Case

Service consuming/producing events via message broker (async architecture).

Technology Stack

• Message broker: Kafka (or RabbitMQ, AWS SQS/SNS)
• Schema registry: Avro or Protobuf
• Consumer pattern: At-least-once or exactly-once delivery

Kafka Consumer Example

src/consumer.py:

from confluent_kafka import Consumer, KafkaException
from prometheus_client import Counter, Histogram
import json
import logging

# Metrics
messages_consumed = Counter('messages_consumed_total', 'Total messages consumed', ['topic'])
processing_latency = Histogram('message_processing_seconds', 'Message processing time')

consumer = Consumer({
    'bootstrap.servers': 'kafka:9092',
    'group.id': 'my-consumer-group',
    'auto.offset.reset': 'earliest',
    'enable.auto.commit': False,
})

consumer.subscribe(['my-topic'])

def process_message(message):
    """Process individual message"""
    with processing_latency.time():
        try:
            data = json.loads(message.value().decode('utf-8'))
            # Business logic here
            logging.info(f"Processed message: {data}")
            messages_consumed.labels(topic=message.topic()).inc()
            return True
        except Exception as e:
            logging.error(f"Error processing message: {e}")
            return False

def run_consumer():
    try:
        while True:
            msg = consumer.poll(timeout=1.0)
            if msg is None:
                continue
            if msg.error():
                raise KafkaException(msg.error())

            success = process_message(msg)
            if success:
                consumer.commit(asynchronous=False)
    except KeyboardInterrupt:
        pass
    finally:
        consumer.close()

if __name__ == '__main__':
    run_consumer()

Scheduled Job Template

Use Case

Cron jobs, batch processing, periodic tasks.

Technology Stack

• Kubernetes CronJob (or external scheduler like Airflow)
• Execution timeout enforcement
• Success/failure notifications

Kubernetes CronJob Example

k8s/cronjob.yaml:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: my-scheduled-job
spec:
  schedule: "0 2 * * *"  # Daily at 2 AM
  concurrencyPolicy: Forbid  # Don't run if previous job still running
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2  # Retry twice
      activeDeadlineSeconds: 3600  # Kill after 1 hour
      template:
        spec:
          containers:
          - name: job
            image: my-registry/my-job:latest
            env:
            - name: JOB_TYPE
              value: "daily-cleanup"
            resources:
              requests:
                memory: "512Mi"
                cpu: "250m"
              limits:
                memory: "1Gi"
                cpu: "500m"
          restartPolicy: OnFailure

Job Script Example:

#!/usr/bin/env python3
import logging
import sys
from datetime import datetime

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

def main():
    try:
        logging.info("Starting scheduled job")

        # Job logic here
        logging.info("Processing data...")

        # Success
        logging.info("Job completed successfully")
        sys.exit(0)
    except Exception as e:
        logging.error(f"Job failed: {e}")
        sys.exit(1)

if __name__ == '__main__':
    main()

Template Versioning and Migration

Version Strategy

Template Versions:

• v1.0: Initial release
• v1.1: Minor improvements (backward compatible)
• v2.0: Breaking changes (require migration)

Deprecation Policy:

• Announce deprecation 6 months before removal
• Provide migration guide
• Support old versions during transition

Migration Guide Example

Migrating from v1.x to v2.0:

Breaking Changes: 1. Health check paths changed: /health → /health/liveness and /health/readiness 2. Metrics endpoint moved: /metrics no longer included by default, add manually 3. Environment variable naming: DB_HOST → DATABASE_HOST

Migration Steps: 1. Update health check paths in Kubernetes manifests 2. Add metrics endpoint if needed 3. Update environment variable names 4. Test in development environment 5. Deploy to staging 6. Deploy to production with rollback plan

Internal Developer Platform (IDP) Architecture

Table of Contents

1. Three-Layer Architecture 2. Developer Portal Layer 3. Platform Orchestration Layer 4. Integration Layer 5. Reference Architecture Diagrams

Three-Layer Architecture

An Internal Developer Platform consists of three architectural layers that work together to provide a comprehensive developer experience:

┌─────────────────────────────────────────────────────────┐
│         Developer Portal (Frontend Layer)              │
│  - Service Catalog    - Software Templates             │
│  - Documentation Hub  - Workflow Automation             │
└─────────────────────────────────────────────────────────┘
                          ↓ ↑
┌─────────────────────────────────────────────────────────┐
│         Integration Layer (Glue)                        │
│  - API Gateway      - CI/CD Integration                 │
│  - Observability    - Security Integration              │
│  - FinOps           - Secrets Management                │
└─────────────────────────────────────────────────────────┘
                          ↓ ↑
┌─────────────────────────────────────────────────────────┐
│         Platform Orchestration (Backend Layer)          │
│  - Infrastructure Provisioning                          │
│  - Environment Management                               │
│  - Deployment Automation (GitOps)                       │
└─────────────────────────────────────────────────────────┘

Developer Portal Layer

Purpose

Provide a unified interface for developers to discover, create, and manage services without needing to understand underlying infrastructure complexity.

Core Components

1. Service Catalog

Function: Centralized inventory of all software assets with ownership, dependencies, and health status.

Key Features:

• Software inventory (services, APIs, libraries, data pipelines, ML models)
• Ownership tracking (teams, individuals, on-call rotation)
• Dependency mapping (upstream/downstream relationships, service mesh topology)
• Health and SLO status (real-time metrics, historical trends)
• Documentation links (architecture diagrams, runbooks, API specifications)
• Metadata (tech stack, cloud provider, cost attribution)

Implementation Examples:

Backstage Software Catalog:

• YAML metadata files in service repositories
• API for programmatic access
• Plugins for enrichment (pull deployment status, cost data)
• Discovery: Automatic scanning of repositories

Port Service Catalog:

• Entity modeling (define custom entities beyond services)
• Blueprints for entity types
• Scorecard system (track service quality, compliance)

Cortex Service Catalog:

• Compliance scoring (based on best practices)
• Integration with incident management (PagerDuty, Opsgenie)
• Service health scoring

2. Software Templates / Scaffolders

Function: Project initialization with best practices, infrastructure, and CI/CD configurations baked in.

Template Components:

• Repository structure (monorepo vs. polyrepo, directory layout)
• Application boilerplate code (framework setup, starter files)
• Infrastructure as code (Kubernetes manifests, Terraform modules)
• CI/CD pipeline configurations (GitHub Actions workflows, GitLab CI YAML)
• Observability instrumentation (Prometheus metrics, structured logging, OpenTelemetry tracing)
• Security configurations (RBAC, network policies, secrets references)
• Documentation templates (README, runbooks, architecture diagrams, API docs)

Implementation Examples:

Backstage Scaffolder:

• Cookiecutter/Yeoman-style templates
• Template inputs (service name, tech stack, deployment environment)
• Actions: Create repository, scaffold code, configure CI/CD, register in catalog
• Template composition (base template + mixins)

Humanitec Resource Packs:

• Infrastructure abstractions (database, cache, message queue)
• Cloud-agnostic resource definitions
• Automatic provisioning on deployment

Template Versioning:

• Version templates (v1.0, v2.0)
• Migration guides when templates evolve
• Deprecation notices for old templates

3. Documentation Hub

Function: Centralized, searchable, version-controlled documentation.

Features:

• Centralized docs (architecture, tutorials, runbooks, API specs)
• Version-controlled (docs-as-code, Git-backed, PR workflow)
• Search and discovery (full-text search, tagging, categorization)
• Auto-generated docs (API specs from OpenAPI, dependency graphs)
• Metrics (doc views, staleness detection)

Implementation Examples:

Backstage TechDocs:

• Markdown + MkDocs
• Docs live alongside code in repositories
• Automatic build and publish on commit
• Search across all docs

GitBook / Confluence Integrations:

• Embed external docs in portal
• Single pane of glass for all documentation

Swagger/OpenAPI Integrations:

• Auto-generate API docs from specs
• Interactive API exploration

4. Workflow Automation

Function: Self-service actions for common platform operations.

Capabilities:

• Self-service actions (create environment, deploy service, request database access)
• Approval workflows (production access requires manager approval)
• Integration with ticketing (Jira, ServiceNow for non-self-service requests)
• ChatOps integrations (Slack, Microsoft Teams for bot-driven workflows)

Implementation Examples:

Port Self-Service Actions:

• Define actions in UI
• Trigger via API, portal, or Slack
• Approval steps configurable

Backstage Custom Plugins:

• React components for custom actions
• API calls to platform services (Kubernetes API, Terraform Cloud)
• Feedback to user (action status, logs)

Humanitec Deployment Pipelines:

• Deploy to environment with one click
• Environment diff before deploy
• Automated rollback on failure

Platform Orchestration Layer

Purpose

Manage infrastructure provisioning, environment lifecycle, and deployment automation without exposing raw cloud APIs to developers.

Core Components

1. Infrastructure Orchestration

Function: Multi-cloud resource provisioning with platform-agnostic APIs.

Capabilities:

• Multi-cloud resource provisioning (AWS, Azure, GCP, on-prem)
• Kubernetes cluster management (namespaces, RBAC, network policies, resource quotas)
• Resource abstraction (developer requests "database", platform provisions RDS/Cloud SQL/etc.)
• Configuration management (separation of app config vs. infrastructure config)

Implementation Examples:

Crossplane:

• Universal control plane for Kubernetes
• Providers for clouds (AWS, Azure, GCP), services (GitHub, Datadog)
• Compositions: Reusable infrastructure abstractions
• Kubernetes-native (CRDs, controllers)

Humanitec Platform Orchestrator:

• Resource management (databases, caches, message queues)
• Resource matching (dev uses in-cluster Postgres, prod uses managed RDS)
• Live Resource Graph: Single source of truth for all resources

Terraform Cloud:

• Workspace orchestration (separate state per environment/service)
• Remote execution (consistent environment, audit trail)
• Policy as code (Sentinel policies for compliance)

Pulumi Automation API:

• Programmatic infrastructure management
• Multi-language support (TypeScript, Python, Go)
• State management and diffs

2. Environment Management

Function: Standardized environments with automated provisioning and promotion pipelines.

Environment Types:

• Development: Developer-owned, permissive policies, frequent deployments, auto-cleanup
• Staging: Pre-production testing, production-like configuration, automated integration tests
• Production: Strict policies, approval workflows, automated rollback, immutable infrastructure

Capabilities:

• Ephemeral environments (per-PR preview deployments, on-demand test environments)
• Environment promotion pipelines (dev → staging → prod, automated or manual)
• Configuration drift detection (compare actual vs. desired state)
• Configuration drift remediation (auto-fix or alert)

Implementation Examples:

Humanitec Environment Manager:

• Environment types with inheritance (base config + overrides)
• Automatic environment creation
• Environment diff (see what changes between dev and prod)

Kubernetes Namespaces + GitOps:

• Namespace per environment
• Argo CD ApplicationSets (template environments)
• Kustomize overlays (base + environment-specific overrides)

Cloud Provider Environments:

• AWS Environments (isolated accounts)
• Azure Management Groups (hierarchy of subscriptions)
• GCP Folders (project organization)

3. Deployment Automation

Function: GitOps-based continuous delivery with progressive deployment strategies.

Principles:

• GitOps-based continuous delivery (Git as source of truth)
• Progressive delivery (canary, blue-green, rolling updates)
• Deployment policies (require approval, automated testing, gradual rollout)
• Rollback capabilities (automated on failure, manual trigger, retain last N versions)

Implementation Examples:

Argo CD:

• GitOps for Kubernetes (declarative desired state in Git)
• Multi-cluster application management
• ApplicationSets for templated deployments
• Sync waves for ordered rollouts
• Health assessment (check app is actually working, not just deployed)

Flux:

• Toolkit approach (composable controllers)
• Image automation (auto-update image tags in Git)
• Notification system (alerts to Slack, GitHub, etc.)

Humanitec Deployment Pipelines:

• Environment-aware deployments
• Approval gates (manual approval before prod)
• Automated testing integration (deploy → run tests → promote)

Spinnaker:

• Advanced deployment strategies (canary analysis, blue-green, rolling)
• Multi-cloud support
• Pipeline as code (JSON/YAML)

Integration Layer

Purpose

Connect developer portal with underlying infrastructure, CI/CD, observability, security, and FinOps tools.

Core Components

1. API Gateway / Abstraction Layer

Function: Unified API for platform capabilities with authentication and rate limiting.

Features:

• Unified API for platform capabilities (abstract underlying tool complexity)
• Authentication and authorization (SSO integration, RBAC, API keys)
• Rate limiting and quotas (prevent abuse, fair usage)
• API versioning and deprecation (backward compatibility, migration paths)

Implementation Examples:

Kubernetes API Server:

• Platform-native for Kubernetes resources
• RBAC for fine-grained permissions
• Admission controllers for policy enforcement

Backstage Backend API:

• REST API for catalog, scaffolder, auth
• Plugin backend APIs

Custom API Gateways:

• Kong, Tyk, AWS API Gateway
• Aggregate multiple backends
• Transform requests/responses

2. CI/CD Integration

Function: Pipeline visibility and control from developer portal.

Features:

• Pipeline status visibility (in-progress, failed, succeeded)
• Trigger builds/deployments from portal
• Artifact management (container registry, package repositories)
• Test results and quality gates (unit tests, integration tests, security scans)

Implementation Examples:

Backstage CI/CD Plugins:

• GitHub Actions plugin (show workflow runs)
• GitLab CI plugin (pipeline status)
• Jenkins plugin (job history)
• CircleCI plugin

Argo Workflows:

• Kubernetes-native CI
• DAG-based workflows
• Artifact passing between steps

Tekton:

• Cloud-native CI/CD (Kubernetes CRDs)
• Reusable tasks and pipelines

3. Observability Integration

Function: Metrics, logs, and traces surfaced in developer portal.

Features:

• Metrics dashboards in portal (per-service, per-team, per-environment)
• Log aggregation and search (centralized logging, grep/filter)
• Distributed tracing (request path visualization, latency breakdown)
• Alerting and on-call management (PagerDuty, Opsgenie integration)

Implementation Examples:

Backstage Prometheus/Grafana Plugins:

• Embed Grafana dashboards
• Show Prometheus metrics per service

Commercial Observability:

• Datadog plugin (APM, logs, metrics in one place)
• New Relic plugin
• Splunk integration

OpenTelemetry:

• Standard instrumentation across languages
• Vendor-agnostic telemetry collection
• Automatic instrumentation in golden paths

4. Security Integration

Function: Vulnerability scanning, policy enforcement, secrets management.

Features:

• Vulnerability scanning (container images, dependencies, IaC)
• Policy enforcement (OPA, Kyverno for Kubernetes)
• Secrets management (Vault integration, encrypted configs, rotation)
• Compliance reporting (SOC2, GDPR, HIPAA dashboards)

Implementation Examples:

Backstage Security Insights Plugins:

• Show vulnerability scan results
• Dependency security scores
• Compliance status per service

Scanning Tools:

• Snyk (container and dependency scanning)
• Trivy (comprehensive vulnerability scanner)
• Aqua, Twistlock (runtime security)

Policy Engines:

• OPA Gatekeeper (Kubernetes admission control)
• Kyverno (policy as Kubernetes resources)
• Cloud Custodian (cloud resource policies)

Secrets Management:

• HashiCorp Vault (dynamic secrets, encryption as a service)
• AWS Secrets Manager, Azure Key Vault, GCP Secret Manager
• External Secrets Operator (sync cloud secrets to Kubernetes)

5. FinOps Integration

Function: Cost visibility, budgets, and optimization recommendations.

Features:

• Cost visibility (per-service, per-team, per-environment, per-deployment)
• Budget alerts (spending thresholds, trend analysis)
• Right-sizing recommendations (over-provisioned resources, unused resources)
• Resource lifecycle management (auto-delete unused resources, hibernation)

Implementation Examples:

Backstage Cost Insights Plugin:

• Show cost per service
• Compare cost across teams
• Trend analysis

Cloud Provider Cost Tools:

• AWS Cost Explorer (detailed cost breakdown)
• Azure Cost Management (budgets, recommendations)
• GCP Cost Management (cost attribution, forecasting)

Kubernetes Cost Tools:

• Kubecost (cost allocation per namespace, pod, label)
• OpenCost (open-source Kubernetes cost monitoring)

Infrastructure Cost Estimation:

• Infracost (IaC cost estimation before deployment)
• Terraform cost estimation modules

Reference Architecture Diagrams

Full-Stack IDP Architecture (Backstage + Crossplane + Argo CD)

┌─────────────────────────────────────────────────────────────────┐
│                        Developer Portal (Backstage)             │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐          │
│  │   Service    │  │   Software   │  │  TechDocs    │          │
│  │   Catalog    │  │   Templates  │  │  (Markdown)  │          │
│  └──────────────┘  └──────────────┘  └──────────────┘          │
│                                                                 │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐          │
│  │   CI/CD      │  │ Observability│  │  Cost        │          │
│  │   Status     │  │  Dashboards  │  │  Insights    │          │
│  └──────────────┘  └──────────────┘  └──────────────┘          │
└─────────────────────────────────────────────────────────────────┘
                            ↓ API Calls
┌─────────────────────────────────────────────────────────────────┐
│                     Kubernetes Clusters                          │
│  ┌─────────────────────────────────────────────────────┐        │
│  │  Crossplane (Infrastructure Orchestration)          │        │
│  │  - Providers: AWS, Azure, GCP                       │        │
│  │  - Compositions: Reusable resource abstractions     │        │
│  └─────────────────────────────────────────────────────┘        │
│  ┌─────────────────────────────────────────────────────┐        │
│  │  Argo CD (GitOps Continuous Delivery)              │        │
│  │  - Sync Git → Cluster                               │        │
│  │  - ApplicationSets for multi-env                    │        │
│  └─────────────────────────────────────────────────────┘        │
│  ┌─────────────────────────────────────────────────────┐        │
│  │  Application Workloads                              │        │
│  │  - Deployments, Services, Ingresses                 │        │
│  └─────────────────────────────────────────────────────┘        │
└─────────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────────┐
│                     Cloud Providers                              │
│  AWS (RDS, S3, EKS)  │  Azure (CosmosDB, AKS)  │  GCP (GKE)     │
└─────────────────────────────────────────────────────────────────┘

Hybrid Architecture (Backstage + Humanitec)

┌─────────────────────────────────────────────────────────────────┐
│             Developer Portal (Backstage or Port)                │
│  - Service Catalog    - Documentation    - Self-Service UI     │
└─────────────────────────────────────────────────────────────────┘
                            ↓ API
┌─────────────────────────────────────────────────────────────────┐
│             Platform Orchestrator (Humanitec)                   │
│  ┌─────────────────────────────────────────────────────┐        │
│  │  Live Resource Graph (Single Source of Truth)      │        │
│  │  - Applications, Resources, Dependencies            │        │
│  └─────────────────────────────────────────────────────┘        │
│  ┌─────────────────────────────────────────────────────┐        │
│  │  Resource Management                                │        │
│  │  - Resource Definitions (databases, caches, etc.)   │        │
│  │  - Resource Matching (dev vs. prod resources)       │        │
│  └─────────────────────────────────────────────────────┘        │
│  ┌─────────────────────────────────────────────────────┐        │
│  │  Deployment Pipelines                               │        │
│  │  - Environment orchestration                        │        │
│  │  - Approval workflows                               │        │
│  └─────────────────────────────────────────────────────┘        │
└─────────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────────┐
│     Infrastructure Provisioning (Terraform, Crossplane)         │
└─────────────────────────────────────────────────────────────────┘

Data Flow: Developer Creates New Service

1. Developer opens Backstage portal
   ↓
2. Selects "Create Component" → chooses template (e.g., "Node.js API")
   ↓
3. Fills in template parameters (service name, team, database type)
   ↓
4. Backstage Scaffolder executes template actions:
   a. Create Git repository (GitHub/GitLab API)
   b. Scaffold code (Cookiecutter template)
   c. Add CI/CD pipeline (GitHub Actions workflow file)
   d. Register in service catalog (catalog-info.yaml)
   ↓
5. Developer pushes first commit
   ↓
6. CI/CD pipeline triggers:
   a. Build container image
   b. Security scan (Trivy)
   c. Push to container registry
   ↓
7. Developer creates deployment via portal:
   a. Selects environment (dev, staging, prod)
   b. Crossplane provisions infrastructure (database, cache)
   c. Argo CD syncs application manifests to cluster
   ↓
8. Service is deployed and visible in:
   - Service catalog (ownership, dependencies)
   - Observability dashboards (metrics, logs)
   - Cost insights (resource costs)

Best Practices

Portal Design

Navigation:

• Clear information architecture (catalog, docs, create, admin)
• Search as primary navigation (developers know what they want)
• Personalization (show my services, my team's services)

Performance:

• Fast load times (<3s for catalog page)
• Pagination for large catalogs (100 services per page)
• Caching for frequently accessed data

Accessibility:

• Keyboard navigation
• Screen reader support
• High contrast mode

Orchestration Layer

Resource Abstraction:

• Platform-agnostic APIs (request "database", not "RDS instance")
• Environment-aware resource matching (dev: in-cluster Postgres, prod: managed RDS)
• Cost-aware defaults (right-size by default, allow overrides)

Idempotency:

• Infrastructure operations should be idempotent
• Retry-safe (multiple applies don't create duplicates)
• Rollback-safe (can undo without side effects)

Integration Layer

API Design:

• Versioned APIs (v1, v2 with deprecation notices)
• Pagination for list endpoints
• Filtering and sorting
• Clear error messages

Monitoring:

• Instrument all integrations (metrics for API calls, latency, errors)
• Circuit breakers (fail fast if backend is down)
• Timeouts (don't wait forever for slow backends)

Implementing a Backstage-Based Platform

Table of Contents

1. Prerequisites 2. Installation and Setup 3. Core Configuration 4. Creating the First Golden Path 5. Integration Examples 6. Production Deployment

Prerequisites

Infrastructure Requirements

Development Environment:

• Node.js 18+ and Yarn
• Docker and Docker Compose
• Git
• Text editor / IDE

Production Environment:

• Kubernetes cluster (EKS, GKE, AKS, or self-managed)
• PostgreSQL database (managed or self-hosted)
• Object storage for TechDocs (S3, GCS, Azure Blob)
• Authentication provider (GitHub, GitLab, Okta, Azure AD)

Team Requirements

Minimum Team:

• 1-2 platform engineers (Backstage setup and customization)
• 1 frontend engineer (optional, for custom plugins/UI)

Recommended Team:

• 1 platform lead (strategy, roadmap)
• 2-3 platform engineers (Backstage development, integrations)
• 1 SRE (production operations, monitoring)

Installation and Setup

Quick Start (Development)

# Create new Backstage app
npx @backstage/create-app@latest

# Follow prompts
App name: my-platform
# Choose GitHub authentication

cd my-platform

# Start development server
yarn dev

Access at http://localhost:3000

Directory Structure

my-platform/
├── app-config.yaml              # Main configuration
├── app-config.production.yaml   # Production overrides
├── packages/
│   ├── app/                     # Frontend React app
│   │   ├── src/
│   │   │   ├── App.tsx          # Main app component
│   │   │   └── components/      # Custom components
│   ├── backend/                 # Backend Node.js app
│   │   ├── src/
│   │   │   ├── index.ts         # Backend entry point
│   │   │   └── plugins/         # Backend plugins
│   └── common/                  # Shared code
├── plugins/                     # Custom plugins
└── catalog-info.yaml            # Backstage itself in catalog

Core Configuration

app-config.yaml

app:
  title: My Platform
  baseUrl: http://localhost:3000

organization:
  name: My Company

backend:
  baseUrl: http://localhost:7007
  listen:
    port: 7007
  database:
    client: pg
    connection:
      host: ${POSTGRES_HOST}
      port: ${POSTGRES_PORT}
      user: ${POSTGRES_USER}
      password: ${POSTGRES_PASSWORD}

auth:
  environment: development
  providers:
    github:
      development:
        clientId: ${GITHUB_CLIENT_ID}
        clientSecret: ${GITHUB_CLIENT_SECRET}

catalog:
  import:
    entityFilename: catalog-info.yaml
  rules:
    - allow: [Component, System, API, Resource, Location]
  locations:
    # Register components from Git repositories
    - type: url
      target: https://github.com/my-org/service-catalog/blob/main/all.yaml
      rules:
        - allow: [User, Group]

techdocs:
  builder: 'local'
  generator:
    runIn: 'local'
  publisher:
    type: 'local'

Authentication Setup (GitHub)

1. Create GitHub OAuth App:

• Go to GitHub Settings → Developer settings → OAuth Apps → New OAuth App
• Homepage URL: http://localhost:3000
• Callback URL: http://localhost:7007/api/auth/github/handler/frame

2. Add credentials to app-config.local.yaml (gitignored):

auth:
  providers:
    github:
      development:
        clientId: your-client-id
        clientSecret: your-client-secret

Database Setup

Development (SQLite):

backend:
  database:
    client: better-sqlite3
    connection: ':memory:'

Production (PostgreSQL):

# Create database
createdb backstage

# Set environment variables
export POSTGRES_HOST=localhost
export POSTGRES_PORT=5432
export POSTGRES_USER=backstage
export POSTGRES_PASSWORD=secure-password

Creating the First Golden Path

Software Template Example: Node.js Microservice

Create templates/nodejs-service/template.yaml:

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: nodejs-service
  title: Node.js Microservice
  description: Create a new Node.js microservice with Express and TypeScript
  tags:
    - recommended
    - nodejs
    - api
spec:
  owner: platform-team
  type: service

  parameters:
    - title: Service Information
      required:
        - name
        - owner
      properties:
        name:
          title: Name
          type: string
          description: Unique name for this service
          ui:autofocus: true
        description:
          title: Description
          type: string
          description: What does this service do?
        owner:
          title: Owner
          type: string
          description: Team or individual owner
          ui:field: OwnerPicker
          ui:options:
            catalogFilter:
              kind: [Group, User]

    - title: Deployment Configuration
      required:
        - environment
      properties:
        environment:
          title: Initial Environment
          type: string
          enum:
            - development
            - staging
            - production
          default: development

  steps:
    - id: fetch-base
      name: Fetch Base Template
      action: fetch:template
      input:
        url: ./skeleton
        values:
          name: ${{ parameters.name }}
          description: ${{ parameters.description }}
          owner: ${{ parameters.owner }}
          environment: ${{ parameters.environment }}

    - id: publish
      name: Publish to GitHub
      action: publish:github
      input:
        allowedHosts: ['github.com']
        description: ${{ parameters.description }}
        repoUrl: github.com?owner=my-org&repo=${{ parameters.name }}
        defaultBranch: main

    - id: register
      name: Register in Catalog
      action: catalog:register
      input:
        repoContentsUrl: ${{ steps['publish'].output.repoContentsUrl }}
        catalogInfoPath: '/catalog-info.yaml'

  output:
    links:
      - title: Repository
        url: ${{ steps['publish'].output.remoteUrl }}
      - title: Open in catalog
        icon: catalog
        entityRef: ${{ steps['register'].output.entityRef }}

Template Skeleton Structure

Create templates/nodejs-service/skeleton/:

skeleton/
├── catalog-info.yaml
├── package.json
├── tsconfig.json
├── src/
│   ├── index.ts
│   ├── app.ts
│   └── routes/
│       └── health.ts
├── .github/
│   └── workflows/
│       └── ci.yaml
├── k8s/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── ingress.yaml
├── docs/
│   └── index.md
└── mkdocs.yml

catalog-info.yaml:

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: ${{values.name}}
  description: ${{values.description}}
  annotations:
    github.com/project-slug: my-org/${{values.name}}
    backstage.io/techdocs-ref: dir:.
spec:
  type: service
  lifecycle: experimental
  owner: ${{values.owner}}
  system: platform

package.json:

{
  "name": "${{values.name}}",
  "version": "1.0.0",
  "description": "${{values.description}}",
  "main": "dist/index.js",
  "scripts": {
    "start": "node dist/index.js",
    "dev": "ts-node-dev src/index.ts",
    "build": "tsc",
    "test": "jest"
  },
  "dependencies": {
    "express": "^4.18.0"
  },
  "devDependencies": {
    "@types/express": "^4.17.0",
    "@types/node": "^20.0.0",
    "typescript": "^5.0.0",
    "ts-node-dev": "^2.0.0"
  }
}

Integration Examples

GitHub Actions Plugin

Install plugin:

yarn workspace app add @backstage/plugin-github-actions

Configure in app/src/components/catalog/EntityPage.tsx:

import {
  EntityGithubActionsContent,
  isGithubActionsAvailable,
} from '@backstage/plugin-github-actions';

const cicdContent = (
  <EntitySwitch>
    <EntitySwitch.Case if={isGithubActionsAvailable}>
      <EntityGithubActionsContent />
    </EntitySwitch.Case>
  </EntitySwitch>
);

Add to app-config.yaml:

integrations:
  github:
    - host: github.com
      token: ${GITHUB_TOKEN}

Kubernetes Plugin

Install:

yarn workspace app add @backstage/plugin-kubernetes
yarn workspace backend add @backstage/plugin-kubernetes-backend

Configure backend (packages/backend/src/plugins/kubernetes.ts):

import { KubernetesBuilder } from '@backstage/plugin-kubernetes-backend';
import { Router } from 'express';
import { PluginEnvironment } from '../types';

export default async function createPlugin(
  env: PluginEnvironment,
): Promise<Router> {
  const { router } = await KubernetesBuilder.createBuilder({
    logger: env.logger,
    config: env.config,
  }).build();
  return router;
}

app-config.yaml:

kubernetes:
  serviceLocatorMethod:
    type: 'multiTenant'
  clusterLocatorMethods:
    - type: 'config'
      clusters:
        - url: https://k8s-cluster.example.com
          name: production
          authProvider: 'serviceAccount'
          serviceAccountToken: ${K8S_TOKEN}

Prometheus/Grafana Plugin

Install:

yarn workspace app add @backstage/plugin-prometheus

Add annotation to catalog-info.yaml:

metadata:
  annotations:
    prometheus.io/rule: memUsage|component,|instance,|job
    prometheus.io/alert: all

Production Deployment

Kubernetes Deployment

Create namespace:

kubectl create namespace backstage

Create PostgreSQL (using Helm):

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install postgres bitnami/postgresql \
  --namespace backstage \
  --set auth.database=backstage \
  --set auth.username=backstage

Build Docker image:

# packages/backend/Dockerfile
FROM node:18-bullseye-slim
WORKDIR /app

# Copy package files
COPY package.json yarn.lock ./
COPY packages/backend/package.json ./packages/backend/

# Install dependencies
RUN yarn install --frozen-lockfile --production --network-timeout 300000

# Copy backend code
COPY packages/backend/dist ./packages/backend/dist

CMD ["node", "packages/backend"]
EXPOSE 7007

# Build
docker build -t my-registry/backstage:1.0.0 -f packages/backend/Dockerfile .

# Push
docker push my-registry/backstage:1.0.0

Kubernetes manifests:

deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: backstage
  namespace: backstage
spec:
  replicas: 2
  selector:
    matchLabels:
      app: backstage
  template:
    metadata:
      labels:
        app: backstage
    spec:
      containers:
      - name: backstage
        image: my-registry/backstage:1.0.0
        ports:
        - containerPort: 7007
        env:
        - name: POSTGRES_HOST
          value: postgres-postgresql
        - name: POSTGRES_PORT
          value: "5432"
        - name: POSTGRES_USER
          value: backstage
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-postgresql
              key: password
        - name: GITHUB_TOKEN
          valueFrom:
            secretKeyRef:
              name: backstage-secrets
              key: github-token
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"

service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: backstage
  namespace: backstage
spec:
  selector:
    app: backstage
  ports:
  - port: 80
    targetPort: 7007
  type: ClusterIP

ingress.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: backstage
  namespace: backstage
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
  - hosts:
    - platform.mycompany.com
    secretName: backstage-tls
  rules:
  - host: platform.mycompany.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: backstage
            port:
              number: 80

Deploy:

kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
kubectl apply -f ingress.yaml

Monitoring and Observability

Add Prometheus metrics:

// packages/backend/src/index.ts
import { useHotCleanup } from '@backstage/backend-common';
import { metricsHandler } from '@backstage/backend-defaults/prometheus';

async function main() {
  // ... existing code ...

  // Add metrics endpoint
  apiRouter.use('/metrics', metricsHandler());
}

ServiceMonitor (for Prometheus Operator):

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: backstage
  namespace: backstage
spec:
  selector:
    matchLabels:
      app: backstage
  endpoints:
  - port: http
    path: /metrics

Scaling Considerations

Horizontal Pod Autoscaling:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: backstage
  namespace: backstage
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: backstage
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Database Connection Pooling:

# app-config.production.yaml
backend:
  database:
    client: pg
    connection:
      host: ${POSTGRES_HOST}
      port: ${POSTGRES_PORT}
      user: ${POSTGRES_USER}
      password: ${POSTGRES_PASSWORD}
    pool:
      min: 5
      max: 20

Backup and Disaster Recovery

PostgreSQL Backups:

# Automated backup script
pg_dump -h $POSTGRES_HOST -U backstage backstage | gzip > backstage-$(date +%Y%m%d).sql.gz

# Upload to S3
aws s3 cp backstage-$(date +%Y%m%d).sql.gz s3://backups/backstage/

Restore:

# Download from S3
aws s3 cp s3://backups/backstage/backstage-20250101.sql.gz .

# Restore
gunzip < backstage-20250101.sql.gz | psql -h $POSTGRES_HOST -U backstage backstage