Building Ci Pipelines

Building CI Pipelines

Purpose

CI/CD pipelines automate testing, building, and deploying software. This skill provides patterns for constructing robust, secure, and efficient pipelines across GitHub Actions, GitLab CI, Argo Workflows, and Jenkins. Focus areas: supply chain security (SLSA), monorepo optimization, caching, and parallelization.

When to Use This Skill

Invoke when:

• Setting up continuous integration for new projects
• Implementing automated testing workflows
• Building container images with security provenance
• Optimizing slow CI pipelines (especially monorepos)
• Implementing SLSA supply chain security
• Configuring multi-platform builds
• Setting up GitOps automation
• Migrating from legacy CI systems

Platform Selection

GitHub-hosted → GitHub Actions (SLSA native, 10K+ actions, OIDC) GitLab-hosted → GitLab CI (parent-child pipelines, built-in security) Kubernetes → Argo Workflows (DAG-based, event-driven) Legacy → Jenkins (migrate when possible)

Platform Comparison

Quick Start Patterns

Pattern 1: Basic CI (Lint → Test → Build)

# GitHub Actions
name: CI
on: [push, pull_request]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm run lint

  test:
    needs: lint
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm test

  build:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm run build

Pattern 2: Matrix Strategy (Multi-Platform)

test:
  runs-on: ${{ matrix.os }}
  strategy:
    matrix:
      os: [ubuntu-latest, windows-latest, macos-latest]
      node-version: [18, 20, 22]
  steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-node@v4
      with:
        node-version: ${{ matrix.node-version }}
    - run: npm test

9 jobs (3 OS × 3 versions) in parallel: 5 min vs 45 min sequential.

Pattern 3: Monorepo Affected (Turborepo)

build:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
      with:
        fetch-depth: 0  # Required for affected detection

    - uses: actions/setup-node@v4
      with:
        node-version: 20

    - name: Build affected
      run: npx turbo run build --filter='...[origin/main]'
      env:
        TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
        TURBO_TEAM: ${{ vars.TURBO_TEAM }}

60-80% CI time reduction for monorepos.

Pattern 4: SLSA Level 3 Provenance

name: SLSA Build
on:
  push:
    tags: ['v*']

permissions:
  id-token: write
  contents: read
  packages: write

jobs:
  build:
    runs-on: ubuntu-latest
    outputs:
      digest: ${{ steps.build.outputs.digest }}
    steps:
      - uses: actions/checkout@v4
      - name: Build container
        id: build
        uses: docker/build-push-action@v5
        with:
          push: true
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}

  provenance:
    needs: build
    permissions:
      id-token: write
      actions: read
      packages: write
    uses: slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@v1.10.0
    with:
      image: ghcr.io/${{ github.repository }}
      digest: ${{ needs.build.outputs.digest }}
      registry-username: ${{ github.actor }}
    secrets:
      registry-password: ${{ secrets.GITHUB_TOKEN }}

Verification:

cosign verify-attestation --type slsaprovenance \
  --certificate-identity-regexp "^https://github.com/slsa-framework" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  ghcr.io/myorg/myapp@sha256:abcd...

Pattern 5: OIDC Federation (No Credentials)

deploy:
  runs-on: ubuntu-latest
  permissions:
    id-token: write
    contents: read
  steps:
    - uses: actions/checkout@v4

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v4
      with:
        role-to-assume: arn:aws:iam::123456789012:role/GitHubActionsRole
        aws-region: us-east-1

    - name: Deploy
      run: aws s3 sync ./dist s3://my-bucket

Benefits: No stored credentials, 1-hour lifetime, full audit trail.

Pattern 6: Security Scanning

security:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
      with:
        fetch-depth: 0

    - name: Gitleaks (secret detection)
      uses: gitleaks/gitleaks-action@v2

    - name: Snyk (vulnerability scan)
      uses: snyk/actions/node@master
      env:
        SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}

    - name: SBOM generation
      uses: anchore/sbom-action@v0
      with:
        format: spdx-json
        output-file: sbom.spdx.json

Caching

Automatic Dependency Caching

- uses: actions/setup-node@v4
  with:
    node-version: 20
    cache: 'npm'  # Auto-caches ~/.npm
- run: npm ci

Supported: npm, yarn, pnpm, pip, poetry, cargo, go

Manual Cache Control

- uses: actions/cache@v4
  with:
    path: |
      ~/.cargo/bin
      ~/.cargo/registry
      target/
    key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
    restore-keys: |
      ${{ runner.os }}-cargo-

Multi-Layer Caching (Nx)

- name: Nx Cloud (build outputs)
  run: npx nx affected -t build
  env:
    NX_CLOUD_ACCESS_TOKEN: ${{ secrets.NX_CLOUD_ACCESS_TOKEN }}

- name: Vite Cache
  uses: actions/cache@v4
  with:
    path: '**/node_modules/.vite'
    key: vite-${{ hashFiles('package-lock.json') }}

- name: TypeScript Cache
  uses: actions/cache@v4
  with:
    path: '**/tsconfig.tsbuildinfo'
    key: tsc-${{ hashFiles('tsconfig.json') }}

Result: 70-90% build time reduction.

Parallelization

Job-Level Parallelization

jobs:
  unit-tests:
    steps:
      - run: npm run test:unit

  integration-tests:
    steps:
      - run: npm run test:integration

  e2e-tests:
    steps:
      - run: npm run test:e2e

All three run simultaneously.

Test Sharding

test:
  strategy:
    matrix:
      shard: [1, 2, 3, 4]
  steps:
    - run: npm test -- --shard=${{ matrix.shard }}/4

20min test suite → 5min (4x speedup).

Language Examples

Python

test:
  strategy:
    matrix:
      python-version: ['3.10', '3.11', '3.12']
  steps:
    - uses: actions/setup-python@v5
      with:
        python-version: ${{ matrix.python-version }}
    - run: pipx install poetry
    - run: poetry install
    - run: poetry run ruff check .
    - run: poetry run mypy .
    - run: poetry run pytest --cov

Rust

test:
  strategy:
    matrix:
      os: [ubuntu-latest, windows-latest, macos-latest]
      rust: [stable, nightly]
  steps:
    - uses: dtolnay/rust-toolchain@master
      with:
        toolchain: ${{ matrix.rust }}
        components: rustfmt, clippy
    - uses: Swatinem/rust-cache@v2
    - run: cargo fmt -- --check
    - run: cargo clippy -- -D warnings
    - run: cargo test

Go

test:
  steps:
    - uses: actions/setup-go@v5
      with:
        go-version: '1.23'
        cache: true
    - run: go mod verify
    - uses: golangci/golangci-lint-action@v4
    - run: go test -v -race -coverprofile=coverage.txt ./...

TypeScript

test:
  strategy:
    matrix:
      node-version: [18, 20, 22]
  steps:
    - uses: pnpm/action-setup@v3
      with:
        version: 8
    - uses: actions/setup-node@v4
      with:
        node-version: ${{ matrix.node-version }}
        cache: 'pnpm'
    - run: pnpm install --frozen-lockfile
    - run: pnpm run lint
    - run: pnpm run type-check
    - run: pnpm test

Best Practices

Security

DO:

• Use OIDC instead of long-lived credentials
• Pin actions to commit SHA: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
• Restrict permissions: permissions: { contents: read }
• Scan secrets (Gitleaks) on every commit
• Generate SLSA provenance for releases

DON'T:

• Expose secrets in logs
• Use pull_request_target without validation
• Trust unverified third-party actions

Performance

DO:

• Use affected detection for monorepos
• Cache dependencies and build outputs
• Parallelize independent jobs
• Fail fast: strategy.fail-fast: true
• Use remote caching (Turborepo/Nx Cloud)

DON'T:

• Rebuild everything on every commit
• Run long tests in PR checks
• Use generic cache keys

Debugging

# Enable debug logging
env:
  ACTIONS_STEP_DEBUG: true
  ACTIONS_RUNNER_DEBUG: true

# SSH into runner
- uses: mxschmitt/action-tmate@v3

Advanced Patterns

For detailed guides, see references:

• github-actions-patterns.md - Reusable workflows, composite actions, matrix strategies, OIDC setup
• gitlab-ci-patterns.md - Parent-child pipelines, dynamic generation, runner configuration
• argo-workflows-guide.md - DAG templates, artifact passing, event-driven triggers
• slsa-security-framework.md - SLSA Levels 1-4, provenance generation, cosign verification
• monorepo-ci-strategies.md - Turborepo/Nx/Bazel affected detection algorithms
• caching-strategies.md - Multi-layer caching, Docker optimization, cache invalidation
• parallelization-patterns.md - Test sharding, job dependencies, DAG design
• secrets-management.md - OIDC for AWS/GCP/Azure, Vault integration, rotation

Examples

Complete runnable workflows:

• examples/github-actions-basic/ - Starter template (lint/test/build)
• examples/github-actions-monorepo/ - Turborepo with remote caching
• examples/github-actions-slsa/ - SLSA Level 3 provenance
• examples/gitlab-ci-monorepo/ - Parent-child dynamic pipeline
• examples/argo-workflows-dag/ - Diamond DAG parallelization
• examples/multi-language-matrix/ - Cross-platform testing

Utility Scripts

Token-free execution:

• scripts/validate_workflow.py - Validate YAML syntax and best practices
• scripts/generate_github_workflow.py - Generate workflow from template
• scripts/analyze_ci_performance.py - CI metrics analysis
• scripts/setup_oidc_aws.py - Automate AWS OIDC setup

Related Skills

testing-strategies - Test execution strategies (unit, integration, E2E) deploying-applications - Deployment automation and GitOps auth-security - Secrets management and authentication observability - Pipeline monitoring and alerting

Basic GitHub Actions CI Example

This example demonstrates a standard CI workflow for a Node.js project with linting, testing, building, and security scanning.

Workflow Structure

Jobs

1. lint - Code quality checks

• ESLint
• Prettier formatting

2. test - Unit and integration tests

• Matrix strategy (Node.js 18, 20, 22)
• Coverage reporting to Codecov
• Parallel execution across versions

3. build - Production build

• Depends on lint and test passing
• Uploads build artifacts
• Single Node.js version (20)

4. security - Security scanning

• Gitleaks (secret detection)
• Snyk (vulnerability scanning)
• Runs in parallel with other jobs

Features

• Parallel Execution: lint, test, and security run simultaneously
• Matrix Strategy: Tests run across Node.js 18, 20, and 22
• Dependency Caching: npm cache speeds up installs
• Job Dependencies: Build only runs after lint and test pass
• Artifact Upload: Build artifacts saved for 7 days
• Coverage Reporting: Codecov integration for code coverage

Setup

Required Secrets

Configure these secrets in repository settings:

• SNYK_TOKEN - Snyk API token (optional)
• CODECOV_TOKEN - Codecov token (optional)

Required package.json Scripts

{
  "scripts": {
    "lint": "eslint src/**/*.ts",
    "format:check": "prettier --check 'src/**/*.ts'",
    "test": "jest --coverage",
    "build": "tsc && vite build"
  }
}

Customization

Add More Node.js Versions

strategy:
  matrix:
    node-version: [16, 18, 20, 22]  # Add Node 16

Add Type Checking

- name: Type check
  run: npm run type-check

Add E2E Tests

e2e:
  name: E2E Tests
  needs: build
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - uses: actions/download-artifact@v4
      with:
        name: build-artifacts
        path: dist/
    - run: npm run test:e2e

Performance

Typical Runtime:

• Lint: ~1 minute
• Test (3 versions): ~3 minutes (parallel)
• Build: ~2 minutes
• Security: ~1 minute

Total wall-clock time: ~4 minutes (parallelization)

Sequential time: ~7 minutes (without parallelization)

Triggers

• Pushes to main and develop branches
• Pull requests targeting main branch

Permissions

• contents: read - Read repository files
• pull-requests: write - Comment on PRs (for coverage reports)

GitHub Actions Monorepo CI with Turborepo

This example demonstrates an efficient CI/CD pipeline for a JavaScript/TypeScript monorepo using Turborepo's affected detection and remote caching.

Features

• Affected Detection: Only builds, tests, and deploys packages that changed
• Remote Caching: Turborepo cloud caching for 70-90% cache hit rate
• Parallel Execution: Independent jobs run simultaneously
• Selective Deployment: Preview deploys for PRs, production for main branch
• Performance Optimization: 60-80% reduction in CI time vs naive approach

Workflow Structure

Jobs

1. setup - Install dependencies and cache setup

• Installs npm dependencies
• Prepares Turborepo cache

2. lint - Lint affected packages

• Runs in parallel with test
• Only affected packages since main

3. test - Test affected packages

• Runs in parallel with lint
• Only affected packages since main

4. build - Build affected packages

• Depends on lint and test passing
• Only affected packages since main
• Uploads build artifacts

5. deploy-preview - Deploy preview (PRs only)

• Deploys affected apps to preview environment
• Only runs on pull requests

6. deploy-production - Deploy production (main branch only)

• Deploys affected apps to production
• Only runs on pushes to main
• Uses GitHub environment protection

Turborepo Configuration

turbo.json

{
  "pipeline": {
    "build": {
      "dependsOn": ["^build"],  // Build dependencies first
      "outputs": ["dist/**"],    // Cache these outputs
      "cache": true
    },
    "test": {
      "dependsOn": ["build"],    // Tests depend on build
      "cache": true
    },
    "deploy:production": {
      "dependsOn": ["build", "test"],
      "cache": false             // Never cache deploys
    }
  }
}

Performance Gains

Before (Naive Approach)

Building all packages on every commit:

# 100 packages, 30 minutes total
- run: npm run build --workspaces

Cost:

• 30 minutes per commit
• $0.008/minute = $0.24 per commit
• 100 commits/week = $24/week = $1,248/year

After (Affected Detection)

Building only affected packages:

# Only 10 packages changed = 6 minutes
- run: npx turbo run build --filter='...[origin/main]'

Cost:

• 6 minutes per commit (80% reduction)
• $0.008/minute = $0.048 per commit
• 100 commits/week = $4.80/week = $250/year

Savings: $998/year (80%)

Setup

1. Install Turborepo

npm install turbo --save-dev

2. Configure Remote Caching

# Login to Vercel (free tier available)
npx turbo login

# Link repository
npx turbo link

This generates:

• TURBO_TOKEN - Add to GitHub Secrets
• TURBO_TEAM - Add to GitHub Variables

3. Create turbo.json

See turbo.json in this directory.

4. Add Workflow

Copy .github/workflows/ci.yml to your repository.

5. Configure Secrets/Variables

GitHub Secrets:

• TURBO_TOKEN - Turborepo remote cache token
• VERCEL_TOKEN - Vercel deployment token (if deploying)

GitHub Variables:

• TURBO_TEAM - Turborepo team name

Repository Structure

monorepo/
├── apps/
│   ├── web/              # Next.js app
│   ├── api/              # Express API
│   └── mobile/           # React Native
├── packages/
│   ├── ui-components/    # Shared UI library
│   ├── api-client/       # API client
│   └── utils/            # Shared utilities
├── turbo.json            # Turborepo config
├── package.json
└── .github/
    └── workflows/
        └── ci.yml

Affected Detection Examples

Scenario 1: API Client Change

Changed files:

• packages/api-client/src/client.ts

Affected packages:

• api-client (changed)
• web (depends on api-client)
• api (depends on api-client)
• mobile (depends on api-client)

Not affected:

• ui-components (no dependency on api-client)
• utils (no dependency on api-client)

CI runs:

• Lint: api-client, web, api, mobile (4 packages)
• Test: api-client, web, api, mobile (4 packages)
• Build: api-client, web, api, mobile (4 packages)
• Deploy: web, api, mobile (3 apps)

Scenario 2: UI Component Change

Changed files:

• packages/ui-components/src/Button.tsx

Affected packages:

• ui-components (changed)
• web (depends on ui-components)
• mobile (depends on ui-components)

Not affected:

• api-client, api, utils

CI runs:

• Lint: ui-components, web, mobile (3 packages)
• Test: ui-components, web, mobile (3 packages)
• Build: ui-components, web, mobile (3 packages)
• Deploy: web, mobile (2 apps)

Turborepo Filter Patterns

# All packages changed since main
npx turbo run build --filter='...[origin/main]'

# Specific package and its dependencies
npx turbo run build --filter='web...'

# Specific package and its dependents
npx turbo run build --filter='...api-client'

# Changed packages only (no dependents)
npx turbo run build --filter='[origin/main]'

# Multiple filters
npx turbo run build --filter='web...' --filter='api...'

Cache Strategy

Local Cache

Turborepo caches task outputs locally in .turbo/:

.turbo/
├── cache/
│   ├── abc123def456...  # Hash-based cache entries
│   └── ...

Remote Cache

Turborepo uploads cache to Vercel:

• Same inputs → Same hash → Cached output
• Shared across team members and CI runs
• Typical 70-90% cache hit rate

Cache Hit Example

# First run (cache miss)
npx turbo run build
# Building packages... (3 minutes)

# Second run (cache hit)
npx turbo run build
# ✓ build:ui-components (cached)
# ✓ build:web (cached)
# ✓ build:api (cached)
# (0.5 seconds)

Troubleshooting

Cache Not Working

Check cache configuration:

npx turbo run build --dry-run

Verify environment variables:

env:
  TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
  TURBO_TEAM: ${{ vars.TURBO_TEAM }}

Affected Detection Not Working

Ensure full git history:

- uses: actions/checkout@v4
  with:
    fetch-depth: 0  # Important!

Slow Dependency Installation

Enable npm cache:

- uses: actions/setup-node@v4
  with:
    cache: 'npm'

Advanced: Dynamic Matrix

Generate matrix based on affected packages:

jobs:
  detect-affected:
    outputs:
      packages: ${{ steps.detect.outputs.packages }}
    steps:
      - id: detect
        run: |
          PACKAGES=$(npx turbo run build --filter='...[origin/main]' --dry-run=json | jq -c '.packages')
          echo "packages=$PACKAGES" >> $GITHUB_OUTPUT

  build:
    needs: detect-affected
    strategy:
      matrix:
        package: ${{ fromJSON(needs.detect-affected.outputs.packages) }}
    steps:
      - run: npm run build --workspace=${{ matrix.package }}

Resources

• Turborepo Documentation: https://turbo.build/repo/docs
• Turborepo Remote Caching: https://turbo.build/repo/docs/core-concepts/remote-caching
• Filter Patterns: https://turbo.build/repo/docs/core-concepts/monorepos/filtering

Caching Strategies for CI Pipelines

Comprehensive guide to caching strategies for faster CI/CD pipelines across multiple platforms.

Table of Contents

1. Cache Fundamentals 2. GitHub Actions Caching 3. GitLab CI Caching 4. Docker Layer Caching 5. Remote Caching (Turborepo/Nx) 6. Multi-Layer Strategies 7. Cache Invalidation

Cache Fundamentals

What to Cache

Dependencies:

• node_modules (Node.js)
• ~/.cargo (Rust)
• ~/.cache/pip (Python)
• ~/go/pkg/mod (Go)

Build Outputs:

• dist/, build/ (compiled code)
• .next/, .nuxt/ (framework builds)
• target/ (Rust)

Intermediate Artifacts:

• tsconfig.tsbuildinfo (TypeScript)
• .vite/ (Vite)
• Coverage data

Cache Key Design

Good cache keys:

# Dependencies - invalidate on lock file change
key: deps-${{ hashFiles('**/package-lock.json') }}

# Build outputs - invalidate on source change
key: build-${{ hashFiles('src/**/*.ts') }}

# OS-specific
key: ${{ runner.os }}-deps-${{ hashFiles('**/package-lock.json') }}

Bad cache keys:

# Too generic - never invalidates
key: node-modules

# Too specific - low cache hit rate
key: build-${{ github.sha }}

Cache Restore Keys

Provide fallback keys:

key: ${{ runner.os }}-deps-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
  ${{ runner.os }}-deps-
  ${{ runner.os }}-

GitHub Actions Caching

Automatic Language Caching

# Node.js
- uses: actions/setup-node@v4
  with:
    node-version: 20
    cache: 'npm'  # Automatically caches ~/.npm

# Python
- uses: actions/setup-python@v5
  with:
    python-version: '3.12'
    cache: 'pip'

# Go
- uses: actions/setup-go@v5
  with:
    go-version: '1.23'
    cache: true

# Rust (use external action)
- uses: Swatinem/rust-cache@v2

Manual Cache Control

- name: Cache dependencies
  uses: actions/cache@v4
  with:
    path: |
      ~/.npm
      ~/.cache
      node_modules
    key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-node-

Cache Limits

• Size limit: 10GB per repository
• Retention: 7 days for inactive caches
• Entries: Unlimited (until 10GB total)

Cache Hit Reporting

- name: Cache dependencies
  id: cache
  uses: actions/cache@v4
  with:
    path: node_modules
    key: deps-${{ hashFiles('package-lock.json') }}

- name: Install dependencies
  if: steps.cache.outputs.cache-hit != 'true'
  run: npm ci

- name: Report cache hit
  run: echo "Cache hit: ${{ steps.cache.outputs.cache-hit }}"

GitLab CI Caching

Basic Cache

cache:
  key: "${CI_COMMIT_REF_SLUG}"
  paths:
    - node_modules/
    - .npm/

build:
  script:
    - npm ci
    - npm run build

Per-Job Cache

test:
  cache:
    key: test-cache
    paths:
      - node_modules/
  script:
    - npm test

build:
  cache:
    key: build-cache
    paths:
      - node_modules/
      - dist/
  script:
    - npm run build

Cache Policies

# Only download cache (don't update)
cache:
  key: deps
  paths:
    - node_modules/
  policy: pull

# Only upload cache (don't download)
cache:
  key: deps
  paths:
    - node_modules/
  policy: push

Fallback Keys

cache:
  key:
    files:
      - package-lock.json
  fallback_keys:
    - default-npm-cache
  paths:
    - node_modules/

Docker Layer Caching

BuildKit Cache

- name: Set up Docker Buildx
  uses: docker/setup-buildx-action@v3

- name: Build with cache
  uses: docker/build-push-action@v5
  with:
    context: .
    cache-from: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache
    cache-to: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache,mode=max

Multi-Stage Dockerfile Optimization

# Base stage - changes infrequently
FROM node:20-alpine AS base
WORKDIR /app

# Dependencies stage - changes when package.json changes
FROM base AS deps
COPY package.json package-lock.json ./
RUN npm ci

# Build stage - changes when source changes
FROM base AS builder
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build

# Production stage - minimal final image
FROM base AS runner
COPY --from=builder /app/dist ./dist
COPY --from=deps /app/node_modules ./node_modules
CMD ["node", "dist/index.js"]

Layer reuse:

• Layer 1: Base image (rarely changes)
• Layer 2: Dependencies (changes when package.json changes)
• Layer 3: Source code (changes frequently)

GitHub Actions Docker Cache

- name: Docker meta
  id: meta
  uses: docker/metadata-action@v5
  with:
    images: ghcr.io/${{ github.repository }}

- name: Build and push
  uses: docker/build-push-action@v5
  with:
    context: .
    push: true
    tags: ${{ steps.meta.outputs.tags }}
    cache-from: |
      type=registry,ref=ghcr.io/${{ github.repository }}:buildcache
      type=registry,ref=ghcr.io/${{ github.repository }}:latest
    cache-to: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache,mode=max

Remote Caching (Turborepo/Nx)

Turborepo Remote Cache

Setup:

npx turbo login
npx turbo link

CI Usage:

- name: Build with remote cache
  run: npx turbo run build
  env:
    TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
    TURBO_TEAM: ${{ vars.TURBO_TEAM }}

Benefits:

• Shared across team and CI
• 70-90% cache hit rate
• Zero configuration after setup

Nx Cloud

Setup:

npx nx connect-to-nx-cloud

CI Usage:

- name: Build with Nx Cloud
  run: npx nx affected -t build
  env:
    NX_CLOUD_ACCESS_TOKEN: ${{ secrets.NX_CLOUD_ACCESS_TOKEN }}

Features:

• Distributed task execution
• Smart cache invalidation
• Performance analytics

Multi-Layer Strategies

Layer 1: Package Manager Cache

- uses: actions/setup-node@v4
  with:
    cache: 'npm'  # Caches ~/.npm

Layer 2: Node Modules Cache

- uses: actions/cache@v4
  with:
    path: node_modules
    key: deps-${{ hashFiles('package-lock.json') }}

Layer 3: Build Tool Cache (Nx)

- uses: actions/cache@v4
  with:
    path: .nx/cache
    key: nx-${{ hashFiles('nx.json') }}

Layer 4: Build Outputs Cache

- uses: actions/cache@v4
  with:
    path: |
      apps/**/dist
      packages/**/dist
    key: build-${{ hashFiles('src/**/*.ts') }}

Layer 5: Remote Cache (Turborepo/Nx)

env:
  TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}

Result: 90%+ cache hit rate across all layers

Cache Invalidation

When to Invalidate

Dependency changes:

key: deps-${{ hashFiles('**/package-lock.json') }}

Source code changes:

key: build-${{ hashFiles('src/**/*.ts') }}

Configuration changes:

key: config-${{ hashFiles('tsconfig.json', 'vite.config.ts') }}

Manual Invalidation

GitHub Actions: Delete cache via API:

gh api repos/{owner}/{repo}/actions/caches/{cache_id} --method DELETE

GitLab CI: Clear project cache:

curl --request POST "https://gitlab.com/api/v4/projects/{id}/jobs/{job_id}/erase"

Time-Based Invalidation

key: cache-${{ runner.os }}-${{ github.run_number }}
restore-keys: |
  cache-${{ runner.os }}-

Uses run number as cache key, falls back to previous runs.

Performance Benchmarks

Node.js Project (50 packages)

No caching:

• npm install: 3 minutes
• Build: 5 minutes
• Total: 8 minutes

With dependency caching:

• npm ci (cache hit): 30 seconds
• Build: 5 minutes
• Total: 5.5 minutes (31% faster)

With dependency + build caching:

• npm ci (cache hit): 30 seconds
• Build (partial cache): 2 minutes
• Total: 2.5 minutes (69% faster)

With Turborepo remote cache:

• npm ci (cache hit): 30 seconds
• Build (full cache): 10 seconds
• Total: 40 seconds (92% faster)

Best Practices

DO:

• Use hash-based cache keys
• Provide restore-keys for fallback
• Cache dependencies separately from builds
• Use remote caching for monorepos
• Monitor cache hit rates

DON'T:

• Use generic cache keys (e.g., "cache")
• Cache sensitive data
• Exceed 10GB limit (GitHub Actions)
• Cache generated files in git
• Forget to invalidate on config changes

Troubleshooting

Low Cache Hit Rate

Check cache key specificity:

# Too specific (low hits)
key: build-${{ github.sha }}

# Better
key: build-${{ hashFiles('src/**') }}

Cache Size Issues

Check cache size:

gh api repos/{owner}/{repo}/actions/cache/usage

Reduce cache size:

# Only cache necessary files
path: |
  node_modules
  !node_modules/.cache

Stale Cache

Force cache refresh:

key: v2-deps-${{ hashFiles('package-lock.json') }}

Increment version prefix to invalidate old caches.

GitHub Actions Patterns

This reference provides detailed patterns for GitHub Actions workflows including reusable workflows, composite actions, matrix strategies, and advanced techniques.

Table of Contents

1. Workflow Syntax Reference 2. Reusable Workflows 3. Composite Actions 4. Matrix Strategies 5. OIDC Federation Setup 6. Secrets Management 7. Artifact Management 8. Concurrency Control

Workflow Syntax Reference

Basic Workflow Structure

name: Workflow Name
on: [push, pull_request]

permissions:
  contents: read
  pull-requests: write

env:
  NODE_VERSION: '20'

jobs:
  job-name:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
      - run: npm test

Trigger Events

on:
  # Simple events
  push:
  pull_request:
  workflow_dispatch:  # Manual trigger

  # Event with filters
  push:
    branches: [main, develop]
    paths:
      - 'src/**'
      - '!**.md'

  # Scheduled runs
  schedule:
    - cron: '0 2 * * *'  # 2 AM UTC daily

  # Release events
  release:
    types: [published, created]

  # Pull request events
  pull_request:
    types: [opened, synchronize, reopened]
    branches: [main]

  # Workflow call (for reusable workflows)
  workflow_call:
    inputs:
      environment:
        required: true
        type: string

Job Dependencies

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - run: npm run build

  test:
    needs: build  # Wait for build
    runs-on: ubuntu-latest
    steps:
      - run: npm test

  deploy:
    needs: [build, test]  # Wait for both
    runs-on: ubuntu-latest
    steps:
      - run: npm run deploy

Conditional Execution

jobs:
  deploy:
    # Only run on main branch pushes
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest
    steps:
      - run: deploy.sh

  # Step-level conditions
  notify:
    runs-on: ubuntu-latest
    steps:
      - name: Notify on failure
        if: failure()
        run: echo "Workflow failed"

      - name: Notify on success
        if: success()
        run: echo "Workflow succeeded"

Reusable Workflows

Creating Reusable Workflow

File: .github/workflows/reusable-deploy.yml

name: Reusable Deploy
on:
  workflow_call:
    inputs:
      environment:
        description: 'Target environment'
        required: true
        type: string
      node-version:
        description: 'Node.js version'
        required: false
        type: string
        default: '20'
      run-tests:
        description: 'Run tests before deploy'
        required: false
        type: boolean
        default: true
    secrets:
      deploy-token:
        required: true
      slack-webhook:
        required: false
    outputs:
      deployment-url:
        description: 'URL of deployed application'
        value: ${{ jobs.deploy.outputs.url }}

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: ${{ inputs.environment }}
    outputs:
      url: ${{ steps.deploy.outputs.url }}
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: ${{ inputs.node-version }}
          cache: 'npm'

      - run: npm ci

      - name: Run tests
        if: inputs.run-tests
        run: npm test

      - name: Build
        run: npm run build

      - name: Deploy
        id: deploy
        run: |
          npm run deploy
          echo "url=https://app.example.com" >> $GITHUB_OUTPUT
        env:
          DEPLOY_TOKEN: ${{ secrets.deploy-token }}

      - name: Notify Slack
        if: always() && secrets.slack-webhook != ''
        uses: slackapi/slack-github-action@v1
        with:
          webhook-url: ${{ secrets.slack-webhook }}
          payload: |
            {
              "text": "Deployment to ${{ inputs.environment }}: ${{ job.status }}"
            }

Calling Reusable Workflow

File: .github/workflows/production.yml

name: Production Deploy
on:
  push:
    branches: [main]

jobs:
  deploy-staging:
    uses: ./.github/workflows/reusable-deploy.yml
    with:
      environment: staging
      node-version: '20'
      run-tests: true
    secrets:
      deploy-token: ${{ secrets.STAGING_DEPLOY_TOKEN }}

  deploy-production:
    needs: deploy-staging
    uses: ./.github/workflows/reusable-deploy.yml
    with:
      environment: production
      node-version: '20'
      run-tests: false  # Already tested in staging
    secrets:
      deploy-token: ${{ secrets.PRODUCTION_DEPLOY_TOKEN }}
      slack-webhook: ${{ secrets.SLACK_WEBHOOK }}

  verify:
    needs: deploy-production
    runs-on: ubuntu-latest
    steps:
      - name: Verify deployment
        run: |
          URL="${{ needs.deploy-production.outputs.deployment-url }}"
          curl -f "$URL/health" || exit 1

Composite Actions

Creating Composite Action

File: .github/actions/setup-node-with-cache/action.yml

name: Setup Node with Cache
description: Setup Node.js with dependency caching and installation
author: Your Team

inputs:
  node-version:
    description: Node.js version
    required: false
    default: '20'
  package-manager:
    description: Package manager (npm, yarn, pnpm)
    required: false
    default: 'npm'
  install-dependencies:
    description: Install dependencies after setup
    required: false
    default: 'true'

outputs:
  cache-hit:
    description: Whether cache was hit
    value: ${{ steps.cache.outputs.cache-hit }}

runs:
  using: composite
  steps:
    - name: Setup Node.js
      uses: actions/setup-node@v4
      with:
        node-version: ${{ inputs.node-version }}
        cache: ${{ inputs.package-manager }}

    - name: Cache node modules
      id: cache
      uses: actions/cache@v4
      with:
        path: node_modules
        key: ${{ runner.os }}-${{ inputs.package-manager }}-${{ hashFiles('**/package-lock.json', '**/yarn.lock', '**/pnpm-lock.yaml') }}

    - name: Install dependencies
      if: inputs.install-dependencies == 'true' && steps.cache.outputs.cache-hit != 'true'
      shell: bash
      run: |
        case "${{ inputs.package-manager }}" in
          npm) npm ci ;;
          yarn) yarn install --frozen-lockfile ;;
          pnpm) pnpm install --frozen-lockfile ;;
        esac

    - name: Report status
      shell: bash
      run: |
        echo "✅ Node.js ${{ inputs.node-version }} setup complete"
        echo "📦 Package manager: ${{ inputs.package-manager }}"
        echo "💾 Cache hit: ${{ steps.cache.outputs.cache-hit }}"

Using Composite Action

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: ./.github/actions/setup-node-with-cache
        with:
          node-version: '20'
          package-manager: 'pnpm'

      - run: pnpm run build
      - run: pnpm test

Matrix Strategies

Basic Matrix

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]
        node-version: [18, 20, 22]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
      - run: npm test

Result: 9 jobs (3 OS × 3 versions)

Matrix with Include/Exclude

strategy:
  matrix:
    os: [ubuntu-latest, windows-latest, macos-latest]
    node-version: [18, 20, 22]
    # Exclude specific combinations
    exclude:
      - os: macos-latest
        node-version: 18
    # Add specific combinations
    include:
      - os: ubuntu-latest
        node-version: 22
        experimental: true

Matrix with Custom Variables

strategy:
  matrix:
    config:
      - { os: ubuntu-latest, python: '3.10', toxenv: py310 }
      - { os: ubuntu-latest, python: '3.11', toxenv: py311 }
      - { os: ubuntu-latest, python: '3.12', toxenv: py312 }
      - { os: windows-latest, python: '3.12', toxenv: py312-windows }
      - { os: macos-latest, python: '3.12', toxenv: py312-macos }

steps:
  - uses: actions/setup-python@v5
    with:
      python-version: ${{ matrix.config.python }}

  - run: tox -e ${{ matrix.config.toxenv }}

Dynamic Matrix (JSON)

jobs:
  generate-matrix:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
      - uses: actions/checkout@v4
      - id: set-matrix
        run: |
          # Generate matrix from changed packages
          MATRIX=$(python scripts/get_changed_packages.py)
          echo "matrix=$MATRIX" >> $GITHUB_OUTPUT

  test:
    needs: generate-matrix
    runs-on: ubuntu-latest
    strategy:
      matrix: ${{ fromJSON(needs.generate-matrix.outputs.matrix) }}
    steps:
      - run: npm test --workspace=${{ matrix.package }}

Matrix with Fail-Fast Control

strategy:
  fail-fast: false  # Continue other jobs even if one fails
  matrix:
    os: [ubuntu-latest, windows-latest, macos-latest]
    node-version: [18, 20, 22]
  max-parallel: 4  # Limit concurrent jobs

OIDC Federation Setup

AWS OIDC Configuration

Step 1: Create IAM OIDC Identity Provider

# Via AWS CLI
aws iam create-open-id-connect-provider \
  --url https://token.actions.githubusercontent.com \
  --client-id-list sts.amazonaws.com \
  --thumbprint-list 6938fd4d98bab03faadb97b34396831e3780aea1

Step 2: Create IAM Role with Trust Policy

Trust policy JSON:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repo:myorg/myrepo:*"
        }
      }
    }
  ]
}

More restrictive conditions:

"Condition": {
  "StringEquals": {
    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
    "token.actions.githubusercontent.com:sub": "repo:myorg/myrepo:ref:refs/heads/main"
  }
}

Step 3: Attach Permission Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::my-deploy-bucket",
        "arn:aws:s3:::my-deploy-bucket/*"
      ]
    }
  ]
}

Step 4: Use in Workflow

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write  # Required for OIDC
      contents: read
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/GitHubActionsRole
          role-session-name: GitHubActions-${{ github.run_id }}
          aws-region: us-east-1

      - name: Verify credentials
        run: aws sts get-caller-identity

      - name: Deploy
        run: aws s3 sync ./dist s3://my-deploy-bucket

GCP OIDC Configuration

Step 1: Create Workload Identity Pool

gcloud iam workload-identity-pools create github-pool \
  --location="global" \
  --description="GitHub Actions pool"

Step 2: Create Workload Identity Provider

gcloud iam workload-identity-pools providers create-oidc github-provider \
  --location="global" \
  --workload-identity-pool="github-pool" \
  --issuer-uri="https://token.actions.githubusercontent.com" \
  --attribute-mapping="google.subject=assertion.sub,attribute.actor=assertion.actor,attribute.repository=assertion.repository" \
  --attribute-condition="assertion.repository=='myorg/myrepo'"

Step 3: Grant Service Account Access

gcloud iam service-accounts add-iam-policy-binding deploy-sa@myproject.iam.gserviceaccount.com \
  --role="roles/iam.workloadIdentityUser" \
  --member="principalSet://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/github-pool/attribute.repository/myorg/myrepo"

Step 4: Use in Workflow

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: actions/checkout@v4

      - name: Authenticate to Google Cloud
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: 'projects/123456789/locations/global/workloadIdentityPools/github-pool/providers/github-provider'
          service_account: 'deploy-sa@myproject.iam.gserviceaccount.com'

      - name: Deploy to Cloud Run
        run: gcloud run deploy myapp --image gcr.io/myproject/myapp:latest

Azure OIDC Configuration

Step 1: Create Azure AD Application

az ad app create --display-name github-actions-app

Step 2: Create Service Principal

az ad sp create --id <APP_ID>

Step 3: Create Federated Credential

az ad app federated-credential create \
  --id <APP_ID> \
  --parameters '{
    "name": "github-actions-credential",
    "issuer": "https://token.actions.githubusercontent.com",
    "subject": "repo:myorg/myrepo:ref:refs/heads/main",
    "audiences": ["api://AzureADTokenExchange"]
  }'

Step 4: Use in Workflow

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: actions/checkout@v4

      - name: Azure Login
        uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

      - name: Deploy to Azure
        run: az webapp up --name myapp --resource-group myrg

Secrets Management

GitHub Encrypted Secrets

Store secrets at repository, environment, or organization level.

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Use secret
        run: ./deploy.sh
        env:
          API_KEY: ${{ secrets.API_KEY }}

Never expose secrets in logs:

# BAD - secret exposed
- run: echo "API_KEY=${{ secrets.API_KEY }}"

# GOOD - secret used in env
- run: ./script.sh
  env:
    API_KEY: ${{ secrets.API_KEY }}

Vault Integration

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: hashicorp/vault-action@v3
        with:
          url: https://vault.example.com
          method: jwt
          role: github-actions
          secrets: |
            secret/data/production/api API_KEY | API_KEY ;
            secret/data/production/db DB_PASSWORD | DB_PASSWORD

      - name: Use secrets from Vault
        run: ./deploy.sh
        env:
          API_KEY: ${{ env.API_KEY }}
          DB_PASSWORD: ${{ env.DB_PASSWORD }}

Artifact Management

Upload Artifacts

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - run: npm run build

      - uses: actions/upload-artifact@v4
        with:
          name: build-artifacts
          path: |
            dist/
            build/
          retention-days: 30
          if-no-files-found: error

Download Artifacts

jobs:
  deploy:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: build-artifacts
          path: dist/

      - run: ls -R dist/
      - run: ./deploy.sh

Share Between Jobs

jobs:
  build:
    outputs:
      version: ${{ steps.version.outputs.version }}
    steps:
      - id: version
        run: echo "version=1.2.3" >> $GITHUB_OUTPUT

  deploy:
    needs: build
    steps:
      - run: echo "Deploying version ${{ needs.build.outputs.version }}"

Concurrency Control

Cancel In-Progress Runs

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

For PRs, this cancels previous runs when new commits are pushed.

Environment-Specific Concurrency

jobs:
  deploy-production:
    runs-on: ubuntu-latest
    environment: production
    concurrency:
      group: production-deploy
      cancel-in-progress: false  # Never cancel production deploys

Branch-Specific Concurrency

concurrency:
  group: deploy-${{ github.ref_name }}
  cancel-in-progress: ${{ github.ref_name != 'main' }}

Main branch deploys never cancel; feature branch deploys cancel previous runs.

Monorepo CI Strategies

Strategies for building efficient CI pipelines in monorepos with affected detection, intelligent caching, and parallel execution.

Table of Contents

1. Monorepo Challenges 2. Affected Detection 3. Turborepo Strategy 4. Nx Strategy 5. Bazel Strategy 6. Change Detection Algorithms 7. Performance Optimization

Monorepo Challenges

Problems with Naive CI

Build Everything:

# BAD: Rebuilds all packages on every commit
- run: npm run build --workspaces

Impact:

• 100+ packages = 30+ minute builds
• Wastes CI minutes (60-80% unnecessary builds)
• Slow feedback loop for developers
• Expensive cloud CI costs

Monorepo Scale Examples

Affected Detection

Core Concept

Affected Packages = Changed Packages + Dependent Packages

Example dependency graph:

@myorg/api-client (changed)
    ├── @myorg/frontend (depends on api-client) ← affected
    └── @myorg/mobile (depends on api-client) ← affected

@myorg/backend (changed)
    └── @myorg/admin (depends on backend) ← affected

@myorg/docs (unchanged) ← NOT affected

Changed files:

• packages/api-client/src/index.ts
• packages/backend/src/server.ts

Affected packages:

• api-client (changed)
• frontend (depends on api-client)
• mobile (depends on api-client)
• backend (changed)
• admin (depends on backend)

Not affected:

• docs (no changes, no dependencies on changed packages)

Dependency Graph Construction

Package Dependencies (package.json):

{
  "name": "@myorg/frontend",
  "dependencies": {
    "@myorg/api-client": "workspace:*",
    "@myorg/ui-components": "workspace:*"
  }
}

Build Dependencies (turbo.json / nx.json):

{
  "pipeline": {
    "build": {
      "dependsOn": ["^build"]  // Build dependencies first
    },
    "test": {
      "dependsOn": ["build"]   // Test depends on build
    }
  }
}

Full Graph:

ui-components#build → frontend#build → frontend#test
api-client#build    ↗

Turborepo Strategy

Setup

npm install turbo --save-dev

turbo.json:

{
  "$schema": "https://turbo.build/schema.json",
  "pipeline": {
    "build": {
      "dependsOn": ["^build"],
      "outputs": ["dist/**", ".next/**", "build/**"]
    },
    "test": {
      "dependsOn": ["build"],
      "outputs": ["coverage/**"],
      "cache": false  // Don't cache test results (always run)
    },
    "lint": {
      "outputs": []
    },
    "deploy": {
      "dependsOn": ["build", "test"],
      "cache": false
    }
  },
  "remoteCache": {
    "signature": true
  }
}

CI Integration (GitHub Actions)

name: CI
on:
  pull_request:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Required for comparison

      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Install dependencies
        run: npm ci

      # Build only affected packages since main
      - name: Build affected
        run: npx turbo run build --filter='...[origin/main]'
        env:
          TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
          TURBO_TEAM: ${{ vars.TURBO_TEAM }}

      # Test only affected packages
      - name: Test affected
        run: npx turbo run test --filter='...[origin/main]'
        env:
          TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
          TURBO_TEAM: ${{ vars.TURBO_TEAM }}

Filter Patterns

# All packages changed since main
npx turbo run build --filter='...[origin/main]'

# Specific package and its dependencies
npx turbo run build --filter='@myorg/frontend...'

# Specific package and its dependents
npx turbo run build --filter='...@myorg/api-client'

# Changed packages only (no dependents)
npx turbo run build --filter='[origin/main]'

# Multiple filters
npx turbo run build --filter='@myorg/frontend...' --filter='@myorg/mobile...'

Remote Caching

Benefits:

• Share build artifacts across CI runs
• Share artifacts across team members
• 70-90% cache hit rate in practice

Setup (Vercel):

npx turbo login
npx turbo link

In CI:

env:
  TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}  # From Vercel
  TURBO_TEAM: ${{ vars.TURBO_TEAM }}       # From Vercel

Cache is automatically used for matching builds.

Turborepo + Docker

FROM node:20-alpine AS base

FROM base AS builder
WORKDIR /app
RUN npm install -g turbo
COPY . .
RUN turbo prune --scope=@myorg/api --docker

FROM base AS installer
WORKDIR /app
COPY --from=builder /app/out/json/ .
COPY --from=builder /app/out/package-lock.json ./package-lock.json
RUN npm ci

COPY --from=builder /app/out/full/ .
RUN npx turbo run build --filter=@myorg/api...

FROM base AS runner
WORKDIR /app
COPY --from=installer /app .
CMD ["node", "apps/api/dist/index.js"]

Nx Strategy

Setup

npx create-nx-workspace@latest myorg --preset=npm

nx.json:

{
  "targetDefaults": {
    "build": {
      "dependsOn": ["^build"],
      "outputs": ["{projectRoot}/dist"],
      "cache": true
    },
    "test": {
      "dependsOn": ["build"],
      "cache": true
    }
  },
  "tasksRunnerOptions": {
    "default": {
      "runner": "nx-cloud",
      "options": {
        "cacheableOperations": ["build", "test", "lint"],
        "accessToken": "NX_CLOUD_TOKEN"
      }
    }
  }
}

CI Integration (GitHub Actions)

name: CI
on: [pull_request]

jobs:
  affected:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      # Derive base/head for comparison
      - name: Derive SHAs
        id: setSHAs
        uses: nrwl/nx-set-shas@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: 'npm'

      - run: npm ci

      # Run tasks only for affected projects
      - name: Affected build
        run: npx nx affected -t build --base=${{ steps.setSHAs.outputs.base }} --head=${{ steps.setSHAs.outputs.head }}
        env:
          NX_CLOUD_ACCESS_TOKEN: ${{ secrets.NX_CLOUD_ACCESS_TOKEN }}

      - name: Affected test
        run: npx nx affected -t test --base=${{ steps.setSHAs.outputs.base }} --head=${{ steps.setSHAs.outputs.head }}
        env:
          NX_CLOUD_ACCESS_TOKEN: ${{ secrets.NX_CLOUD_ACCESS_TOKEN }}

      - name: Affected lint
        run: npx nx affected -t lint --base=${{ steps.setSHAs.outputs.base }} --head=${{ steps.setSHAs.outputs.head }}
        env:
          NX_CLOUD_ACCESS_TOKEN: ${{ secrets.NX_CLOUD_ACCESS_TOKEN }}

Nx Commands

# Show affected projects
npx nx affected:graph --base=main --head=HEAD

# Run target for affected projects
npx nx affected -t build --base=main

# Run multiple targets
npx nx affected -t build,test,lint --base=main

# Run specific project
npx nx run @myorg/frontend:build

# Run target for all projects
npx nx run-many -t build --all

Nx Cloud Features

Distributed Task Execution:

• Nx Cloud distributes tasks across multiple agents
• 10x speedup for large monorepos

# .github/workflows/ci.yml
- name: Start Nx Cloud agents
  run: npx nx-cloud start-ci-run --distribute-on="8 linux-medium-js"

- name: Run affected tasks
  run: npx nx affected -t build,test,lint --base=main --parallel=3

- name: Stop Nx Cloud agents
  run: npx nx-cloud stop-all-agents

Computation Caching:

• Hash inputs (source files, dependencies, environment)
• Store outputs in cloud
• Restore exact outputs on cache hit

Performance:

• 70-90% cache hit rate
• 50-80% reduction in CI time

Bazel Strategy

When to Use Bazel

Use Bazel if:

• 1000+ packages (Google-scale monorepo)
• Multi-language (Java, Go, Rust, Python, C++)
• Need hermetic builds (fully reproducible)
• Have dedicated build team

Skip Bazel if:

• <100 packages
• JavaScript/TypeScript only (use Turborepo/Nx)
• Prototype/startup (steep learning curve)

Bazel Basics

BUILD file:

# packages/api-client/BUILD
load("@npm//@bazel/typescript:index.bzl", "ts_library")

ts_library(
    name = "api-client",
    srcs = glob(["src/**/*.ts"]),
    deps = [
        "@npm//axios",
        "@npm//@types/node",
    ],
    module_name = "@myorg/api-client",
    visibility = ["//visibility:public"],
)

WORKSPACE file:

workspace(name = "myorg")

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

# Node.js rules
http_archive(
    name = "build_bazel_rules_nodejs",
    sha256 = "...",
    urls = ["https://github.com/bazelbuild/rules_nodejs/releases/..."],
)

Affected Targets

# Show changed targets
bazel query 'rdeps(//..., set($(git diff --name-only main...HEAD | sed "s|^|//|" | sed "s|/BUILD$||")))'

# Build affected targets
bazel build $(bazel query '...')

Remote Execution

Bazel can execute builds on remote servers:

bazel build //... \
  --remote_executor=grpcs://remote.buildbuddy.io \
  --remote_cache=grpcs://remote.buildbuddy.io \
  --remote_upload_local_results=true

Benefits:

• Consistent build environment
• Massive parallelization (100+ machines)
• Shared remote cache across team

Change Detection Algorithms

Git-Based Detection

1. Get Changed Files:

git diff --name-only origin/main...HEAD

Output:

packages/api-client/src/index.ts
packages/api-client/package.json
packages/backend/src/server.ts

2. Map Files to Packages:

function getAffectedPackages(changedFiles: string[]): string[] {
  const packages = new Set<string>();

  for (const file of changedFiles) {
    // Find nearest package.json
    const pkg = findNearestPackageJson(file);
    if (pkg) packages.add(pkg.name);
  }

  return Array.from(packages);
}

Result: ['@myorg/api-client', '@myorg/backend']

3. Build Dependency Graph:

function buildDependencyGraph(): Map<string, Set<string>> {
  const graph = new Map();

  for (const pkg of allPackages) {
    const deps = new Set();

    // Add package dependencies
    for (const dep of pkg.dependencies) {
      if (isWorkspacePackage(dep)) {
        deps.add(dep);
      }
    }

    graph.set(pkg.name, deps);
  }

  return graph;
}

4. Find Affected Packages (BFS):

function findAffectedPackages(
  changedPackages: string[],
  graph: Map<string, Set<string>>
): string[] {
  const affected = new Set(changedPackages);
  const queue = [...changedPackages];

  while (queue.length > 0) {
    const current = queue.shift()!;

    // Find packages that depend on current
    for (const [pkg, deps] of graph.entries()) {
      if (deps.has(current) && !affected.has(pkg)) {
        affected.add(pkg);
        queue.push(pkg);
      }
    }
  }

  return Array.from(affected);
}

Content-Based Detection

Turborepo/Nx approach:

1. Compute task hash:

function computeTaskHash(task: Task): string {
  const inputs = [
    hashSourceFiles(task.package),
    hashDependencies(task.package),
    hashBuildConfig(task),
    hashEnvironment(task),
  ];

  return sha256(inputs.join(''));
}

2. Check cache:

async function runTask(task: Task) {
  const hash = computeTaskHash(task);

  // Check local cache
  const cached = await getFromCache(hash);
  if (cached) {
    console.log(`Cache hit for ${task.name}`);
    return cached;
  }

  // Execute task
  const output = await executeTask(task);

  // Store in cache
  await putInCache(hash, output);

  return output;
}

3. Cache hit = skip execution

Same inputs → Same hash → Cached output restored

Performance Optimization

Parallel Execution

Turborepo:

# Run tasks in parallel (default)
npx turbo run build test --parallel

# Limit parallelism
npx turbo run build --concurrency=4

Nx:

# Parallel execution (default)
npx nx affected -t build --parallel=3

# Max parallelism
npx nx affected -t build --parallel=true

Pipeline Optimization

Sequential (slow):

lint (2m) → test (5m) → build (3m) = 10m total

Parallel (fast):

lint (2m) ──┐
            ├─→ build (3m) = 5m total
test (5m) ──┘

turbo.json (parallel lint/test):

{
  "pipeline": {
    "lint": {},
    "test": {},
    "build": {
      "dependsOn": ["lint", "test"]
    }
  }
}

Incremental Builds

TypeScript incremental compilation:

{
  "compilerOptions": {
    "incremental": true,
    "tsBuildInfoFile": ".tsbuildinfo"
  }
}

Cache .tsbuildinfo in CI:

- uses: actions/cache@v4
  with:
    path: '**/tsconfig.tsbuildinfo'
    key: tsc-${{ hashFiles('**/tsconfig.json') }}

CI-Specific Optimizations

Shallow clones:

# Fetch minimal history
- uses: actions/checkout@v4
  with:
    fetch-depth: 1  # Only latest commit

Sparse checkouts (large monorepos):

- uses: actions/checkout@v4
  with:
    sparse-checkout: |
      packages/api-client
      packages/frontend

Split jobs by affected packages:

jobs:
  detect-affected:
    outputs:
      packages: ${{ steps.detect.outputs.packages }}
    steps:
      - id: detect
        run: |
          PACKAGES=$(npx turbo run build --filter='...[origin/main]' --dry-run | jq -c '.packages')
          echo "packages=$PACKAGES" >> $GITHUB_OUTPUT

  build-package:
    needs: detect-affected
    strategy:
      matrix:
        package: ${{ fromJSON(needs.detect-affected.outputs.packages) }}
    steps:
      - run: npm run build --workspace=${{ matrix.package }}

Best Practices

DO:

• Use affected detection for all monorepo CI
• Enable remote caching (Turborepo/Nx Cloud)
• Cache build artifacts between CI runs
• Parallelize independent tasks
• Use shallow clones when possible

DON'T:

• Build all packages on every commit
• Skip dependency graph construction
• Ignore cache optimization
• Run serial pipelines unnecessarily
• Use Bazel for small JavaScript monorepos

Tool Comparison

SLSA Security Framework

Supply-chain Levels for Software Artifacts (SLSA) is a security framework for ensuring the integrity of software artifacts throughout the software supply chain.

Table of Contents

1. SLSA Overview 2. SLSA Levels 3. Provenance Generation 4. Attestation Verification 5. GitHub Actions Integration 6. GitLab CI Integration 7. Cosign Integration

SLSA Overview

SLSA (pronounced "salsa") prevents tampering and unauthorized modifications to software packages. It provides:

• Provenance: Record of how software was built
• Integrity: Cryptographic verification
• Non-falsifiability: Build system generates provenance, not developers
• Auditability: Complete build trail

Why SLSA Matters

Supply Chain Attacks Are Real:

• SolarWinds (2020): Malicious code injected in build process
• Codecov (2021): Bash Uploader script compromised
• Log4Shell (2021): Vulnerability in widely-used library

Compliance Requirements:

• NIST SSDF (Secure Software Development Framework)
• PCI DSS 4.0 (Payment Card Industry Data Security Standard)
• Executive Order 14028 (Federal software procurement)

SLSA Levels

Level 1: Provenance Exists

Requirements:

• Build process generates provenance
• Provenance is available to artifact consumers

Provides:

• Basic record that artifact was built
• Minimal security assurance

Example:

# GitHub Actions - Basic provenance
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm run build
      - name: Generate build record
        run: |
          echo "{\"artifact\": \"myapp.tar.gz\", \"built_at\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"}" > provenance.json
      - uses: actions/upload-artifact@v4
        with:
          name: provenance
          path: provenance.json

Level 2: Hosted Build Service

Requirements:

• Build service (not local machine) generates provenance
• Service-generated provenance (tamper-resistant)
• Authenticated provenance

Provides:

• Provenance cannot be modified by developers
• Build environment is known

Example: GitHub Actions automatically satisfies Level 2 (hosted build service)

Level 3: Hardened Build Platforms (RECOMMENDED)

Requirements:

• Hardened build environment
• Non-falsifiable provenance (cryptographically signed)
• Isolated build execution
• Parameterless build definition

Provides:

• High confidence in artifact integrity
• Verifiable chain of custody
• Industry standard for production

Example: See GitHub Actions Integration

Level 4: Two-Party Review + Hermetic Builds

Requirements:

• Two-person review for all changes
• Hermetic, reproducible builds
• Dependencies fetched only from trusted sources

Provides:

• Maximum supply chain security
• Fully reproducible builds

Status: Future state, limited tooling availability

Provenance Generation

SLSA Provenance Format

Provenance is a JSON document describing how an artifact was built:

{
  "_type": "https://in-toto.io/Statement/v0.1",
  "subject": [
    {
      "name": "ghcr.io/myorg/myapp",
      "digest": {
        "sha256": "abcd1234..."
      }
    }
  ],
  "predicateType": "https://slsa.dev/provenance/v0.2",
  "predicate": {
    "builder": {
      "id": "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v1.10.0"
    },
    "buildType": "https://slsa.dev/container-based-build/v0.1",
    "invocation": {
      "configSource": {
        "uri": "git+https://github.com/myorg/myapp@refs/heads/main",
        "digest": {
          "sha1": "efgh5678..."
        },
        "entryPoint": ".github/workflows/release.yml"
      }
    },
    "metadata": {
      "buildStartedOn": "2025-12-04T10:30:00Z",
      "buildFinishedOn": "2025-12-04T10:35:00Z",
      "completeness": {
        "parameters": true,
        "environment": true,
        "materials": true
      }
    },
    "materials": [
      {
        "uri": "git+https://github.com/myorg/myapp",
        "digest": {
          "sha1": "efgh5678..."
        }
      }
    ]
  }
}

Key Provenance Fields

subject: What was built

• name: Artifact identifier (image name, package name)
• digest: Content hash (sha256)

builder.id: Who built it

• URI identifying the build system
• Includes version/commit for reproducibility

invocation.configSource: Build instructions

• Repository URI
• Commit hash
• Workflow file path

materials: Build inputs

• Source code commit
• Base images
• Dependencies

Attestation Verification

Verification Process

1. Download artifact and attestation 2. Verify attestation signature (cryptographic) 3. Check builder identity (trusted build system) 4. Verify subject matches artifact (digest comparison) 5. Validate materials (expected source repo, commit)

Manual Verification Example

# Download container image
docker pull ghcr.io/myorg/myapp@sha256:abcd1234...

# Verify SLSA provenance with cosign
cosign verify-attestation \
  --type slsaprovenance \
  --certificate-identity-regexp "^https://github.com/slsa-framework" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  ghcr.io/myorg/myapp@sha256:abcd1234...

# Output shows verified provenance

Automated Verification in Deployment

# Kubernetes admission controller
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sImageProvenance
metadata:
  name: require-slsa-l3
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    minimumLevel: 3
    allowedBuilders:
      - "https://github.com/slsa-framework/slsa-github-generator"

GitHub Actions Integration

Container Image with SLSA Level 3

name: SLSA Build
on:
  push:
    tags: ['v*']

permissions: read-all

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    outputs:
      digest: ${{ steps.build.outputs.digest }}
    steps:
      - uses: actions/checkout@v4

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push container
        id: build
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}

      - name: Extract digest
        run: |
          echo "digest=${{ steps.build.outputs.digest }}" >> $GITHUB_OUTPUT

  provenance:
    needs: build
    permissions:
      id-token: write  # For signing
      actions: read    # For reading workflow context
      packages: write  # For attaching attestation
    uses: slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@v1.10.0
    with:
      image: ghcr.io/${{ github.repository }}
      digest: ${{ needs.build.outputs.digest }}
      registry-username: ${{ github.actor }}
    secrets:
      registry-password: ${{ secrets.GITHUB_TOKEN }}

Binary Artifacts with SLSA Level 3

name: SLSA Binary Build
on:
  push:
    tags: ['v*']

permissions: read-all

jobs:
  build:
    runs-on: ubuntu-latest
    outputs:
      hashes: ${{ steps.hash.outputs.hashes }}
    steps:
      - uses: actions/checkout@v4

      - name: Build binaries
        run: |
          make build-linux
          make build-windows
          make build-macos

      - name: Generate hashes
        id: hash
        run: |
          cd dist/
          sha256sum * > checksums.txt
          echo "hashes=$(cat checksums.txt | base64 -w0)" >> $GITHUB_OUTPUT

      - uses: actions/upload-artifact@v4
        with:
          name: binaries
          path: dist/*

  provenance:
    needs: build
    permissions:
      id-token: write
      actions: read
      contents: write  # For attaching to release
    uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v1.10.0
    with:
      base64-subjects: "${{ needs.build.outputs.hashes }}"
      upload-assets: true

NPM Package with SLSA Level 3

name: SLSA NPM Publish
on:
  release:
    types: [published]

permissions: read-all

jobs:
  build:
    runs-on: ubuntu-latest
    outputs:
      package-name: ${{ steps.detect.outputs.package-name }}
      package-digest: ${{ steps.hash.outputs.digest }}
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - run: npm ci
      - run: npm run build
      - run: npm pack

      - name: Detect package
        id: detect
        run: |
          PKG=$(ls *.tgz)
          echo "package-name=$PKG" >> $GITHUB_OUTPUT

      - name: Hash package
        id: hash
        run: |
          DIGEST=$(sha256sum ${{ steps.detect.outputs.package-name }} | awk '{print $1}')
          echo "digest=$DIGEST" >> $GITHUB_OUTPUT

      - uses: actions/upload-artifact@v4
        with:
          name: package
          path: "*.tgz"

  provenance:
    needs: build
    permissions:
      id-token: write
      actions: read
    uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v1.10.0
    with:
      base64-subjects: "${{ needs.build.outputs.package-digest }}"

  publish:
    needs: [build, provenance]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: package

      - uses: actions/download-artifact@v4
        with:
          name: provenance

      - uses: actions/setup-node@v4
        with:
          node-version: 20
          registry-url: 'https://registry.npmjs.org'

      - name: Publish to NPM
        run: npm publish ${{ needs.build.outputs.package-name }}
        env:
          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

GitLab CI Integration

GitLab CI does not have native SLSA Level 3 support. Manual implementation required.

Manual Provenance Generation

# .gitlab-ci.yml
stages:
  - build
  - attest

build:
  stage: build
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - echo "DIGEST=$(docker inspect --format='{{.Id}}' $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA)" >> build.env
  artifacts:
    reports:
      dotenv: build.env

attest:
  stage: attest
  image: gcr.io/projectsigstore/cosign:latest
  script:
    - |
      cat > provenance.json <<EOF
      {
        "_type": "https://in-toto.io/Statement/v0.1",
        "subject": [{
          "name": "$CI_REGISTRY_IMAGE",
          "digest": {"sha256": "$DIGEST"}
        }],
        "predicateType": "https://slsa.dev/provenance/v0.2",
        "predicate": {
          "builder": {"id": "https://gitlab.com/$CI_PROJECT_PATH"},
          "buildType": "https://gitlab.com/gitlab-ci",
          "invocation": {
            "configSource": {
              "uri": "git+$CI_PROJECT_URL",
              "digest": {"sha1": "$CI_COMMIT_SHA"}
            }
          }
        }
      }
      EOF
    - cosign attest --key cosign.key --predicate provenance.json $CI_REGISTRY_IMAGE@sha256:$DIGEST

Cosign Integration

Cosign is a tool for signing and verifying container images and other artifacts.

Signing with Cosign

# Generate key pair (one-time)
cosign generate-key-pair

# Sign container image
cosign sign --key cosign.key ghcr.io/myorg/myapp@sha256:abcd1234...

# Sign with OIDC (keyless)
cosign sign ghcr.io/myorg/myapp@sha256:abcd1234...

Attaching Attestations

# Attach SLSA provenance
cosign attest --key cosign.key \
  --predicate provenance.json \
  --type slsaprovenance \
  ghcr.io/myorg/myapp@sha256:abcd1234...

# Attach SBOM
cosign attest --key cosign.key \
  --predicate sbom.spdx.json \
  --type spdx \
  ghcr.io/myorg/myapp@sha256:abcd1234...

Verifying Signatures

# Verify with public key
cosign verify --key cosign.pub ghcr.io/myorg/myapp@sha256:abcd1234...

# Verify with OIDC (keyless)
cosign verify \
  --certificate-identity-regexp "^https://github.com/myorg" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  ghcr.io/myorg/myapp@sha256:abcd1234...

Policy Enforcement

# Verify attestation exists
cosign verify-attestation \
  --type slsaprovenance \
  --key cosign.pub \
  ghcr.io/myorg/myapp@sha256:abcd1234...

# Extract and validate provenance content
cosign verify-attestation \
  --type slsaprovenance \
  --key cosign.pub \
  ghcr.io/myorg/myapp@sha256:abcd1234... | jq '.payload | @base64d | fromjson'

GitHub Actions with Cosign

jobs:
  sign:
    runs-on: ubuntu-latest
    permissions:
      id-token: write  # For keyless signing
      packages: write
    steps:
      - name: Install Cosign
        uses: sigstore/cosign-installer@v3

      - name: Login to registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build image
        run: docker build -t ghcr.io/${{ github.repository }}:${{ github.sha }} .

      - name: Push image
        run: docker push ghcr.io/${{ github.repository }}:${{ github.sha }}

      - name: Sign image (keyless)
        run: |
          cosign sign --yes \
            ghcr.io/${{ github.repository }}@$(docker inspect --format='{{index .RepoDigests 0}}' ghcr.io/${{ github.repository }}:${{ github.sha }} | cut -d@ -f2)

      - name: Generate and attach SBOM
        run: |
          syft ghcr.io/${{ github.repository }}:${{ github.sha }} -o spdx-json > sbom.spdx.json
          cosign attest --yes --predicate sbom.spdx.json --type spdx \
            ghcr.io/${{ github.repository }}@$(docker inspect --format='{{index .RepoDigests 0}}' ghcr.io/${{ github.repository }}:${{ github.sha }} | cut -d@ -f2)

Best Practices

DO:

• Achieve SLSA Level 3 for production artifacts
• Use OIDC (keyless signing) when possible
• Verify provenance before deployment
• Include SBOMs alongside provenance
• Store keys in hardware security modules (HSMs) if using key-based signing

DON'T:

• Skip provenance generation for public releases
• Use developer-generated provenance (Level 1 only)
• Store signing keys in repositories
• Trust artifacts without verification
• Implement custom provenance formats (use SLSA standard)

Resources

• SLSA Specification: https://slsa.dev
• GitHub SLSA Generator: https://github.com/slsa-framework/slsa-github-generator
• Cosign Documentation: https://docs.sigstore.dev/cosign/overview
• In-Toto: https://in-toto.io
• Sigstore: https://www.sigstore.dev