Deploying On Gcp

GCP Patterns

Build applications and infrastructure using Google Cloud Platform services with appropriate service selection, architecture patterns, and best practices.

Purpose

This skill provides decision frameworks and implementation patterns for Google Cloud Platform (GCP) services across compute, storage, databases, data analytics, machine learning, networking, and security. It guides service selection based on workload requirements and demonstrates production-ready patterns using Terraform, Python SDKs, and gcloud CLI.

When to Use

Use this skill when:

• Selecting GCP compute services (Cloud Run, GKE, Cloud Functions, Compute Engine, App Engine)
• Choosing storage or database services (Cloud Storage, Cloud SQL, Spanner, Firestore, Bigtable, BigQuery)
• Designing data analytics pipelines (BigQuery, Pub/Sub, Dataflow, Dataproc, Composer)
• Implementing ML workflows (Vertex AI, AutoML, pre-trained APIs)
• Architecting network infrastructure (VPC, Load Balancing, CDN, Cloud Armor)
• Setting up IAM, security, and cost optimization
• Migrating from AWS or Azure to GCP
• Building multi-cloud or GCP-first architectures

Core Concepts

GCP Service Categories

Compute Options:

• Cloud Run: Serverless containers for stateless HTTP services (auto-scale to zero)
• GKE (Google Kubernetes Engine): Managed Kubernetes for complex orchestration
• Cloud Functions: Event-driven functions for simple processing
• Compute Engine: Virtual machines for full OS control
• App Engine: Platform-as-a-Service for web applications

Storage & Databases:

• Cloud Storage: Object storage with Standard/Nearline/Coldline/Archive tiers
• Cloud SQL: Managed PostgreSQL/MySQL/SQL Server (up to 96TB)
• Cloud Spanner: Global distributed SQL with 99.999% SLA
• Firestore: NoSQL document database with real-time sync
• Bigtable: Wide-column NoSQL for time-series and IoT (petabyte scale)
• AlloyDB: PostgreSQL-compatible with 4x performance improvement

Data & Analytics:

• BigQuery: Serverless data warehouse (petabyte-scale SQL analytics)
• Pub/Sub: Global messaging and event streaming
• Dataflow: Apache Beam for stream and batch processing
• Dataproc: Managed Spark and Hadoop clusters
• Cloud Composer: Managed Apache Airflow for workflows

AI/ML Services:

• Vertex AI: Unified ML platform (training, deployment, monitoring)
• AutoML: No-code ML for standard tasks
• Pre-trained APIs: Vision, Natural Language, Speech, Translation
• TPUs: Tensor Processing Units for large model training

Decision Framework: Compute Service Selection

Need to run code in GCP?
├─ HTTP service?
│  ├─ YES → Stateless?
│  │  ├─ YES → Cloud Run (auto-scale to zero)
│  │  └─ NO → Need Kubernetes? → GKE | Compute Engine
│  └─ NO (Event-driven)
│     ├─ Simple function? → Cloud Functions
│     └─ Complex orchestration? → GKE | Cloud Run Jobs

Selection Guide:

• First choice: Cloud Run (unless state or Kubernetes required)
• Need Kubernetes: GKE Autopilot (managed) or Standard (full control)
• Simple events: Cloud Functions (60-min max execution)
• Full control: Compute Engine (VMs with custom configuration)

Decision Framework: Database Selection

Choose database type:
├─ Relational (SQL)
│  ├─ Multi-region required? → Cloud Spanner
│  ├─ PostgreSQL + high performance? → AlloyDB
│  └─ Standard RDBMS → Cloud SQL (PostgreSQL/MySQL/SQL Server)
│
├─ Document (NoSQL)
│  ├─ Mobile/web with offline sync? → Firestore
│  └─ Flexible schema, no offline? → MongoDB Atlas (Marketplace)
│
├─ Key-Value
│  ├─ Time-series or IoT data? → Bigtable
│  └─ Caching layer? → Memorystore (Redis/Memcached)
│
└─ Analytics
   └─ Petabyte-scale SQL analytics → BigQuery

Decision Framework: Storage Selection

Storage type needed?
├─ Objects/Files
│  ├─ Frequent access → Cloud Storage (Standard)
│  ├─ Monthly access → Cloud Storage (Nearline)
│  ├─ Quarterly access → Cloud Storage (Coldline)
│  └─ Yearly access → Cloud Storage (Archive)
│
├─ Block storage → Persistent Disk (SSD/Standard/Extreme)
└─ Shared filesystem → Filestore (NFS)

GCP vs AWS vs Azure Service Mapping

Architecture Patterns

Pattern 1: Serverless Web Application

Use Case: Stateless HTTP API with database and caching

Architecture:

Internet → Cloud Load Balancer → Cloud Run → Cloud SQL (PostgreSQL)
                                            → Memorystore (Redis)
                                            → Cloud Storage

Key Services:

• Cloud Run for API service (auto-scaling containers)
• Cloud SQL for transactional data
• Memorystore for caching
• Cloud Storage for file uploads

For detailed Terraform configuration, see references/compute-services.md.

Pattern 2: Data Analytics Platform

Use Case: Real-time event processing and analytics

Architecture:

Data Sources → Pub/Sub → Dataflow → BigQuery → Looker/Tableau
                          ↓
                     Cloud Storage (staging)

Key Services:

• Pub/Sub for event ingestion (at-least-once delivery)
• Dataflow for stream processing (Apache Beam)
• BigQuery for analytics (partitioned tables, clustering)
• Cloud Storage for staging and backups

For BigQuery optimization patterns, see references/data-analytics.md.

Pattern 3: ML Pipeline

Use Case: End-to-end machine learning workflow

Architecture:

Training Data (GCS) → Vertex AI Training → Model Registry → Vertex AI Endpoints
                                                              ↓
                                                         Predictions

Key Services:

• Vertex AI Workbench for notebook development
• Vertex AI Training for custom models (GPU/TPU support)
• Vertex AI Endpoints for model serving (auto-scaling)
• Vertex AI Pipelines for orchestration (Kubeflow)

For ML implementation examples, see references/ml-ai-services.md.

Pattern 4: GKE Microservices Platform

Use Case: Complex orchestration with multiple services

Architecture:

Internet → Cloud Load Balancer → GKE Cluster
                                   ├─ Ingress Controller
                                   ├─ Service Mesh (optional)
                                   ├─ Microservice A
                                   ├─ Microservice B
                                   └─ Microservice C

Key Features:

• GKE Autopilot (fully managed nodes) or Standard (custom configuration)
• Workload Identity for secure GCP service access
• Private cluster with Private Google Access
• Config Connector for managing GCP resources via Kubernetes

For GKE setup and best practices, see references/compute-services.md.

Best Practices

Cost Optimization

Compute:

• Use Committed Use Discounts for predictable workloads (57% off)
• Use Spot VMs for fault-tolerant workloads (60-91% off)
• Cloud Run scales to zero when idle (no charges)
• GKE Autopilot charges only for pod resources, not nodes

Storage:

• Use appropriate Cloud Storage classes (Standard/Nearline/Coldline/Archive)
• Enable Object Lifecycle Management to transition cold data
• Archive backups with Coldline or Archive (99% cheaper than Standard)

Data:

• BigQuery: Use partitioned and clustered tables
• Query only needed columns (avoid SELECT *)
• Use BI Engine for caching (up to 10TB free)
• Consider flat-rate pricing for heavy BigQuery usage

For detailed cost strategies, see references/cost-optimization.md.

Security Fundamentals

IAM Best Practices:

• Follow principle of least privilege
• Use service accounts, not user accounts for applications
• Enable Workload Identity for GKE workloads (no service account keys)
• Use Secret Manager for secrets, not environment variables

Network Security:

• Use Private Google Access (access GCP services without public IPs)
• Enable Cloud NAT for outbound internet from private instances
• Implement VPC Service Controls for data exfiltration protection
• Use Identity-Aware Proxy (IAP) for zero-trust access

Data Security:

• Enable encryption at rest (default) and in transit
• Use Customer-Managed Encryption Keys (CMEK) for sensitive data
• Implement VPC Service Controls perimeter for data protection
• Enable audit logging for all projects

For comprehensive security patterns, see references/security-iam.md.

High Availability

Multi-Region Strategy:

• Cloud Storage: Use multi-region locations (US, EU, ASIA)
• Cloud SQL: Enable Regional HA (automatic failover)
• Cloud Spanner: Use multi-region configurations (99.999% SLA)
• Global Load Balancing: Route traffic to nearest healthy backend

Backup and Disaster Recovery:

• Cloud SQL: Enable automated backups and point-in-time recovery
• Persistent Disk: Schedule snapshot backups
• Cloud Storage: Enable versioning for critical data
• BigQuery: Use table snapshots for time travel

For networking and HA patterns, see references/networking.md.

Quick Reference

Common gcloud Commands

# Project management
gcloud projects list
gcloud config set project PROJECT_ID

# Cloud Run
gcloud run deploy SERVICE_NAME --image IMAGE_URL --region REGION
gcloud run services list

# GKE
gcloud container clusters create-auto CLUSTER_NAME --region REGION
gcloud container clusters get-credentials CLUSTER_NAME --region REGION

# Cloud Storage
gsutil mb gs://BUCKET_NAME
gsutil cp FILE gs://BUCKET_NAME/

# BigQuery
bq mk DATASET_NAME
bq query --use_legacy_sql=false 'SELECT * FROM dataset.table LIMIT 10'

# Cloud SQL
gcloud sql instances create INSTANCE_NAME --database-version=POSTGRES_15 --region=REGION
gcloud sql connect INSTANCE_NAME --user=postgres

For complete command reference, see examples/gcloud/common-commands.sh.

Python SDK Quick Start

# Cloud Storage
from google.cloud import storage
client = storage.Client()
bucket = client.bucket('my-bucket')
blob = bucket.blob('file.txt')
blob.upload_from_filename('local-file.txt')

# BigQuery
from google.cloud import bigquery
client = bigquery.Client()
query = "SELECT * FROM `project.dataset.table` LIMIT 10"
results = client.query(query).result()

# Pub/Sub
from google.cloud import pubsub_v1
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path('project', 'topic-name')
future = publisher.publish(topic_path, b'message data')

For complete Python examples, see examples/python/.

Terraform Quick Start

# Provider configuration
terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.0"
    }
  }
}

provider "google" {
  project = "my-project-id"
  region  = "us-central1"
}

# Cloud Run service
resource "google_cloud_run_service" "api" {
  name     = "api-service"
  location = "us-central1"

  template {
    spec {
      containers {
        image = "gcr.io/project/api:latest"
      }
    }
  }
}

For complete Terraform examples, see examples/terraform/.

Service Selection Cheatsheet

Integration with Other Skills

Related Skills:

• infrastructure-as-code: Use Terraform to provision GCP resources (see examples/terraform/)
• kubernetes-operations: Deploy and manage applications on GKE
• building-ci-pipelines: Use Cloud Build for CI/CD to Cloud Run or GKE
• secret-management: Use Secret Manager for sensitive configuration
• observability: Use Cloud Monitoring and Cloud Logging for metrics and logs
• data-architecture: Design data lakes and warehouses using BigQuery and Cloud Storage
• mlops-patterns: Implement ML pipelines using Vertex AI
• aws-patterns: Compare AWS and GCP service equivalents for multi-cloud
• azure-patterns: Compare Azure and GCP service equivalents

Progressive Disclosure

For detailed documentation:

• Compute services: See references/compute-services.md for Cloud Run, GKE, Cloud Functions, Compute Engine, and App Engine patterns
• Storage & databases: See references/storage-databases.md for detailed service selection and configuration
• Data analytics: See references/data-analytics.md for BigQuery, Pub/Sub, Dataflow, and Dataproc patterns
• ML/AI services: See references/ml-ai-services.md for Vertex AI, AutoML, and pre-trained API usage
• Networking: See references/networking.md for VPC, Load Balancing, CDN, and Cloud Armor patterns
• Security & IAM: See references/security-iam.md for IAM patterns, Workload Identity, and Secret Manager
• Cost optimization: See references/cost-optimization.md for detailed cost reduction strategies

For working examples:

• Terraform configurations: See examples/terraform/ for infrastructure templates
• Python SDK usage: See examples/python/ for client library examples
• gcloud CLI commands: See examples/gcloud/common-commands.sh for command reference

Key Decisions Summary

When choosing GCP:

• Data analytics workloads (BigQuery is best-in-class)
• ML/AI applications (Vertex AI, TPUs, Google Research backing)
• Kubernetes-native applications (GKE invented by Kubernetes creators)
• Serverless containers (Cloud Run is mature and cost-effective)
• Real-time streaming (Pub/Sub + Dataflow)

GCP's unique advantages:

• BigQuery: Serverless, petabyte-scale, fastest data warehouse
• Cloud Run: Most mature serverless container platform
• GKE: Most advanced managed Kubernetes (Autopilot mode)
• Vertex AI: Unified ML platform (training, deployment, monitoring)
• Per-second billing and sustained use discounts (automatic cost savings)

Multi-region recommendations:

• Production workloads: Use multi-region for 99.95%+ SLA
• Cloud Storage: Multi-region for global access
• Cloud Spanner: Multi-region for global transactions
• Global Load Balancing: Route to nearest healthy backend

GCP Compute Services Reference

Detailed patterns and configurations for Google Cloud Platform compute services.

Table of Contents

1. Cloud Run (Serverless Containers) 2. Google Kubernetes Engine (GKE) 3. Cloud Functions 4. Compute Engine (VMs) 5. App Engine 6. Service Comparison Matrix

---

Cloud Run (Serverless Containers)

When to Use Cloud Run

Ideal For:

• Stateless HTTP services and APIs
• Microservices architecture
• Variable or unpredictable traffic
• Services that should scale to zero when idle
• Quick deployment without infrastructure management

Not Ideal For:

• Long-running jobs (use Cloud Run Jobs)
• Stateful applications (use GKE or Compute Engine)
• Applications requiring persistent connections (use GKE)

Cloud Run Configuration

Terraform Example:

resource "google_cloud_run_service" "api" {
  name     = "api-service"
  location = "us-central1"

  template {
    spec {
      containers {
        image = "gcr.io/project-id/api:latest"

        # Resource limits
        resources {
          limits = {
            cpu    = "2"      # Up to 8 CPUs
            memory = "2Gi"    # Up to 32GB
          }
        }

        # Environment variables
        env {
          name  = "DATABASE_URL"
          value = "postgresql://..."
        }

        # Secret from Secret Manager
        env {
          name = "API_KEY"
          value_from {
            secret_key_ref {
              name = google_secret_manager_secret.api_key.secret_id
              key  = "latest"
            }
          }
        }

        # Port configuration
        ports {
          container_port = 8080
        }

        # Startup probe
        startup_probe {
          http_get {
            path = "/health"
          }
          initial_delay_seconds = 0
          timeout_seconds       = 1
          period_seconds        = 3
          failure_threshold     = 3
        }
      }

      # Concurrency control
      container_concurrency = 80  # Max concurrent requests per instance

      # Timeout
      timeout_seconds = 300  # Max 60 minutes

      # Service account for Workload Identity
      service_account_name = google_service_account.api.email
    }

    metadata {
      annotations = {
        # Auto-scaling
        "autoscaling.knative.dev/minScale" = "1"
        "autoscaling.dev/maxScale" = "100"

        # CPU allocation (always or request-based)
        "run.googleapis.com/cpu-throttling" = "true"  # CPU allocated only during requests

        # Cloud SQL connection
        "run.googleapis.com/cloudsql-instances" = google_sql_database_instance.main.connection_name

        # VPC connector (for private resources)
        "run.googleapis.com/vpc-access-connector" = google_vpc_access_connector.connector.id
        "run.googleapis.com/vpc-access-egress"    = "private-ranges-only"
      }

      labels = {
        environment = "production"
        team        = "api"
      }
    }
  }

  traffic {
    percent         = 100
    latest_revision = true
  }

  # Prevent accidental deletion
  lifecycle {
    prevent_destroy = true
  }
}

# Public access
data "google_iam_policy" "noauth" {
  binding {
    role    = "roles/run.invoker"
    members = ["allUsers"]
  }
}

resource "google_cloud_run_service_iam_policy" "noauth" {
  location = google_cloud_run_service.api.location
  service  = google_cloud_run_service.api.name
  policy_data = data.google_iam_policy.noauth.policy_data
}

# Private access (requires authentication)
resource "google_cloud_run_service_iam_member" "auth" {
  location = google_cloud_run_service.api.location
  service  = google_cloud_run_service.api.name
  role     = "roles/run.invoker"
  member   = "serviceAccount:caller@project.iam.gserviceaccount.com"
}

Cloud Run Jobs

For batch processing and scheduled tasks:

resource "google_cloud_run_v2_job" "batch_processor" {
  name     = "batch-processor"
  location = "us-central1"

  template {
    template {
      containers {
        image = "gcr.io/project-id/batch:latest"

        resources {
          limits = {
            cpu    = "4"
            memory = "8Gi"
          }
        }
      }

      # Max execution time
      timeout = "3600s"  # 1 hour

      # Task configuration
      max_retries = 3
    }

    # Parallelism
    task_count = 10
    parallelism = 5
  }
}

# Schedule with Cloud Scheduler
resource "google_cloud_scheduler_job" "trigger" {
  name        = "trigger-batch"
  description = "Trigger batch processing job"
  schedule    = "0 2 * * *"  # Daily at 2am
  time_zone   = "America/New_York"

  http_target {
    http_method = "POST"
    uri         = "https://${google_cloud_run_v2_job.batch_processor.location}-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/${var.project_id}/jobs/${google_cloud_run_v2_job.batch_processor.name}:run"

    oauth_token {
      service_account_email = google_service_account.scheduler.email
    }
  }
}

---

Google Kubernetes Engine (GKE)

When to Use GKE

Ideal For:

• Complex multi-service orchestration
• Stateful applications (databases, caches)
• Applications requiring persistent connections
• Existing Kubernetes deployments
• Advanced networking requirements
• Multi-tenant platforms

Not Ideal For:

• Simple stateless services (use Cloud Run)
• No Kubernetes expertise (use Cloud Run or App Engine)

GKE Autopilot vs Standard

GKE Autopilot Configuration

resource "google_container_cluster" "autopilot" {
  name     = "gke-autopilot-cluster"
  location = "us-central1"

  # Enable Autopilot
  enable_autopilot = true

  # Release channel (RAPID, REGULAR, STABLE)
  release_channel {
    channel = "REGULAR"
  }

  # IP allocation policy
  ip_allocation_policy {
    cluster_secondary_range_name  = "pods"
    services_secondary_range_name = "services"
  }

  # Workload Identity (required for Autopilot)
  workload_identity_config {
    workload_pool = "${var.project_id}.svc.id.goog"
  }

  # Private cluster
  private_cluster_config {
    enable_private_nodes    = true
    enable_private_endpoint = false
    master_ipv4_cidr_block  = "172.16.0.0/28"
  }

  # Master authorized networks
  master_authorized_networks_config {
    cidr_blocks {
      cidr_block   = "10.0.0.0/8"
      display_name = "internal-network"
    }
  }

  # Maintenance window
  maintenance_policy {
    daily_maintenance_window {
      start_time = "03:00"
    }
  }

  # Binary authorization
  binary_authorization {
    evaluation_mode = "PROJECT_SINGLETON_POLICY_ENFORCE"
  }

  # Network policy
  network_policy {
    enabled  = true
    provider = "PROVIDER_UNSPECIFIED"
  }

  # Monitoring and logging
  monitoring_config {
    enable_components = ["SYSTEM_COMPONENTS", "WORKLOADS"]
    managed_prometheus {
      enabled = true
    }
  }

  logging_config {
    enable_components = ["SYSTEM_COMPONENTS", "WORKLOADS"]
  }
}

GKE Standard Configuration

resource "google_container_cluster" "standard" {
  name     = "gke-standard-cluster"
  location = "us-central1"

  # Remove default node pool (create custom pool separately)
  remove_default_node_pool = true
  initial_node_count       = 1

  release_channel {
    channel = "REGULAR"
  }

  workload_identity_config {
    workload_pool = "${var.project_id}.svc.id.goog"
  }

  private_cluster_config {
    enable_private_nodes    = true
    enable_private_endpoint = false
    master_ipv4_cidr_block  = "172.16.0.0/28"
  }

  ip_allocation_policy {
    cluster_secondary_range_name  = "pods"
    services_secondary_range_name = "services"
  }

  network_policy {
    enabled  = true
    provider = "PROVIDER_UNSPECIFIED"
  }

  addons_config {
    http_load_balancing {
      disabled = false
    }
    horizontal_pod_autoscaling {
      disabled = false
    }
    network_policy_config {
      disabled = false
    }
    gcp_filestore_csi_driver_config {
      enabled = true
    }
    gce_persistent_disk_csi_driver_config {
      enabled = true
    }
  }
}

# Custom node pool
resource "google_container_node_pool" "primary" {
  name       = "primary-pool"
  location   = "us-central1"
  cluster    = google_container_cluster.standard.name
  node_count = 1

  autoscaling {
    min_node_count = 1
    max_node_count = 10
  }

  management {
    auto_repair  = true
    auto_upgrade = true
  }

  node_config {
    preemptible  = false
    machine_type = "e2-standard-4"
    disk_size_gb = 100
    disk_type    = "pd-standard"

    # Service account with minimal permissions
    service_account = google_service_account.gke_nodes.email
    oauth_scopes = [
      "https://www.googleapis.com/auth/cloud-platform"
    ]

    # Workload Identity
    workload_metadata_config {
      mode = "GKE_METADATA"
    }

    # Shielded instance
    shielded_instance_config {
      enable_secure_boot          = true
      enable_integrity_monitoring = true
    }

    labels = {
      environment = "production"
    }

    tags = ["gke-node"]
  }
}

# Spot VM node pool (for fault-tolerant workloads)
resource "google_container_node_pool" "spot" {
  name       = "spot-pool"
  location   = "us-central1"
  cluster    = google_container_cluster.standard.name
  node_count = 0

  autoscaling {
    min_node_count = 0
    max_node_count = 20
  }

  node_config {
    spot         = true  # 60-91% cheaper
    machine_type = "e2-standard-4"

    service_account = google_service_account.gke_nodes.email
    oauth_scopes = [
      "https://www.googleapis.com/auth/cloud-platform"
    ]

    workload_metadata_config {
      mode = "GKE_METADATA"
    }

    labels = {
      workload-type = "batch"
    }

    taint {
      key    = "workload-type"
      value  = "batch"
      effect = "NO_SCHEDULE"
    }
  }
}

Workload Identity Setup

# Enable Workload Identity on cluster
gcloud container clusters update CLUSTER_NAME \
    --workload-pool=PROJECT_ID.svc.id.goog

# Create Kubernetes service account
kubectl create serviceaccount KSA_NAME \
    --namespace NAMESPACE

# Create GCP service account
gcloud iam service-accounts create GSA_NAME

# Bind Kubernetes SA to GCP SA
gcloud iam service-accounts add-iam-policy-binding \
    GSA_NAME@PROJECT_ID.iam.gserviceaccount.com \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]"

# Annotate Kubernetes SA
kubectl annotate serviceaccount KSA_NAME \
    --namespace NAMESPACE \
    iam.gke.io/gcp-service-account=GSA_NAME@PROJECT_ID.iam.gserviceaccount.com

---

Cloud Functions

When to Use Cloud Functions

Ideal For:

• Event-driven processing (Cloud Storage, Pub/Sub, Firestore triggers)
• Lightweight HTTP endpoints
• Single-purpose functions
• Integration glue between GCP services

Not Ideal For:

• Long-running tasks (10-minute max for gen2)
• Complex application logic (use Cloud Run)
• Stateful operations

Cloud Functions Gen2 (Recommended)

# Storage bucket for function source
resource "google_storage_bucket" "functions" {
  name     = "${var.project_id}-functions"
  location = "US"
}

# Upload function source
resource "google_storage_bucket_object" "function_zip" {
  name   = "function-source.zip"
  bucket = google_storage_bucket.functions.name
  source = "path/to/function.zip"
}

# Cloud Function with HTTP trigger
resource "google_cloudfunctions2_function" "http_function" {
  name     = "http-function"
  location = "us-central1"

  build_config {
    runtime     = "python39"
    entry_point = "hello_http"
    source {
      storage_source {
        bucket = google_storage_bucket.functions.name
        object = google_storage_bucket_object.function_zip.name
      }
    }
  }

  service_config {
    max_instance_count = 100
    min_instance_count = 0
    available_memory   = "256Mi"
    timeout_seconds    = 60

    environment_variables = {
      API_KEY = "value"
    }

    # Secret from Secret Manager
    secret_environment_variables {
      key        = "DB_PASSWORD"
      project_id = var.project_id
      secret     = google_secret_manager_secret.db_password.secret_id
      version    = "latest"
    }

    service_account_email = google_service_account.function.email
  }
}

# Cloud Function with Pub/Sub trigger
resource "google_cloudfunctions2_function" "pubsub_function" {
  name     = "pubsub-function"
  location = "us-central1"

  build_config {
    runtime     = "python39"
    entry_point = "process_message"
    source {
      storage_source {
        bucket = google_storage_bucket.functions.name
        object = google_storage_bucket_object.function_zip.name
      }
    }
  }

  service_config {
    max_instance_count = 100
    available_memory   = "512Mi"
    timeout_seconds    = 300
  }

  event_trigger {
    trigger_region = "us-central1"
    event_type     = "google.cloud.pubsub.topic.v1.messagePublished"
    pubsub_topic   = google_pubsub_topic.events.id
    retry_policy   = "RETRY_POLICY_RETRY"
  }
}

# Cloud Function with Storage trigger
resource "google_cloudfunctions2_function" "storage_function" {
  name     = "storage-function"
  location = "us-central1"

  build_config {
    runtime     = "python39"
    entry_point = "process_file"
    source {
      storage_source {
        bucket = google_storage_bucket.functions.name
        object = google_storage_bucket_object.function_zip.name
      }
    }
  }

  service_config {
    max_instance_count = 50
    available_memory   = "1Gi"
    timeout_seconds    = 540
  }

  event_trigger {
    trigger_region = "us-central1"
    event_type     = "google.cloud.storage.object.v1.finalized"
    event_filters {
      attribute = "bucket"
      value     = google_storage_bucket.uploads.name
    }
  }
}

---

Compute Engine (VMs)

When to Use Compute Engine

Ideal For:

• Full OS control required
• GPU/TPU workloads
• Windows applications
• Legacy application lift-and-shift
• Custom kernel or networking

Not Ideal For:

• Simple web services (use Cloud Run)
• Don't want to manage infrastructure (use Cloud Run or App Engine)

Compute Engine Configuration

# Instance template for managed instance group
resource "google_compute_instance_template" "web" {
  name_prefix  = "web-template-"
  machine_type = "e2-medium"
  region       = "us-central1"

  disk {
    source_image = "debian-cloud/debian-11"
    auto_delete  = true
    boot         = true
    disk_size_gb = 20
    disk_type    = "pd-balanced"
  }

  network_interface {
    network    = google_compute_network.main.id
    subnetwork = google_compute_subnetwork.private.id

    # No external IP (use Cloud NAT)
    access_config {
      network_tier = "PREMIUM"
    }
  }

  metadata_startup_script = file("startup.sh")

  service_account {
    email  = google_service_account.instance.email
    scopes = ["cloud-platform"]
  }

  shielded_instance_config {
    enable_secure_boot          = true
    enable_integrity_monitoring = true
    enable_vtpm                 = true
  }

  tags = ["web-server"]

  lifecycle {
    create_before_destroy = true
  }
}

# Managed instance group with autoscaling
resource "google_compute_region_instance_group_manager" "web" {
  name   = "web-igm"
  region = "us-central1"

  base_instance_name = "web"

  version {
    instance_template = google_compute_instance_template.web.id
  }

  target_size = 3

  named_port {
    name = "http"
    port = 80
  }

  auto_healing_policies {
    health_check      = google_compute_health_check.web.id
    initial_delay_sec = 300
  }
}

# Autoscaler
resource "google_compute_region_autoscaler" "web" {
  name   = "web-autoscaler"
  region = "us-central1"
  target = google_compute_region_instance_group_manager.web.id

  autoscaling_policy {
    min_replicas    = 2
    max_replicas    = 10
    cooldown_period = 60

    cpu_utilization {
      target = 0.6
    }
  }
}

---

App Engine

When to Use App Engine

Ideal For:

• Simple web applications
• Quick prototyping
• Legacy App Engine applications
• No infrastructure management desired

Not Ideal For:

• New projects (Cloud Run is preferred)
• Need containerization (use Cloud Run)
• Complex orchestration (use GKE)

App Engine Configuration

# app.yaml
runtime: python39
entrypoint: gunicorn -b :$PORT main:app

instance_class: F2

automatic_scaling:
  target_cpu_utilization: 0.65
  min_instances: 1
  max_instances: 10
  min_pending_latency: 30ms
  max_pending_latency: automatic
  max_concurrent_requests: 50

env_variables:
  API_KEY: "value"

handlers:
- url: /static
  static_dir: static

- url: /.*
  script: auto

---

Service Comparison Matrix

Selection Decision Tree

Need to run application in GCP?
├─ Need Kubernetes? → YES
│  ├─ Need node customization? → YES → GKE Standard
│  └─ NO (want managed) → GKE Autopilot
│
├─ HTTP service? → YES
│  ├─ Stateless? → YES
│  │  ├─ Container-based? → YES → Cloud Run
│  │  └─ NO (simple web app) → App Engine
│  └─ NO (stateful) → Compute Engine or GKE
│
├─ Event-driven? → YES
│  ├─ Simple function? → YES → Cloud Functions
│  └─ Complex processing → Cloud Run Jobs
│
└─ Need full OS control? → YES → Compute Engine

Recommendation: Start with Cloud Run for most HTTP services. Move to GKE only if you need Kubernetes-specific features or complex orchestration.

GCP Cost Optimization Reference

Strategies for reducing Google Cloud Platform costs.

Table of Contents

• Compute Cost Optimization
• Committed Use Discounts (57% off)
• Spot VMs (60-91% off)
• Right-Sizing VMs
• Cloud Run Cost Optimization
• Storage Cost Optimization
• Object Lifecycle Management
• Compression and Deduplication
• BigQuery Cost Optimization
• Partitioned and Clustered Tables
• Query Best Practices
• Flat-Rate Pricing
• Database Cost Optimization
• Cloud SQL Instance Scheduling
• Use Read Replicas Instead of Scaling Up
• Network Cost Optimization
• Use Regional Resources
• Cloud CDN for Egress
• Premium vs Standard Network Tier
• Monitoring and Alerts
• Cost Budget Alerts
• Cost Anomaly Detection
• Cost Optimization Summary
• Cost Monitoring Tools
• Best Practices Checklist

Compute Cost Optimization

Committed Use Discounts (57% off)

# Reserve compute resources for 1 or 3 years
resource "google_compute_commitment" "commitment" {
  name   = "compute-commitment"
  region = "us-central1"
  plan   = "THIRTY_SIX_MONTH"  # 1-year or 3-year

  resources {
    type   = "VCPU"
    amount = "100"
  }

  resources {
    type   = "MEMORY"
    amount = "400"  # GB
  }
}

Spot VMs (60-91% off)

resource "google_compute_instance_template" "spot" {
  name_prefix  = "spot-template-"
  machine_type = "n2-standard-4"

  scheduling {
    preemptible                 = true
    automatic_restart           = false
    on_host_maintenance         = "TERMINATE"
    provisioning_model          = "SPOT"
    instance_termination_action = "STOP"
  }

  disk {
    source_image = "debian-cloud/debian-11"
  }

  network_interface {
    network = "default"
  }
}

Right-Sizing VMs

# Get recommendations
gcloud recommender recommendations list \
    --project=PROJECT_ID \
    --location=us-central1 \
    --recommender=google.compute.instance.MachineTypeRecommender

# Apply recommendation
gcloud compute instances set-machine-type INSTANCE_NAME \
    --machine-type=n2-standard-2 \
    --zone=us-central1-a

Cloud Run Cost Optimization

resource "google_cloud_run_service" "optimized" {
  name     = "optimized-service"
  location = "us-central1"

  template {
    spec {
      containers {
        image = "gcr.io/project/app:latest"

        resources {
          limits = {
            cpu    = "1"      # Right-size CPU
            memory = "512Mi"  # Right-size memory
          }
        }
      }

      # Scale to zero when idle
      container_concurrency = 80
    }

    metadata {
      annotations = {
        "autoscaling.knative.dev/minScale" = "0"  # Scale to zero
        "autoscaling.knative.dev/maxScale" = "10"

        # CPU allocated only during requests
        "run.googleapis.com/cpu-throttling" = "true"
      }
    }
  }
}

Storage Cost Optimization

Object Lifecycle Management

resource "google_storage_bucket" "optimized" {
  name     = "optimized-bucket"
  location = "US"

  # Transition to cheaper storage classes
  lifecycle_rule {
    condition {
      age = 30
    }
    action {
      type          = "SetStorageClass"
      storage_class = "NEARLINE"  # $0.01/GB/month vs $0.02/GB for Standard
    }
  }

  lifecycle_rule {
    condition {
      age = 90
    }
    action {
      type          = "SetStorageClass"
      storage_class = "COLDLINE"  # $0.004/GB/month
    }
  }

  lifecycle_rule {
    condition {
      age = 365
    }
    action {
      type          = "SetStorageClass"
      storage_class = "ARCHIVE"  # $0.0012/GB/month
    }
  }

  # Delete old data
  lifecycle_rule {
    condition {
      age = 730  # 2 years
    }
    action {
      type = "Delete"
    }
  }

  # Delete old versions
  lifecycle_rule {
    condition {
      num_newer_versions = 3
    }
    action {
      type = "Delete"
    }
  }
}

Compression and Deduplication

from google.cloud import storage
import gzip

client = storage.Client()
bucket = client.bucket('my-bucket')

# Upload compressed objects
blob = bucket.blob('data.json.gz')
blob.content_encoding = 'gzip'

with gzip.open('data.json', 'rb') as f:
    blob.upload_from_file(f, content_type='application/json')

BigQuery Cost Optimization

Partitioned and Clustered Tables

resource "google_bigquery_table" "optimized" {
  dataset_id = google_bigquery_dataset.analytics.dataset_id
  table_id   = "events_optimized"

  # Partitioning reduces query costs
  time_partitioning {
    type          = "DAY"
    field         = "event_timestamp"
    expiration_ms = 7776000000  # Auto-delete after 90 days
  }

  # Clustering further reduces costs
  clustering = ["user_id", "event_type", "country"]

  # Range partitioning (alternative)
  range_partitioning {
    field = "user_id"
    range {
      start    = 0
      end      = 1000000
      interval = 10000
    }
  }

  schema = jsonencode([
    { name = "event_timestamp", type = "TIMESTAMP", mode = "REQUIRED" },
    { name = "user_id", type = "INTEGER", mode = "REQUIRED" },
    { name = "event_type", type = "STRING", mode = "REQUIRED" },
    { name = "country", type = "STRING", mode = "NULLABLE" }
  ])
}

Query Best Practices

-- BAD: Full table scan ($$$)
SELECT * FROM `project.dataset.table`;

-- GOOD: Query specific columns and partitions ($)
SELECT user_id, event_type
FROM `project.dataset.table`
WHERE event_timestamp >= '2025-01-01'
  AND event_timestamp < '2025-01-02'
  AND country = 'US';

-- Use LIMIT for exploration
SELECT *
FROM `project.dataset.table`
WHERE event_timestamp >= '2025-01-01'
LIMIT 1000;

-- Preview without cost
SELECT *
FROM `project.dataset.table`
TABLESAMPLE SYSTEM (1 PERCENT);

Flat-Rate Pricing

For heavy BigQuery usage:

resource "google_bigquery_reservation" "reservation" {
  name     = "production-reservation"
  location = "US"

  # 100 slots = ~$2,000/month (vs on-demand at $5/TB)
  # Break-even: ~400 TB queried per month
  slot_capacity = 100
}

resource "google_bigquery_reservation_assignment" "assignment" {
  assignee      = "projects/${var.project_id}"
  job_type      = "QUERY"
  reservation   = google_bigquery_reservation.reservation.id
}

Database Cost Optimization

Cloud SQL Instance Scheduling

# Stop instance during off-hours
gcloud sql instances patch INSTANCE_NAME \
    --activation-policy=NEVER

# Start instance
gcloud sql instances patch INSTANCE_NAME \
    --activation-policy=ALWAYS

# Automate with Cloud Scheduler
gcloud scheduler jobs create http stop-db \
    --schedule="0 18 * * *" \
    --uri="https://sqladmin.googleapis.com/v1/projects/PROJECT_ID/instances/INSTANCE_NAME/stop" \
    --http-method=POST \
    --oauth-service-account-email=SERVICE_ACCOUNT

Use Read Replicas Instead of Scaling Up

# Read replica in same region (cheaper than larger primary)
resource "google_sql_database_instance" "replica" {
  name                 = "read-replica"
  master_instance_name = google_sql_database_instance.main.name
  region               = "us-central1"

  replica_configuration {
    failover_target = false
  }

  settings {
    tier = "db-custom-2-8192"  # Smaller than primary
  }
}

Network Cost Optimization

Use Regional Resources

# Multi-region costs more for data processing
# Use regional unless global distribution required

resource "google_compute_backend_service" "regional" {
  name                  = "regional-backend"
  load_balancing_scheme = "INTERNAL_MANAGED"  # Regional, cheaper
  protocol              = "HTTP"
}

Cloud CDN for Egress

resource "google_compute_backend_service" "cdn_enabled" {
  name = "cdn-backend"

  # Enable CDN to reduce egress costs
  cdn_policy {
    cache_mode  = "CACHE_ALL_STATIC"
    default_ttl = 3600
  }
}

Premium vs Standard Network Tier

# Standard tier: 25-40% cheaper, acceptable for most workloads
resource "google_compute_address" "standard_tier" {
  name         = "standard-ip"
  region       = "us-central1"
  network_tier = "STANDARD"  # Cheaper than PREMIUM
}

Monitoring and Alerts

Cost Budget Alerts

resource "google_billing_budget" "budget" {
  billing_account = var.billing_account
  display_name    = "Monthly Budget"

  budget_filter {
    projects = ["projects/${var.project_id}"]
  }

  amount {
    specified_amount {
      currency_code = "USD"
      units         = "1000"
    }
  }

  threshold_rules {
    threshold_percent = 0.5
  }

  threshold_rules {
    threshold_percent = 0.9
  }

  threshold_rules {
    threshold_percent = 1.0
  }

  all_updates_rule {
    pubsub_topic = google_pubsub_topic.budget_alerts.id
  }
}

Cost Anomaly Detection

# Enable cost anomaly detection
gcloud billing accounts describe BILLING_ACCOUNT \
    --format="value(anomalyConfig.state)"

# List anomalies
gcloud billing accounts anomalies list \
    --billing-account=BILLING_ACCOUNT

Cost Optimization Summary

Cost Monitoring Tools

# Export billing to BigQuery
gcloud alpha billing accounts update BILLING_ACCOUNT \
    --export-enabled \
    --export-dataset=PROJECT_ID:billing_export

# Query costs
SELECT
  service.description AS service,
  SUM(cost) AS total_cost
FROM `project.billing_export.gcp_billing_export_v1_*`
WHERE _TABLE_SUFFIX = FORMAT_DATE('%Y%m%d', CURRENT_DATE())
GROUP BY service
ORDER BY total_cost DESC;

Best Practices Checklist

• [ ] Use committed use discounts for predictable workloads
• [ ] Use Spot VMs for batch/fault-tolerant workloads
• [ ] Right-size all resources (use Recommender)
• [ ] Enable Cloud Run CPU throttling and scale-to-zero
• [ ] Implement Storage lifecycle policies
• [ ] Use partitioned and clustered tables in BigQuery
• [ ] Stop non-production databases during off-hours
• [ ] Enable Cloud CDN for static content
• [ ] Use Standard network tier where acceptable
• [ ] Set up budget alerts and anomaly detection
• [ ] Export billing data to BigQuery for analysis
• [ ] Review cost recommendations monthly

GCP Data Analytics Services Reference

Patterns for BigQuery, Pub/Sub, Dataflow, Dataproc, and Cloud Composer.

Table of Contents

1. BigQuery 2. Pub/Sub 3. Dataflow 4. Dataproc 5. Cloud Composer

---

BigQuery

Optimized Table Configuration

resource "google_bigquery_dataset" "analytics" {
  dataset_id    = "analytics"
  location      = "US"
  description   = "Analytics dataset"

  default_table_expiration_ms = 7776000000  # 90 days

  access {
    role          = "OWNER"
    user_by_email = google_service_account.analytics.email
  }
}

resource "google_bigquery_table" "events" {
  dataset_id = google_bigquery_dataset.analytics.dataset_id
  table_id   = "events"

  # Partitioning by day (reduces query cost)
  time_partitioning {
    type  = "DAY"
    field = "event_timestamp"
    expiration_ms = 7776000000  # 90 days
  }

  # Clustering (further reduces query cost)
  clustering = ["user_id", "event_type", "country"]

  schema = jsonencode([
    {
      name = "event_id"
      type = "STRING"
      mode = "REQUIRED"
    },
    {
      name = "event_timestamp"
      type = "TIMESTAMP"
      mode = "REQUIRED"
    },
    {
      name = "user_id"
      type = "STRING"
      mode = "REQUIRED"
    },
    {
      name = "event_type"
      type = "STRING"
      mode = "REQUIRED"
    },
    {
      name = "country"
      type = "STRING"
      mode = "NULLABLE"
    },
    {
      name = "properties"
      type = "JSON"
      mode = "NULLABLE"
    }
  ])
}

Query Optimization

from google.cloud import bigquery

client = bigquery.Client()

# Use parameterized queries
query = """
    SELECT
        event_type,
        COUNT(*) as event_count,
        COUNT(DISTINCT user_id) as unique_users
    FROM `project.analytics.events`
    WHERE event_timestamp >= @start_time
        AND event_timestamp < @end_time
        AND country = @country
    GROUP BY event_type
    ORDER BY event_count DESC
"""

job_config = bigquery.QueryJobConfig(
    query_parameters=[
        bigquery.ScalarQueryParameter("start_time", "TIMESTAMP", "2025-01-01"),
        bigquery.ScalarQueryParameter("end_time", "TIMESTAMP", "2025-01-31"),
        bigquery.ScalarQueryParameter("country", "STRING", "US"),
    ]
)

# Use query caching (24 hours by default)
query_job = client.query(query, job_config=job_config)
results = query_job.result()

# BigQuery ML
ml_query = """
    CREATE OR REPLACE MODEL `project.analytics.user_churn_model`
    OPTIONS(
      model_type='LOGISTIC_REG',
      input_label_cols=['churned']
    ) AS
    SELECT
      user_id,
      activity_count,
      last_active_days_ago,
      churned
    FROM `project.analytics.user_features`
"""

---

Pub/Sub

Topic and Subscription Configuration

# Pub/Sub topic with schema
resource "google_pubsub_schema" "events" {
  name = "events-schema"
  type = "AVRO"
  definition = jsonencode({
    type = "record"
    name = "Event"
    fields = [
      { name = "event_id", type = "string" },
      { name = "timestamp", type = "long" },
      { name = "user_id", type = "string" },
      { name = "event_type", type = "string" }
    ]
  })
}

resource "google_pubsub_topic" "events" {
  name = "events-topic"

  schema_settings {
    schema   = google_pubsub_schema.events.id
    encoding = "JSON"
  }

  message_retention_duration = "604800s"  # 7 days
}

# Pull subscription
resource "google_pubsub_subscription" "events_pull" {
  name  = "events-pull"
  topic = google_pubsub_topic.events.name

  ack_deadline_seconds = 20
  message_retention_duration = "604800s"

  retry_policy {
    minimum_backoff = "10s"
    maximum_backoff = "600s"
  }

  dead_letter_policy {
    dead_letter_topic     = google_pubsub_topic.dead_letter.id
    max_delivery_attempts = 5
  }

  expiration_policy {
    ttl = ""  # Never expire
  }
}

# Push subscription
resource "google_pubsub_subscription" "events_push" {
  name  = "events-push"
  topic = google_pubsub_topic.events.name

  push_config {
    push_endpoint = "https://example.com/webhook"

    oidc_token {
      service_account_email = google_service_account.pubsub.email
    }

    attributes = {
      x-goog-version = "v1"
    }
  }
}

Python Publisher/Subscriber

from google.cloud import pubsub_v1
import json

# Publisher
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path('project-id', 'events-topic')

def publish_message(event_data):
    message_json = json.dumps(event_data).encode('utf-8')
    future = publisher.publish(
        topic_path,
        message_json,
        event_type=event_data['event_type']  # Custom attribute
    )
    return future.result()

# Subscriber (pull)
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path('project-id', 'events-pull')

def callback(message):
    print(f'Received: {message.data}')
    message.ack()  # Acknowledge receipt

streaming_pull_future = subscriber.subscribe(subscription_path, callback=callback)

---

Dataflow

Terraform Configuration

resource "google_dataflow_job" "stream_to_bq" {
  name              = "stream-events-to-bigquery"
  template_gcs_path = "gs://dataflow-templates-us-central1/latest/PubSub_to_BigQuery"
  temp_gcs_location = google_storage_bucket.dataflow_temp.url

  parameters = {
    inputTopic      = google_pubsub_topic.events.id
    outputTableSpec = "${var.project_id}:${google_bigquery_dataset.analytics.dataset_id}.${google_bigquery_table.events.table_id}"
  }

  on_delete = "cancel"
}

Apache Beam Pipeline (Python)

import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions

def run_pipeline():
    options = PipelineOptions(
        project='project-id',
        runner='DataflowRunner',
        region='us-central1',
        temp_location='gs://bucket/temp',
        staging_location='gs://bucket/staging',
        streaming=True
    )

    with beam.Pipeline(options=options) as pipeline:
        (pipeline
         | 'Read from Pub/Sub' >> beam.io.ReadFromPubSub(
             topic='projects/project-id/topics/events-topic')
         | 'Parse JSON' >> beam.Map(lambda msg: json.loads(msg))
         | 'Transform' >> beam.Map(transform_event)
         | 'Write to BigQuery' >> beam.io.WriteToBigQuery(
             'project-id:analytics.events',
             schema='event_id:STRING,timestamp:TIMESTAMP,user_id:STRING',
             write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND,
             create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED)
        )

def transform_event(event):
    return {
        'event_id': event['id'],
        'timestamp': event['timestamp'],
        'user_id': event['user']
    }

---

Dataproc

Cluster Configuration

resource "google_dataproc_cluster" "spark" {
  name   = "spark-cluster"
  region = "us-central1"

  cluster_config {
    staging_bucket = google_storage_bucket.dataproc.name

    master_config {
      num_instances = 1
      machine_type  = "n2-standard-4"
      disk_config {
        boot_disk_type    = "pd-ssd"
        boot_disk_size_gb = 100
      }
    }

    worker_config {
      num_instances = 3
      machine_type  = "n2-standard-4"
      disk_config {
        boot_disk_size_gb = 100
      }
    }

    preemptible_worker_config {
      num_instances = 5
    }

    software_config {
      image_version = "2.1-debian11"
      properties = {
        "spark:spark.executor.memory" = "4g"
      }
    }

    gce_cluster_config {
      zone = "us-central1-a"
      service_account_scopes = ["cloud-platform"]
    }

    autoscaling_config {
      policy_uri = google_dataproc_autoscaling_policy.policy.name
    }
  }
}

resource "google_dataproc_autoscaling_policy" "policy" {
  policy_id = "dataproc-policy"
  location  = "us-central1"

  worker_config {
    max_instances = 10
    min_instances = 2
  }

  basic_algorithm {
    yarn_config {
      graceful_decommission_timeout = "30s"
      scale_up_factor               = 0.05
      scale_down_factor             = 1.0
    }
  }
}

---

Cloud Composer

Airflow Environment

resource "google_composer_environment" "airflow" {
  name   = "production-airflow"
  region = "us-central1"

  config {
    software_config {
      airflow_config_overrides = {
        core-dags_are_paused_at_creation = "True"
        webserver-rbac                   = "True"
      }

      pypi_packages = {
        pandas    = ""
        numpy     = ""
        requests  = ">=2.28.0"
      }

      env_variables = {
        ENVIRONMENT = "production"
      }
    }

    node_config {
      zone         = "us-central1-a"
      machine_type = "n1-standard-4"
      disk_size_gb = 100

      service_account = google_service_account.composer.email
    }

    workloads_config {
      scheduler {
        cpu        = 2
        memory_gb  = 7.5
        storage_gb = 5
        count      = 2
      }
      web_server {
        cpu        = 2
        memory_gb  = 7.5
        storage_gb = 5
      }
      worker {
        cpu        = 2
        memory_gb  = 7.5
        storage_gb = 5
        min_count  = 2
        max_count  = 6
      }
    }
  }
}

Sample DAG

from datetime import datetime, timedelta
from airflow import DAG
from airflow.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperator
from airflow.providers.google.cloud.transfers.gcs_to_bigquery import GCSToBigQueryOperator

default_args = {
    'owner': 'data-team',
    'depends_on_past': False,
    'start_date': datetime(2025, 1, 1),
    'email': ['alerts@example.com'],
    'email_on_failure': True,
    'retries': 3,
    'retry_delay': timedelta(minutes=5),
}

with DAG(
    'daily_analytics_pipeline',
    default_args=default_args,
    schedule_interval='0 2 * * *',  # Daily at 2am
    catchup=False,
) as dag:

    load_data = GCSToBigQueryOperator(
        task_id='load_data_from_gcs',
        bucket='data-bucket',
        source_objects=['data/*.csv'],
        destination_project_dataset_table='project.analytics.raw_events',
        write_disposition='WRITE_TRUNCATE',
    )

    transform = BigQueryInsertJobOperator(
        task_id='transform_data',
        configuration={
            'query': {
                'query': """
                    INSERT INTO `project.analytics.events`
                    SELECT * FROM `project.analytics.raw_events`
                    WHERE timestamp >= CURRENT_DATE()
                """,
                'useLegacySql': False,
            }
        },
    )

    load_data >> transform

Best Practices

BigQuery

• Use partitioned and clustered tables
• Query only needed columns (avoid SELECT *)
• Use LIMIT for exploratory queries
• Cache results (24 hours)
• Use BI Engine for dashboards

Pub/Sub

• Use dead letter topics
• Set appropriate ack deadlines
• Implement exponential backoff
• Use message ordering when needed
• Monitor subscription backlog

Dataflow

• Use windowing for streaming
• Implement exactly-once processing
• Monitor pipeline metrics
• Use Flex templates for custom pipelines
• Enable autoscaling

Dataproc

• Use preemptible workers (60-91% cheaper)
• Enable autoscaling
• Store data in Cloud Storage (ephemeral clusters)
• Use initialization actions
• Monitor YARN metrics

Composer

• Use Workload Identity
• Set appropriate resource limits
• Monitor DAG performance
• Use XCom for small data only
• Implement alerting

GCP ML/AI Services Reference

Patterns for Vertex AI, AutoML, and pre-trained AI services.

Table of Contents

1. Vertex AI 2. AutoML 3. Pre-trained APIs 4. TPUs

---

Vertex AI

Custom Training Job

from google.cloud import aiplatform

aiplatform.init(
    project='project-id',
    location='us-central1',
    staging_bucket='gs://my-bucket'
)

# Create custom training job
job = aiplatform.CustomTrainingJob(
    display_name='custom-training-job',
    script_path='train.py',
    container_uri='gcr.io/cloud-aiplatform/training/pytorch-gpu.1-13:latest',
    requirements=['pandas==2.0.0', 'scikit-learn==1.3.0'],
    model_serving_container_image_uri='gcr.io/cloud-aiplatform/prediction/pytorch-gpu.1-13:latest',
)

# Run training
model = job.run(
    dataset=dataset,
    replica_count=1,
    machine_type='n1-standard-8',
    accelerator_type='NVIDIA_TESLA_V100',
    accelerator_count=1,
    model_display_name='my-model',
    training_fraction_split=0.8,
    validation_fraction_split=0.1,
    test_fraction_split=0.1,
)

# Deploy model
endpoint = model.deploy(
    deployed_model_display_name='deployed-model',
    machine_type='n1-standard-4',
    min_replica_count=1,
    max_replica_count=10,
    accelerator_type='NVIDIA_TESLA_T4',
    accelerator_count=1,
)

# Make prediction
prediction = endpoint.predict(instances=[[5.1, 3.5, 1.4, 0.2]])

Vertex AI Pipelines

from kfp.v2 import dsl
from kfp.v2.dsl import component, pipeline

@component(base_image='python:3.9')
def preprocess_data(input_path: str, output_path: str):
    import pandas as pd
    df = pd.read_csv(input_path)
    # Preprocessing logic
    df.to_csv(output_path, index=False)

@component(base_image='gcr.io/cloud-aiplatform/training/pytorch-gpu.1-13:latest')
def train_model(data_path: str, model_path: str):
    # Training logic
    pass

@pipeline(name='ml-pipeline')
def ml_pipeline(input_path: str):
    preprocess_task = preprocess_data(input_path=input_path, output_path='/tmp/processed.csv')
    train_task = train_model(data_path=preprocess_task.output, model_path='/tmp/model')

# Compile and run
from kfp.v2 import compiler
compiler.Compiler().compile(pipeline_func=ml_pipeline, package_path='pipeline.json')

from google.cloud import aiplatform
aiplatform.PipelineJob(
    display_name='ml-pipeline-run',
    template_path='pipeline.json',
    parameter_values={'input_path': 'gs://bucket/data.csv'}
).run()

---

AutoML

AutoML Tables (Structured Data)

from google.cloud import aiplatform

# Create dataset
dataset = aiplatform.TabularDataset.create(
    display_name='sales-predictions',
    gcs_source='gs://bucket/sales_data.csv',
)

# Train AutoML model
job = aiplatform.AutoMLTabularTrainingJob(
    display_name='automl-sales-job',
    optimization_prediction_type='regression',
    optimization_objective='minimize-rmse',
)

model = job.run(
    dataset=dataset,
    target_column='sales_amount',
    training_fraction_split=0.8,
    validation_fraction_split=0.1,
    test_fraction_split=0.1,
    model_display_name='sales-prediction-model',
    budget_milli_node_hours=8000,  # 8 hours
)

AutoML Vision

# Create image dataset
dataset = aiplatform.ImageDataset.create(
    display_name='product-classification',
    gcs_source='gs://bucket/images.csv',  # CSV with image paths and labels
)

# Train image classification model
job = aiplatform.AutoMLImageTrainingJob(
    display_name='automl-image-job',
    prediction_type='classification',
    multi_label=False,
)

model = job.run(
    dataset=dataset,
    model_display_name='product-classifier',
    training_fraction_split=0.8,
    validation_fraction_split=0.1,
    test_fraction_split=0.1,
    budget_milli_node_hours=20000,  # 20 hours
)

# Deploy and predict
endpoint = model.deploy(machine_type='n1-standard-4')
prediction = endpoint.predict(instances=[{'content': 'gs://bucket/test_image.jpg'}])

---

Pre-trained APIs

Vision API

from google.cloud import vision

client = vision.ImageAnnotatorClient()

# Image from GCS
image = vision.Image()
image.source.image_uri = 'gs://bucket/image.jpg'

# Or from local file
with open('image.jpg', 'rb') as f:
    content = f.read()
image = vision.Image(content=content)

# Label detection
response = client.label_detection(image=image)
for label in response.label_annotations:
    print(f'{label.description}: {label.score}')

# Text detection (OCR)
response = client.text_detection(image=image)
print(response.full_text_annotation.text)

# Face detection
response = client.face_detection(image=image)
for face in response.face_annotations:
    print(f'Joy: {face.joy_likelihood}')
    print(f'Sorrow: {face.sorrow_likelihood}')

# Object localization
response = client.object_localization(image=image)
for obj in response.localized_object_annotations:
    print(f'{obj.name}: {obj.score}')

Natural Language API

from google.cloud import language_v1

client = language_v1.LanguageServiceClient()

text = "Google Cloud Platform provides powerful AI services."
document = language_v1.Document(content=text, type_=language_v1.Document.Type.PLAIN_TEXT)

# Sentiment analysis
sentiment = client.analyze_sentiment(document=document).document_sentiment
print(f'Sentiment score: {sentiment.score}')
print(f'Sentiment magnitude: {sentiment.magnitude}')

# Entity extraction
response = client.analyze_entities(document=document)
for entity in response.entities:
    print(f'{entity.name}: {entity.type_}')

# Syntax analysis
response = client.analyze_syntax(document=document)
for token in response.tokens:
    print(f'{token.text.content}: {token.part_of_speech.tag}')

Translation API

from google.cloud import translate_v2

client = translate_v2.Client()

# Translate text
result = client.translate('Hello, world!', target_language='es')
print(result['translatedText'])  # "¡Hola Mundo!"

# Detect language
result = client.detect_language('Bonjour')
print(result['language'])  # 'fr'

# Get supported languages
languages = client.get_languages()
for language in languages:
    print(f'{language["name"]}: {language["language"]}')

Speech-to-Text

from google.cloud import speech

client = speech.SpeechClient()

# Audio from GCS
audio = speech.RecognitionAudio(uri='gs://bucket/audio.wav')

# Or from local file
with open('audio.wav', 'rb') as f:
    content = f.read()
audio = speech.RecognitionAudio(content=content)

config = speech.RecognitionConfig(
    encoding=speech.RecognitionConfig.AudioEncoding.LINEAR16,
    sample_rate_hertz=16000,
    language_code='en-US',
    enable_automatic_punctuation=True,
)

# Synchronous recognition (< 60 seconds)
response = client.recognize(config=config, audio=audio)
for result in response.results:
    print(result.alternatives[0].transcript)

# Long audio recognition
operation = client.long_running_recognize(config=config, audio=audio)
response = operation.result(timeout=300)

---

TPUs

Cloud TPU Configuration

resource "google_tpu_node" "tpu" {
  name               = "ml-tpu"
  zone               = "us-central1-a"
  accelerator_type   = "v3-8"  # 8 cores
  tensorflow_version = "2.12.0"
  network            = google_compute_network.main.id
  cidr_block         = "10.0.0.0/29"

  scheduling_config {
    preemptible = false
  }

  labels = {
    environment = "production"
  }
}

TPU Training Example

import tensorflow as tf

# TPU initialization
resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='ml-tpu')
tf.config.experimental_connect_to_cluster(resolver)
tf.tpu.experimental.initialize_tpu_system(resolver)
strategy = tf.distribute.TPUStrategy(resolver)

# Model training with TPU
with strategy.scope():
    model = tf.keras.Sequential([
        tf.keras.layers.Dense(128, activation='relu'),
        tf.keras.layers.Dense(10, activation='softmax')
    ])

    model.compile(
        optimizer='adam',
        loss='sparse_categorical_crossentropy',
        metrics=['accuracy']
    )

# Train on TPU
model.fit(train_dataset, epochs=10, validation_data=val_dataset)

Service Selection Guide

Best Practices

• Start with pre-trained APIs for standard tasks
• Use AutoML for custom models with limited ML expertise
• Use Vertex AI custom training for advanced use cases
• Enable Vertex AI Model Monitoring for production models
• Use Feature Store for reusable features
• Implement ML pipelines for reproducibility
• Use TPUs for large-scale transformer model training
• Monitor prediction costs and latency

GCP Networking Reference

VPC, Load Balancing, CDN, and Cloud Armor patterns.

Table of Contents

• VPC Configuration
• Global Load Balancing
• Cloud Armor (WAF)
• VPC Peering
• Private Service Connection

VPC Configuration

resource "google_compute_network" "main" {
  name                    = "main-vpc"
  auto_create_subnetworks = false
  routing_mode            = "GLOBAL"
}

resource "google_compute_subnetwork" "private" {
  name          = "private-subnet"
  ip_cidr_range = "10.0.1.0/24"
  region        = "us-central1"
  network       = google_compute_network.main.id

  private_ip_google_access = true

  secondary_ip_range {
    range_name    = "gke-pods"
    ip_cidr_range = "10.1.0.0/16"
  }

  secondary_ip_range {
    range_name    = "gke-services"
    ip_cidr_range = "10.2.0.0/20"
  }

  log_config {
    aggregation_interval = "INTERVAL_5_SEC"
    flow_sampling        = 0.5
  }
}

# Cloud NAT for private instances
resource "google_compute_router" "router" {
  name    = "nat-router"
  region  = "us-central1"
  network = google_compute_network.main.id
}

resource "google_compute_router_nat" "nat" {
  name                               = "nat-gateway"
  router                             = google_compute_router.router.name
  region                             = google_compute_router.router.region
  nat_ip_allocate_option             = "AUTO_ONLY"
  source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"

  log_config {
    enable = true
    filter = "ERRORS_ONLY"
  }
}

Global Load Balancing

# Global HTTP(S) Load Balancer
resource "google_compute_global_address" "lb_ip" {
  name = "global-lb-ip"
}

resource "google_compute_global_forwarding_rule" "https" {
  name       = "https-forwarding-rule"
  target     = google_compute_target_https_proxy.default.id
  port_range = "443"
  ip_address = google_compute_global_address.lb_ip.address
}

resource "google_compute_target_https_proxy" "default" {
  name             = "https-proxy"
  url_map          = google_compute_url_map.default.id
  ssl_certificates = [google_compute_managed_ssl_certificate.default.id]
}

resource "google_compute_managed_ssl_certificate" "default" {
  name = "ssl-cert"
  managed {
    domains = ["example.com", "www.example.com"]
  }
}

resource "google_compute_url_map" "default" {
  name            = "url-map"
  default_service = google_compute_backend_service.default.id

  host_rule {
    hosts        = ["example.com"]
    path_matcher = "allpaths"
  }

  path_matcher {
    name            = "allpaths"
    default_service = google_compute_backend_service.default.id

    path_rule {
      paths   = ["/api/*"]
      service = google_compute_backend_service.api.id
    }
  }
}

resource "google_compute_backend_service" "default" {
  name                  = "backend-service"
  protocol              = "HTTP"
  timeout_sec           = 30
  health_checks         = [google_compute_health_check.default.id]
  load_balancing_scheme = "EXTERNAL_MANAGED"

  backend {
    group           = google_compute_region_network_endpoint_group.cloudrun.id
    balancing_mode  = "UTILIZATION"
    capacity_scaler = 1.0
  }

  cdn_policy {
    cache_mode                   = "CACHE_ALL_STATIC"
    default_ttl                  = 3600
    client_ttl                   = 7200
    max_ttl                      = 86400
    negative_caching             = true
    serve_while_stale            = 86400

    cache_key_policy {
      include_protocol = true
      include_host     = true
    }
  }

  iap {
    oauth2_client_id     = google_iap_client.default.client_id
    oauth2_client_secret = google_iap_client.default.secret
  }
}

Cloud Armor (WAF)

resource "google_compute_security_policy" "policy" {
  name = "security-policy"

  # Default rule
  rule {
    action   = "allow"
    priority = "2147483647"
    match {
      versioned_expr = "SRC_IPS_V1"
      config {
        src_ip_ranges = ["*"]
      }
    }
  }

  # Block specific countries
  rule {
    action   = "deny(403)"
    priority = "1000"
    match {
      expr {
        expression = "origin.region_code == 'CN' || origin.region_code == 'RU'"
      }
    }
  }

  # Rate limiting
  rule {
    action   = "rate_based_ban"
    priority = "2000"
    match {
      versioned_expr = "SRC_IPS_V1"
      config {
        src_ip_ranges = ["*"]
      }
    }
    rate_limit_options {
      conform_action = "allow"
      exceed_action  = "deny(429)"

      rate_limit_threshold {
        count        = 100
        interval_sec = 60
      }

      ban_duration_sec = 600
    }
  }

  # SQL injection protection
  rule {
    action   = "deny(403)"
    priority = "3000"
    match {
      expr {
        expression = "evaluatePreconfiguredExpr('sqli-stable')"
      }
    }
  }

  # XSS protection
  rule {
    action   = "deny(403)"
    priority = "3001"
    match {
      expr {
        expression = "evaluatePreconfiguredExpr('xss-stable')"
      }
    }
  }

  adaptive_protection_config {
    layer_7_ddos_defense_config {
      enable = true
    }
  }
}

# Attach to backend service
resource "google_compute_backend_service_security_policy_attachment" "policy" {
  backend_service = google_compute_backend_service.default.id
  security_policy = google_compute_security_policy.policy.id
}

VPC Peering

resource "google_compute_network_peering" "peering1" {
  name         = "peering-to-network2"
  network      = google_compute_network.network1.id
  peer_network = google_compute_network.network2.id

  export_custom_routes = true
  import_custom_routes = true
}

resource "google_compute_network_peering" "peering2" {
  name         = "peering-to-network1"
  network      = google_compute_network.network2.id
  peer_network = google_compute_network.network1.id

  export_custom_routes = true
  import_custom_routes = true
}

Private Service Connection

resource "google_compute_global_address" "private_ip_alloc" {
  name          = "private-ip-alloc"
  purpose       = "VPC_PEERING"
  address_type  = "INTERNAL"
  prefix_length = 16
  network       = google_compute_network.main.id
}

resource "google_service_networking_connection" "private_vpc_connection" {
  network                 = google_compute_network.main.id
  service                 = "servicenetworking.googleapis.com"
  reserved_peering_ranges = [google_compute_global_address.private_ip_alloc.name]
}