Infrastructure Cicd Data Engineering

# Infrastructure CI/CD for Data Engineering

> Skill by [ara.so](https://ara.so) — Data Skills collection

This project demonstrates practical CI/CD patterns for deploying data infrastructure changes using GitHub Actions, Terraform, and AWS. It uses OpenID Connect (OIDC) for secure, keyless authentication between GitHub Actions and AWS, eliminating the need for long-lived AWS credentials.

## What This Project Does

- **Bootstraps infrastructure**: Creates S3 backend for Terraform state and OIDC provider for GitHub Actions
- **Automates deployments**: Uses GitHub Actions workflows to plan and apply Terraform changes
- **Enforces reviews**: Requires manual approval before production deployments
- **Validates code**: Runs Terraform formatting and validation checks on PRs

## Project Structure

```
.
├── terraform/
│   ├── bootstrap/          # Initial setup (S3 backend, OIDC)
│   │   └── main.tf
│   └── main/              # Main infrastructure definitions
│       └── main.tf
├── .github/
│   └── workflows/
│       ├── ci.yml         # Format and validation checks
│       └── deploy.yml     # Deployment workflow
└── tear-down.sh           # Cleanup script
```

## Prerequisites

1. **AWS Account** with appropriate permissions
2. **Terraform** installed locally (v1.0+)
3. **GitHub Account** and repository access
4. **AWS CLI** configured with credentials

```bash
# Verify Terraform installation
terraform version

# Verify AWS credentials
aws sts get-caller-identity
```

## Bootstrap Setup

### Step 1: Create S3 Backend and OIDC Provider

The bootstrap process creates:
- S3 bucket for Terraform state storage
- DynamoDB table for state locking
- IAM OIDC provider for GitHub Actions
- IAM role that GitHub Actions will assume

```bash
# Initialize and apply bootstrap configuration
terraform -chdir=terraform/bootstrap init
terraform -chdir=terraform/bootstrap apply

# Capture the outputs
terraform -chdir=terraform/bootstrap output
```

**Expected output:**
```
github_actions_role_arn = "arn:aws:iam::123456789012:role/github-actions-role"
state_bucket_name = "my-terraform-state-bucket"
```

### Step 2: Configure GitHub Repository Secrets

Create a repository secret named `AWS_ROLE_ARN`:

1. Navigate to: `Settings → Secrets and variables → Actions → New repository secret`
2. Name: `AWS_ROLE_ARN`
3. Value: The ARN output from bootstrap (without quotes)

```bash
# Example ARN format (don't include quotes when pasting)
arn:aws:iam::123456789012:role/github-actions-role
```

### Step 3: Create GitHub Environment

Set up a production environment with manual approval:

1. Navigate to: `Settings → Environments → New environment`
2. Name: `production`
3. Configure protection rules:
   - ✅ Required reviewers (minimum 1)
   - Add yourself or team members as reviewers

## Bootstrap Terraform Configuration

**terraform/bootstrap/main.tf** (simplified example):

```hcl
terraform {
  required_version = ">= 1.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

# S3 bucket for Terraform state
resource "aws_s3_bucket" "terraform_state" {
  bucket = "${var.project_name}-terraform-state-${var.environment}"
  
  tags = {
    Name        = "Terraform State Bucket"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_buck