Iac Terraform Data Engineering

# IaC for Data Engineering with Terraform

> Skill by [ara.so](https://ara.so) — Data Skills collection.

This project provides Infrastructure-as-Code (IaC) templates and patterns for data engineers using Terraform to provision and manage AWS resources. It focuses on creating reproducible, version-controlled infrastructure for data platforms including S3 storage, EC2 compute instances, and IAM permissions.

## What This Project Does

- Provides Terraform configurations for common data engineering infrastructure on AWS
- Demonstrates IaC best practices for S3 buckets, EC2 instances, and IAM roles
- Shows state management and lifecycle operations for data infrastructure
- Teaches reproducible infrastructure provisioning for data pipelines

## Prerequisites

Before using this project, ensure you have:

1. **AWS Account** with root or admin access
2. **Terraform CLI** installed ([installation guide](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli))
3. **AWS CLI** installed and configured ([setup guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html))
4. **AWS Credentials** configured via `aws configure`

## AWS IAM Setup

Create an IAM user with appropriate permissions:

1. **Create IAM User**: Navigate to AWS Console → IAM → Users → Create user
2. **Create Inline Policy**: Attach a custom policy to the user
3. **Grant Permissions**: For development/learning, grant full access to:
   - Amazon S3
   - Amazon EC2
   - AWS IAM

**⚠️ Security Note**: Full service access is NOT recommended for production. Use least-privilege policies in production environments.

## Project Structure

```
terraform/
├── main.tf           # Main Terraform configuration
├── variables.tf      # Input variables (if present)
├── outputs.tf        # Output values (if present)
└── terraform.tfstate # State file (generated)
```

## Key Terraform Commands

### Initialize Terraform

Initialize the working directory and download provider plugins:

```bash
terraform -chdir=terraform init
```

### Validate Configuration

Check if the configuration is syntactically valid:

```bash
terraform -chdir=terraform validate
```

### Format Code

Automatically format Terraform files to canonical style:

```bash
terraform -chdir=terraform fmt
```

### Plan Infrastructure Changes

Preview what Terraform will create/modify/destroy:

```bash
terraform -chdir=terraform plan
```

### Apply Configuration

Create or update infrastructure:

```bash
terraform -chdir=terraform apply
```

Terraform will show a plan and ask for confirmation. Type `yes` to proceed.

### Auto-approve (for automation)

```bash
terraform -chdir=terraform apply -auto-approve
```

### Destroy Infrastructure

Remove all resources managed by Terraform:

```bash
terraform -chdir=terraform destroy
```

## Configuration

### Basic Terraform Configuration Example

Before applying, modify `terraform/main.tf` to customize resource names:

```hcl
# terraform/main.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

# S3 bucket for data storage
resource "aws_s3_bucket" "data_bucket" {
  bucket = "my-unique-data-engineering-bucket-12345"
  
  tags = {
    Name        = "Data Engineering Bucket"
    Environment = "dev"
    ManagedBy   = "Terraform"
  }
}

# EC2 instance for data processing
resource "aws_instance" "data_processor" {