Networking

You are an AWS networking architect. Design, review, and troubleshoot VPC architectures and network configurations.

VPC Design Principles

Subnet Tiers

Always design with three tiers:

• Public subnets: Resources that need direct internet access (ALBs, NAT Gateways, bastion hosts). Route table has 0.0.0.0/0 -> Internet Gateway.
• Private subnets: Application workloads (EC2, ECS, Lambda). Route table has 0.0.0.0/0 -> NAT Gateway. Can reach the internet but are not reachable from it.
• Isolated subnets: Databases and sensitive workloads. No route to the internet at all. Access AWS services only through VPC endpoints.

CIDR Planning

• Use /16 for the VPC (65,536 IPs) unless you have a reason not to
• Use /20 or /24 per subnet depending on expected scale
• Reserve CIDR space for future expansion — you cannot resize a VPC CIDR easily
• Avoid overlapping CIDRs across VPCs if you ever plan to peer them or use Transit Gateway
• Use RFC 1918 ranges: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16

Availability Zones

• Minimum 2 AZs for production. 3 AZs is the standard for high availability.
• Each tier gets one subnet per AZ (e.g., 3 AZs x 3 tiers = 9 subnets)

Security Groups vs NACLs

Opinionated guidance:

• Security groups are your primary network control. Use them for everything.
• NACLs are defense-in-depth only. Do not use NACLs as your main firewall — they are harder to manage and debug.
• Reference security groups by ID (not CIDR) to allow traffic between resources. This is more maintainable and self-documenting.
• One security group per logical role (e.g., alb-sg, app-sg, db-sg). Chain them: ALB -> App -> DB.

VPC Endpoints

Gateway Endpoints (free)

• S3 and DynamoDB only
• Added to route tables — no ENI, no security group
• Always create these — they are free (no hourly charge, no per-GB data processing fee), they keep S3/DynamoDB traffic on the AWS backbone instead of traversing NAT Gateways (which charge $0.045/GB processed), and they reduce latency by avoiding the extra hop through NAT. The only cost is a route table entry.

Interface Endpoints (cost per hour + data)

• All other AWS services (STS, Secrets Manager, ECR, CloudWatch, KMS, etc.)
• Creates an ENI in your subnet — requires a security group
• Enable Private DNS so the default service endpoint resolves to the private IP
• Prioritize these for isolated subnets: ecr.api, ecr.dkr, s3 (gateway), logs, sts, secretsmanager, kms

Transit Gateway

Use Transit Gateway when:

• You have more than 2 VPCs that need to communicate
• You need hub-and-spoke or any-to-any connectivity
• You need centralized egress or ingress through a shared services VPC

Do NOT use VPC peering for more than 2-3 VPCs — it does not scale (N*(N-1)/2 connections).

Key Transit Gateway patterns:

• Shared Services VPC: Central VPC with DNS, logging, security tools. All spoke VPCs route through TGW.
• Centralized Egress: Single NAT Gateway in a shared VPC. All private subnets route 0.0.0.0/0 through TGW to the shared VPC.
• Segmentation via route tables: Use separate TGW route tables for prod, staging, dev to isolate environments.

VPC Peering

• Point-to-point only. Not transitive — if A peers with B and B peers with C, A cannot reach C.
• Works cross-region and cross-account
• Good for 2-3 VPCs. Beyond that, use Transit Gateway.
• CIDRs must not overlap

Route53

Hosted Zones

• Public hosted zone: DNS for internet-facing resources. NS records must be registered with your domain registrar.
• Private hosted zone: DNS for internal resources. Associated with one or more VPCs. Not resolvable from the internet.

Routing Policies

• Simple: Single resource. Default.
• Weighted: Split traffic by percentage. Good for canary deployments.
• Latency-based: Route to the lowest-latency region. Use for multi-region apps.
• Failover: Active/passive. Requires health checks.
• Geolocation: Route by user's country/continent. Good for compliance (data residency).
• Geoproximity: Route by geographic distance with bias. Use Traffic Flow.
• Multivalue Answer: Return multiple healthy IPs. Poor man's load balancer (use ALB instead).

Health Checks

• Always attach health checks to failover and latency records
• Health checks can monitor an endpoint, a CloudWatch alarm, or other health checks (calculated)
• Health check interval: 30s standard, 10s fast (costs more)

NAT Gateway

• One per AZ for high availability. A single NAT Gateway is a single point of failure.
• Placed in public subnets
• Costs: per-hour charge + per-GB data processing. This adds up fast.
• For cost savings in dev/staging: use a single NAT Gateway (accept the AZ risk) or use NAT instances
• If you only need AWS service access (not general internet), use VPC endpoints instead — cheaper and more secure

Common CLI Commands

# Describe VPCs
aws ec2 describe-vpcs --query 'Vpcs[*].{ID:VpcId,CIDR:CidrBlock,Name:Tags[?Key==`Name`].Value|[0]}'

# Describe subnets in a VPC
aws ec2 describe-subnets --filters "Name=vpc-id,Values=vpc-xxx" --query 'Subnets[*].{ID:SubnetId,AZ:AvailabilityZone,CIDR:CidrBlock,Public:MapPublicIpOnLaunch}'

# List security group rules
aws ec2 describe-security-group-rules --filter "Name=group-id,Values=sg-xxx"

# List VPC endpoints
aws ec2 describe-vpc-endpoints --filters "Name=vpc-id,Values=vpc-xxx" --query 'VpcEndpoints[*].{ID:VpcEndpointId,Service:ServiceName,Type:VpcEndpointType}'

# Check route tables
aws ec2 describe-route-tables --filters "Name=vpc-id,Values=vpc-xxx" --query 'RouteTables[*].{ID:RouteTableId,Routes:Routes}'

# List Transit Gateway attachments
aws ec2 describe-transit-gateway-attachments --query 'TransitGatewayAttachments[*].{ID:TransitGatewayAttachmentId,ResourceType:ResourceType,State:State}'

# Test connectivity (VPC Reachability Analyzer)
aws ec2 create-network-insights-path --source eni-xxx --destination eni-yyy --protocol TCP --destination-port 443

# Route53 — list hosted zones
aws route53 list-hosted-zones --query 'HostedZones[*].{Name:Name,ID:Id,Private:Config.PrivateZone}'

# Route53 — list records
aws route53 list-resource-record-sets --hosted-zone-id /hostedzone/ZXXXXX

Output Format

Reference Files

• references/cidr-planning.md — CIDR allocation strategies, worked examples for three-tier VPCs, multi-account planning, EKS/Lambda IP considerations, secondary CIDRs, and AWS VPC IPAM
• references/vpc-endpoint-catalog.md — Catalog of commonly used VPC endpoints organized by priority, with configuration guidance, security groups, cost analysis, and endpoint policies

Related Skills

• security-review — Network security posture, security group audits, NACLs
• iam — VPC endpoint policies, resource-based access control
• ec2 — Instance placement, security groups, and subnet selection
• ecs — awsvpc networking, task-level security groups, service discovery, ECR endpoint requirements
• eks — Pod networking, secondary CIDRs, CNI configuration, IP address planning
• lambda — Lambda VPC configuration, ENI usage, endpoint requirements
• rds-aurora — Database subnet groups, isolated subnet placement

Anti-Patterns

• Single AZ NAT Gateway in production: One AZ goes down, all private subnets lose internet access. Use one NAT per AZ.
• Using NACLs as primary firewall: Stateless rules are error-prone. Use security groups. NACLs are backup only.
• Overly permissive security groups: 0.0.0.0/0 on port 22 or 3389 is never acceptable in production. Use Systems Manager Session Manager instead.
• No VPC endpoints for S3/DynamoDB: Gateway endpoints are free. Always create them.
• Overlapping CIDRs: Makes peering and Transit Gateway impossible later. Plan CIDR allocation upfront.
• Public subnets for everything: Databases, application servers, and internal services belong in private or isolated subnets. Only load balancers and NAT Gateways need public subnets.
• Hardcoding IPs instead of using DNS: Use Route53 private hosted zones and service discovery. IPs change; DNS names persist.
• Not enabling VPC Flow Logs: Essential for security auditing and debugging. Enable at minimum at the VPC level with a 14-day retention in CloudWatch Logs.
• Using VPC peering for 5+ VPCs: The mesh becomes unmanageable. Switch to Transit Gateway.

CIDR Planning for AWS VPCs

Strategies and worked examples for VPC and subnet CIDR allocation.

CIDR Fundamentals

AWS reserves 5 IPs per subnet: network address, VPC router, DNS, future use, broadcast.

Strategy 1: Standard Three-Tier, Three-AZ VPC

The default starting point for most production workloads.

VPC CIDR: 10.0.0.0/16 (65,531 usable IPs)

Key points:

• Each subnet has 4,091 usable IPs, enough for most workloads
• Reserved space (10.0.144.0 - 10.0.255.255) for future tiers (e.g., caching layer, additional AZs)
• Total used: 9 subnets, ~36,819 IPs. Remaining: ~28,700 IPs.

Strategy 2: Compact VPC for Dev/Test

Smaller allocation to conserve address space. Suitable for non-production environments.

VPC CIDR: 10.1.0.0/20 (4,091 usable IPs)

Key points:

• 2 AZs for dev/test (cost savings)
• 251 IPs per subnet is sufficient for most non-production workloads
• Fits inside a /20, leaving room for many dev VPCs in the 10.1.0.0/16 range

Strategy 3: Multi-Account CIDR Allocation

When using AWS Organizations with multiple accounts, plan CIDR ranges at the organization level to avoid overlap.

Organization supernet: 10.0.0.0/8

Account Allocation:
  Production    : 10.0.0.0/16   (65K IPs)
  Staging       : 10.1.0.0/16   (65K IPs)
  Development   : 10.2.0.0/16   (65K IPs)
  Shared Svcs   : 10.3.0.0/16   (65K IPs)
  Security      : 10.4.0.0/16   (65K IPs)
  Sandbox       : 10.5.0.0/16   (65K IPs)
  Reserved      : 10.6.0.0/15 through 10.255.0.0/16

Within each account:

10.0.0.0/16 (Production Account)
  ├── us-east-1 VPC  : 10.0.0.0/18   (16K IPs)
  ├── us-west-2 VPC  : 10.0.64.0/18  (16K IPs)
  ├── eu-west-1 VPC  : 10.0.128.0/18 (16K IPs)
  └── Reserved       : 10.0.192.0/18 (16K IPs)

Rules:

• No CIDR overlap across any account or region
• Each VPC gets a /18 within its account's /16
• Transit Gateway or VPC peering works without conflicts
• Document the allocation in a central IPAM or spreadsheet

Strategy 4: AWS VPC IPAM

For organizations with 10+ VPCs, use AWS VPC IPAM (IP Address Manager) instead of spreadsheets.

IPAM hierarchy:

IPAM Pool (Organization level): 10.0.0.0/8
  ├── Regional Pool (us-east-1): 10.0.0.0/12
  │     ├── Production Pool: 10.0.0.0/14
  │     ├── Non-Prod Pool:   10.4.0.0/14
  │     └── Reserved:        10.8.0.0/13
  └── Regional Pool (eu-west-1): 10.16.0.0/12
        ├── Production Pool: 10.16.0.0/14
        └── Non-Prod Pool:   10.20.0.0/14

Benefits:

• Automatic CIDR allocation (no manual tracking)
• Prevents overlapping allocations
• Integrates with AWS Organizations and RAM
• Compliance rules enforce minimum/maximum CIDR sizes

EKS-Specific CIDR Considerations

EKS consumes IPs aggressively. Each pod gets its own IP from the subnet by default (VPC CNI plugin).

IP consumption calculation:

IPs needed = (max pods per node) x (max nodes) + (node IPs) + (overhead)

Example:
  30 pods/node x 50 nodes = 1,500 pod IPs
  + 50 node IPs
  + services, DaemonSets, buffer
  ≈ 2,000 IPs minimum → /21 per AZ (2,043 usable)

Mitigation strategies when IPs are limited:

• Secondary CIDR: Add a 100.64.0.0/16 (CGNAT range) secondary CIDR to the VPC for pod networking
• Prefix delegation: Assign /28 prefixes to ENIs instead of individual IPs (increases pod density per node)
• Custom networking: Use a separate subnet CIDR for pods vs. nodes
• IPv6: Dual-stack VPC eliminates IPv4 address exhaustion entirely

Lambda-Specific CIDR Considerations

Lambda functions in a VPC consume ENIs (and thus IPs) from your subnets. Since 2019, Lambda uses Hyperplane ENIs that are shared across invocations, but you still need adequate capacity.

Rule of thumb: Allocate at least a /24 per AZ for Lambda-heavy workloads. Monitor ENI usage.

Common Mistakes

Secondary CIDR Blocks

If you run out of IPs in your primary CIDR, you can add secondary CIDR blocks to a VPC.

Constraints:

• Up to 5 IPv4 CIDRs per VPC (adjustable)
• The secondary CIDR must not overlap with the primary or any peered VPC CIDRs
• 100.64.0.0/10 (CGNAT range) is commonly used for secondary CIDRs, especially for EKS pod networking
• Secondary CIDRs can be from different RFC 1918 ranges than the primary

# Add secondary CIDR
aws ec2 associate-vpc-cidr-block --vpc-id vpc-xxx --cidr-block 100.64.0.0/16

# Create subnets in the secondary CIDR
aws ec2 create-subnet --vpc-id vpc-xxx --cidr-block 100.64.0.0/20 --availability-zone us-east-1a

VPC Endpoint Catalog

Commonly used VPC endpoints with configuration guidance. Organized by priority.

Always Create (Free Gateway Endpoints)

These are free. There is no reason not to create them in every VPC.

Gateway endpoint notes:

• Added to route tables (not subnet ENIs)
• No security group required
• Use VPC endpoint policies to restrict which buckets/tables can be accessed
• Must be associated with the route tables for subnets that need access

# Create S3 gateway endpoint
aws ec2 create-vpc-endpoint \
  --vpc-id vpc-xxx \
  --service-name com.amazonaws.us-east-1.s3 \
  --route-table-ids rtb-aaa rtb-bbb rtb-ccc

# Create DynamoDB gateway endpoint
aws ec2 create-vpc-endpoint \
  --vpc-id vpc-xxx \
  --service-name com.amazonaws.us-east-1.dynamodb \
  --route-table-ids rtb-aaa rtb-bbb rtb-ccc

Priority 1: Essential for Isolated Subnets

If you have subnets with no internet access (isolated/private without NAT), you need these interface endpoints for basic AWS service connectivity.

Cost per interface endpoint: ~$0.01/hour per AZ (~$7.20/month per AZ) + $0.01/GB data processed.

For 3 AZs: ~$21.60/month per endpoint. 8 endpoints = ~$173/month. Compare against NAT Gateway cost.

Priority 2: Container Workloads (ECS/EKS)

Required when running containers in private/isolated subnets.

All three are required for ECR image pulls. Missing any one causes pull failures.

# Create ECR endpoints (both required)
for svc in ecr.api ecr.dkr; do
  aws ec2 create-vpc-endpoint \
    --vpc-id vpc-xxx \
    --vpc-endpoint-type Interface \
    --service-name com.amazonaws.us-east-1.$svc \
    --subnet-ids subnet-aaa subnet-bbb subnet-ccc \
    --security-group-ids sg-xxx \
    --private-dns-enabled
done

EKS additional endpoints:

Priority 3: Lambda in VPC

Lambda in a VPC needs these endpoints to call AWS services without NAT Gateway.

Note: Lambda functions in a VPC already need the Priority 1 endpoints (STS, Logs, KMS, etc.).

Priority 4: Data and Analytics

Priority 5: Security and Compliance

Interface Endpoint Configuration

All interface endpoints share these configuration requirements:

Security Group

Create a dedicated security group for VPC endpoints:

# Create endpoint security group
aws ec2 create-security-group \
  --group-name vpc-endpoints-sg \
  --description "Security group for VPC interface endpoints" \
  --vpc-id vpc-xxx

# Allow HTTPS from VPC CIDR
aws ec2 authorize-security-group-ingress \
  --group-id sg-xxx \
  --protocol tcp \
  --port 443 \
  --cidr 10.0.0.0/16

All AWS API calls go over HTTPS (port 443). The security group only needs inbound 443 from your VPC CIDR.

Private DNS

• Enable Private DNS on interface endpoints so the default AWS service endpoint (e.g., sqs.us-east-1.amazonaws.com) resolves to the private endpoint IP
• Without Private DNS, you must configure your SDK/application to use the VPC endpoint DNS name
• Private DNS requires enableDnsSupport and enableDnsHostnames on the VPC

Subnet Placement

• Place endpoints in the same subnets as the resources that use them
• For high availability, create the endpoint in all AZs where you have workloads
• Each AZ creates one ENI per endpoint

Cost Optimization

VPC endpoints vs. NAT Gateway:

Break-even analysis:

• If you use <5 AWS services from private subnets, endpoints are cheaper
• If you use 5+ services AND need general internet access, NAT Gateway may be simpler
• If you have isolated subnets (no internet), endpoints are your only option
• Combining both is common: endpoints for high-volume AWS services (S3, ECR, Logs), NAT for occasional internet access

Cost-saving tips:

• Gateway endpoints (S3, DynamoDB) are always free. Create them first.
• Share endpoints across subnets in the same AZ. One endpoint serves all resources in its AZ.
• Review endpoint data processing costs. High-volume services (S3, ECR pulls) benefit most from endpoints.

Endpoint Policies

Restrict what actions can be performed through an endpoint. Defense in depth.

{
  "Statement": [
    {
      "Sid": "AllowSpecificBucket",
      "Effect": "Allow",
      "Principal": "*",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::my-app-bucket/*"
    }
  ]
}

Use endpoint policies to:

• Restrict S3 access to specific buckets (prevent data exfiltration)
• Restrict ECR access to your account's repositories
• Limit KMS to specific key ARNs
• Prevent calling services in other accounts