Aws Containers

AWS Containers

Service Overview

When a developer says "deploy my container" without naming a service: recommend ECS Express Mode for simple HTTP apps (replaces App Runner for new customers). Recommend ECS Fargate for everything else. Never recommend EKS unless they explicitly ask for Kubernetes.

Overview

Provides expertise for building, deploying, and operating containerized workloads using Amazon ECS, AWS Fargate, Amazon ECR, and AWS App Runner.

Recommended setup: Install the AWS MCP server for sandboxed execution, audit logging, and enterprise controls. See: aws.amazon.com/mcp

Without AWS MCP: This skill works with any agent that has AWS CLI access. All commands use standard AWS CLI syntax.

When NOT to use this skill:

• Kubernetes or EKS workloads → use the kubernetes skill
• CI/CD pipeline setup for container deployments → use the deploy skill
• VPC subnet design and security group architecture → use the networking skill
• Running code without containers (Lambda, Step Functions) → use the serverless skill

Before executing any commands:

• You MUST verify AWS CLI v2 is installed and configured before running commands
• You MUST inform the user if required tools (AWS CLI, Docker, Session Manager plugin) are missing
• You MUST respect the user's decision to abort at any point

Gotchas

Apply these every time. Each corrects a mistake agents make without explicit instruction.

1. Fargate CPU/memory must be valid combinations. Arbitrary values cause Invalid 'cpu' setting for task:

• 256 (0.25 vCPU): 512 MiB, 1 GB, 2 GB
• 512 (0.5 vCPU): 1–4 GB (1 GB increments)
• 1024 (1 vCPU): 2–8 GB (1 GB increments)
• 2048 (2 vCPU): 4–16 GB (1 GB increments)
• 4096 (4 vCPU): 8–30 GB (1 GB increments)
• 8192 (8 vCPU): 16–60 GB (4 GB increments)
• 16384 (16 vCPU): 32–120 GB (8 GB increments)

If the user requests an invalid combination, tell them and recommend the nearest valid option. You MUST NOT silently produce an invalid task definition.

2. Fargate requires `awsvpc` networking mode — no exceptions. Agents frequently suggest bridge or host mode for Fargate tasks, which causes immediate registration failure. You MUST set networkMode to awsvpc for all Fargate task definitions. On EC2, awsvpc is recommended; bridge is legacy only.

3. Execution role vs task role — never confuse them. executionRoleArn: ECS agent uses it to pull images, fetch secrets, write logs. taskRoleArn: application code uses it to call AWS APIs. ECS Exec permissions (ssmmessages:*) go on the task role. ECR pull permissions go on the execution role. ecr:GetAuthorizationToken MUST use Resource: "*" (registry-level action).

4. Secrets are injected at task launch only — no hot-reload. Changed secrets require aws ecs update-service --force-new-deployment. To reference a specific JSON key in Secrets Manager: arn:aws:secretsmanager:region:account:secret:name-hash:json-key:: — the trailing colons are required (they represent empty version-stage and version-id fields). You can also use SSM Parameter Store with valueFrom pointing to the parameter ARN — the execution role needs ssm:GetParameters permission.

5. ALB deregistration delay defaults to 300s — reduce to 30–60s. This is the #1 cause of slow deployments. Set it on the target group. It SHOULD exceed your longest request duration.

6. Set `healthCheckGracePeriodSeconds` on every ECS service behind an ALB. Without it, the ALB marks tasks unhealthy before they're ready, the circuit breaker counts failures, and the deployment rolls back. JVM/Spring Boot apps need 60–120s.

7. Always enable deployment circuit breaker with rollback. Without it, bad deployments stay "in progress" for 30+ minutes. In CDK: circuitBreaker: { rollback: true } (specifying the property implicitly enables it; enable defaults to true).

8. Private subnet Fargate tasks need NAT or all four VPC endpoints. Required endpoints: ecr.dkr (interface), ecr.api (interface), s3 (gateway — ECR stores layers in S3), logs (interface — for CloudWatch). The S3 gateway endpoint is the most commonly missed. For ECS Exec, also add ssmmessages.

9. ECR lifecycle policies evaluate within 24 hours — not immediately. Multi-architecture images referenced by a manifest list cannot be expired until the manifest list is deleted first. Preview before applying: first aws ecr start-lifecycle-policy-preview --repository-name $REPO, then aws ecr get-lifecycle-policy-preview --repository-name $REPO --output json to see which images would be affected.

10. ECS Exec requires task role permissions, NOT execution role. The task role needs ssmmessages:CreateControlChannel, CreateDataChannel, OpenControlChannel, OpenDataChannel. Tasks launched before enabling enableExecuteCommand do NOT support ECS Exec — force a new deployment. The container image must include the binary specified in --command (e.g., /bin/sh for interactive sessions). For command logging to S3 or CloudWatch Logs, script and cat must also be installed. Fargate platform version MUST be 1.4.0+.

11. `awslogs` log driver mode — check your account's default. Per ECS docs, the ECS service defaults to non-blocking mode, which drops logs when the buffer fills. The defaultLogDriverMode account setting can override this per account. For guaranteed log delivery (audit/compliance), explicitly set "mode": "blocking" in logConfiguration.options. Check your effective default: aws ecs list-account-settings --name defaultLogDriverMode --effective-settings --output json.

12. App Runner VPC connector routes ALL application-initiated outbound traffic through the VPC. (App Runner is sunset — new customers should use ECS Express Mode instead.) Without a NAT gateway, external API calls and AWS service calls from your application code break. App Runner's own managed traffic (pulling images, pushing logs, retrieving secrets) is NOT routed through the VPC and is unaffected. Implement retry logic with backoff for database connections at startup.

13. For `desiredCount=1` zero-downtime deploys: `minimumHealthyPercent=100, maximumPercent=200`. This requires capacity for 2 tasks during deployment. You MUST NOT set minimumHealthyPercent=0 if zero downtime is required.

14. 502 Bad Gateway from ALB — check in this order: (a) Container not listening on the port in the target group. (b) Container crashing before responding. (c) Task security group doesn't allow inbound from ALB security group on the container port. (d) Health check path returns non-200. (e) Health check timeout exceeds response time.

15. Fargate platform version: always use `LATEST` or `1.4.0`. Version 1.3.0 is being retired June 15, 2026 and terminated June 30, 2026.

16. SQS worker scaling: use a custom backlog-per-task metric. Raw ApproximateNumberOfMessagesVisible with target tracking doesn't work because adding tasks doesn't reduce queue depth proportionally. Use custom metric (ApproximateNumberOfMessagesVisible / RunningTaskCount) with target tracking, or use step scaling. CDK QueueProcessingFargateService handles this automatically via scalingSteps. Workers MUST handle SIGTERM gracefully within stopTimeout (default 30s, max 120s on Fargate).

17. Blue/green deployments: use native ECS blue/green (July 2025+) for new services. Supports all-at-once, canary, and linear traffic shifting (canary/linear added October 2025), plus Service Connect, headless services, EBS volumes, and lifecycle hooks. CodeDeploy blue/green is now legacy — native ECS blue/green has full feature parity.

18. Container dependency `HEALTHY` condition requires a health check on the dependency container. Without a configured health check, the dependent container never starts — ECS does not progress it to its next state. If startTimeout is set (max 120s), the dependency times out and the task fails; if not set, the dependent container blocks indefinitely. For init containers, use SUCCESS condition instead.

Quick-Start: CDK Fargate Web App

import * as cdk from 'aws-cdk-lib';
import * as ecs from 'aws-cdk-lib/aws-ecs';
import * as ecsPatterns from 'aws-cdk-lib/aws-ecs-patterns';

const service = new ecsPatterns.ApplicationLoadBalancedFargateService(this, 'WebApp', {
  taskImageOptions: {
    image: ecs.ContainerImage.fromEcrRepository(repo, 'latest'),
    containerPort: 8080,
    secrets: { DB_PASSWORD: ecs.Secret.fromSecretsManager(dbSecret) },
  },
  cpu: 512,
  memoryLimitMiB: 1024,
  desiredCount: 2,
  publicLoadBalancer: true,
  circuitBreaker: { rollback: true },
  minHealthyPercent: 100,
});

service.targetGroup.setAttribute('deregistration_delay.timeout_seconds', '30');

const scaling = service.service.autoScaleTaskCount({ minCapacity: 2, maxCapacity: 10 });
scaling.scaleOnCpuUtilization('CpuScaling', { targetUtilizationPercent: 70 });

CDK L3 patterns auto-create VPC, cluster, ALB, target group, and security groups. For production, create these separately and pass them in. ApplicationLoadBalancedFargateService defaults to assignPublicIp: false — tasks in public subnets need assignPublicIp: true for internet access, or use private subnets with NAT.

Quick-Start: ECS Exec

# 1. Enable on the service (existing tasks won't support it — force new deployment)
aws ecs update-service --cluster $CLUSTER --service $SERVICE \
  --enable-execute-command --force-new-deployment --output json

# 2. Connect (task role must have ssmmessages:* permissions)
aws ecs execute-command --cluster $CLUSTER --task $TASK_ID \
  --container $CONTAINER --interactive --command "/bin/sh"

If TargetNotConnectedException: wait 30–60s for SSM agent startup, check NAT/VPC endpoint for ssmmessages, verify task role (not execution role) has permissions.

Common Workflows

Use the best available tool for AWS operations (MCP server, AWS CLI, or SDK). The commands below show the AWS CLI form.

Read reference files only when the conversation requires deeper detail.

• Read references/task-definition-authoring.md if the user needs to author a task definition, configure CPU/memory, set up networking modes, inject secrets, mount volumes, or configure container dependencies.
• Read references/fargate-service-deployment.md if the user needs to deploy a Fargate service behind an ALB, configure health checks, tune deregistration delay, set up path-based routing, or handle private subnet networking.
• Read references/ecr-repository-management.md if the user needs ECR lifecycle policies, image scanning, cross-account image pulls, or is debugging image pull errors.
• Read references/ecs-exec-debugging.md if the user needs to set up ECS Exec, debug TargetNotConnectedException, configure session logging, or validate ECS Exec prerequisites.
• Read references/service-scaling-and-updates.md if the user needs auto-scaling, deployment strategies (rolling, blue/green), circuit breaker configuration, or Service Connect setup.
• Read references/app-runner-guide.md if the user has an existing App Runner service, needs to troubleshoot App Runner connectivity, or wants to migrate from App Runner to ECS Express Mode.
• Read references/ecs-infrastructure-patterns.md if the user needs CDK or CloudFormation examples for Fargate services, SQS workers, scheduled tasks, EFS volumes, ECS Exec, path-based routing, private subnets, or FireLens.
• Read references/ecs-logging-and-firelens.md if the user needs awslogs configuration, FireLens/Fluent Bit setup, multiline log handling, or guaranteed log delivery.
• Read references/ecs-troubleshooting-guide.md if the user is debugging task placement failures, OOM kills (exit code 137), health check failures, image pull errors, or networking issues in private subnets.
• Read references/fargate-spot.md if the user asks about Fargate Spot pricing, capacity provider strategies, or interruption handling.

Decision Guide: ECS Express Mode vs ECS Fargate

App Runner: Sunset April 30, 2026 — no new customers, no new features. Existing customers should migrate to ECS Express Mode. See App Runner Availability Change.

Default recommendation: Use ECS Fargate for production workloads. Use ECS Express Mode for the simplest path (new customers).

Troubleshooting

CannotPullContainerError

Cause: Task cannot reach ECR. In private subnets, tasks need NAT gateway or VPC endpoints (ecr.api, ecr.dkr, s3 gateway, logs). Fix: Verify route table has a route to NAT gateway or create the required VPC endpoints. Verify the execution role has ecr:GetDownloadUrlForLayer, ecr:BatchGetImage, ecr:GetAuthorizationToken (Resource: "*"). Check security group allows outbound HTTPS (443).

Task failed ELB health checks

Cause: Health check path returns non-200, container not listening on the configured port, or health check grace period too short. Fix: Verify the container responds on the health check path and port. Set healthCheckGracePeriodSeconds to at least 60s (longer for JVM apps). Ensure the security group allows traffic from the ALB security group on the container port.

OutOfMemoryError / exit code 137

Cause: Container exceeded its memory hard limit (SIGKILL). On Fargate, task-level memory is the hard limit. Fix: Increase task-level memory. For JVM apps, use -XX:MaxRAMPercentage=75 instead of fixed -Xmx — this automatically adapts to the container's memory allocation. Check container-level memory (hard limit) vs memoryReservation (soft limit).

AccessDeniedException on AWS API calls from container

Cause: Permissions are on the execution role instead of the task role, or the task role is missing. Fix: Verify the task definition has taskRoleArn set (not just executionRoleArn). Add the required permissions to the task role.

Service stuck deploying / tasks keep restarting

Cause: Deployment circuit breaker not enabled, or health check failing on new tasks. Fix: Enable circuit breaker with rollback. Check service events: aws ecs describe-services --cluster $CLUSTER --services $SERVICE --output json. Check stopped task reasons: aws ecs describe-tasks --cluster $CLUSTER --tasks $TASK_ID --output json.

ECS Exec TargetNotConnectedException

Cause: SSM agent not running, missing task role permissions, or missing VPC endpoint. Fix: Verify enableExecuteCommand is true on the service. Check the task role has SSM permissions. For private subnets, create the ssmmessages VPC endpoint. Verify with aws ecs describe-tasks that ExecuteCommandAgent status is RUNNING.

Error retry classification

Security Considerations

• You MUST use IAM roles (execution role + task role) — never embed credentials in container images or environment variables
• You MUST use Secrets Manager or SSM Parameter Store for sensitive configuration, injected via the secrets field in the task definition
• You SHOULD enable ECR image scanning on push for vulnerability detection
• You SHOULD use private subnets with NAT gateway or VPC endpoints for production workloads
• You MUST enable CloudTrail for ECS API audit logging
• You SHOULD configure CloudWatch Container Insights for monitoring
• You SHOULD use readonlyRootFilesystem: true in container definitions where possible (note: incompatible with ECS Exec)
• You MUST scope task role permissions to specific resources — avoid * wildcards and *FullAccess policies
• You MUST confirm with the user before executing destructive operations: --force-new-deployment (replaces all running tasks), delete-service, deregister-task-definition. ECS does not support --dry-run — use the plan-validate-execute pattern: explain what will happen, get confirmation, then execute
• You SHOULD use ACM certificates with HTTPS listeners on ALBs fronting ECS services — per ECS network security best practices: "provision certificates for the load balancer using AWS Certificate Manager (ACM)"
• You SHOULD avoid logging sensitive data (secrets, PII, tokens) in container stdout/stderr — these flow to CloudWatch Logs via the awslogs driver. If sensitive data may appear in logs, enable CloudWatch Logs encryption with a KMS key
• You SHOULD attach an AWS WAF WebACL to internet-facing ALBs for defense in depth against common web exploits
• You SHOULD include aws:SourceArn and aws:SourceAccount condition keys in ECR repository policies for cross-account access to prevent confused deputy attacks

Additional Resources

• Amazon ECS Developer Guide
• Amazon ECS API Reference
• Amazon ECS Best Practices Guide
• Amazon ECR User Guide
• AWS Fargate Documentation
• ECS Express Mode Getting Started
• ECS Security Best Practices
• App Runner Developer Guide (existing customers)
• App Runner Availability Change (Sunset)

App Runner Guide

⚠️ App Runner was sunset April 30, 2026. No new customers. No new features. Existing customers should migrate to ECS Express Mode. See: App Runner Availability Change

This reference file is for existing App Runner customers who need to operate their current services or migrate to ECS Express Mode. Do NOT recommend App Runner for new projects.

Table of Contents

• Verify Dependencies
• Critical: App Runner Sunset Notice
• ECS Express Mode as Replacement
• Comparison: App Runner vs ECS Express Mode vs ECS Fargate
• Auto Scaling Behavior
• VPC Connector Gotchas
• Migration Guide: App Runner to ECS Express Mode
• Security Considerations

---

Verify Dependencies

Operators MUST confirm the following before proceeding:

---

Critical: App Runner Sunset Notice

App Runner is no longer accepting new customers after April 30, 2026.

Existing customers MAY continue using the service, but SHOULD plan migration.

See: <https://docs.aws.amazon.com/apprunner/latest/dg/apprunner-availability-change.html>

Key implications:

• New AWS accounts created on or after April 30, 2026 are not expected to have access to create App Runner services. AWS documentation states the service will be "closed to new customers" but does not document the specific API-level behavior.
• Existing services continue to run but SHOULD be migrated to ECS Express Mode or ECS Fargate.
• AWS has not announced an end-of-life date for existing services, but operators SHOULD NOT start new projects on App Runner.

---

ECS Express Mode as Replacement

ECS Express Mode (announced November 2025) provisions a complete ECS stack with a single API call:

• ECS cluster + Fargate service
• Application Load Balancer
• Auto scaling policy
• Security groups and networking

# Create an ECS Express Mode service
aws ecs create-express-gateway-service \
  --service-name $SERVICE_NAME \
  --execution-role-arn $EXECUTION_ROLE_ARN \
  --infrastructure-role-arn $INFRA_ROLE_ARN \
  --primary-container "{\"image\":\"$IMAGE_URI\",\"containerPort\":$CONTAINER_PORT,\"secrets\":[{\"name\":\"DB_PASSWORD\",\"valueFrom\":\"$SECRET_ARN\"}]}" \
  --region $REGION \
  --output json

Security note: Use the secrets field (referencing AWS Secrets Manager or SSM Parameter Store ARNs) for sensitive values. Do NOT pass secrets via the environment field — environment variables are visible in plaintext in the ECS task definition. See: ExpressGatewayContainer API

>

This example shows minimum required parameters. For production deployments, operators SHOULD also configure: a task role with least-privilege permissions (--task-role-arn), private subnets for internal services (--network-configuration), WAF association on the ALB, and ALB access logging.

ECS Express Mode is designed as the direct migration path for App Runner workloads. It preserves the simplicity of App Runner while providing full ECS capabilities when needed.

---

Comparison: App Runner vs ECS Express Mode vs ECS Fargate

---

Auto Scaling Behavior

App Runner uses concurrency-based auto scaling:

• Metric: Number of concurrent requests per instance.
• Default concurrency target: 100 concurrent requests per instance.
• Minimum instances: 1 — App Runner MUST NOT scale to zero. At least one instance is always running and billed.
• Maximum instances: Configurable (default 25).

# Describe current auto scaling configuration
aws apprunner describe-auto-scaling-configuration \
  --auto-scaling-configuration-arn $AUTO_SCALING_ARN \
  --region $REGION \
  --output json

Operators SHOULD note:

• Because App Runner cannot scale to zero, idle services still incur cost for the minimum instance.
• Concurrency-based scaling differs from CPU/memory-based scaling in ECS — workloads with high CPU but low concurrency MAY not scale correctly.

---

VPC Connector Gotchas

When a VPC connector is attached to an App Runner service, operators MUST understand these behaviors:

1. Routes ALL Outbound Traffic Through VPC

The VPC connector routes all outbound traffic from the service through the specified subnets. There is no split-tunneling — public internet access is lost unless the VPC has a NAT gateway.

2. No Static Outbound IP

App Runner with a VPC connector does NOT provide a static outbound IP address. If downstream services require IP allowlisting, operators MUST place a NAT gateway with an Elastic IP in the VPC.

3. Boot-Time Dependency Failures

If your application code depends on AWS APIs or external endpoints during startup (e.g., fetching configuration from DynamoDB, calling an external API), and the VPC lacks proper routing, the service WILL fail to start with timeout errors.

Important: App Runner's own managed actions — pulling source code and container images, pushing logs, and retrieving secrets referenced in the service configuration — are NOT routed through your VPC connector. This traffic traverses AWS-managed networking. You do NOT need VPC endpoints for ECR, CloudWatch Logs, or Secrets Manager to support App Runner's internal operations.

>

Source: Enabling VPC access for outgoing traffic: "App Runner traffic — App Runner manages several actions on your behalf, such as pulling source code and images, pushing logs, and retrieving secrets. The traffic that these actions generate isn't routed through your VPC."

VPC endpoints or a NAT gateway are required ONLY for traffic originating from your application code at runtime. The following apply only if your container code calls these services:

4. AWS Services Need VPC Endpoints or NAT

With a VPC connector, calls to AWS services (DynamoDB, SQS, S3, etc.) MUST route through either:

• A VPC endpoint for that service, OR
• A NAT gateway

Without one of these, API calls to AWS services WILL time out.

---

Migration Guide: App Runner to ECS Express Mode

Overview

The recommended migration strategy uses DNS weighted routing to shift traffic gradually from App Runner to ECS Express Mode.

High-Level Steps

1. Deploy ECS Express Mode service with the same container image and environment variables. 2. Validate the ECS Express Mode service independently (health checks, functional tests). 3. Configure Route 53 weighted routing:

• Create a weighted record for the App Runner custom domain endpoint (weight: 100).
• Create a weighted record for the ECS Express Mode ALB endpoint (weight: 0).

4. Gradually shift traffic by adjusting weights (e.g., 90/10 → 70/30 → 50/50 → 0/100). 5. Monitor error rates, latency, and logs at each step before increasing ECS weight. 6. Decommission the App Runner service once 100% traffic is on ECS Express Mode.

# Example: Update Route 53 weighted record to shift 20% traffic to ECS
aws route53 change-resource-record-sets \
  --hosted-zone-id $HOSTED_ZONE_ID \
  --change-batch '{
    "Changes": [
      {
        "Action": "UPSERT",
        "ResourceRecordSet": {
          "Name": "'"$DOMAIN_NAME"'",
          "Type": "A",
          "SetIdentifier": "ecs-express",
          "Weight": 20,
          "AliasTarget": {
            "HostedZoneId": "'"$ALB_HOSTED_ZONE_ID"'",
            "DNSName": "'"$ALB_DNS_NAME"'",
            "EvaluateTargetHealth": true
          }
        }
      }
    ]
  }' \
  --region $REGION \
  --output json

Operators SHOULD:

• Run both services in parallel for at least one full traffic cycle before completing cutover.
• Compare App Runner and ECS Express Mode metrics side-by-side during migration.
• Keep the App Runner service running (but at minimum scale) as a rollback target until confident.

---

Security Considerations

Both App Runner and ECS Express Mode expose public HTTPS endpoints by default with no built-in authentication. Operators MUST address the following security controls.

Source: App Runner security, ECS security best practices, Express Mode best practices

Authentication and Authorization

• App Runner and ECS Express Mode provide no built-in authentication. Services are publicly accessible by default. Source: Enabling Private endpoint for incoming traffic: "By default when you create an AWS App Runner service, the service is accessible over the internet."
• Operators MUST implement authentication at the application layer (e.g., JWT validation, OAuth 2.0) or place an API Gateway with authorizers in front of the service.
• For internal-only services, use private subnets with an internal ALB. ECS Express Mode provisions an internal ALB when private subnets are provided via --network-configuration. Source: Express Mode network configuration defaults: "If you provide private subnets (subnets without an internet gateway in their route table), Express Mode will provision an internal ALB."

Secret Management

• MUST NOT pass secrets via the environment field in container definitions — environment variables are visible in plaintext in ECS task definitions.
• MUST use the secrets field in primaryContainer, referencing AWS Secrets Manager or SSM Parameter Store:

"secrets": [{"name": "DB_PASSWORD", "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:my-secret"}]

• Source: ExpressGatewayContainer API — `secrets` field: "The secrets to pass to the container. Type: Array of Secret objects."
• App Runner supports managed secret injection via service configuration — these secrets are retrieved by App Runner's managed infrastructure, not through your VPC. Source: Enabling VPC access for outgoing traffic
• Operators SHOULD enable automatic secret rotation in Secrets Manager. Source: Express Mode best practices — Secrets management

IAM Least Privilege

• The task execution role SHOULD use the AWS-managed AmazonECSTaskExecutionRolePolicy. Avoid broader policies.
• The infrastructure role SHOULD use the AWS-managed AmazonECSInfrastructureRoleforExpressGatewayServices policy.
• The task role (--task-role-arn) MUST follow least privilege — grant only the specific actions and resources the application requires. Avoid *FullAccess policies and service:* wildcards.
• Source: Express Mode IAM role defaults

Encryption

• In transit: Both App Runner and ECS Express Mode enforce HTTPS/TLS by default. Express Mode auto-provisions an ACM certificate and configures an HTTPS listener on port 443. Source: Express Mode ALB defaults: "listener-configurations.protocol: https"
• At rest: Operators SHOULD enable KMS encryption on CloudWatch Logs log groups, ECR repositories, and any data stores the application uses. Secrets Manager encrypts secrets at rest by default using either an AWS-managed or customer-provided KMS key.

Network Security

• ECS Express Mode auto-creates security groups scoped to ALB → task traffic. The LB Security Group allows inbound HTTPS (443) and outbound to the task on the container port only. Source: Express Mode network configuration defaults
• When providing custom security groups via --network-configuration, operators MUST NOT use 0.0.0.0/0 for inbound rules on non-public services. Scope inbound to specific CIDR ranges or security group references.
• Operators SHOULD enable VPC Flow Logs for network traffic monitoring. Source: Express Mode best practices — Network security

AWS WAF

• Operators SHOULD attach an AWS WAF WebACL for defense in depth against common web exploits:
• App Runner: Supports direct WAF web ACL association. Source: Associating an AWS WAF web ACL with your service
• ECS Express Mode: Associate a WAF WebACL to the ALB via aws wafv2 associate-web-acl --resource-arn <alb-arn>. Source: Express Mode best practices — Network security

Security Headers

• Applications SHOULD return standard security headers in HTTP responses:
• Strict-Transport-Security (HSTS) — prevents protocol downgrade attacks
• Content-Security-Policy (CSP) — mitigates XSS attacks
• X-Frame-Options — prevents clickjacking
• X-Content-Type-Options: nosniff — prevents MIME-type sniffing
• These headers are set at the application level. Neither App Runner nor the Express Mode ALB adds them automatically.

Input Validation and Rate Limiting

• Operators SHOULD implement input validation and rate limiting at the application layer.
• App Runner's MaxConcurrency setting (default: 100) provides per-instance request throttling but is not a substitute for application-level rate limiting.
• For stricter controls, operators MAY place API Gateway in front of the service for managed throttling, or use AWS WAF rate-based rules.

Logging and Monitoring

• ALB access logs: Disabled by default in Express Mode (access-logs.enabled: false). Operators SHOULD enable access logs on the ALB and direct them to an S3 bucket with encryption. Source: Express Mode ALB defaults
• CloudWatch alarms: Operators SHOULD create alarms for 5XX error rates, latency P99, and unhealthy host count. Express Mode auto-creates a metric alarm for detecting faulty deployments.
• CloudTrail: Verify CloudTrail is enabled for API-level audit logging in the target account and region.
• Sensitive data: Operators MUST NOT log sensitive data (credentials, PII, tokens) in application logs. SHOULD enable KMS encryption on CloudWatch Logs log groups.
• Source: Express Mode best practices — Monitoring and logging

ECR Repository Management Reference

Table of Contents

• Verify Dependencies
• Create Repository
• Authenticate and Push Images
• Lifecycle Policies
• Image Scanning
• Cross-Account Image Pulls
• Common Image Pull Errors
• Security Considerations

---

Verify Dependencies

Before managing ECR repositories, the operator MUST confirm:

1. Docker is installed and the Docker daemon is running. 2. The caller has the specific IAM permissions needed for the operation (e.g., ecr:CreateRepository, ecr:GetAuthorizationToken, ecr:PutImage). Avoid granting ecr:* in production — scope permissions to the actions and repositories required.

aws sts get-caller-identity --output json
docker info --format '{{.ServerVersion}}'

---

Create Repository

aws ecr create-repository \
  --repository-name "$REPO_NAME" \
  --image-scanning-configuration scanOnPush=true \
  --image-tag-mutability IMMUTABLE \
  --encryption-configuration encryptionType=AES256 \
  --region "$REGION" \
  --output json

Deprecation notice: --image-scanning-configuration is being deprecated in favor of registry-level scanning configuration via put-registry-scanning-configuration (see Image Scanning section). The parameter still works but prefer the registry-level approach for new setups.

The operator SHOULD set:

• scanOnPush=true to automatically scan images for vulnerabilities on push (or configure scanning at the registry level — see Image Scanning).
• image-tag-mutability IMMUTABLE to prevent tag overwriting. This ensures a given tag always refers to the same image digest. Use IMMUTABLE_WITH_EXCLUSION with --image-tag-mutability-exclusion-filters if specific tags (e.g., latest) must remain mutable.

---

Authenticate and Push Images

Authenticate Docker to ECR

aws ecr get-login-password --region "$REGION" \
  | docker login --username AWS \
    --password-stdin "$ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com"

Warning: The authentication token expires after 12 hours. The operator MUST re-authenticate before pushing if the token has expired. CI/CD pipelines SHOULD call get-login-password at the start of every build.

Build, Tag, and Push

docker build -t "$REPO_NAME:$IMAGE_TAG" .
docker tag "$REPO_NAME:$IMAGE_TAG" \
  "$ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com/$REPO_NAME:$IMAGE_TAG"
docker push \
  "$ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com/$REPO_NAME:$IMAGE_TAG"

Verify the Push

aws ecr describe-images \
  --repository-name "$REPO_NAME" \
  --image-ids imageTag="$IMAGE_TAG" \
  --region "$REGION" \
  --output json

---

Lifecycle Policies

Lifecycle policies automatically expire old images. ECR evaluates rules approximately every 24 hours — images are not removed immediately after a rule matches.

Policy JSON Structure

{
  "rules": [
    {
      "rulePriority": 1,
      "description": "Keep only the last 10 tagged images",
      "selection": {
        "tagStatus": "tagged",
        "tagPrefixList": ["v"],
        "countType": "imageCountMoreThan",
        "countNumber": 10
      },
      "action": {
        "type": "expire"
      }
    },
    {
      "rulePriority": 2,
      "description": "Expire untagged images older than 7 days",
      "selection": {
        "tagStatus": "untagged",
        "countType": "sinceImagePushed",
        "countUnit": "days",
        "countNumber": 7
      },
      "action": {
        "type": "expire"
      }
    }
  ]
}

Key Fields

Apply a Lifecycle Policy

aws ecr put-lifecycle-policy \
  --repository-name "$REPO_NAME" \
  --lifecycle-policy-text file://lifecycle-policy.json \
  --region "$REGION" \
  --output json

Verify the policy was applied:

aws ecr get-lifecycle-policy \
  --repository-name "$REPO_NAME" \
  --region "$REGION" \
  --output json

Preview Before Applying

The operator SHOULD preview the policy to see which images would be affected before applying:

aws ecr start-lifecycle-policy-preview \
  --repository-name "$REPO_NAME" \
  --lifecycle-policy-text file://lifecycle-policy.json \
  --region "$REGION" \
  --output json

Poll the preview status with get-lifecycle-policy-preview until it completes.

aws ecr get-lifecycle-policy-preview \
  --repository-name "$REPO_NAME" \
  --region "$REGION" \
  --output json

Manifest List Blocking

Lifecycle policies do not delete images referenced by a manifest list (multi-architecture images). The operator MUST account for this when designing policies for multi-arch repositories.

CDK addLifecycleRule

import * as ecr from 'aws-cdk-lib/aws-ecr';

const repo = new ecr.Repository(this, 'Repo', {
  repositoryName: '$REPO_NAME',
  imageScanOnPush: true,
  imageTagMutability: ecr.TagMutability.IMMUTABLE,
});

repo.addLifecycleRule({
  tagPrefixList: ['v'],
  maxImageCount: 10,
  description: 'Keep only the last 10 tagged images',
});

repo.addLifecycleRule({
  maxImageAge: cdk.Duration.days(7),
  tagStatus: ecr.TagStatus.UNTAGGED,
  description: 'Expire untagged images older than 7 days',
});

---

Image Scanning

Basic Scanning

Basic scanning has no separate ECR charge (only enhanced scanning incurs Inspector charges).

# Trigger a manual scan
aws ecr start-image-scan \
  --repository-name "$REPO_NAME" \
  --image-id imageTag="$IMAGE_TAG" \
  --region "$REGION" \
  --output json

# Retrieve scan findings
aws ecr describe-image-scan-findings \
  --repository-name "$REPO_NAME" \
  --image-id imageTag="$IMAGE_TAG" \
  --region "$REGION" \
  --output json

Enhanced Scanning with Amazon Inspector

Enhanced scanning provides continuous, automated scanning using Amazon Inspector. It covers OS packages and programming language packages.

The operator MUST enable enhanced scanning at the registry level:

aws ecr put-registry-scanning-configuration \
  --scan-type ENHANCED \
  --rules '[{"scanFrequency":"CONTINUOUS_SCAN","repositoryFilters":[{"filter":"*","filterType":"WILDCARD"}]}]' \
  --region "$REGION" \
  --output json

Enhanced scanning incurs additional Inspector charges.

---

Cross-Account Image Pulls

To allow account $CONSUMER_ACCOUNT_ID to pull images from a repository in account $ACCOUNT_ID:

Step 1: Set Repository Policy (Source Account)

aws ecr set-repository-policy \
  --repository-name "$REPO_NAME" \
  --policy-text '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "AllowCrossAccountPull",
        "Effect": "Allow",
        "Principal": {
          "AWS": "arn:aws:iam::$CONSUMER_ACCOUNT_ID:root"
        },
        "Action": [
          "ecr:GetDownloadUrlForLayer",
          "ecr:BatchGetImage"
        ]
      }
    ]
  }' \
  --region "$REGION" \
  --output json

Security: For tighter control, replace the :root principal with a specific IAM role ARN (e.g., the consumer's ECS execution role). For organizations using AWS Organizations, use a Condition with aws:PrincipalOrgID to allow all accounts in the organization without listing each account ID.

Note: The minimum pull permissions are ecr:BatchGetImage and ecr:GetDownloadUrlForLayer (per ECR on ECS docs). Omit ecr:BatchCheckLayerAvailability — it is not required for pulling images (it is a Read action used by the ECR proxy primarily during push to check if layers already exist). ecr:GetAuthorizationToken is registry-level and must be on the consumer's identity-based policy, not the repository policy.

Step 2: Execution Role Permissions (Consumer Account)

The ECS execution role in the consumer account MUST have ecr:GetAuthorizationToken and the pull actions listed above. The execution role's trust policy MUST allow ecs-tasks.amazonaws.com to assume it.

---

Common Image Pull Errors

---

Security Considerations

• Encryption at rest: Use KMS via --encryption-configuration when you need key-level audit trail (KMS logs GenerateDataKey, Decrypt calls in CloudTrail) and customer-managed key rotation. AES256 (S3-managed keys) is the default. All ECR API calls are logged by CloudTrail regardless of encryption type.
• Image tag immutability: Set IMMUTABLE to prevent tag overwriting attacks (supply chain security). Use IMMUTABLE_WITH_EXCLUSION only when specific tags must remain mutable.
• Least-privilege IAM: Scope ECR permissions to specific repository ARNs. Separate push (CI/CD) from pull (execution role) permissions. ecr:GetAuthorizationToken requires Resource: "*" — it cannot be scoped to a repository.
• Cross-account access: Use aws:PrincipalOrgID conditions in repository policies. Grant only ecr:BatchGetImage and ecr:GetDownloadUrlForLayer for pull-only access. Prefer specific role ARNs over :root principals.
• Logging and monitoring: ECR API calls are logged by CloudTrail. Set CloudWatch alarms on ECR API usage metrics to detect unusual pull patterns or approaching quota limits. See ECR usage metrics.
• Lifecycle policies: Expire untagged and old images to reduce attack surface from unpatched images.

ECS Exec Debugging Reference

Table of Contents

• Verify Dependencies
• Enable ECS Exec on a Service
• Task Role SSM Permissions
• Caller IAM Permissions
• Run an Interactive Command
• Common Errors
• Session Logging
• Considerations and Limitations
• Security Considerations

---

Verify Dependencies

Before using ECS Exec, the operator MUST confirm:

1. The Session Manager plugin is installed locally. Verify with:

   session-manager-plugin

If installed, this returns: The Session Manager plugin is installed successfully. Use the AWS CLI to start a session. 2. The ECS service uses Fargate platform version 1.4.0 or later (Linux) or 1.0.0 (Windows), or EC2 with ECS agent 1.50.2+. 3. The task role has SSM permissions (see below). 4. The container image includes /bin/sh (or the shell specified in the --command flag).

Constraints for parameter acquisition:

• You MUST verify all required parameters ($CLUSTER, $SERVICE) are provided. If any are missing, ask for them upfront in a single prompt.
• If all required parameters are provided, proceed to enable ECS Exec — do not ask the user to confirm what they already specified.
• For $TASK_ID and $CONTAINER, you SHOULD discover them via aws ecs list-tasks and aws ecs describe-tasks if not provided, inform the user what you found, and proceed.

aws sts get-caller-identity --output json
aws ecs describe-services \
  --cluster "$CLUSTER" \
  --services "$SERVICE_NAME" \
  --region "$REGION" \
  --query "services[0].platformVersion" \
  --output json

---

Enable ECS Exec on a Service

ECS Exec MUST be enabled on the service. Enabling it on an existing service requires --force-new-deployment to replace running tasks with new tasks that have the SSM agent binaries bind-mounted into the container.

aws ecs update-service \
  --cluster "$CLUSTER" \
  --service "$SERVICE_NAME" \
  --enable-execute-command \
  --force-new-deployment \
  --region "$REGION" \
  --output json

Verify that enableExecuteCommand is true:

aws ecs describe-services \
  --cluster "$CLUSTER" \
  --services "$SERVICE_NAME" \
  --region "$REGION" \
  --query "services[0].enableExecuteCommand" \
  --output json

The --force-new-deployment flag triggers a rolling replacement of all tasks. The operator SHOULD perform this during a maintenance window for services with tight availability requirements.

---

Task Role SSM Permissions

The task role (not the execution role) MUST have the following SSM permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssmmessages:CreateControlChannel",
        "ssmmessages:CreateDataChannel",
        "ssmmessages:OpenControlChannel",
        "ssmmessages:OpenDataChannel"
      ],
      "Resource": "*"
    }
  ]
}

If session logging is enabled (see Session Logging), the task role MUST also have permissions for the logging destination:

• CloudWatch Logs:
• logs:DescribeLogGroups (Resource: *)
• logs:CreateLogStream (on the log group ARN)
• logs:DescribeLogStreams (on the log group ARN)
• logs:PutLogEvents (on the log group ARN)
• S3:
• s3:GetBucketLocation (Resource: *)
• s3:GetEncryptionConfiguration (on the bucket ARN)
• s3:PutObject (on the bucket ARN/*)
• KMS (if encrypted): kms:Decrypt on the KMS key.

---

Caller IAM Permissions

The IAM principal running ecs execute-command MUST have:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ecs:ExecuteCommand",
      "Resource": [
        "arn:aws:ecs:$REGION:$ACCOUNT_ID:task/$CLUSTER/*",
        "arn:aws:ecs:$REGION:$ACCOUNT_ID:cluster/$CLUSTER"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "ecs:DescribeTasks",
      "Resource": "arn:aws:ecs:$REGION:$ACCOUNT_ID:task/$CLUSTER/*"
    }
  ]
}

Least-privilege tip: Use condition keys such as ecs:cluster, ecs:container-name, ecs:task, ecs:ResourceTag/${TagKey}, and aws:ResourceTag/${TagKey} to further restrict which clusters, containers, or tagged tasks a principal can exec into. See Using IAM policies to limit access to ECS Exec.

KMS encryption: If the cluster's executeCommandConfiguration specifies a kmsKeyId, the caller MUST also have kms:GenerateDataKey on that KMS key ARN.

---

Run an Interactive Command

aws ecs execute-command \
  --cluster "$CLUSTER" \
  --task "$TASK_ID" \
  --container "$CONTAINER_NAME" \
  --interactive \
  --command "/bin/sh" \
  --region "$REGION"

For a specific diagnostic command (single command, not a shell):

aws ecs execute-command \
  --cluster "$CLUSTER" \
  --task "$TASK_ID" \
  --container "$CONTAINER_NAME" \
  --interactive \
  --command "cat /etc/resolv.conf" \
  --region "$REGION"

Amazon ECS only supports initiating interactive sessions, so the --interactive flag is always required.

---

Common Errors

Tip: Use the ECS Exec Checker script to verify that your cluster and task meet all prerequisites for ECS Exec. It checks your AWS CLI environment, cluster, and task configuration.

TargetNotConnectedException

This is the most common error. It means the SSM agent in the task cannot establish a connection.

Debugging steps (check in order):

1. SSM agent startup delay — After a new deployment with --enable-execute-command, the SSM agent inside the task needs time to start and register. Verify the agent is running by checking that ExecuteCommandAgent lastStatus is RUNNING in describe-tasks output before retrying. In practice, this typically takes 30–60 seconds after the task reaches RUNNING status.

2. Private subnet networking — If the task runs in a private subnet, it MUST have a route to the ssmmessages endpoint. Either:

• A NAT gateway in the route table, OR
• A VPC interface endpoint for com.amazonaws.$REGION.ssmmessages with a security group allowing inbound HTTPS (port 443) from the task security group. Do NOT use 0.0.0.0/0 — scope the inbound rule to the task security group or the VPC CIDR.

   aws ec2 describe-vpc-endpoints \
     --filters Name=service-name,Values="com.amazonaws.$REGION.ssmmessages" \
     --region "$REGION" \
     --output json

3. Task role permissions — Verify the task role has all four ssmmessages:* actions. A missing permission causes a silent connection failure.

   aws iam list-attached-role-policies \
     --role-name "$TASK_ROLE_NAME" \
     --output json

   aws iam list-role-policies \
     --role-name "$TASK_ROLE_NAME" \
     --output json

4. Platform version — Confirm the task is running on Fargate platform version 1.4.0 or later:

   aws ecs describe-tasks \
     --cluster "$CLUSTER" \
     --tasks "$TASK_ID" \
     --region "$REGION" \
     --query "tasks[0].platformVersion" \
     --output json

5. Container has a shell — The container image MUST include /bin/sh. Minimal or distroless images may not have a shell. Use a debug sidecar or rebuild the image with a shell for debugging.

InvalidParameterException: Execute command not enabled

The service does not have ECS Exec enabled. Run update-service with --enable-execute-command --force-new-deployment.

SessionManagerPlugin is not found

The Session Manager plugin is not installed or not in the system PATH. Install it from the AWS documentation.

---

Session Logging

ECS Exec sessions SHOULD be logged to S3 or CloudWatch Logs for audit purposes. AWS CloudTrail automatically records ExecuteCommand API calls, but session content (commands and output) is only captured when logging is explicitly configured below.

The container image requires script and cat to be installed in order to have command logs uploaded correctly to Amazon S3 or CloudWatch Logs. Some minimal or distroless images may not include these utilities.

Configure Logging

aws ecs create-cluster \
  --cluster-name "$CLUSTER" \
  --configuration '{
    "executeCommandConfiguration": {
      "kmsKeyId": "$KMS_KEY_ID",
      "logging": "OVERRIDE",
      "logConfiguration": {
        "cloudWatchLogGroupName": "/ecs/exec/$CLUSTER",
        "cloudWatchEncryptionEnabled": true,
        "s3BucketName": "$LOGGING_BUCKET",
        "s3EncryptionEnabled": true,
        "s3KeyPrefix": "ecs-exec-logs"
      }
    }
  }' \
  --region "$REGION" \
  --output json

Security: The kmsKeyId encrypts the data channel between the local client and the container (in addition to the default TLS 1.2). The cloudWatchEncryptionEnabled and s3EncryptionEnabled flags encrypt session logs at rest. The CloudWatch log group MUST be encrypted with a KMS customer managed key when cloudWatchEncryptionEnabled is true.

Warning: ECS Exec session logs may capture sensitive data such as environment variables, secrets, database queries, and command output. Ensure logging destinations are encrypted and access is restricted to authorized personnel.

For existing clusters, use update-cluster with the same --configuration parameter.

The task role MUST have write permissions to the configured logging destination.

---

Considerations and Limitations

---

Security Considerations

ECS Exec provides powerful break-glass access to running containers. The following security controls SHOULD be applied:

• Root access risk: All ECS Exec commands run as root regardless of the container's user configuration. Limit who can call ecs:ExecuteCommand via IAM policies with condition keys (ecs:cluster, ecs:container-name, aws:ResourceTag).
• Prevent SSM session hijacking: Deny ssm:StartSession directly on ECS task ARNs to prevent unlogged sessions that bypass ECS Exec auditing. See Limiting access to the Start Session action.
• Encrypt the data channel: Provide a kmsKeyId in the cluster's executeCommandConfiguration to encrypt data between the local client and the container beyond the default TLS 1.2.
• Enable and encrypt session logging: Configure session logging to S3 or CloudWatch Logs with encryption enabled. Session logs may contain sensitive data (environment variables, secrets, query results).
• Audit with CloudTrail: ExecuteCommand API calls are recorded in AWS CloudTrail. Ensure CloudTrail is enabled and that trails cover the regions where ECS Exec is used.
• Task role trust policy: When creating the task IAM role, use aws:SourceAccount and aws:SourceArn condition keys in the trust policy to prevent the confused deputy problem.
• Disable ECS Exec in production when not needed: Use the ecs:enable-execute-command condition key to prevent services from being launched with ECS Exec enabled unless explicitly authorized.

For more information, see ECS Exec security and Amazon ECS security best practices.

ECS Infrastructure Patterns

Table of Contents

• Verify Dependencies
• L3 Construct Overview
• Web App on Fargate
• SQS Worker
• Scheduled Task
• Path-Based Routing
• EFS Volume
• ECS Exec Setup
• Private Subnets with VPC Endpoints
• FireLens Logging
• Secrets with Explicit Role Separation
• CloudFormation YAML Template for Fargate
• Security Considerations

---

Verify Dependencies

Operators MUST confirm the following before proceeding:

---

L3 Construct Overview

When to drop to L2 constructs: Use L2 (ecs.FargateService + elbv2.ApplicationLoadBalancer) when you need multiple services behind one ALB, custom task definitions with multiple containers, fine-grained log driver configuration (mode: blocking), or EFS volumes. L3 patterns don't expose these.

---

Web App on Fargate

import * as ecs from 'aws-cdk-lib/aws-ecs';
import * as ecsPatterns from 'aws-cdk-lib/aws-ecs-patterns';

const service = new ecsPatterns.ApplicationLoadBalancedFargateService(this, 'WebApp', {
  cluster,
  taskImageOptions: {
    image: ecs.ContainerImage.fromRegistry('$IMAGE_URI'),
    containerPort: $CONTAINER_PORT,
    environment: {
      NODE_ENV: 'staging',
    },
  },
  desiredCount: 2,
  circuitBreaker: { rollback: true },
  publicLoadBalancer: true,
});

// Reduce deregistration delay for faster deployments
service.targetGroup.setAttribute('deregistration_delay.timeout_seconds', '30');

// Auto scaling
const scaling = service.service.autoScaleTaskCount({
  minCapacity: 2,
  maxCapacity: 10,
});

scaling.scaleOnCpuUtilization('CpuScaling', {
  targetUtilizationPercent: 60,
});

scaling.scaleOnRequestCount('RequestScaling', {
  requestsPerTarget: 1000,
  targetGroup: service.targetGroup,
});

Key points:

• circuitBreaker: { rollback: true } MUST be set — this automatically rolls back failed deployments instead of leaving the service in a degraded state. In CDK, specifying the circuitBreaker property implicitly enables it (enable is optional and defaults to true).
• Operators SHOULD reduce deregistration_delay.timeout_seconds from the default 300s. A value of 30s is appropriate for most web services.
• setAttribute is used because the L3 pattern does not expose deregistration delay in its props (the underlying ApplicationTargetGroup has a deregistrationDelay property, but the L3 pattern doesn't pass it through).

Validate before deploying: cdk synth to catch type errors and missing props → cdk diff to review changes → cdk deploy only after validation passes.

• To set mode: blocking for guaranteed log delivery (see CloudFormation section for rationale), use a custom task definition instead of taskImageOptions:

const taskDef = new ecs.FargateTaskDefinition(this, 'TaskDef', { cpu: 512, memoryLimitMiB: 1024 });
taskDef.addContainer('App', {
  image: ecs.ContainerImage.fromRegistry('$IMAGE_URI'),
  portMappings: [{ containerPort: 8080 }],
  logging: ecs.LogDrivers.awsLogs({
    streamPrefix: 'app',
    mode: ecs.AwsLogDriverMode.BLOCKING,
  }),
});

---

SQS Worker

import * as ecsPatterns from 'aws-cdk-lib/aws-ecs-patterns';

const worker = new ecsPatterns.QueueProcessingFargateService(this, 'Worker', {
  cluster,
  image: ecs.ContainerImage.fromRegistry('$IMAGE_URI'),
  environment: {
    WORKER_TYPE: 'processor',
  },
  minScalingCapacity: 1,
  maxScalingCapacity: 20,
  scalingSteps: [
    { upper: 0, change: -1 },
    { lower: 1, change: +1 },
    { lower: 50, change: +3 },
    { lower: 200, change: +5 },
  ],
  cpu: 512,
  memoryLimitMiB: 1024,
  circuitBreaker: { rollback: true },
});

Key points:

• scalingSteps defines step scaling based on the ApproximateNumberOfMessagesVisible metric on the SQS queue.

---

Scheduled Task

import * as ecsPatterns from 'aws-cdk-lib/aws-ecs-patterns';
import * as appscaling from 'aws-cdk-lib/aws-applicationautoscaling';

new ecsPatterns.ScheduledFargateTask(this, 'NightlyJob', {
  cluster,
  scheduledFargateTaskImageOptions: {
    image: ecs.ContainerImage.fromRegistry('$IMAGE_URI'),
    memoryLimitMiB: 2048,
    cpu: 1024,
    environment: {
      JOB_NAME: 'nightly-report',
    },
  },
  schedule: appscaling.Schedule.expression('cron(0 3 * * ? *)'),
  platformVersion: ecs.FargatePlatformVersion.LATEST,
});

---

Path-Based Routing

import * as ecs from 'aws-cdk-lib/aws-ecs';
import * as elbv2 from 'aws-cdk-lib/aws-elasticloadbalancingv2';

const alb = new elbv2.ApplicationLoadBalancer(this, 'ALB', {
  vpc,
  internetFacing: true,
});

const listener = alb.addListener('Listener', { port: 80 });

// Service A: /api/*
const serviceA = new ecs.FargateService(this, 'ApiService', {
  cluster,
  taskDefinition: apiTaskDef,
  healthCheckGracePeriod: cdk.Duration.seconds(60),
});

const targetGroupA = listener.addTargets('ApiTarget', {
  port: $CONTAINER_PORT,
  targets: [serviceA],
  conditions: [elbv2.ListenerCondition.pathPatterns(['/api/*'])],
  priority: 10,
  healthCheck: {
    path: '/api/health',
    interval: cdk.Duration.seconds(30),
  },
});

// Service B: /* (default)
const serviceB = new ecs.FargateService(this, 'WebService', {
  cluster,
  taskDefinition: webTaskDef,
  healthCheckGracePeriod: cdk.Duration.seconds(60),
});

listener.addTargets('WebTarget', {
  port: $CONTAINER_PORT,
  targets: [serviceB],
  healthCheck: {
    path: '/health',
    interval: cdk.Duration.seconds(30),
  },
});

Key points:

• Rules with conditions MUST have a priority — lower numbers evaluate first.
• healthCheckGracePeriod SHOULD be tuned on each service if the default 60 seconds is insufficient for the application's startup time. CDK defaults to 60s when a load balancer is attached.

---

EFS Volume

import * as efs from 'aws-cdk-lib/aws-efs';
import * as ecs from 'aws-cdk-lib/aws-ecs';

const fileSystem = new efs.FileSystem(this, 'SharedFS', {
  vpc,
  encrypted: true,
  performanceMode: efs.PerformanceMode.GENERAL_PURPOSE,
  removalPolicy: cdk.RemovalPolicy.RETAIN,
});

const taskDef = new ecs.FargateTaskDefinition(this, 'TaskDef', {
  cpu: 512,
  memoryLimitMiB: 1024,
});

taskDef.addVolume({
  name: 'efs-volume',
  efsVolumeConfiguration: {
    fileSystemId: fileSystem.fileSystemId,
  },
});

const container = taskDef.addContainer('App', {
  image: ecs.ContainerImage.fromRegistry('$IMAGE_URI'),
});

container.addMountPoints({
  sourceVolume: 'efs-volume',
  containerPath: '/mnt/data',
  readOnly: false,
});

const service = new ecs.FargateService(this, 'Service', {
  cluster,
  taskDefinition: taskDef,
});

// CRITICAL: Allow ECS tasks to connect to EFS on port 2049
fileSystem.connections.allowDefaultPortFrom(service);

Key points:

• allowDefaultPortFrom opens NFS port 2049 from the ECS service security group to the EFS security group. Without this, tasks WILL hang on mount with timeout errors.
• removalPolicy: RETAIN prevents accidental deletion of persistent data.

---

ECS Exec Setup

const taskDef = new ecs.FargateTaskDefinition(this, 'TaskDef', {
  cpu: 512,
  memoryLimitMiB: 1024,
});

const service = new ecs.FargateService(this, 'Service', {
  cluster,
  taskDefinition: taskDef,
  enableExecuteCommand: true, // Automatically grants the 4 required ssmmessages actions to the task role
});

CRITICAL: enableExecuteCommand: true automatically grants the task role the 4 required ssmmessages actions (CreateControlChannel, CreateDataChannel, OpenControlChannel, OpenDataChannel). No manual policy attachment is needed in CDK. For CloudFormation, add an inline policy with these 4 actions on the task role.

CRITICAL: SSM permissions MUST be on the task role, NOT the execution role. The execution role is used by the ECS agent to pull images and write logs. The task role is assumed by the running container — ECS Exec runs inside the container and therefore needs SSM permissions on the task role.

Common mistake:

// WRONG — this will NOT work for ECS Exec
taskDef.executionRole.addManagedPolicy(
  iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonSSMManagedInstanceCore')
);

Verify ECS Exec after deployment:

aws ecs execute-command \
  --cluster $CLUSTER \
  --task $TASK_ID \
  --container $CONTAINER_NAME \
  --interactive \
  --command "/bin/sh" \
  --region $REGION

---

Private Subnets with VPC Endpoints

When running ECS tasks in private subnets without a NAT gateway, operators MUST create these 4 VPC endpoints:

// 1. ECR Docker — pull container images
vpc.addInterfaceEndpoint('EcrDockerEndpoint', {
  service: ec2.InterfaceVpcEndpointAwsService.ECR_DOCKER,
});

// 2. ECR API — authenticate with ECR
vpc.addInterfaceEndpoint('EcrApiEndpoint', {
  service: ec2.InterfaceVpcEndpointAwsService.ECR,
});

// 3. CloudWatch Logs — push container logs
vpc.addInterfaceEndpoint('CloudWatchLogsEndpoint', {
  service: ec2.InterfaceVpcEndpointAwsService.CLOUDWATCH_LOGS,
});

// 4. S3 Gateway — ECR stores image layers in S3
vpc.addGatewayEndpoint('S3Endpoint', {
  service: ec2.GatewayVpcEndpointAwsService.S3,
});

Additional endpoints MAY be needed:

---

FireLens Logging

// Log router sidecar — SHOULD be essential:true (AWS recommended)
const logRouter = taskDef.addFirelensLogRouter('LogRouter', {
  image: ecs.ContainerImage.fromRegistry('amazon/aws-for-fluent-bit:latest'),
  essential: true,
  firelensConfig: {
    type: ecs.FirelensLogRouterType.FLUENTBIT,
  },
  // Log router's OWN logs MUST use awslogs, NOT awsfirelens
  logging: ecs.LogDrivers.awsLogs({
    streamPrefix: 'firelens',
    logGroup,
  }),
});

// Application container uses awsfirelens driver
const appContainer = taskDef.addContainer('App', {
  image: ecs.ContainerImage.fromRegistry('$IMAGE_URI'),
  essential: true,
  logging: ecs.LogDrivers.firelens({
    options: {
      Name: 'cloudwatch_logs',
      region: '$REGION',
      log_group_name: '$LOG_GROUP',
      log_stream_prefix: 'app/',
      auto_create_group: 'true',
    },
  }),
});

Key rules:

• The log router container SHOULD have essential: true (AWS recommends this). If it crashes and is not essential, logs are silently lost with no indication.
• The log router MUST use awslogs for its own logs, NOT awsfirelens. Using awsfirelens for the log router creates a circular dependency that prevents the task from starting.
• Application containers use awsfirelens to route logs through the FireLens sidecar.

---

Secrets with Explicit Role Separation

import * as secretsmanager from 'aws-cdk-lib/aws-secretsmanager';
import * as iam from 'aws-cdk-lib/aws-iam';

const dbSecret = secretsmanager.Secret.fromSecretNameV2(this, 'DbSecret', '$SECRET_NAME');

const taskDef = new ecs.FargateTaskDefinition(this, 'TaskDef', {
  cpu: 512,
  memoryLimitMiB: 1024,
});

const container = taskDef.addContainer('App', {
  image: ecs.ContainerImage.fromRegistry('$IMAGE_URI'),
  secrets: {
    DB_PASSWORD: ecs.Secret.fromSecretsManager(dbSecret, 'password'),
  },
});

// CDK automatically grants the execution role read access to secrets
// specified in the secrets block (via ContainerDefinition.addSecret).
// An explicit grantRead is only needed if the secret is fetched at
// runtime by the task role and not referenced in the task definition.

Role separation:

• If secrets are injected via the task definition secrets block, grantRead MUST target the execution role.
• If the application fetches secrets at runtime via SDK calls, grantRead MUST target the task role.
• Operators SHOULD NOT grant secret access to both roles unless both access patterns are used.

---

CloudFormation YAML Template for Fargate

For operators who need raw CloudFormation instead of CDK:

AWSTemplateFormatVersion: '2010-09-09'
Description: ECS Fargate service with ALB

Parameters:
  ClusterName:
    Type: String
  ImageUri:
    Type: String
  ContainerPort:
    Type: Number
    Default: 8080
  VpcId:
    Type: AWS::EC2::VPC::Id
  PublicSubnetIds:
    Type: List<AWS::EC2::Subnet::Id>
    Description: Public subnets for the internet-facing ALB
  PrivateSubnetIds:
    Type: List<AWS::EC2::Subnet::Id>
    Description: Private subnets for ECS tasks (must have NAT gateway or VPC endpoints)
  CertificateArn:
    Type: String
    Description: ARN of the ACM certificate for HTTPS
  DesiredCount:
    Type: Number
    Default: 2

Resources:
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: !Sub '${ClusterName}-task'
      Cpu: '512'
      Memory: '1024'
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      ExecutionRoleArn: !GetAtt ExecutionRole.Arn
      TaskRoleArn: !GetAtt TaskRole.Arn
      ContainerDefinitions:
        - Name: app
          Image: !Ref ImageUri
          PortMappings:
            - ContainerPort: !Ref ContainerPort
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: !Ref LogGroup
              awslogs-region: !Ref 'AWS::Region'
              awslogs-stream-prefix: app
              mode: blocking

  Service:
    Type: AWS::ECS::Service
    DependsOn: ListenerRule
    Properties:
      Cluster: !Ref ClusterName
      TaskDefinition: !Ref TaskDefinition
      DesiredCount: !Ref DesiredCount
      LaunchType: FARGATE
      DeploymentConfiguration:
        DeploymentCircuitBreaker:
          Enable: true
          Rollback: true
      NetworkConfiguration:
        AwsvpcConfiguration:
          Subnets: !Ref PrivateSubnetIds
          SecurityGroups:
            - !Ref ServiceSG
      LoadBalancers:
        - ContainerName: app
          ContainerPort: !Ref ContainerPort
          TargetGroupArn: !Ref TargetGroup
      HealthCheckGracePeriodSeconds: 60

  ServiceSG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: ECS service security group
      VpcId: !Ref VpcId
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: !Ref ContainerPort
          ToPort: !Ref ContainerPort
          SourceSecurityGroupId: !Ref AlbSG

  AlbSG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: ALB security group
      VpcId: !Ref VpcId
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 443
          ToPort: 443
          CidrIp: 0.0.0.0/0

  LogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub '/ecs/${ClusterName}'
      RetentionInDays: 30

  ExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy

  TaskRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
            Action: sts:AssumeRole

  TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Port: !Ref ContainerPort
      Protocol: HTTP
      VpcId: !Ref VpcId
      TargetType: ip
      HealthCheckPath: /health
      HealthCheckIntervalSeconds: 30
      TargetGroupAttributes:
        - Key: deregistration_delay.timeout_seconds
          Value: '30'

  ALB:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Scheme: internet-facing
      SecurityGroups:
        - !Ref AlbSG
      Subnets: !Ref PublicSubnetIds

  Listener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      LoadBalancerArn: !Ref ALB
      Port: 443
      Protocol: HTTPS
      SslPolicy: ELBSecurityPolicy-TLS13-1-2-2021-06
      Certificates:
        - CertificateArn: !Ref CertificateArn
      DefaultActions:
        - Type: forward
          TargetGroupArn: !Ref TargetGroup

  # This rule is functionally redundant with the Listener's DefaultActions (both forward to the same TargetGroup).
  # It exists so the Service resource can use DependsOn: ListenerRule to ensure listener infrastructure is ready
  # before ECS registers targets. To remove it, change Service DependsOn to reference the Listener instead.
  ListenerRule:
    Type: AWS::ElasticLoadBalancingV2::ListenerRule
    Properties:
      ListenerArn: !Ref Listener
      Priority: 1
      Conditions:
        - Field: path-pattern
          Values:
            - '/*'
      Actions:
        - Type: forward
          TargetGroupArn: !Ref TargetGroup

Key points:

• DeploymentCircuitBreaker with Rollback: true MUST be enabled.
• mode: blocking MUST be set in log configuration for guaranteed log delivery. The ECS defaultLogDriverMode account setting defaults to non-blocking, which drops logs when the buffer fills. Without an explicit mode: blocking, tasks inherit the account default and may silently drop logs under backpressure.
• Security group ingress uses SourceSecurityGroupId (ALB → service) rather than open CIDR ranges.
• The ALB security group uses 0.0.0.0/0 per AWS recommended rules for internet-facing ALBs. For internal-only services, use Scheme: internal with VPC CIDR instead.
• For production internet-facing ALBs, attach an AWS WAF WebACL for defense in depth against common web exploits.
• Operators SHOULD NOT log sensitive data (secrets, PII, tokens) to container stdout/stderr — these flow to CloudWatch Logs via the awslogs driver. Enable CloudWatch Logs encryption with a KMS key if sensitive data may appear in logs.
• HealthCheckGracePeriodSeconds SHOULD be set when using a load balancer (CDK defaults to 60s when a load balancer is attached).
• Validate before deploying: aws cloudformation validate-template --template-body file://template.yaml

---

Security Considerations

• Encryption at rest: EFS volumes MUST use encrypted: true. CloudWatch Log Groups SHOULD use a KMS key for encryption when logs may contain sensitive data. ECR repositories encrypt images at rest by default (AES-256).
• Encryption in transit: ALBs SHOULD use HTTPS listeners with ACM certificates and a modern TLS policy (ELBSecurityPolicy-TLS13-1-2-2021-06 or newer). EFS traffic is encrypted in transit when using the TLS mount helper.
• IAM least privilege: Task roles MUST be scoped to specific resources — avoid * wildcards and *FullAccess policies. The execution role should use AmazonECSTaskExecutionRolePolicy (managed, scoped) plus only the additional permissions needed (e.g., Secrets Manager access for specific secrets).
• Secrets management: Use ecs.Secret.fromSecretsManager() or ecs.Secret.fromSsmParameter() — never pass secrets via environment variables in plain text.
• Network security: Use private subnets with VPC endpoints for production workloads. The service security group should only allow inbound from the ALB security group (via SourceSecurityGroupId), not open CIDRs.
• Web application protection: Attach AWS WAF to internet-facing ALBs. Add security headers (CSP, HSTS, X-Frame-Options) at the application level or via ALB response header insertion.
• Monitoring: Enable CloudWatch Container Insights for cluster and service metrics. Enable CloudTrail for ECS API audit logging.
• Reference: ECS Security Best Practices

ECS Logging

Table of Contents

• Verify Dependencies
• awslogs Driver
• Blocking vs Non-Blocking Mode
• Multiline Logs
• FireLens / Fluent Bit Setup
• When to Use Which

---

Verify Dependencies

---

awslogs Driver

The awslogs driver sends container stdout/stderr directly to CloudWatch Logs.

Required and Optional Options

CLI Example

aws ecs register-task-definition \
  --family $TASK_FAMILY \
  --network-mode awsvpc \
  --requires-compatibilities FARGATE \
  --cpu 512 \
  --memory 1024 \
  --execution-role-arn $EXECUTION_ROLE_ARN \
  --container-definitions '[
    {
      "name": "app",
      "image": "'$IMAGE_URI'",
      "essential": true,
      "portMappings": [{"containerPort": '$CONTAINER_PORT'}],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "'$LOG_GROUP'",
          "awslogs-region": "'$REGION'",
          "awslogs-stream-prefix": "app",
          "mode": "blocking"
        }
      }
    }
  ]' \
  --region $REGION \
  --output json

---

Blocking vs Non-Blocking Mode

IMPORTANT: ECS defaults to non-blocking log driver mode, which silently drops logs when the buffer fills (per API_LogConfiguration.html). The `defaultLogDriverMode` account setting can override this per account. For guaranteed log delivery, explicitly set "mode": "blocking" in logConfiguration.options.

Behavior Comparison

Recommendation

Operators MUST set mode to blocking when log completeness is required:

• Audit trails
• Financial transaction logs
• Security event logs
• Debugging intermittent failures

Operators MAY use non-blocking mode when:

• Application availability is more important than log completeness
• High-throughput logging would cause backpressure issues
• Logs are supplementary (metrics are the primary observability signal)

Setting Blocking Mode Explicitly

Because the default changed, operators MUST explicitly set mode: blocking in all task definitions where guaranteed log delivery is required. Do NOT rely on the default.

"logConfiguration": {
  "logDriver": "awslogs",
  "options": {
    "awslogs-group": "$LOG_GROUP",
    "awslogs-region": "$REGION",
    "awslogs-stream-prefix": "app",
    "mode": "blocking"
  }
}

Non-Blocking Buffer Tuning

When using non-blocking mode, operators SHOULD tune max-buffer-size based on log volume:

• Default 10m is sufficient for low-throughput services.
• High-throughput services SHOULD increase to 25m or higher (AWS uses 25m in its FireLens example).
• When logs are dropped in non-blocking mode, they are silently lost — there is no built-in CloudWatch metric for dropped logs. Monitor IncomingLogEvents and compare against expected application log volume to detect gaps.

---

Multiline Logs

Stack traces and multi-line log entries are split across multiple CloudWatch log events by default. Use these options to group them:

awslogs-datetime-format

Matches the timestamp at the start of each log entry. Lines without a matching timestamp are appended to the previous entry.

"logConfiguration": {
  "logDriver": "awslogs",
  "options": {
    "awslogs-group": "$LOG_GROUP",
    "awslogs-region": "$REGION",
    "awslogs-stream-prefix": "app",
    "awslogs-datetime-format": "%Y-%m-%d %H:%M:%S",
    "mode": "blocking"
  }
}

Common datetime patterns:

awslogs-multiline-pattern

A regex pattern that matches the start of a new log entry. More flexible than awslogs-datetime-format but MUST NOT be used together with it.

"logConfiguration": {
  "logDriver": "awslogs",
  "options": {
    "awslogs-group": "$LOG_GROUP",
    "awslogs-region": "$REGION",
    "awslogs-stream-prefix": "app",
    "awslogs-multiline-pattern": "^(INFO|WARN|ERROR|DEBUG|FATAL)",
    "mode": "blocking"
  }
}

• awslogs-datetime-format and awslogs-multiline-pattern MUST NOT be used together. If both are set, awslogs-datetime-format takes precedence and awslogs-multiline-pattern is ignored.
• Operators SHOULD prefer awslogs-datetime-format when log entries start with a timestamp.

---

FireLens / Fluent Bit Setup

FireLens routes container logs through a Fluent Bit (or Fluentd) sidecar, enabling delivery to multiple destinations (CloudWatch, S3, Elasticsearch, Datadog, etc.).

Architecture

┌─────────────┐     stdout/stderr     ┌──────────────┐     ┌─────────────────┐
│ App Container│ ──────────────────── │ Log Router   │ ──► │ CloudWatch Logs  │
│ (awsfirelens)│                      │ (Fluent Bit) │ ──► │ S3              │
└─────────────┘                       │ (awslogs)    │ ──► │ Elasticsearch   │
                                      └──────────────┘     └─────────────────┘

Task Definition Structure

{
  "family": "$TASK_FAMILY",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "512",
  "memory": "1024",
  "executionRoleArn": "$EXECUTION_ROLE_ARN",
  "taskRoleArn": "$TASK_ROLE_ARN",
  "containerDefinitions": [
    {
      "name": "log-router",
      "image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:3",
      "essential": true,
      "firelensConfiguration": {
        "type": "fluentbit"
      },
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "$LOG_GROUP",
          "awslogs-region": "$REGION",
          "awslogs-stream-prefix": "firelens",
          "mode": "blocking"
        }
      }
    },
    {
      "name": "app",
      "image": "$IMAGE_URI",
      "essential": true,
      "portMappings": [{"containerPort": $CONTAINER_PORT}],
      "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
          "Name": "cloudwatch_logs",
          "region": "$REGION",
          "log_group_name": "$LOG_GROUP",
          "log_stream_prefix": "app/",
          "auto_create_group": "true"
        }
      }
    }
  ]
}

Critical Rules

1. The log router container SHOULD have "essential": true (AWS recommendation). If the log router crashes and is not essential, the task continues running but all logs are silently lost.

2. The log router SHOULD use awslogs for its own logs, NOT awsfirelens. All AWS examples follow this pattern. Using awsfirelens on the log router would route its own logs through itself, which can prevent the task from starting.

3. The application container uses awsfirelens as its log driver to route logs through the FireLens sidecar.

4. The task role (not execution role) MUST have permissions for the destination services (CloudWatch Logs, S3, Kinesis, etc.) because Fluent Bit runs as the task role.

---

Security Considerations

• CloudWatch Logs log groups SHOULD be encrypted with a KMS key for sensitive workloads (audit, financial, security logs). Use aws logs associate-kms-key --log-group-name $LOG_GROUP --kms-key-id $KMS_KEY_ARN.
• Containers may log sensitive data (credentials, tokens, PII) to stdout/stderr. Consider CloudWatch Logs data protection policies to detect and mask sensitive patterns.
• Scope IAM log permissions to specific log group ARNs instead of Resource: "*" where possible.
• FireLens listens on port 24224. Do NOT allow inbound traffic on this port in the task's security group to prevent external access to the log router.

---

When to Use Which

ECS Troubleshooting

Table of Contents

• Verify Dependencies
• Exit Code Reference
• OOM Kills Deep Dive
• Task Placement Failures
• Health Check Debugging Checklist
• Image Pull Errors
• Private Subnet Networking
• ENI Trunking for EC2 awsvpc Density
• Security Considerations

---

Verify Dependencies

Operators MUST confirm the following before proceeding:

---

Exit Code Reference

Key Diagnostic Rules

• Exit code 137 means the container received SIGKILL. Check stoppedReason from describe-tasks first: if it contains "OutOfMemoryError", investigate OOM — see OOM Kills Deep Dive. If the task was being stopped (deployment, scale-in) and stoppedReason does NOT mention OOM, the container likely did not handle SIGTERM within stopTimeout — add a SIGTERM handler and verify stopTimeout is sufficient.
• Exit code 143 is expected during normal operations (deployments, scale-in). It means the container handled SIGTERM gracefully. It is NOT an error.
• Exit code 1 requires application log analysis — check CloudWatch Logs for the container's last output.

---

OOM Kills Deep Dive

Exit code 137 commonly indicates the container exceeded its memory limit and was killed by the kernel (OOM killer) or the Docker daemon. It can also occur when a container does not exit within stopTimeout after receiving SIGTERM.

Container Memory Hard Limit vs Task-Level Memory

On Fargate, the task-level memory is the overall ceiling. If a container definition sets a memory hard limit, Fargate enforces it — the container is killed if it exceeds that limit. If no per-container memory is set, a single container MAY consume all task memory, starving sidecars.

Diagnosing OOM Kills

# Step 1: Describe the stopped task to find the stop reason
aws ecs describe-tasks \
  --cluster $CLUSTER \
  --tasks $TASK_ID \
  --region $REGION \
  --output json \
  --query 'tasks[0].{stopCode:stopCode,stoppedReason:stoppedReason,containers:containers[*].{name:name,exitCode:exitCode,reason:reason}}'

Look for:

• stoppedReason containing "OutOfMemoryError" or "oom"
• Container reason containing "OutOfMemoryError: Container killed due to memory usage"
• exitCode: 137 on the affected container

JVM Fix: Use MaxRAMPercentage Instead of Fixed Xmx

# Fixed heap — works but does not adapt when container memory changes
java -Xmx512m -jar app.jar

# Container-aware — heap scales automatically with container memory limit
java -XX:MaxRAMPercentage=75.0 -jar app.jar

• In containerized environments, -XX:MaxRAMPercentage is preferred over fixed -Xmx because the heap scales automatically when the container memory limit changes. Fixed -Xmx values also work but require manual adjustment and must account for non-heap memory.
• A starting value of 75.0 leaves ~25% for JVM non-heap memory (metaspace, thread stacks, direct buffers, GC overhead). Workloads with many threads or large direct buffers may need a lower percentage (e.g., 50–70%); simple applications may safely use 80% or higher.
• On Fargate (Platform 1.4+), HotSpot-based JVMs (OpenJDK, Corretto, Temurin) correctly detect the task memory limit via cgroup. OpenJ9 has a known bug where it may not detect the limit correctly (openj9#11998) — set container-level memory as a workaround if using OpenJ9.

Quick Memory Check

# Check memory utilization for running tasks in a service
aws ecs describe-services \
  --cluster $CLUSTER \
  --services $SERVICE_NAME \
  --region $REGION \
  --output json \
  --query 'services[0].{desiredCount:desiredCount,runningCount:runningCount,deployments:deployments[*].{status:status,desiredCount:desiredCount,runningCount:runningCount,failedTasks:failedTasks}}'

---

Task Placement Failures

When ECS cannot place a task, the service event log shows the reason. Common failures:

Diagnosing Placement Failures

# Check service events for placement failure messages
aws ecs describe-services \
  --cluster $CLUSTER \
  --services $SERVICE_NAME \
  --region $REGION \
  --output json \
  --query 'services[0].events[:10]'

---

Health Check Debugging Checklist

When tasks are being killed by ALB health checks, follow these steps in order:

Step 1: Verify the Health Check Endpoint Responds Locally

Confirm the application responds on the health check path and port. Use ECS Exec if available:

aws ecs execute-command \
  --cluster $CLUSTER \
  --task $TASK_ID \
  --container $CONTAINER_NAME \
  --interactive \
  --command "curl -s -o /dev/null -w '%{http_code}' http://localhost:$CONTAINER_PORT/health" \
  --region $REGION

Step 2: Check healthCheckGracePeriod

If tasks are killed before the application finishes starting, healthCheckGracePeriod is too low or not set.

aws ecs describe-services \
  --cluster $CLUSTER \
  --services $SERVICE_NAME \
  --region $REGION \
  --output json \
  --query 'services[0].healthCheckGracePeriodSeconds'

This value MUST be greater than the application startup time. Operators SHOULD set it to at least 60 seconds.

Step 3: Verify Target Group Health Check Settings

aws elbv2 describe-target-health \
  --target-group-arn $TARGET_GROUP_ARN \
  --region $REGION \
  --output json

Check that:

• Health check path matches the application's actual health endpoint.
• Health check port matches the container port (or is set to traffic-port).
• Healthy threshold, interval, and timeout are reasonable.

Step 4: Check Security Group Rules

The ALB security group MUST be allowed to reach the container port on the task security group.

aws ec2 describe-security-groups \
  --group-ids $TASK_SG_ID \
  --region $REGION \
  --output json \
  --query 'SecurityGroups[0].IpPermissions'

Step 5: Check Container Logs for Startup Errors

aws logs get-log-events \
  --log-group-name $LOG_GROUP \
  --log-stream-name "$STREAM_PREFIX/$CONTAINER_NAME/$TASK_ID" \
  --limit 50 \
  --region $REGION \
  --output json

Step 6: Verify the Container Is Listening on the Correct Interface

The application MUST listen on 0.0.0.0 (all interfaces), not 127.0.0.1 (localhost only). In awsvpc mode, the ALB health check comes from the ALB's IP, not localhost.

---

Image Pull Errors

Diagnosing Image Pull Failures

# Check stopped task for pull error details
aws ecs describe-tasks \
  --cluster $CLUSTER \
  --tasks $TASK_ID \
  --region $REGION \
  --output json \
  --query 'tasks[0].containers[*].{name:name,reason:reason,lastStatus:lastStatus}'

---

Private Subnet Networking

When ECS tasks run in private subnets (no internet gateway route), the following VPC endpoints are required:

Required Endpoints (Minimum for ECS Fargate)

Additional Endpoints by Feature

Verifying Endpoint Connectivity

# List VPC endpoints in the VPC
aws ec2 describe-vpc-endpoints \
  --filters "Name=vpc-id,Values=$VPC_ID" \
  --region $REGION \
  --output json \
  --query 'VpcEndpoints[*].{ServiceName:ServiceName,State:State,VpcEndpointType:VpcEndpointType}'

Operators MUST verify:

1. Endpoints are in available state. 2. Interface endpoints have security groups that allow inbound HTTPS (port 443) from the task security group. 3. Interface endpoints are associated with the same subnets as the ECS tasks. 4. The S3 gateway endpoint route table is associated with the task subnets.

---

ENI Trunking for EC2 awsvpc Density

By default, each ECS task using awsvpc network mode on EC2 consumes one ENI on the host instance. This limits the number of tasks per instance to the instance's ENI limit minus one (reserved for the host).

ENI trunking allows multiple tasks to share a trunk ENI, significantly increasing task density.

Enabling ENI Trunking

# Enable for the entire account (all clusters in the region)
aws ecs put-account-setting-default \
  --name awsvpcTrunking \
  --value enabled \
  --region $REGION \
  --output json

# Or enable for a specific IAM user/role only
aws ecs put-account-setting \
  --name awsvpcTrunking \
  --value enabled \
  --principal-arn $PRINCIPAL_ARN \
  --region $REGION \
  --output json

Requirements

• Instance MUST be launched after the setting is enabled. Existing instances are NOT affected.
• Instance type MUST support ENI trunking (most c5, m5, r5 and newer generation types).
• The ECS agent on the instance MUST be version 1.28.1 or later, with ecs-init version 1.28.1-2 or later.

Verifying ENI Trunking

# Check account setting
aws ecs list-account-settings \
  --name awsvpcTrunking \
  --effective-settings \
  --region $REGION \
  --output json

# Check instance ENI attachment (look for trunk ENI)
aws ecs describe-container-instances \
  --cluster $CLUSTER \
  --container-instances $CONTAINER_INSTANCE_ID \
  --region $REGION \
  --output json \
  --query 'containerInstances[0].{attachments:attachments,remainingResources:remainingResources}'

Task Density Comparison (Example: c5.large)

Exact limits vary by instance type — see Supported instance types for ENI trunking.

Operators SHOULD enable ENI trunking for any EC2 cluster using awsvpc network mode to avoid RESOURCE:ENI placement failures.

---

Security Considerations

• The troubleshooting commands in this guide require read-only permissions (ecs:Describe*, ecs:List*, logs:GetLogEvents, ec2:DescribeSecurityGroups, ec2:DescribeVpcEndpoints, elbv2:DescribeTargetHealth). Do not grant broader permissions for debugging.
• ECS Exec (execute-command) provides shell access to running containers. Restrict ssmmessages:* permissions to authorized operators only and audit usage via CloudTrail.
• VPC endpoint security groups MUST restrict inbound HTTPS (port 443) to the task security group — do not use 0.0.0.0/0.
• When reviewing container logs for errors, be aware that application logs may contain sensitive data. Use CloudWatch Logs encryption with a KMS key for log groups containing sensitive output.
• The 0.0.0.0 listen address in Health Check Step 6 refers to the container's network interface binding, not a security group rule. In awsvpc mode, each task has its own ENI and the ALB health check arrives from the ALB's IP, requiring the application to listen on all interfaces.

Fargate Service Deployment Reference

Table of Contents

• Verify Dependencies
• Create Cluster
• Register Task Definition
• Create Application Load Balancer
• Create Target Group
• Create ALB Listener
• Create ECS Service
• Verify Service Health
• Private Subnet Networking
• 502 Bad Gateway Debugging Checklist
• Path-Based Routing
• Security Considerations

---

Verify Dependencies

Before deploying a Fargate service, the operator MUST confirm:

1. A registered task definition $TASK_DEFINITION exists. 2. A VPC ($VPC_ID) with at least two subnets ($SUBNET_1, $SUBNET_2) in different AZs exists. 3. Security groups for the ALB ($ALB_SG_ID) and tasks ($TASK_SG_ID) exist. 4. The execution role and task role referenced in the task definition exist. 5. An ACM certificate ($ACM_CERT_ARN) exists for the ALB HTTPS listener.

Constraints for parameter acquisition:

• You MUST verify all required parameters ($CLUSTER, $TASK_DEFINITION, $SUBNET_1, $SUBNET_2, $ALB_SG_ID, $TASK_SG_ID, $CONTAINER_NAME, $CONTAINER_PORT) are provided. If any are missing, ask for them upfront in a single prompt.
• If all required parameters are provided, proceed to Step 1 — do not ask the user to confirm what they already specified.
• For optional parameters not specified by the user ($SERVICE_NAME, $CLUSTER name, health check path), you SHOULD select reasonable defaults, inform the user what you chose, and proceed.

aws sts get-caller-identity --output json
aws ecs describe-task-definition \
  --task-definition "$TASK_DEFINITION" \
  --region "$REGION" \
  --output json
aws ec2 describe-subnets \
  --subnet-ids "$SUBNET_1" "$SUBNET_2" \
  --region "$REGION" \
  --output json

---

Create Cluster

aws ecs create-cluster \
  --cluster-name "$CLUSTER" \
  --settings name=containerInsights,value=enabled \
  --region "$REGION" \
  --output json

The operator SHOULD enable Container Insights for observability.

---

Register Task Definition

If not already registered, register the task definition from a JSON file:

aws ecs register-task-definition \
  --cli-input-json file://task-definition.json \
  --region "$REGION" \
  --output json

See task-definition-authoring.md for the task definition structure.

---

Create Application Load Balancer

aws elbv2 create-load-balancer \
  --name "$ALB_NAME" \
  --subnets "$SUBNET_1" "$SUBNET_2" \
  --security-groups "$ALB_SG_ID" \
  --scheme internet-facing \
  --type application \
  --region "$REGION" \
  --output json

The ALB security group MUST allow inbound traffic on the listener ports:

[
  {
    "IpProtocol": "tcp",
    "FromPort": 443,
    "ToPort": 443,
    "IpRanges": [
      { "CidrIp": "$ALLOWED_CIDR", "Description": "Inbound HTTPS from allowed range" }
    ]
  },
  {
    "IpProtocol": "tcp",
    "FromPort": 80,
    "ToPort": 80,
    "IpRanges": [
      { "CidrIp": "$ALLOWED_CIDR", "Description": "Inbound HTTP for HTTPS redirect" }
    ]
  }
]

The task security group MUST allow inbound traffic from the ALB security group on the container port:

{
  "IpProtocol": "tcp",
  "FromPort": $CONTAINER_PORT,
  "ToPort": $CONTAINER_PORT,
  "UserIdGroupPairs": [
    { "GroupId": "$ALB_SG_ID", "Description": "Inbound from ALB" }
  ]
}

---

Create Target Group

For Fargate with awsvpc networking, the target type MUST be ip.

aws elbv2 create-target-group \
  --name "$TG_NAME" \
  --protocol HTTP \
  --port $CONTAINER_PORT \
  --vpc-id "$VPC_ID" \
  --target-type ip \
  --health-check-path "/health" \
  --health-check-interval-seconds 30 \
  --health-check-timeout-seconds 5 \
  --healthy-threshold-count 2 \
  --unhealthy-threshold-count 2 \
  --region "$REGION" \
  --output json

Health Check Configuration

Deregistration Delay

The operator SHOULD set deregistration delay to 30–60 seconds to allow in-flight requests to complete:

aws elbv2 modify-target-group-attributes \
  --target-group-arn "$TG_ARN" \
  --attributes Key=deregistration_delay.timeout_seconds,Value=30 \
  --region "$REGION" \
  --output json

---

Create ALB Listener

The operator MUST create an HTTPS listener with an ACM certificate for encryption in transit. Per AWS ECS Network Security Best Practices: "If your service is fronted by a public facing load balancer, use TLS/SSL to encrypt the traffic from the client's browser to the load balancer."

aws elbv2 create-listener \
  --load-balancer-arn "$ALB_ARN" \
  --protocol HTTPS \
  --port 443 \
  --ssl-policy "ELBSecurityPolicy-TLS13-1-2-2021-06" \
  --certificates CertificateArn="$ACM_CERT_ARN" \
  --default-actions Type=forward,TargetGroupArn="$TG_ARN" \
  --region "$REGION" \
  --output json

The operator SHOULD also create an HTTP-to-HTTPS redirect listener:

aws elbv2 create-listener \
  --load-balancer-arn "$ALB_ARN" \
  --protocol HTTP \
  --port 80 \
  --default-actions 'Type=redirect,RedirectConfig={Protocol=HTTPS,Port=443,StatusCode=HTTP_301}' \
  --region "$REGION" \
  --output json

---

Create ECS Service

aws ecs create-service \
  --cluster "$CLUSTER" \
  --service-name "$SERVICE_NAME" \
  --task-definition "$TASK_DEFINITION" \
  --desired-count 2 \
  --launch-type FARGATE \
  --platform-version "LATEST" \
  --network-configuration "awsvpcConfiguration={subnets=[$SUBNET_1,$SUBNET_2],securityGroups=[$TASK_SG_ID],assignPublicIp=DISABLED}" \
  --load-balancers "targetGroupArn=$TG_ARN,containerName=$CONTAINER_NAME,containerPort=$CONTAINER_PORT" \
  --health-check-grace-period-seconds 90 \
  --deployment-configuration "minimumHealthyPercent=100,maximumPercent=200,deploymentCircuitBreaker={enable=true,rollback=true}" \
  --region "$REGION" \
  --output json

Deployment Configuration

Health Check Grace Period

The healthCheckGracePeriodSeconds SHOULD be set when using a load balancer to prevent ECS from marking tasks unhealthy before the application finishes starting. CDK defaults to 60 seconds when a load balancer is attached.

Circuit Breaker with Rollback

The operator SHOULD enable the deployment circuit breaker with rollback. When enabled, ECS automatically rolls back to the last stable deployment if the new deployment fails to reach a steady state.

---

Verify Service Health

aws ecs describe-services \
  --cluster "$CLUSTER" \
  --services "$SERVICE_NAME" \
  --region "$REGION" \
  --output json

aws ecs list-tasks \
  --cluster "$CLUSTER" \
  --service-name "$SERVICE_NAME" \
  --desired-status RUNNING \
  --region "$REGION" \
  --output json

aws elbv2 describe-target-health \
  --target-group-arn "$TG_ARN" \
  --region "$REGION" \
  --output json

The operator MUST verify:

1. runningCount equals desiredCount in the service description. 2. All targets in the target group report healthy. 3. No deployment events show errors in the service events list.

---

Private Subnet Networking

When tasks run in private subnets with assignPublicIp=DISABLED, they MUST have a path to reach AWS service endpoints.

Option 1: NAT Gateway

Tasks route through a NAT gateway in a public subnet. This is simpler but incurs NAT gateway data processing charges.

Option 2: VPC Endpoints (Recommended for Cost Optimization)

The operator SHOULD create VPC endpoints to avoid NAT gateway costs for AWS service traffic:

Interface endpoints MUST have a security group allowing inbound HTTPS (port 443) from the task security group:

{
  "IpProtocol": "tcp",
  "FromPort": 443,
  "ToPort": 443,
  "UserIdGroupPairs": [
    { "GroupId": "$TASK_SG_ID", "Description": "HTTPS from ECS tasks" }
  ]
}

Without either a NAT gateway or VPC endpoints, tasks in private subnets fail to pull images and push logs.

---

502 Bad Gateway Debugging Checklist

When the ALB returns HTTP 502, the operator MUST check these items in order:

1. Target group health — Run describe-target-health. If targets are unhealthy, the application is not responding on the health check path. Check application logs in CloudWatch. 2. Security group rules — Confirm the task security group allows inbound from the ALB security group on the container port. Confirm the ALB security group allows inbound on the listener ports. 3. Container port mismatch — Verify the containerPort in the task definition matches the port the application listens on, and matches the target group port. 4. Health check grace period — If tasks are being killed before the application starts, increase healthCheckGracePeriodSeconds. 5. Application crash — Check CloudWatch Logs for the task. If the container exits immediately, inspect the stoppedReason:

aws ecs describe-tasks \
  --cluster "$CLUSTER" \
  --tasks "$TASK_ARN" \
  --region "$REGION" \
  --output json

---

Path-Based Routing

To route different URL paths to different target groups, create ALB listener rules.

Create Additional Target Group

aws elbv2 create-target-group \
  --name "$TG_NAME_API" \
  --protocol HTTP \
  --port $CONTAINER_PORT \
  --vpc-id "$VPC_ID" \
  --target-type ip \
  --health-check-path "/api/health" \
  --region "$REGION" \
  --output json

Create Listener Rule

aws elbv2 create-rule \
  --listener-arn "$LISTENER_ARN" \
  --priority 10 \
  --conditions Field=path-pattern,Values='/api/*' \
  --actions Type=forward,TargetGroupArn="$TG_ARN_API" \
  --region "$REGION" \
  --output json

Rules are evaluated in priority order (lowest number first). The default action on the listener acts as a catch-all for unmatched paths.

The operator SHOULD assign priorities with gaps (e.g., 10, 20, 30) to allow inserting new rules later without reordering.

---

Security Considerations

The operator SHOULD review the following security controls for production deployments:

• HTTPS/TLS: The ALB listener MUST use HTTPS with an ACM certificate. HTTP traffic SHOULD redirect to HTTPS (see Create ALB Listener). Per AWS ECS Network Security Best Practices: "use TLS/SSL to encrypt the traffic from the client's browser to the load balancer."
• AWS WAF: The operator SHOULD associate an AWS WAF web ACL with the ALB for defense in depth against common web exploits (SQL injection, XSS, rate limiting).
• ALB access logs: The operator SHOULD enable ALB access logs to an S3 bucket for audit and troubleshooting. See Enable access logs for your ALB.
• VPC Flow Logs: Per AWS ECS best practices: "Use Amazon VPC Flow Logs to analyze the traffic to and from long-running tasks." The operator SHOULD enable VPC Flow Logs for the subnets running Fargate tasks.
• Security headers: The application SHOULD return security headers (Strict-Transport-Security, Content-Security-Policy, X-Content-Type-Options, X-Frame-Options) in HTTP responses.