Step Functions

You are a Step Functions specialist. Help teams design reliable, cost-effective state machine workflows.

Decision Framework: Standard vs Express

Opinionated recommendation:

• Default to Standard for business workflows, orchestration, and anything requiring auditability.
• Use Express for high-volume event processing (>100K executions/day), data transforms, and ETL microbatches where duration is under 5 minutes.
• Express is cheaper at scale but loses execution history -- you must configure CloudWatch Logs.

State Types

Task State (does work)

Opinionated: Always add Retry and Catch to every Task state. Without Retry, a transient failure (Lambda throttle, DynamoDB ProvisionedThroughputExceededException, network timeout) fails the entire execution immediately — even though a retry 2 seconds later would succeed. Without Catch, a permanent failure (invalid input, missing resource) causes an unhandled error that terminates the workflow with no way to log the failure, notify anyone, or run compensating actions. The cost of adding Retry+Catch is a few lines of ASL; the cost of omitting them is silent failures in production.

Direct Service Integrations (prefer over Lambda wrappers)

Step Functions can call 200+ AWS services directly. Do NOT wrap simple API calls in Lambda. Common direct integrations to use instead of Lambda:

• DynamoDB: GetItem, PutItem, UpdateItem, DeleteItem, Query
• SQS: SendMessage
• SNS: Publish
• EventBridge: PutEvents
• ECS/Fargate: RunTask (for long-running containers)
• Glue: StartJobRun
• SageMaker: CreateTransformJob, CreateTrainingJob
• Bedrock: InvokeModel

See references/integrations.md for ASL examples of each integration, plus Choice, Parallel, Map, and Wait state examples.

Other State Types

• Choice: Branch based on input values (string, numeric, boolean comparisons)
• Parallel: Run multiple branches concurrently, Catch on any branch failure
• Map (Inline): Iterate over a collection with configurable MaxConcurrency
• Map (Distributed): Process millions of items from S3 with Express child executions
• Wait: Pause for a duration or until a timestamp

Error Handling: Retry and Catch

Retry Strategy

"Retry": [
  {
    "ErrorEquals": ["States.Timeout"],
    "IntervalSeconds": 5,
    "MaxAttempts": 2,
    "BackoffRate": 2.0
  },
  {
    "ErrorEquals": ["TransientError", "Lambda.ServiceException"],
    "IntervalSeconds": 1,
    "MaxAttempts": 5,
    "BackoffRate": 2.0,
    "JitterStrategy": "FULL"
  },
  {
    "ErrorEquals": ["States.ALL"],
    "MaxAttempts": 0
  }
]

Opinionated: Order retries from specific to general. Use JitterStrategy: FULL to prevent thundering herd. Put States.ALL with MaxAttempts: 0 last to explicitly catch-and-fail on unexpected errors rather than retrying them.

Catch and Error Recovery

"Catch": [
  {
    "ErrorEquals": ["PaymentDeclined"],
    "Next": "NotifyCustomerPaymentFailed",
    "ResultPath": "$.error"
  },
  {
    "ErrorEquals": ["States.ALL"],
    "Next": "GenericErrorHandler",
    "ResultPath": "$.error"
  }
]

Always use `ResultPath` in Catch to preserve the original input alongside the error. Without it, the error replaces your entire state input.

Pattern: Saga (Compensating Transactions)

For distributed transactions across services where you need to undo completed steps on failure. Each step has a compensating action, compensations run in reverse order, and compensations must be idempotent. See references/patterns.md for the full ASL example with compensating transaction flow.

Pattern: Human Approval (Callback)

Use .waitForTaskToken to pause execution until an external system sends a callback via send-task-success or send-task-failure. Always set `TimeoutSeconds` on callback tasks. Without it, the execution waits forever (up to 1 year for Standard). See references/patterns.md for the full ASL and CLI examples.

Pattern: Distributed Map

Process millions of items from S3 using Express child executions for massive parallelism. See references/patterns.md for the ASL example with S3 CSV reader configuration.

Common CLI Commands

# Create state machine
aws stepfunctions create-state-machine \
  --name my-workflow \
  --definition file://definition.json \
  --role-arn arn:aws:iam::123456789:role/step-functions-role

# Start execution
aws stepfunctions start-execution \
  --state-machine-arn arn:aws:states:us-east-1:123456789:stateMachine:my-workflow \
  --input '{"orderId": "12345"}'

# List executions
aws stepfunctions list-executions \
  --state-machine-arn arn:aws:states:us-east-1:123456789:stateMachine:my-workflow \
  --status-filter FAILED

# Get execution details
aws stepfunctions describe-execution \
  --execution-arn arn:aws:states:us-east-1:123456789:execution:my-workflow:exec-123

# Get execution history (debug step-by-step)
aws stepfunctions get-execution-history \
  --execution-arn arn:aws:states:us-east-1:123456789:execution:my-workflow:exec-123 \
  --query 'events[?type==`TaskFailed` || type==`ExecutionFailed`]'

# Update state machine
aws stepfunctions update-state-machine \
  --state-machine-arn arn:aws:states:us-east-1:123456789:stateMachine:my-workflow \
  --definition file://definition.json

# Test a state (local testing)
aws stepfunctions test-state \
  --definition '{"Type":"Task","Resource":"arn:aws:states:::dynamodb:getItem","Parameters":{"TableName":"Orders","Key":{"orderId":{"S":"123"}}}}' \
  --role-arn arn:aws:iam::123456789:role/step-functions-role \
  --input '{"orderId": "123"}'

Workflow Studio

Use Workflow Studio in the AWS Console for:

• Visual design and prototyping (drag-and-drop states)
• Understanding existing workflows
• Quick iteration on state machine logic

Opinionated: Start in Workflow Studio for prototyping, then export to ASL (Amazon States Language) JSON and manage in version control. Never rely solely on the console for production workflows.

Input/Output Processing

Data flows through each state as: InputPath -> Parameters -> Task -> ResultSelector -> ResultPath -> OutputPath

Opinionated: Use ResultPath generously to accumulate data through states. Use ResultSelector to trim large API responses down to only what you need (saves state size and cost on Standard workflows). See references/integrations.md for detailed examples of each processing stage.

Anti-Patterns

1. Lambda wrappers for AWS API calls: Step Functions integrates directly with 200+ services. Don't write a Lambda just to call DynamoDB PutItem or SQS SendMessage. 2. No error handling on Task states: Every Task state should have Retry (for transient errors) and Catch (for permanent failures). No exceptions. 3. Ignoring state payload limits: Standard workflows have a 256 KB payload limit per state. Store large data in S3 and pass references. 4. Using Standard for high-volume short tasks: If you're running >100K executions/day with <5 min duration, Express workflows are dramatically cheaper. 5. Missing TimeoutSeconds on callback tasks: Without a timeout, .waitForTaskToken tasks will hang for up to 1 year if the callback never arrives. 6. Not using Distributed Map for large datasets: Inline Map processes items sequentially or with limited concurrency within one execution. Distributed Map scales to millions of items. 7. Putting business logic in the state machine: ASL is for orchestration, not computation. Complex data transforms and business rules belong in Lambda functions. 8. Not enabling logging for Express workflows: Express workflows have no built-in execution history. You MUST configure CloudWatch Logs or you'll have zero visibility. 9. Monolith state machines: A 50-state workflow is hard to understand and test. Break large workflows into nested state machines using arn:aws:states:::states:startExecution.sync:2. 10. Not using `JitterStrategy` on retries: Without jitter, retried tasks create thundering herd effects that amplify the original failure.

Cost Optimization

• Standard: $0.025 per 1,000 state transitions. Minimize states. Use direct integrations to avoid Lambda invocation costs on top of transition costs.
• Express: Priced by number of requests and duration. Cheaper for high-volume, short workflows.
• Pass states are not free in Standard (they count as transitions). Eliminate unnecessary Pass states.
• Combine simple sequential tasks where possible to reduce transition count.
• Use ResultSelector to trim response payloads -- smaller payloads mean faster processing.

Reference Files

• references/patterns.md -- Saga, callback, and Distributed Map patterns with full ASL examples
• references/integrations.md -- Direct service integration examples (DynamoDB, SQS, SNS, EventBridge, ECS, Bedrock), state type ASL, and input/output processing pipeline details

Related Skills

• aws-plan -- Architecture planning that may include Step Functions workflows
• lambda -- Lambda functions used as Task state targets
• api-gateway -- API Gateway to Step Functions direct integrations (StartExecution, StartSyncExecution)
• observability -- CloudWatch Logs, X-Ray tracing, and monitoring for Step Functions
• aws-debug -- Debugging failed Step Functions executions

Step Functions Service Integrations and Data Flow

Direct Service Integrations

Step Functions can call 200+ AWS services directly. Prefer direct integrations over Lambda wrappers for simple API calls.

DynamoDB PutItem

{
  "PutItem": {
    "Type": "Task",
    "Resource": "arn:aws:states:::dynamodb:putItem",
    "Parameters": {
      "TableName": "Orders",
      "Item": {
        "orderId": {"S.$": "$.orderId"},
        "status": {"S": "PENDING"},
        "createdAt": {"S.$": "$$.State.EnteredTime"}
      }
    },
    "Next": "NotifyCustomer"
  }
}

DynamoDB GetItem

{
  "GetItem": {
    "Type": "Task",
    "Resource": "arn:aws:states:::dynamodb:getItem",
    "Parameters": {
      "TableName": "Orders",
      "Key": {
        "orderId": {"S.$": "$.orderId"}
      }
    },
    "ResultSelector": {
      "orderId.$": "$.Item.orderId.S",
      "status.$": "$.Item.status.S"
    },
    "ResultPath": "$.orderData",
    "Next": "ProcessOrder"
  }
}

SQS SendMessage

{
  "SendMessage": {
    "Type": "Task",
    "Resource": "arn:aws:states:::sqs:sendMessage",
    "Parameters": {
      "QueueUrl": "https://sqs.us-east-1.amazonaws.com/123456789/my-queue",
      "MessageBody": {
        "orderId.$": "$.orderId",
        "action": "process"
      }
    },
    "Next": "Done"
  }
}

SNS Publish

{
  "NotifyCustomer": {
    "Type": "Task",
    "Resource": "arn:aws:states:::sns:publish",
    "Parameters": {
      "TopicArn": "arn:aws:sns:us-east-1:123456789:order-notifications",
      "Message": {
        "orderId.$": "$.orderId",
        "status": "Order confirmed"
      }
    },
    "Next": "Done"
  }
}

EventBridge PutEvents

{
  "EmitEvent": {
    "Type": "Task",
    "Resource": "arn:aws:states:::events:putEvents",
    "Parameters": {
      "Entries": [
        {
          "Source": "order-service",
          "DetailType": "OrderCompleted",
          "Detail": {
            "orderId.$": "$.orderId",
            "amount.$": "$.amount"
          }
        }
      ]
    },
    "Next": "Done"
  }
}

ECS/Fargate RunTask

{
  "RunFargateTask": {
    "Type": "Task",
    "Resource": "arn:aws:states:::ecs:runTask.sync",
    "Parameters": {
      "LaunchType": "FARGATE",
      "Cluster": "arn:aws:ecs:us-east-1:123456789:cluster/my-cluster",
      "TaskDefinition": "arn:aws:ecs:us-east-1:123456789:task-definition/my-task:1",
      "NetworkConfiguration": {
        "AwsvpcConfiguration": {
          "Subnets": ["subnet-abc123"],
          "SecurityGroups": ["sg-abc123"],
          "AssignPublicIp": "DISABLED"
        }
      },
      "Overrides": {
        "ContainerOverrides": [
          {
            "Name": "my-container",
            "Environment": [
              { "Name": "ORDER_ID", "Value.$": "$.orderId" }
            ]
          }
        ]
      }
    },
    "Next": "Done"
  }
}

Bedrock InvokeModel

{
  "InvokeModel": {
    "Type": "Task",
    "Resource": "arn:aws:states:::bedrock:invokeModel",
    "Parameters": {
      "ModelId": "anthropic.claude-3-sonnet-20240229-v1:0",
      "Body": {
        "anthropic_version": "bedrock-2023-05-31",
        "max_tokens": 1024,
        "messages": [
          {
            "role": "user",
            "content.$": "$.prompt"
          }
        ]
      },
      "ContentType": "application/json",
      "Accept": "application/json"
    },
    "ResultSelector": {
      "response.$": "$.Body.content[0].text"
    },
    "ResultPath": "$.modelResult",
    "Next": "Done"
  }
}

Common Direct Integrations Reference

Input/Output Processing Pipeline

Step Functions processes data through a pipeline at each state:

InputPath -> Parameters -> Task -> ResultSelector -> ResultPath -> OutputPath

InputPath

Filters what the state sees from the input. Default: $ (everything).

{
  "ProcessOrder": {
    "Type": "Task",
    "InputPath": "$.orderDetails",
    "Resource": "...",
    "Next": "Done"
  }
}

Parameters

Constructs the payload sent to the task. Use .$ suffix for JSONPath references.

{
  "ProcessOrder": {
    "Type": "Task",
    "Parameters": {
      "orderId.$": "$.orderId",
      "timestamp.$": "$$.State.EnteredTime",
      "staticValue": "PROCESSING"
    },
    "Resource": "...",
    "Next": "Done"
  }
}

ResultSelector

Reshapes the task result before merging back. Use to trim large API responses.

{
  "GetOrder": {
    "Type": "Task",
    "Resource": "arn:aws:states:::dynamodb:getItem",
    "Parameters": {
      "TableName": "Orders",
      "Key": { "orderId": {"S.$": "$.orderId"} }
    },
    "ResultSelector": {
      "orderId.$": "$.Item.orderId.S",
      "status.$": "$.Item.status.S",
      "amount.$": "$.Item.amount.N"
    },
    "ResultPath": "$.orderData",
    "Next": "ProcessOrder"
  }
}

ResultPath

Where to place the result in the original input. Use $.taskResult to preserve original input alongside the result.

{
  "ChargeCard": {
    "Type": "Task",
    "Resource": "...",
    "ResultPath": "$.chargeResult",
    "Next": "ReserveInventory"
  }
}

Without ResultPath, the task result replaces the entire state input. With ResultPath: "$.chargeResult", the result is merged into the input at that path.

OutputPath

Filters what gets passed to the next state.

{
  "GetOrder": {
    "Type": "Task",
    "Resource": "...",
    "ResultPath": "$.orderData",
    "OutputPath": "$.orderData",
    "Next": "ProcessOrder"
  }
}

Best Practices

• Use ResultPath generously to accumulate data through states
• Use ResultSelector to trim large API responses (saves state size and cost on Standard workflows)
• The .$ suffix in Parameters is how you reference JSONPath values vs static strings
• $$. prefix accesses the context object (execution ARN, state name, entered time, task token)

State Type Examples

Choice State (Branching)

{
  "CheckOrderType": {
    "Type": "Choice",
    "Choices": [
      {
        "Variable": "$.orderType",
        "StringEquals": "express",
        "Next": "ExpressShipping"
      },
      {
        "Variable": "$.amount",
        "NumericGreaterThan": 1000,
        "Next": "RequireApproval"
      }
    ],
    "Default": "StandardShipping"
  }
}

Parallel State (Concurrent Branches)

{
  "ProcessInParallel": {
    "Type": "Parallel",
    "Branches": [
      {
        "StartAt": "ChargeCard",
        "States": {
          "ChargeCard": { "Type": "Task", "Resource": "...", "End": true }
        }
      },
      {
        "StartAt": "ReserveInventory",
        "States": {
          "ReserveInventory": { "Type": "Task", "Resource": "...", "End": true }
        }
      }
    ],
    "Catch": [{"ErrorEquals": ["States.ALL"], "Next": "RollbackAll"}],
    "Next": "ConfirmOrder"
  }
}

Inline Map State (Iterate Over Collections)

{
  "ProcessItems": {
    "Type": "Map",
    "ItemsPath": "$.items",
    "MaxConcurrency": 10,
    "ItemProcessor": {
      "ProcessorConfig": {
        "Mode": "INLINE"
      },
      "StartAt": "ProcessItem",
      "States": {
        "ProcessItem": { "Type": "Task", "Resource": "...", "End": true }
      }
    },
    "Next": "Done"
  }
}

Wait State

{
  "WaitForApproval": {
    "Type": "Wait",
    "Seconds": 3600,
    "Next": "CheckApproval"
  }
}

Wait until a specific timestamp:

{
  "WaitUntilDelivery": {
    "Type": "Wait",
    "TimestampPath": "$.deliveryTime",
    "Next": "Deliver"
  }
}

Step Functions Patterns

Saga Pattern (Compensating Transactions)

For distributed transactions across services where you need to undo completed steps on failure.

Flow

StartOrder -> ChargeCard -> ReserveInventory -> ShipOrder -> Done
                |               |                  |
                v               v                  v
           RefundCard    ReleaseInventory     CancelShipment
                |               |                  |
                +--------> OrderFailed <-----------+

Key Principles

1. Each step has a compensating action 2. Compensations run in reverse order 3. Compensations must be idempotent 4. Store step results for compensation context

Full ASL Example

{
  "Comment": "Order saga with compensating transactions",
  "StartAt": "ChargeCard",
  "States": {
    "ChargeCard": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789:function:charge-card",
      "ResultPath": "$.chargeResult",
      "Retry": [
        {
          "ErrorEquals": ["States.TaskFailed"],
          "IntervalSeconds": 2,
          "MaxAttempts": 3,
          "BackoffRate": 2.0
        }
      ],
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "Next": "OrderFailed",
          "ResultPath": "$.error"
        }
      ],
      "Next": "ReserveInventory"
    },
    "ReserveInventory": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789:function:reserve-inventory",
      "ResultPath": "$.inventoryResult",
      "Retry": [
        {
          "ErrorEquals": ["States.TaskFailed"],
          "IntervalSeconds": 2,
          "MaxAttempts": 3,
          "BackoffRate": 2.0
        }
      ],
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "Next": "RefundCard",
          "ResultPath": "$.error"
        }
      ],
      "Next": "ShipOrder"
    },
    "ShipOrder": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789:function:ship-order",
      "ResultPath": "$.shipResult",
      "Retry": [
        {
          "ErrorEquals": ["States.TaskFailed"],
          "IntervalSeconds": 2,
          "MaxAttempts": 3,
          "BackoffRate": 2.0
        }
      ],
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "Next": "ReleaseInventory",
          "ResultPath": "$.error"
        }
      ],
      "Next": "Done"
    },
    "RefundCard": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789:function:refund-card",
      "ResultPath": "$.refundResult",
      "Next": "OrderFailed"
    },
    "ReleaseInventory": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789:function:release-inventory",
      "ResultPath": "$.releaseResult",
      "Next": "RefundCard"
    },
    "CancelShipment": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789:function:cancel-shipment",
      "ResultPath": "$.cancelResult",
      "Next": "ReleaseInventory"
    },
    "Done": {
      "Type": "Succeed"
    },
    "OrderFailed": {
      "Type": "Fail",
      "Error": "OrderFailed",
      "Cause": "One or more steps failed and compensations have been applied"
    }
  }
}

Human Approval / Callback Pattern

Use .waitForTaskToken to pause execution until an external system sends a callback. Common for human approval flows, external system integrations, and async processing.

ASL Example

{
  "WaitForApproval": {
    "Type": "Task",
    "Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken",
    "Parameters": {
      "QueueUrl": "https://sqs.us-east-1.amazonaws.com/123456789/approval-queue",
      "MessageBody": {
        "taskToken.$": "$$.Task.Token",
        "orderId.$": "$.orderId",
        "amount.$": "$.amount"
      }
    },
    "TimeoutSeconds": 86400,
    "Next": "ProcessApproval"
  }
}

Callback Commands

The external system calls back with:

aws stepfunctions send-task-success \
  --task-token "TOKEN" \
  --task-output '{"approved": true}'

# Or on rejection:
aws stepfunctions send-task-failure \
  --task-token "TOKEN" \
  --error "Rejected" \
  --cause "Manager declined the order"

Always set `TimeoutSeconds` on callback tasks. Without it, the execution waits forever (up to 1 year for Standard).

Distributed Map Pattern

For large-scale processing of millions of items from S3. Unlike Inline Map, Distributed Map launches child executions (Express) for massive parallelism.

ASL Example

{
  "ProcessLargeDataset": {
    "Type": "Map",
    "ItemProcessor": {
      "ProcessorConfig": {
        "Mode": "DISTRIBUTED",
        "ExecutionType": "EXPRESS"
      },
      "StartAt": "ProcessBatch",
      "States": {
        "ProcessBatch": { "Type": "Task", "Resource": "...", "End": true }
      }
    },
    "ItemReader": {
      "Resource": "arn:aws:states:::s3:getObject",
      "ReaderConfig": {
        "InputType": "CSV",
        "CSVHeaderLocation": "FIRST_ROW"
      },
      "Parameters": {
        "Bucket": "my-bucket",
        "Key": "data.csv"
      }
    },
    "MaxConcurrency": 1000,
    "Next": "Done"
  }
}

When to Use Distributed Map

• Processing millions of items from S3 (CSV, JSON, manifest)
• Need concurrency beyond what Inline Map offers
• Each item requires non-trivial processing
• Want to leverage Express workflow pricing for child executions