Agentcore

Specialist guidance for Amazon Bedrock AgentCore. Covers the full platform: Runtime, Memory, Gateway, Identity, Policy, Code Interpreter, Browser, Observability, and Evaluations. Framework-agnostic and model-agnostic.

Process

1. Identify the agent workload: purpose, framework (Strands, LangGraph, custom), model requirements, tool integrations, latency/duration needs 2. Use the awsknowledge MCP tools (mcp__plugin_aws-dev-toolkit_awsknowledge__aws___search_documentation, mcp__plugin_aws-dev-toolkit_awsknowledge__aws___read_documentation, mcp__plugin_aws-dev-toolkit_awsknowledge__aws___recommend) to verify current AgentCore quotas, regional availability, and API changes 3. Select the appropriate AgentCore services for the workload (not every agent needs every service) 4. Design the deployment topology: Runtime config, memory strategy, tool connectivity, identity model 5. Configure security: Identity, Policy (Cedar), VPC connectivity, guardrails 6. Set up observability and evaluations from day one 7. Plan the PoC-to-production migration path

AgentCore Service Selection Matrix

When You Need Each Service

Always Start With

• Runtime — every production agent needs managed hosting
• Observability — instrument from day one, not after the first incident

Add Based on Workload

• Memory — when agents need conversation continuity or personalization
• Gateway — when agents call external APIs or Lambda functions (most agents)
• Identity — when agents access third-party services requiring OAuth or API keys
• Policy — when you need deterministic guardrails on tool usage (compliance, financial, PII)

Add for Specialized Capabilities

• Code Interpreter — data analysis agents, code generation agents
• Browser — web scraping, form-filling, UI testing agents
• Evaluations — continuous quality monitoring (should be added before production)

Runtime

AgentCore Runtime is a serverless, purpose-built hosting environment for AI agents.

Key Capabilities

• Framework-agnostic: Strands Agents, LangGraph, custom Python, any framework
• Model-agnostic: any foundation model (Bedrock, self-hosted, third-party)
• Session isolation: each user session runs in its own execution context
• Supports real-time conversations (<1s latency) through to 8-hour async workloads
• Bidirectional streaming (WebSocket) for natural conversations
• Consumption-based pricing: CPU + memory billed per-second (1-second minimum)
• A2A (Agent-to-Agent) protocol support for cross-framework multi-agent systems

Development vs Production Deployment

Development and testing: Use the AgentCore CLI or Starter Toolkit for fast iteration — scaffolding, local dev, quick deploys, and testing.

Production: Define all AgentCore resources in IaC (CDK, Terraform, CloudFormation, or SAM). CLI-created resources are useful for prototyping but should not be the source of truth for production infrastructure. The Starter Toolkit's CDK templates are a solid starting point for production IaC.

Deployment Options

• AgentCore CLI (dev/test): Fastest path — agentcore init → agentcore deploy in minutes
• Starter Toolkit (reference IaC): Full-stack CDK template with auth, frontend, and all services pre-wired — fork and customize for production
• CDK / Terraform / SAM (production): Define resources in IaC, deploy via CI/CD pipeline
• Container image (manual): Docker image pushed to ECR, deployed to Runtime — full control over build

AgentCore CLI

The AgentCore CLI is the preferred tool for scaffolding, local development, and rapid iteration on agents. It abstracts away container builds, ECR pushes, and runtime configuration into simple commands. Use it for dev/test workflows — for production, define the same resources in IaC.

Install

pip install agentcore-cli

Quick Start

# Initialize a new agent project (choose framework: strands, langgraph, or custom)
agentcore init my-agent --framework strands

# Develop locally
cd my-agent
agentcore dev

# Deploy to AgentCore Runtime
agentcore deploy --region us-east-1

# Test the deployed agent
agentcore invoke --agent-name my-agent --input "Hello, what can you do?"

What the CLI Handles

• Project scaffolding: generates agent code, Dockerfile, requirements, and config
• Local development: agentcore dev runs the agent locally with hot-reload
• Build + push: builds the Docker container, pushes to ECR automatically
• Deploy: creates/updates the agent runtime and endpoint
• Invoke: test deployed agents from the command line
• Alias management: create and update aliases for version routing

CLI vs Direct AWS CLI

Use the AgentCore CLI for day-to-day development and testing. For production, define the equivalent resources in CDK, Terraform, or CloudFormation — the CLI is great for proving out configurations quickly, but IaC is the source of truth for production infrastructure.

Starter Toolkit (FAST Template)

The AgentCore Starter Toolkit provides a full-stack CDK reference architecture. Use it when you need a complete production deployment with authentication, frontend, and all AgentCore services wired together.

What It Provides

• CDK infrastructure: Full IaC for Runtime, Gateway, Memory, Code Interpreter, and Observability — one cdk deploy
• Auth integration: Amazon Cognito authentication pre-wired for frontend → Runtime, agents → Gateway, and API Gateway
• Frontend template: React app with streamable HTTP for real-time agent response streaming via CloudFront
• Framework templates: Pre-built agent patterns for Strands Agents and LangGraph (framework-agnostic by design)
• CI/CD patterns: GitHub Actions workflow for build, scan (Amazon Inspector), deploy, and alias management
• Observability: AWS OpenTelemetry Distro auto-instrumentation for traces → X-Ray, metrics/logs → CloudWatch

Quick Start

git clone https://github.com/aws/bedrock-agentcore-starter-toolkit.git
cd bedrock-agentcore-starter-toolkit
pip install -r requirements.txt
cdk deploy --all

Architecture

The Fullstack AgentCore Solution Template (FAST) deploys:

CloudFront (React frontend)
  → Cognito (auth)
    → AgentCore Runtime (agent hosting)
      → AgentCore Memory (conversation + episodic)
      → AgentCore Gateway (MCP-compatible tools)
      → AgentCore Code Interpreter (Python sandbox)
      → AgentCore Observability → CloudWatch + X-Ray

Four authentication integration points are handled automatically: 1. User sign-in to the frontend 2. Frontend → AgentCore Runtime (token-based) 3. Agent → AgentCore Gateway (token-based) 4. API requests → API Gateway (token-based)

Tooling Decision Matrix

Runtime Configuration

Production Deployment Pattern

1. Define all AgentCore resources in IaC (CDK, Terraform, or CloudFormation) — Runtime, Gateway, Memory, Identity, Policy 2. Build agent container with AgentCore SDK decorators (CI/CD pipeline) 3. Push to ECR via pipeline (not manual docker push) 4. Deploy via cdk deploy / terraform apply / CloudFormation changeset 5. Create aliases for version management in IaC (never use TSTALIASID in production) 6. Configure resource-based policies for cross-account access if needed 7. Use the AgentCore CLI's agentcore invoke for smoke testing deployed agents

Memory

Short-Term Memory

• Session-scoped conversation history
• Automatic — enabled by default in Runtime
• Maintains context within a single conversation

Long-Term Memory

• Persists across sessions — agent learns and adapts over time
• Episodic memory: stores extracted insights from past interactions
• Extraction jobs process conversation transcripts into retrievable knowledge
• Consumption-based pricing for storage and retrieval

When to Use Long-Term Memory

• Customer support agents that need to remember past interactions
• Personal assistant agents that build user profiles over time
• Agents that should improve with repeated use

When to Skip Long-Term Memory

• Stateless utility agents (code formatters, calculators)
• Agents where session isolation is a compliance requirement
• Simple single-turn tool-calling agents

Gateway

Converts existing APIs, Lambda functions, and services into MCP-compatible tools that any agent framework can consume.

Key Patterns

• Lambda targets: point Gateway at a Lambda function, it becomes an MCP tool
• API targets: wrap REST/HTTP APIs as agent-callable tools
• MCP server federation: connect to existing MCP servers
• Tools are automatically indexed and discoverable by agents
• Policy enforcement happens at the Gateway boundary

Gateway + Policy Integration

Gateway intercepts all agent-to-tool traffic. Policy evaluates Cedar rules against each request before allowing or denying. This separation means:

• Security teams write policies without touching agent code
• Policies are deterministic (not LLM-based)
• Audit logging captures every allow/deny decision

Identity

Manages how agents authenticate to third-party services and AWS resources.

Workload Identities

• Each agent runtime gets an identity
• Supports IAM role assumption for AWS resources
• OAuth2 token exchange for third-party services (Salesforce, Jira, etc.)
• API key vault for services requiring static credentials
• Custom claims support for enhanced authentication

Best Practice

• Use workload identities instead of embedding credentials in agent code
• Store OAuth client secrets in token vaults, not Secrets Manager (AgentCore manages rotation)
• Use resource-based policies to scope cross-account access

Policy

Deterministic control over agent-tool interactions using Cedar language.

How It Works

1. Create a Policy Engine and attach it to a Gateway 2. Write Cedar policies (or author in natural language — AgentCore converts to Cedar) 3. Gateway intercepts tool calls and evaluates against policies in real-time 4. Allow/deny decisions are logged for audit

Common Policy Patterns

Policy vs Bedrock Guardrails

• Policy: controls what tools an agent can call and with what parameters — deterministic, Cedar-based
• Guardrails: controls what content an agent can produce — LLM-based content filtering, PII detection
• Use both: Policy for tool-level control, Guardrails for content-level control

Multi-Agent Architectures

Bedrock Multi-Agent Collaboration (Managed)

• Supervisor agent orchestrates collaborator agents
• Built-in task delegation and response aggregation
• Each agent has its own tools, knowledge bases, guardrails
• Best for: teams wanting managed orchestration with minimal custom code

A2A Protocol (Agent-to-Agent)

• Cross-framework interoperability (Strands + LangGraph + custom agents can communicate)
• Agents advertise capabilities via Agent Cards
• Task-based request lifecycle with artifacts
• OAuth 2.0 and IAM authentication for secure inter-agent communication
• Best for: heterogeneous agent ecosystems, cross-team agent integration

Agents-as-Tools Pattern

• Specialized agents registered as tools of a supervisor agent
• All agents run within the same AgentCore Runtime
• Supervisor selects and delegates dynamically
• Best for: monolithic deployments where all agents are owned by one team

Architecture Decision

Anti-Patterns

• Using TSTALIASID in production. Create proper aliases with version pinning. Test aliases have no SLA and no rollback capability.
• Skipping observability until "later". Instrument from day one. Debugging an unobservable agent in production is flying blind.
• God agent that does everything. If you need "and" in the agent's job description, you need two agents. Decompose into focused, composable agents.
• Embedding credentials in agent instructions or environment variables. Use AgentCore Identity for OAuth/API keys, IAM roles for AWS resources.
• Not setting session TTLs. Idle sessions consume compute resources. Set appropriate TTLs based on actual usage patterns.
• Skipping Policy for tool access. Without Policy, any agent can call any tool with any parameters. In production, that is a compliance and security gap.
• Over-engineering the PoC. Ship something that works with Runtime + Observability first. Add Memory, Gateway, Policy as needs emerge.
• Ignoring token costs during development. Track token usage per agent/session from the start. Costs compound fast with multi-step reasoning loops.
• Manual prompt management. Treat system prompts like code — version control, review, test. Prompt drift is a production incident waiting to happen.
• Not evaluating before production. Run evals (built-in or DeepEval) in CI/CD. "It looks right" is not a quality gate.
• CLI-deployed resources as production infrastructure. The AgentCore CLI is excellent for dev/test, but production resources should be defined in IaC (CDK, Terraform, CloudFormation). CLI-created resources are not version-controlled, not reproducible, and not auditable.

Pricing Model

AgentCore uses consumption-based pricing across all services — no upfront commitments.

Regional Availability

AgentCore services are available across multiple regions. Core services (Runtime, Memory, Gateway, Identity) are available in: us-east-1, us-east-2, us-west-2, ap-southeast-1, ap-southeast-2, ap-south-1, ap-northeast-1, eu-west-1, eu-central-1. Check the awsknowledge MCP tools (mcp__plugin_aws-dev-toolkit_awsknowledge__aws___search_documentation, mcp__plugin_aws-dev-toolkit_awsknowledge__aws___read_documentation, mcp__plugin_aws-dev-toolkit_awsknowledge__aws___recommend) for the latest regional availability, as new regions are added regularly.

Additional Resources

Reference Files

For detailed operational guidance, consult:

• `references/runtime-deployment.md` — Container setup, SDK decorators, CI/CD with GitHub Actions, alias management, VPC configuration, scaling patterns, and Starter Toolkit usage
• `references/memory-gateway-identity.md` — Memory configuration (short-term and long-term), Gateway setup with Lambda/API targets, Identity OAuth2/API key patterns, and Policy Cedar examples
• `references/observability-evaluations.md` — OpenTelemetry instrumentation, CloudWatch/X-Ray integration, Langfuse for LLM-specific analytics, DeepEval evaluation patterns, CI/CD eval integration, and production monitoring dashboards

Related Skills

• `bedrock` — Bedrock cost modeling and model selection for agent workloads
• `strands-agent` — Strands Agents SDK scaffolding (deploys to AgentCore Runtime)
• `security-review` — IAM, network, and encryption audit for agent infrastructure
• `networking` — VPC design for agents accessing private resources
• `observability` — CloudWatch/X-Ray deep-dive for agent monitoring
• `step-functions` — Alternative orchestration for deterministic multi-step workflows

Output Format

When recommending an AgentCore architecture, include:

Include estimated monthly cost range using the cost-check skill or the awspricing MCP tools.

AgentCore Memory, Gateway, Identity, and Policy Reference

Memory

Short-Term Memory (Session-Scoped)

Short-term memory is enabled by default and maintains conversation history within a session. No additional configuration required.

# Short-term memory is automatic with AgentCore Runtime sessions
# Each session_id maintains its own conversation context
response = bedrock_agentcore_runtime.invoke_agent_runtime(
    agentRuntimeId=agent_id,
    agentRuntimeEndpointName="production",
    sessionId="user-session-123",  # Context persists across calls with same session_id
    payload={"input": "What was my last question?"}
)

Long-Term Memory (Cross-Session)

Long-term memory enables agents to remember information across sessions and build user-specific knowledge.

Create a Memory Resource

aws bedrock-agentcore create-memory \
  --memory-name customer-support-memory \
  --memory-strategies '[
    {
      "strategyName": "user-preferences",
      "description": "Extract and store user preferences from conversations",
      "type": "SEMANTIC",
      "configuration": {
        "semantic": {
          "extractionCriteria": "Extract user preferences, past issues, product ownership, and communication style"
        }
      }
    }
  ]'

Integrate Memory with Agent (Strands)

from strands import Agent
from strands.tools.agentcore import AgentCoreMemoryTool

memory_tool = AgentCoreMemoryTool(
    memory_id="memory-abc123",
    region="us-east-1"
)

agent = Agent(
    model=model,
    system_prompt="You are a customer support agent. Use memory to provide personalized service.",
    tools=[memory_tool, ...other_tools]
)

Memory Extraction Jobs

Process past conversation transcripts into retrievable long-term memory:

aws bedrock-agentcore start-memory-extraction-job \
  --memory-id memory-abc123 \
  --source-session-ids '["session-1", "session-2", "session-3"]'

Memory Strategies

Memory Quotas

---

Gateway

Creating a Gateway

aws bedrock-agentcore create-gateway \
  --gateway-name my-tools-gateway \
  --protocol-type MCP

Adding a Lambda Target

aws bedrock-agentcore create-gateway-target \
  --gateway-id gw-abc123 \
  --name order-lookup \
  --description "Look up customer orders by order ID or customer email" \
  --target-configuration '{
    "lambdaTarget": {
      "functionArn": "arn:aws:lambda:us-east-1:123456789:function:order-lookup",
      "toolSchema": {
        "inputSchema": {
          "type": "object",
          "properties": {
            "orderId": {"type": "string", "description": "The order ID to look up"},
            "customerEmail": {"type": "string", "description": "Customer email for order search"}
          }
        }
      }
    }
  }'

Adding an API Target

aws bedrock-agentcore create-gateway-target \
  --gateway-id gw-abc123 \
  --name crm-api \
  --description "Query the CRM system for customer information" \
  --target-configuration '{
    "apiTarget": {
      "uri": "https://api.example.com/customers",
      "method": "GET",
      "authConfiguration": {
        "oAuth2": {
          "credentialProviderArn": "arn:aws:bedrock-agentcore:us-east-1:123456789:credential-provider/crm-oauth"
        }
      }
    }
  }'

Connecting Existing MCP Servers

Gateway can federate with existing MCP servers, making their tools available to AgentCore agents:

aws bedrock-agentcore create-gateway-target \
  --gateway-id gw-abc123 \
  --name external-mcp \
  --target-configuration '{
    "mcpTarget": {
      "uri": "https://mcp.example.com/sse",
      "transportType": "SSE"
    }
  }'

Syncing Gateway Targets

After adding or modifying targets, sync to update the tool index:

aws bedrock-agentcore sync-gateway-targets \
  --gateway-id gw-abc123

Using Gateway Tools in Agents (Strands)

from strands import Agent
from strands.tools.agentcore import AgentCoreGatewayTool

gateway_tools = AgentCoreGatewayTool(
    gateway_id="gw-abc123",
    region="us-east-1"
)

agent = Agent(
    model=model,
    tools=[gateway_tools]
)

Gateway Quotas

---

Identity

Workload Identities

Each agent runtime can be assigned a workload identity that manages authentication to external services.

OAuth2 Credential Provider

# Create an OAuth2 credential provider for Salesforce
aws bedrock-agentcore create-oauth2-credential-provider \
  --name salesforce-oauth \
  --credential-provider-vendor SALESFORCE \
  --oauth2-provider-config '{
    "authorizationServerUrl": "https://login.salesforce.com/services/oauth2/token",
    "clientId": "your-client-id",
    "clientSecretArn": "arn:aws:secretsmanager:us-east-1:123456789:secret:sf-client-secret",
    "scopes": ["api", "refresh_token"]
  }'

API Key Credential Provider

# Create an API key provider for a third-party service
aws bedrock-agentcore create-api-key-credential-provider \
  --name weather-api \
  --api-key-secret-arn "arn:aws:secretsmanager:us-east-1:123456789:secret:weather-api-key"

Token Vault

For services requiring managed token storage and rotation:

aws bedrock-agentcore create-token-vault \
  --token-vault-name production-tokens

Identity Best Practices

• One credential provider per external service — do not share credentials across services
• Use OAuth2 over API keys when the service supports it — tokens can be scoped and rotated
• Store secrets in Secrets Manager — credential providers reference ARNs, never inline secrets
• Use custom claims for enhanced authorization context in resource-based policies

---

Policy

Creating a Policy Engine

aws bedrock-agentcore create-policy-engine \
  --policy-engine-name production-policies \
  --gateway-id gw-abc123

Writing Cedar Policies

Natural Language Authoring

AgentCore converts natural language to Cedar:

aws bedrock-agentcore start-policy-generation \
  --policy-engine-id pe-abc123 \
  --description "Allow refunds under $1000 for customer support agents.
                 Block all delete operations.
                 Only allow engineering team to access the deployment tool."

Direct Cedar Policies

// Limit refund amounts
forbid (
  principal,
  action == Action::"invoke",
  resource == Tool::"process-refund"
) when {
  resource.input.refundAmount > 1000
};

// Restrict tool access by role
permit (
  principal,
  action == Action::"invoke",
  resource == Tool::"deploy-service"
) when {
  principal.department == "engineering"
};

// Block dangerous operations entirely
forbid (
  principal,
  action == Action::"invoke",
  resource == Tool::"delete-customer-data"
);

// Time-based access control
permit (
  principal,
  action == Action::"invoke",
  resource == Tool::"trading-api"
) when {
  context.currentHour >= 9 && context.currentHour <= 16
};

Attaching Policies

aws bedrock-agentcore create-policy \
  --policy-engine-id pe-abc123 \
  --policy-name refund-limits \
  --policy-document file://policies/refund-limits.cedar

Policy Monitoring

Policy decisions are logged automatically. Query them for audit:

# Check recent policy denials
aws logs filter-log-events \
  --log-group-name /aws/bedrock-agentcore/policy \
  --filter-pattern "DENY"

Policy vs Other Guardrail Mechanisms

Use all four layers together for defense in depth.

AgentCore Observability and Evaluations Reference

Observability

AgentCore Observability provides OpenTelemetry-compatible tracing, metrics, and logging for agent workflows. Traces flow to CloudWatch and X-Ray.

Automatic Instrumentation

Agents deployed on AgentCore Runtime with the AWS OpenTelemetry Distro are automatically instrumented. Traces capture:

• Agent invocation start/end
• Model inference calls (model ID, latency, token usage)
• Tool calls (tool name, parameters, duration, success/failure)
• Memory operations (read/write)
• Session lifecycle events

Manual Instrumentation (Custom Spans)

from opentelemetry import trace

tracer = trace.get_tracer("my-agent")

@tracer.start_as_current_span("custom-business-logic")
def process_order(order_id: str):
    span = trace.get_current_span()
    span.set_attribute("order.id", order_id)
    span.set_attribute("order.type", "refund")

    # Your business logic here
    result = lookup_order(order_id)

    span.set_attribute("order.found", result is not None)
    return result

CloudWatch Metrics

Critical Metrics — Alarm on These

Important Metrics — Review Weekly

CloudWatch Dashboard Template

# Create a comprehensive AgentCore monitoring dashboard
aws cloudwatch put-dashboard \
  --dashboard-name AgentCore-Production \
  --dashboard-body '{
    "widgets": [
      {
        "type": "metric",
        "properties": {
          "title": "Invocations & Errors",
          "metrics": [
            ["AWS/BedrockAgentCore", "InvocationCount", "AgentId", "my-agent"],
            ["AWS/BedrockAgentCore", "InvocationErrors", "AgentId", "my-agent"]
          ],
          "period": 300,
          "stat": "Sum"
        }
      },
      {
        "type": "metric",
        "properties": {
          "title": "Latency (p50/p99)",
          "metrics": [
            ["AWS/BedrockAgentCore", "InvocationLatency", "AgentId", "my-agent", {"stat": "p50"}],
            ["AWS/BedrockAgentCore", "InvocationLatency", "AgentId", "my-agent", {"stat": "p99"}]
          ],
          "period": 300
        }
      },
      {
        "type": "metric",
        "properties": {
          "title": "Active Sessions",
          "metrics": [
            ["AWS/BedrockAgentCore", "SessionCount", "AgentId", "my-agent"]
          ],
          "period": 60,
          "stat": "Maximum"
        }
      }
    ]
  }'

X-Ray Tracing

AgentCore traces integrate with X-Ray for distributed tracing across agent → tool → downstream service calls.

# Query traces for a specific agent
aws xray get-trace-summaries \
  --start-time $(date -v-1H +%s) \
  --end-time $(date +%s) \
  --filter-expression 'service("bedrock-agentcore") AND annotation.agent_id = "my-agent"'

Langfuse Integration (LLM-Specific Analytics)

For deeper LLM-level observability beyond infrastructure metrics, layer Langfuse on top of CloudWatch:

from langfuse import Langfuse
from langfuse.decorators import observe, langfuse_context

langfuse = Langfuse(
    public_key="pk-...",    # Store in Secrets Manager
    secret_key="sk-...",    # Store in Secrets Manager
    host="https://your-langfuse-instance.com"
)

@observe(as_type="generation")
def invoke_model(prompt, model_id):
    """Model invocation with Langfuse tracing."""
    response = bedrock_runtime.invoke_model(
        modelId=model_id,
        body=json.dumps({"messages": [{"role": "user", "content": prompt}]})
    )
    result = json.loads(response['body'].read())

    langfuse_context.update_current_observation(
        model=model_id,
        usage={
            "input_tokens": result['usage']['input_tokens'],
            "output_tokens": result['usage']['output_tokens']
        }
    )
    return result

@observe()
def run_agent(user_input):
    """Full agent execution with nested tracing."""
    classification = invoke_model(f"Classify: {user_input}", "amazon.nova-micro-v1:0")
    response = invoke_model(f"Respond: {user_input}", "anthropic.claude-sonnet-4-20250514")
    return response

Observability Stack Recommendation

---

Evaluations

Built-In Evaluators (13 Available)

AgentCore provides 13 built-in evaluators covering common quality dimensions:

On-Demand Evaluation

# Run an on-demand evaluation against test data
aws bedrock-agentcore create-on-demand-evaluation \
  --evaluation-name weekly-quality-check \
  --evaluator-ids '["answer-relevancy", "faithfulness", "hallucination"]' \
  --test-data-source '{
    "s3Uri": "s3://my-evals-bucket/test-cases.jsonl"
  }'

Online Evaluation (Continuous Monitoring)

# Configure continuous evaluation on sampled live traffic
aws bedrock-agentcore create-online-evaluation-config \
  --config-name production-monitoring \
  --agent-runtime-id $RUNTIME_ID \
  --evaluator-ids '["answer-relevancy", "faithfulness", "tool-selection"]' \
  --sampling-rate 0.1  # Evaluate 10% of live sessions

DeepEval Integration (CI/CD Quality Gate)

For more control and CI/CD integration, use DeepEval alongside AgentCore evaluations:

# tests/agent_evals.py
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    AnswerRelevancyMetric,
    FaithfulnessMetric,
    HallucinationMetric,
    GEval
)

# Answer relevancy — does the agent actually answer the question?
def test_answer_relevancy():
    test_case = LLMTestCase(
        input="What is the refund policy for enterprise customers?",
        actual_output=agent_response,
        retrieval_context=["Enterprise customers can request refunds within 30 days..."]
    )
    metric = AnswerRelevancyMetric(threshold=0.7)
    assert_test(test_case, [metric])

# Faithfulness — is the agent grounded in retrieved context?
def test_faithfulness():
    test_case = LLMTestCase(
        input="What are the SLA terms?",
        actual_output=agent_response,
        retrieval_context=retrieved_docs
    )
    metric = FaithfulnessMetric(threshold=0.8)
    assert_test(test_case, [metric])

# Custom eval — agent-specific quality criteria
def test_tool_use_correctness():
    correctness = GEval(
        name="Tool Use Correctness",
        criteria="The agent selected the appropriate tool and passed correct parameters.",
        evaluation_params=["input", "actual_output"],
        threshold=0.7
    )
    test_case = LLMTestCase(
        input="Look up order #12345",
        actual_output=agent_response
    )
    assert_test(test_case, [correctness])

Running Evals in CI/CD

# In your GitHub Actions workflow
- name: Run agent evaluations
  run: |
    pip install deepeval
    deepeval test run tests/agent_evals.py --report

- name: Run AgentCore built-in evals
  run: |
    aws bedrock-agentcore create-on-demand-evaluation \
      --evaluation-name "ci-$GITHUB_SHA" \
      --evaluator-ids '["answer-relevancy", "faithfulness", "tool-selection"]' \
      --test-data-source '{"s3Uri": "s3://evals/test-cases.jsonl"}'

Eval Strategy by Phase

Building Your Eval Dataset

1. Start with 20-30 representative queries from real users or domain experts 2. Include edge cases: ambiguous queries, out-of-scope requests, adversarial inputs 3. Version your eval dataset alongside your agent code (in Git) 4. Expand as you discover failure modes in production — every production incident should add at least one eval case 5. Separate eval tiers: fast smoke tests (5 cases, every commit) vs full regression (50+ cases, nightly)

Evaluation Quotas

---

Production Monitoring Playbook

Daily Checks

1. Review CloudWatch dashboard for invocation count, error rate, latency trends 2. Check for any Policy DENY spikes (may indicate agent behavior drift) 3. Review Langfuse cost-per-conversation trends

Weekly Checks

1. Review online evaluation scores — any degradation? 2. Audit token usage trends — any unexpected growth? 3. Check session TTL utilization — are sessions timing out prematurely? 4. Review tool call error rates by tool — any upstream service degradation?

On Model Update

1. Run full DeepEval regression suite before switching models 2. Deploy new model version behind canary alias (10% traffic) 3. Monitor online eval scores for canary vs production for 24-48 hours 4. Promote or rollback based on eval scores

Incident Response

1. Check X-Ray traces for the failing session 2. Review Policy decisions — was a tool call incorrectly denied/allowed? 3. Check CloudWatch Logs for agent-level errors 4. Review Langfuse trace for the specific conversation (token usage, tool calls, reasoning steps) 5. Add a new eval case for the failure mode

AgentCore Runtime Deployment Reference

AgentCore CLI (Preferred)

The AgentCore CLI is the fastest way to create, develop, and deploy agents. It handles container builds, ECR pushes, and runtime configuration automatically.

pip install agentcore-cli

# Scaffold a new agent
agentcore init my-agent --framework strands

# Run locally with hot-reload
cd my-agent && agentcore dev

# Deploy to AgentCore Runtime
agentcore deploy --region us-east-1

# Test the deployed agent
agentcore invoke --agent-name my-agent --input "Hello"

# Manage aliases
agentcore alias create --agent-name my-agent --alias-name production --version 1

For full-stack deployments with auth and frontend, use the Starter Toolkit (CDK-based FAST template) instead.

---

Manual Container Setup

Use this approach when you need full control over the build process or are integrating into existing CI/CD infrastructure.

Minimal Dockerfile

FROM python:3.12-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .
EXPOSE 8080
CMD ["python", "agent.py"]

Requirements

boto3>=1.35.0
bedrock-agentcore-runtime>=0.1.0
strands-agents>=0.1.0  # or your framework of choice

AgentCore SDK Decorators (Strands Example)

# agent.py — AgentCore Runtime compatible agent
from bedrock_agentcore_runtime import BedrockAgentCoreApp
from strands import Agent
from strands.models import BedrockModel

app = BedrockAgentCoreApp()

model = BedrockModel(
    model_id="anthropic.claude-sonnet-4-20250514",
    region_name="us-east-1"
)

@app.handler
def handle_request(session_id: str, input_text: str):
    agent = Agent(
        model=model,
        system_prompt="You are a helpful assistant.",
        tools=[...]
    )
    return agent(input_text)

if __name__ == "__main__":
    app.run(port=8080)

The BedrockAgentCoreApp wrapper creates the HTTP server with required health check and invocation endpoints, handles authentication, and integrates with AgentCore's session management.

Starter Toolkit (FAST Template)

For full-stack deployments with Cognito auth, React frontend, and all AgentCore services:

git clone https://github.com/aws/bedrock-agentcore-starter-toolkit.git
cd bedrock-agentcore-starter-toolkit
pip install -r requirements.txt
cdk deploy --all

The FAST template deploys: Runtime + Gateway + Memory + Code Interpreter + Observability + Cognito + CloudFront frontend. See the main SKILL.md for the full architecture diagram.

CI/CD with GitHub Actions

# .github/workflows/deploy-agent.yml
name: Deploy Agent to AgentCore

on:
  push:
    branches: [main]
    paths: ['agents/**']

permissions:
  id-token: write
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::role/agentcore-deploy
          aws-region: us-east-1

      - name: Login to Amazon ECR
        uses: aws-actions/amazon-ecr-login@v2

      - name: Build and push container
        run: |
          docker build -t $ECR_REPO:$GITHUB_SHA .
          docker push $ECR_REPO:$GITHUB_SHA

      - name: Deploy to AgentCore Runtime
        run: |
          aws bedrock-agentcore create-agent-runtime \
            --agent-runtime-name my-agent \
            --agent-runtime-artifact '{"containerImage": {"uri": "'$ECR_REPO:$GITHUB_SHA'"}}'

      - name: Run agent evaluations
        run: |
          deepeval test run tests/agent_evals.py --report

      - name: Update alias to new version
        run: |
          aws bedrock-agentcore update-agent-runtime-endpoint \
            --agent-runtime-endpoint-name production \
            --agent-runtime-id $RUNTIME_ID

Alias Management

Aliases decouple consumers from specific agent versions. Use them for:

Traffic Splitting for Canary Deployments

# Route 90% to v1, 10% to v2
aws bedrock-agentcore update-agent-runtime-endpoint \
  --agent-runtime-endpoint-name production \
  --routing-configuration '[
    {"agentRuntimeVersion": "1", "weight": 90},
    {"agentRuntimeVersion": "2", "weight": 10}
  ]'

Rollback Pattern

# Immediate rollback: point alias back to previous version
aws bedrock-agentcore update-agent-runtime-endpoint \
  --agent-runtime-endpoint-name production \
  --agent-runtime-version 1

VPC Configuration

Enable VPC connectivity when agents need to access:

• Private databases (RDS, DynamoDB via VPC endpoint)
• Internal APIs behind an ALB
• On-premises resources via VPN/Direct Connect

aws bedrock-agentcore update-agent-runtime \
  --agent-runtime-id $RUNTIME_ID \
  --network-configuration '{
    "networkMode": "VPC",
    "vpcConfig": {
      "subnetIds": ["subnet-abc123", "subnet-def456"],
      "securityGroupIds": ["sg-xyz789"]
    }
  }'

VPC Security Group Rules

• Outbound: Allow HTTPS (443) to Bedrock endpoints, your APIs, and any external services
• Inbound: Not required — AgentCore Runtime initiates all connections
• Place in private subnets with NAT Gateway for internet access (model API calls)

Scaling Patterns

Real-Time Conversational Agents

• CPU: 1 vCPU, Memory: 2 GiB
• Session TTL: 300-600s
• Expect sub-second response initiation with streaming

Long-Running Async Agents (Research, Data Processing)

• CPU: 2-4 vCPU, Memory: 4-8 GiB
• Session TTL: up to 28,800s (8 hours)
• Use async invocation API for fire-and-forget patterns

High-Concurrency Agents

• AgentCore auto-scales based on concurrent sessions
• Default quota: 1,000 active session workloads per account (us-east-1), 500 in other regions
• Request quota increase for high-traffic agents before launch

Resource Quotas (Key Limits)

Always verify current limits via awsknowledge MCP tools (mcp__plugin_aws-dev-toolkit_awsknowledge__aws___search_documentation, mcp__plugin_aws-dev-toolkit_awsknowledge__aws___read_documentation, mcp__plugin_aws-dev-toolkit_awsknowledge__aws___recommend) — quotas are updated frequently.