Aws Observability

AWS Observability

Overview

Domain expertise for AWS observability across metrics, logs, and traces. Covers CloudWatch platform capabilities (alarms, dashboards, Log Insights, custom metrics, EMF), X-Ray trace analysis, CloudTrail operational auditing, and ADOT collector configuration.

Works best with the AWS MCP server — enables running CLI commands, querying CloudWatch, and validating configurations directly. All guidance also works with standard AWS CLI access.

Note: Reference files contain specific runtime versions, quota values, and feature matrices that may change. When precision matters (e.g., deploying to production, choosing a runtime, or checking a quota), confirm values against current AWS documentation rather than relying solely on the values in these files.

Routing

Files

CloudWatch Alarms

Configure and manage CloudWatch alarms including metric, composite, and anomaly detection types with evaluation mechanics and recommended defaults.

Contents

• Alarm types
• Missing data treatment
• Evaluation mechanics
• Composite alarms
• Anomaly detection
• Recommended defaults
• Common mistakes
• CDK patterns

---

Alarm types

Metric Alarm

Watches a single metric or metric math expression.

• States: OK, ALARM, INSUFFICIENT_DATA
• Actions: SNS, EC2 (stop/terminate/reboot/recover), Auto Scaling, Lambda, SSM OpsItems, SSM Incident Manager, CloudWatch Investigations
• M-of-N evaluation: DatapointsToAlarm (M) out of EvaluationPeriods (N)
• Rate limit: PutMetricAlarm = 3 TPS (adjustable)

Composite Alarm

Combines states of other alarms with Boolean logic.

• Rule operators: AND, OR, NOT, AT_LEAST(M, STATE, (alarms...))
• AT_LEAST supports percentages: AT_LEAST(50%, ALARM, (a1, a2, a3))
• Actions: SNS, Lambda, SSM — cannot perform EC2 or Auto Scaling actions
• Limits: max 100 underlying alarms per composite, 150 composites per underlying, 500 rule elements
• Composite and all underlying alarms must be in the same account and Region
• Action suppression: ActionsSuppressor alarm can suppress composite alarm actions during known events (deployments, maintenance)

PromQL Alarm (OpenTelemetry metrics)

Monitors OTel metrics using PromQL instant queries with duration-based pending/recovery periods. Use for metrics sent via OTLP (150 labels, 30-day retention).

---

Missing data treatment

Four options — the most misunderstood CloudWatch feature.

Note: The CloudWatch console defaults DynamoDB alarms to ignore instead of the usual missing. The API stores whatever you specify.

Premature alarm transitions

With treatMissingData=missing, the pattern M, M, B, M, M can trigger ALARM even with only 1 breaching datapoint. CloudWatch goes to ALARM when the oldest available breaching datapoint is at least as old as datapointsToAlarm and all more recent points are breaching or missing.

Fix: For non-sparse metrics, explicitly set notBreaching or breaching — don't rely on the default.

---

Evaluation mechanics

Three core settings

1. Period — seconds per data point aggregation (valid: 10, 20, 30, or any multiple of 60) 2. Evaluation Periods (N) — number of most recent periods to evaluate 3. Datapoints to Alarm (M) — how many of N must breach

Evaluation frequency

• Period ≥ 1 min → evaluated every minute
• Period = 10s/20s/30s → evaluated every 10 seconds
• If EvaluationPeriods × Period > 1 day → evaluated once per hour

Evaluation Range

CloudWatch fetches more data points than the configured Evaluation Periods — the actual lookback window is wider than expected.

Example: Alarm with 1-day period, 1 evaluation period, treatMissingData=breaching:

• You expect it to fire after 1 day of no data
• CloudWatch actually looks back ~3 days before firing
• Dead man switch alarms fire later than expected due to hourly evaluation

Evaluation period quotas

• Period ≥ 1 hour → max evaluation window: 7 days
• Period < 1 hour → max evaluation window: 1 day

---

Composite alarms

When to use

• Reduce alert fatigue: only page when BOTH high CPU AND high error rate
• Service-level health: aggregate per-resource alarms into one service alarm
• Suppress during deployments: use ActionsSuppressor to mute during known events

Rule expression syntax

ALARM("error-rate-alarm") AND ALARM("latency-alarm")
ALARM("error-rate-alarm") OR ALARM("throttle-alarm")
NOT ALARM("maintenance-window")
AT_LEAST(2, ALARM, (a1, a2, a3))
AT_LEAST(50%, ALARM, (a1, a2, a3, a4))

Limitations

• Cannot perform EC2 actions (stop, terminate, reboot, recover)
• Cannot perform Auto Scaling actions
• Composite and all underlying alarms must be in the same account and Region (underlying alarms must be same account + Region; monitoring accounts via OAM can watch source account metrics)
• Cross-account observability monitoring account CAN watch source account alarms

---

Anomaly detection

• Uses ANOMALY_DETECTION_BAND function as threshold
• Band width = anomaly detection threshold value (configurable; higher value = thicker band of expected values)
• Trains on up to 2 weeks of metric data (works with less, accuracy improves over time)
• Cost: Higher than a regular alarm — see CloudWatch pricing for current anomaly detection alarm rates
• Rate limit: 1,000 ANOMALY_DETECTION_BAND usages in GetMetricData per second
• Use when: baselines are unknown, workloads are seasonal/variable

---

Recommended defaults

WARNING: Never use Average for duration/latency alarms. Average hides tail latency — use p99 or p90. A function averaging 100ms but with p99 at 5s has a serious problem that Average won't catch.

---

Common mistakes

1. M=N=1 with 1-minute periods — Too sensitive. The most recent datapoint may not have full information. Use "1 out of 2" or "1 out of 3" minimum.

2. Relying on default `missing` treatment — Explicitly configure for your metric type. Error metrics should use notBreaching. Health checks should use breaching.

3. Not understanding Evaluation Range — Alarms look back further than configured. Dead man switches with multi-day periods are evaluated once per hour, causing significant delay.

4. Metric math alarms for EC2 actions — Alarms based on metric math expressions cannot perform EC2 actions (stop, terminate, reboot, recover). Use a simple metric alarm instead.

5. High-resolution alarms without need — 10-second evaluation costs more. Each metric in a math expression is billed separately.

6. Using Average statistic for duration/latency alarms — Average hides tail latency. A function averaging 100ms with p99 at 5s has a serious problem Average won't catch. Always use p99 or p90 via --extended-statistic p99.

7. Ignoring DynamoDB's default override — DynamoDB alarms default to ignore for missing data, not the global missing.

8. Alarms on INSUFFICIENT_DATA state — Alarms invoke actions only on state changes, except Auto Scaling actions which continue invoking while in the new state.

---

CDK patterns

Error rate alarm (production pattern)

Note: Alarm on error rate (percentage via math expression), not raw error count. Raw counts trigger on a single error even during 10,000 successful invocations.

For CLI:

aws cloudwatch put-metric-alarm --alarm-name MyFunc-ErrorRate \
  --metrics '[
    {"Id":"errors","MetricStat":{"Metric":{"Namespace":"AWS/Lambda","MetricName":"Errors","Dimensions":[{"Name":"FunctionName","Value":"MyFunc"}]},"Period":60,"Stat":"Sum"},"ReturnData":false},
    {"Id":"invocations","MetricStat":{"Metric":{"Namespace":"AWS/Lambda","MetricName":"Invocations","Dimensions":[{"Name":"FunctionName","Value":"MyFunc"}]},"Period":60,"Stat":"Sum"},"ReturnData":false},
    {"Id":"error_rate","Expression":"IF(invocations > 0, errors * 100 / invocations, 0)","Label":"Error Rate %"}
  ]' \
  --threshold 5 --comparison-operator GreaterThanThreshold \
  --evaluation-periods 3 --datapoints-to-alarm 2 \
  --treat-missing-data notBreaching

For CDK:

import { Alarm, ComparisonOperator, MathExpression, TreatMissingData } from 'aws-cdk-lib/aws-cloudwatch';
import { Duration } from 'aws-cdk-lib';

const errorRateAlarm = new Alarm(this, 'ErrorRateAlarm', {
  metric: new MathExpression({
    expression: 'IF(invocations > 0, errors * 100 / invocations, 0)',
    usingMetrics: {
      errors: fn.metricErrors({ period: Duration.minutes(1) }),
      invocations: fn.metricInvocations({ period: Duration.minutes(1) }),
    },
  }),
  threshold: 5,
  evaluationPeriods: 3,
  datapointsToAlarm: 2,
  comparisonOperator: ComparisonOperator.GREATER_THAN_THRESHOLD,
  treatMissingData: TreatMissingData.NOT_BREACHING,
});

Duration/latency alarm (use p99, never Average)

const durationAlarm = new Alarm(this, 'DurationP99Alarm', {
  metric: fn.metricDuration({ statistic: 'p99', period: Duration.minutes(1) }),
  threshold: 3000, // 3 seconds
  evaluationPeriods: 3,
  datapointsToAlarm: 2,
  comparisonOperator: ComparisonOperator.GREATER_THAN_THRESHOLD,
  treatMissingData: TreatMissingData.NOT_BREACHING,
});

For CLI:

aws cloudwatch put-metric-alarm --alarm-name MyFunc-Duration-P99 \
  --namespace AWS/Lambda --metric-name Duration \
  --dimensions Name=FunctionName,Value=MyFunc \
  --extended-statistic p99 --period 60 \
  --evaluation-periods 3 --datapoints-to-alarm 2 \
  --threshold 3000 --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching

Composite alarm

import { CompositeAlarm, AlarmRule, AlarmState } from 'aws-cdk-lib/aws-cloudwatch';

const serviceHealthAlarm = new CompositeAlarm(this, 'ServiceHealth', {
  alarmRule: AlarmRule.anyOf(
    AlarmRule.fromAlarm(errorRateAlarm, AlarmState.ALARM),
    AlarmRule.fromAlarm(latencyAlarm, AlarmState.ALARM),
    AlarmRule.fromAlarm(throttleAlarm, AlarmState.ALARM),
  ),
});

Anomaly detection alarm (CloudFormation)

Resources:
  AnomalyDetector:
    Type: AWS::CloudWatch::AnomalyDetector
    Properties:
      MetricName: Invocations
      Namespace: AWS/Lambda
      Stat: Sum
  AnomalyAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      ComparisonOperator: LessThanLowerOrGreaterThanUpperThreshold
      # Anomaly detection band already models expected variability, so EvaluationPeriods: 1 is acceptable
      EvaluationPeriods: 1
      Metrics:
        - Expression: ANOMALY_DETECTION_BAND(m1, 2)
          Id: ad1
        - Id: m1
          MetricStat:
            Metric:
              MetricName: Invocations
              Namespace: AWS/Lambda
            Period: 86400
            Stat: Sum
      ThresholdMetricId: ad1
      TreatMissingData: breaching

CloudTrail Operational Auditing

Using CloudTrail for operational debugging: who changed what, when. Not for security threat detection.

Contents

• Event types
• Event history
• Common operational queries
• Querying CloudTrail logs
• CloudTrail → CloudWatch integration

---

Event types

---

Event history

• 90 days of management events retained by default, no trail required
• Searchable in console by event name, resource type, user name, time range
• 200,000 event limit when downloading
• Single account, single Region only
• Cannot view data events, Insights events, or network activity events

Common lookups

# Who deleted an S3 bucket?
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteBucket \
  --start-time 2026-04-20T00:00:00Z

# Who modified a security group?
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=AuthorizeSecurityGroupIngress

# Who stopped an EC2 instance?
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=i-1234567890abcdef0

---

Common operational queries

"Who deleted my resource?"

1. Check Event History (90 days) for Delete* events 2. Filter by resource name or resource type 3. Look at userIdentity.arn for the actor and sourceIPAddress for origin

"Who changed this configuration?"

1. Search for Update*, Modify*, Put* events on the resource 2. Compare requestParameters across events to see what changed

"What happened during the incident?"

1. Filter by time range of the incident 2. Look for errorCode fields (AccessDenied, ThrottlingException) 3. Correlate with CloudWatch metrics/logs for the same time window

"Who accessed my data?" (requires data events)

Data events must be explicitly enabled on the trail:

aws cloudtrail put-event-selectors --trail-name my-trail \
  --advanced-event-selectors '[{
    "Name": "S3DataEvents",
    "FieldSelectors": [
      {"Field": "eventCategory", "Equals": ["Data"]},
      {"Field": "resources.type", "Equals": ["AWS::S3::Object"]}
    ]
  }]'

---

Querying CloudTrail logs

Recommended: Trail → S3 → Athena

For new setups, deliver CloudTrail logs to S3 and query with Amazon Athena:

SELECT eventTime, userIdentity.arn, sourceIPAddress, eventName
FROM cloudtrail_logs
WHERE eventName = 'DeleteBucket'
  AND eventTime > '2026-04-20'
ORDER BY eventTime DESC
LIMIT 100;

This is the long-term supported approach — works with standard SQL, scales to any volume, and integrates with existing S3-based analytics.

---

CloudTrail → CloudWatch integration

Alert on specific API calls

CloudTrail → Trail → CloudWatch Logs → Metric Filter → CloudWatch Alarm → SNS

1. Configure trail to deliver events to a CloudWatch Logs log group 2. Create metric filter for the event pattern (e.g., { $.eventName = "DeleteBucket" }) 3. Create alarm on the metric filter 4. Configure SNS notification

Event selectors

• Basic: simple include/exclude for management and data events
• Advanced: fine-grained filtering by event source, resource type, resource ARN
• Exclude high-volume management event sources on trails: AWS KMS, RDS Data API
• Max 250 data resources across all basic event selectors per trail (does not apply to advanced event selectors)

CloudWatch Dashboards

Widget types, cross-account/region patterns, dynamic labels, and recommended defaults.

Contents

• Widget types
• Cross-account and cross-region
• Dynamic labels
• Dashboard variables
• Sharing constraints
• Recommended defaults
• CDK patterns

---

Widget types

---

Cross-account and cross-region

Prerequisites

• CloudWatch Observability Access Manager (OAM) configured
• Monitoring account + source account links established
• IAM roles for cross-account access

Dashboard body JSON

Each widget supports accountId and region parameters:

{
  "type": "metric",
  "properties": {
    "metrics": [["AWS/Lambda", "Errors", "FunctionName", "my-fn"]],
    "region": "us-west-2",
    "accountId": "123456789012"
  }
}

Limitations

• Search expressions operate within the widget's configured region (set region per widget for cross-region search)
• Cross-account composite alarms are not supported. However, with OAM, metric alarms in a monitoring account can watch metrics from source accounts.
• Cross-account alarms do NOT support ANOMALY_DETECTION_BAND, INSIGHT_RULE, or SERVICE_QUOTA functions

---

Dynamic labels

Use dynamic values in metric widget labels (common tokens shown; AWS supports 28+ tokens including time-based variants like ${MAX_TIME}, ${LAST_TIME_RELATIVE}, and property tokens like ${PROP('MetricName')}, ${PROP('Region')}):

Example: "label": "${PROP('FunctionName')} p99=${MAX}ms"

Max 6 dynamic values per label. ${LABEL} can only be used once per label.

---

Dashboard variables

Variables add dropdown/radio/text inputs that dynamically filter all widgets on a dashboard. Up to 25 variables per dashboard.

Two types:

• Property variables: Populate from CloudWatch dimension values (e.g., all FunctionName values in AWS/Lambda)
• Pattern variables: Free-text input matched against metric patterns

Variables are a top-level variables array in the dashboard body JSON, peer to widgets. They eliminate the need for per-function or per-instance dashboards.

Shared dashboard viewers cannot change variable values — the dashboard renders with the default value only.

---

Sharing constraints

• Shared users cannot see composite alarm widgets, Logs Insights widgets, or custom widgets unless you add the corresponding permissions (DescribeAlarms, CloudWatch Logs query permissions, Lambda invoke) to the sharing IAM policy
• cloudwatch:GetMetricData and ec2:DescribeTags cannot be scoped — shared users can query all metrics and EC2 tags in the account
• Cognito resources are created in us-east-1 regardless of dashboard region

---

Best-practice defaults

Dashboard structure pattern

1. Row 1: Markdown header + alarm status widgets (24-wide) 2. Row 2: Key business metrics (Number widgets, 6-wide each) 3. Row 3: Request/error rate graphs (Line widgets, 12-wide) 4. Row 4: Latency percentiles (Line widget, 24-wide) 5. Row 5: Log Insights query results (Logs table, 24-wide)

Sharing

• Share publicly or with specific email addresses via Amazon Cognito
• Shared dashboards accessible via URL without AWS console login
• Check the CloudWatch pricing page for current dashboard costs

API limits

• PutDashboard, GetDashboard, ListDashboards, DeleteDashboards: all 10 TPS (adjustable)

---

CDK patterns

Dashboard with alarm and graph widgets

import { Dashboard, AlarmWidget, GraphWidget, TextWidget, PeriodOverride } from 'aws-cdk-lib/aws-cloudwatch';

const dashboard = new Dashboard(this, 'ServiceDashboard', {
  dashboardName: `${serviceName}-${stage}`,
  start: '-PT8H',
  periodOverride: PeriodOverride.INHERIT,
});

dashboard.addWidgets(
  new TextWidget({ width: 24, height: 1, markdown: '# Service Health' }),
  new AlarmWidget({ width: 12, height: 6, title: 'Error Rate', alarm: errorRateAlarm }),
  new AlarmWidget({ width: 12, height: 6, title: 'Latency P99', alarm: latencyAlarm }),
  new GraphWidget({
    width: 24, height: 6,
    title: 'Invocations & Errors',
    left: [fn.metricInvocations({ period: Duration.minutes(1) })],
    right: [fn.metricErrors({ period: Duration.minutes(1) })],
  }),
);

Automatic dashboards

Pre-built per-service dashboards are available by default (EC2, Lambda, S3, etc.). No setup required. Use these as starting points, then customize.

CloudWatch Logs Insights

Complete query syntax reference, performance tips, and reusable query library.

Contents

• Commands
• Filter syntax
• Parse command
• Stats and aggregation
• Time functions
• Advanced commands
• Known issues
• Reusable query library

---

Commands

Auto-discovered fields: @timestamp, @message, @logStream, @log (account-id:log-group-name), @ingestionTime, @entity. JSON fields auto-flattened with dot notation.

---

Filter syntax

# Comparison: =, !=, <, <=, >, >=
filter statusCode >= 400

# Boolean: and, or, not
filter statusCode >= 400 and statusCode < 500

# Set membership
filter statusCode in [400, 401, 403, 404]

# Substring
filter @message like "ERROR"

# Regex
filter @message like /(?i)error/        # case-insensitive
filter @message =~ /timeout after \d+/  # regex match

# Negation
filter @message not like "DEBUG"

Field index optimization: Only filter field = value and filter field IN [...] use indexes. filter field like does NOT use indexes.

---

Parse command

Glob mode (wildcards)

parse @message "User * performed * on *" as user, action, resource

Regex mode (named groups)

parse @message /User (?<user>\w+) performed (?<action>\w+)/

Chaining for complex logs

# XML parsing
parse @message "<EventData>*</EventData>" as @EventData
| parse @EventData "<Data Name='ObjectName'>*</Data>" as ObjectName

---

Stats and aggregation

# Basic aggregation
stats count(*), sum(duration), avg(duration), min(duration), max(duration)

# Percentiles
stats pct(duration, 50) as p50, pct(duration, 95) as p95, pct(duration, 99) as p99

# Time bucketing
stats count(*) as cnt by bin(5m)

# Group by field
stats count(*) as cnt by statusCode

# Combined
stats avg(duration) as avg_ms, pct(duration, 99) as p99 by serviceName, bin(1h)

---

Time functions

• bin(period) — time bucketing: bin(5m), bin(1h), bin(1d)
• datefloor(ts, period), dateceil(ts, period) — truncate/round
• fromMillis(num), toMillis(ts) — epoch conversion
• now() — time query processing was started, in epoch seconds

bin() caps:

• ms → max 1000, s → max 60, m → max 60, h → max 24
• Use bin(5m) NOT bin(300s) — 300 exceeds the s→60 cap

---

Advanced commands

JOIN

Correlate events across log groups by a shared key:

filter status >= 500
| join type=inner left=api right=infra
    where api.requestId=infra.requestId
    (SOURCE '/aws/infra-logs')

Subqueries

Use nested queries to filter the outer query:

filter requestId in (
    SOURCE '/aws/lambda/database-service'
    | filter errorType = "DatabaseConnectionTimeout"
    | fields requestId
)

Anomaly detection

fields @timestamp, @message
| filter @message like /ERROR/
| pattern @message
| anomaly

Scheduled queries

Recurring queries with results delivered to S3 and EventBridge. Configure via console or API.

---

Known issues

1. Backtick-escape field names with special characters: event-name is interpreted as event minus name. Use ` event-name ` instead.

2. 100 concurrent query limit per account (not adjustable). Partition queries by time range instead of parallelizing beyond this limit.

3. JSON structured logs only ~10% faster than unstructured text search. The real speedup comes from parallelizing across time ranges.

4. Parallelization strategy: Break queries into time-range chunks and run in parallel (14 × 12h instead of 1 × 7d). Reduces 84-minute query to ~6 minutes.

5. `pattern`, `diff`, `unmask`, `anomaly`, and `filterIndex` don't work on Infrequent Access log class.

6. `head` and `tail` are deprecated — use limit instead.

7. StartQuery API: 10 TPS (most regions). GetQueryResults: 10 TPS.

8. Max 50 log groups per query (API-level limit on logGroupNames/logGroupIdentifiers).

9. No nested subqueries or correlated subqueries — only simple subqueries.

10. Subquery inner execution is limited to 30 seconds. The overall query timeout is 60 minutes.

---

Reusable query library

Error analysis

# Recent errors with context
fields @timestamp, @message, @logStream
| filter @message like /ERROR/
| sort @timestamp desc
| limit 100

# Error rate by time bucket
fields @timestamp, @message
| filter @message like /ERROR/
| stats count(*) as errorCount by bin(5m)
| sort errorCount desc

# Top error patterns (ML clustering)
fields @timestamp, @message
| filter @message like /ERROR/
| pattern @message

Lambda-specific

# Cold start analysis
filter @type = "REPORT"
| stats avg(@duration) as avg_ms, max(@duration) as max_ms,
        count(*) as invocations,
        sum(strcontains(@message, "Init Duration")) as coldStarts
  by bin(1h)

# Memory utilization
filter @type = "REPORT"
| stats max(@memorySize / 1000 / 1000) as provisioned_mb,
        max(@maxMemoryUsed / 1000 / 1000) as used_mb,
        avg(@maxMemoryUsed * 100 / @memorySize) as utilization_pct
  by bin(1h)

# Timeout detection
filter @message like /Task timed out/
| fields @timestamp, @requestId, @message
| sort @timestamp desc
| limit 20

API Gateway

# 5xx errors by endpoint
fields @timestamp, httpMethod, resourcePath, status
| filter status >= 500
| stats count(*) as errors by resourcePath, httpMethod
| sort errors desc

# Latency percentiles by endpoint
fields @timestamp, resourcePath, responseLatency
| stats pct(responseLatency, 50) as p50,
        pct(responseLatency, 90) as p90,
        pct(responseLatency, 99) as p99
  by resourcePath
| sort p99 desc

Cross-service correlation

# Multi-log-group error correlation (using SOURCE)
SOURCE logGroups(namePrefix: ['/app-logs', '/api-gateway-logs'])
| fields @timestamp, @message, @log
| filter @message like /ERROR/ or status >= 500
| sort @timestamp desc
| limit 200

CloudWatch Custom Metrics

Publishing, querying, and managing custom metrics — EMF, PutMetricData, metric filters, and retention.

Contents

• EMF vs PutMetricData
• Embedded Metric Format (EMF)
• PutMetricData API
• Metric filters
• Metric retention
• Dimension design
• Metric math
• EMF constraints

---

EMF vs PutMetricData

Default recommendation: Use EMF for Lambda and containerized workloads. Use PutMetricData for batch jobs or when you need synchronous confirmation.

---

Embedded Metric Format (EMF)

JSON structure

{
  "_aws": {
    "Timestamp": 1574109732004,
    "CloudWatchMetrics": [{
      "Namespace": "MyService",
      "Dimensions": [["ServiceName", "Environment"]],
      "Metrics": [
        { "Name": "Latency", "Unit": "Milliseconds", "StorageResolution": 60 },
        { "Name": "RequestCount", "Unit": "Count" }
      ]
    }]
  },
  "ServiceName": "OrderService",
  "Environment": "Production",
  "Latency": 100,
  "RequestCount": 1,
  "RequestId": "abc-123"
}

EMF limits

• Max 100 metrics per MetricDirective
• Max 30 dimensions per DimensionSet (may be empty)
• Dimension value: max 1024 characters, must be string
• Metric value: must be numeric or array of numerics (max 100 values)
• Max log event size: 1 MB
• Namespace: 1–1024 characters, should not start with AWS/
• Timestamp in _aws is required per the EMF spec and JSON schema (milliseconds since epoch). In practice, if omitted, CloudWatch uses the log event's ingestion time — but explicitly setting it is recommended to avoid clock-skew issues.

EMF libraries

For Lambda/containers, use a library that handles EMF serialization (e.g., Lambda Powertools Metrics, aws-embedded-metrics). These libraries manage the _aws metadata block, dimension limits, and metric flushing automatically.

---

PutMetricData API

Limits

• 500 TPS per account per region (adjustable via Service Quotas) — NOT 150 TPS
• Up to 1,000 MetricDatum items per request
• Up to 150 values per MetricDatum (for percentile statistics support)
• Max 30 dimensions per metric
• Metric name: max 255 characters
• Namespace: max 255 characters, should not start with AWS/

StatisticSets (batch optimization)

Instead of publishing individual data points, aggregate into StatisticSets:

{
  "MetricName": "Latency",
  "StatisticValues": {
    "SampleCount": 100,
    "Sum": 5000,
    "Minimum": 10,
    "Maximum": 200
  },
  "Unit": "Milliseconds"
}

Reduces API calls and cost.

---

Metric filters

Extract metrics from log events automatically.

• Max 100 metric filters per log group
• Filter pattern: space-delimited terms or JSON property matching
• PutMetricFilter API: 5 TPS
• Metric filter → CloudWatch metric → alarm pipeline is the standard log-to-alert pattern

Example: count 5xx errors from access logs

{ $.statusCode >= 500 }

Publishes a metric with value 1 for each matching log event.

---

Metric retention

Automatic aggregation cascade

Key insight: You cannot query 1-minute data from 2 months ago. It has been automatically aggregated to 5-minute resolution. High-resolution (1-second) data is only available for 3 hours.

OTel metrics: Only 30 days retention (public preview) — significantly shorter than traditional CloudWatch metrics (15 months).

Metric expiry

• Metrics with no new data for 15 months expire
• Metrics with no data for 2 weeks are not listed by ListMetrics (but still exist)

---

Dimension design

Note: Each unique dimension combination = separate metric = separate cost.

Anti-patterns

• Do not use requestId, userId, sessionId as dimensions — creates millions of metrics
• Do not publish {InstanceId, InstanceType} and expect to query by InstanceId alone — must publish both combinations separately
• Do not use inconsistent units — metrics with different units are separate data streams

Best practices

• Use low-cardinality dimensions: ServiceName, Environment, Operation, StatusCode
• Use the SEARCH function for cross-dimension queries
• Always specify units consistently
• Audit custom metrics regularly — remove unused ones

---

Metric math

Combine metrics using expressions in alarms and dashboards.

Functions

SUM, AVG, MIN, MAX, STDDEV, PERIOD, SEARCH, IF, FILL, ANOMALY_DETECTION_BAND

Error rate pattern

errors * 100 / invocations

SEARCH expression (dynamic metrics)

SEARCH('{AWS/Lambda,FunctionName} MetricName="Errors"', 'Sum', 300)

Automatically includes new functions matching the pattern — useful in dashboards and graphs (SEARCH cannot be used in alarms).

Limits

• Max 10 metrics in a metric math alarm expression
• Use Metrics Insights queries for more (max 10,000 metrics, 500 time series returned)
• Metrics Insights alarm data window: 3 hours only
• Max 500 metrics+expressions per dashboard graph

Metric math in alarms — constraints

• `FILL` can permanently stick an alarm: If a metric is published with slight delay, FILL replaces the missing latest point with the fill value, keeping the alarm in a fixed state. Use M-of-N alarms instead.
• `RATE` on sparse metrics is unpredictable: The evaluation range varies, causing inconsistent rate calculations. Avoid RATE in alarms on metrics that don't publish every period.
• Anomaly detection restrictions (non-exhaustive): Cannot use more than one ANOMALY_DETECTION_BAND per expression, cannot combine with METRICS() or SEARCH, cannot use high-resolution metrics. See CloudWatch metric math docs for full list.

---

EMF constraints

• Flush interval affects alarms: Flush EMF logs to CloudWatch at ≤5 second intervals. Longer intervals cause alarms to evaluate partial or missing data. In Lambda (where flush is automatic), use M-of-N alarms to compensate.
• Monitor EMF parsing failures: AWS/Logs namespace publishes EMFValidationErrors and EMFParsingErrors metrics. Check these if metrics aren't appearing.
• Target values cannot be nested: "A.a" matches { "A.a": 1 }, NOT { "A": { "a": 1 } }. Metric and dimension values must be on the root node.
• Multiple DimensionSets multiply metrics: Dimensions: [["Service"], ["Service", "Operation"]] creates 2 metrics per data point, not 1. Libraries like Powertools do this by default.
• Dimension key max 250 chars (per EMF schema); dimension value max 1024 chars.

CloudWatch Synthetics

Runtime constraints, blueprint compatibility, and common pitfalls for CloudWatch Synthetics canaries.

Contents

• Runtime and blueprint compatibility
• CDK pattern
• VPC canaries
• Common failures
• Limits

---

Runtime and blueprint compatibility

Playwright cannot use 4 of 7 blueprints. Java has no browser — API-only.

Run aws synthetics describe-runtime-versions for the latest runtime versions.

Deprecated runtimes continue running but you cannot update code or config without upgrading first.

---

Key flags

CDK:

const canary = new synthetics.Canary(this, 'ApiCanary', {
  // ... standard props ...
  activeTracing: true,              // X-Ray — adds 2.5-7% to run time
  provisionedResourceCleanup: true, // delete Lambda on canary delete
  artifactsBucketLifecycleRules: [{ expiration: Duration.days(30) }], // prevent S3 accumulation
});

// BREACHING — canary not running IS the problem
canary.metricSuccessPercent().createAlarm(this, 'CanaryAlarm', {
  threshold: 90,
  evaluationPeriods: 3,
  datapointsToAlarm: 2,
  comparisonOperator: ComparisonOperator.LESS_THAN_THRESHOLD,
  treatMissingData: TreatMissingData.BREACHING,
});

maxRetries (via Schedule.RetryConfig) and dryRunAndUpdate are not exposed in the CDK L2 construct — use CfnCanary escape hatch or CLI.

CLI — alarm on canary success rate:

aws cloudwatch put-metric-alarm \
  --alarm-name my-api-canary-success \
  --namespace CloudWatchSynthetics \
  --metric-name SuccessPercent \
  --dimensions Name=CanaryName,Value=my-api-canary \
  --statistic Average --period 300 \
  --evaluation-periods 3 --datapoints-to-alarm 2 \
  --threshold 90 --comparison-operator LessThanThreshold \
  --treat-missing-data breaching

CLI — safe update via dry run:

aws synthetics start-canary-dry-run --name my-api-canary --runtime-version syn-nodejs-puppeteer-15.0
aws synthetics get-canary --name my-api-canary --dry-run-id $DRY_RUN_ID
aws synthetics update-canary --name my-api-canary --dry-run-id $DRY_RUN_ID

Key CDK/CloudFormation constraints:

• ExecutionRoleArn is required — CloudFormation does not auto-create roles (unlike the console)
• Changing Name triggers replacement (delete + create), causing monitoring gaps
• Without provisionedResourceCleanup: true, deleting the stack orphans Lambda functions and layers
• Editing any canary property resets the schedule — next run happens immediately

---

VPC canaries

Canaries in VPCs must run in private subnets (Lambda ENIs don't get public IPs, even in public subnets).

Internet access (required for uploading metrics to CloudWatch and artifacts to S3):

• Option A: NAT Gateway in a public subnet + route from private subnet
• Option B: VPC endpoints — Interface endpoint for monitoring, Gateway endpoint for s3

VPC endpoint policy constraint: The S3 gateway endpoint policy must include s3:ListAllMyBuckets, s3:GetBucketLocation, and s3:PutObject — separate from the IAM role policy.

DNS: Both DNS Resolution and DNS Hostnames must be enabled on the VPC.

Silent failure mode: If the VPC has no internet access and no VPC endpoints, the canary runs but cannot upload metrics or artifacts — it appears as if it never ran.

---

Common failures

---

Limits

Distributed Tracing: X-Ray and ADOT

X-Ray SDK is in maintenance mode. Use ADOT (OpenTelemetry) for all new projects.

Contents

• ADOT vs X-Ray SDK
• Trace structure
• Annotations vs metadata
• Sampling rules
• ADOT collector configuration
• Instrumentation patterns
• Migration constraints
• Common mistakes

---

ADOT vs X-Ray SDK

Migration path: AWS provides migration guides from X-Ray SDK to OpenTelemetry SDK. The CloudWatch agent now also supports sending traces to X-Ray — no separate daemon needed.

---

Trace structure

• Trace — collection of all segments from a single request, identified by trace ID
• Segment — JSON document with a 64 KB documented limit representing work done by a service. Do not exceed this; behavior above 64 KB is undocumented and may change.
• Subsegment — granular detail within a segment (downstream calls, custom code blocks)
• Inferred segment — generated by X-Ray from subsegments for uninstrumented downstream services

Trace ID format

X-Amzn-Trace-Id: Root=1-58406520-a006649127e371903a2de979;Parent=53995c3f42cd8ad8;Sampled=1

Format: 1-{8 hex epoch}-{24 hex unique}. W3C trace IDs are supported (reformatted).

Retention

• Trace data: 30 days (not configurable)
• Service graph: 30 days

---

Annotations vs metadata

Rule of thumb: If you need to search for it → annotation. If you just need to store it → metadata.

WARNING: 50 annotations per trace is a hard limit. Plan your annotation schema carefully.

---

Sampling rules

Default rule

• Reservoir: 1 request per second (shared across all instances)
• Rate: 5% of additional requests
• Conservative default to control costs

Rule evaluation

• Rules evaluated in ascending priority order (1–9999, lower = higher priority)
• Default rule priority = 10000 (always last)
• First matching rule wins

Rule parameters

Parent-based sampling (critical concept)

Sampling decision is made once by the root service. Downstream services honor the upstream decision regardless of their own rules. Custom rules only apply where no sampling decision exists yet.

Adaptive sampling (newer)

• SamplingRateBoost — auto-increases rate during anomalies
• MaxRate — ceiling for boosted rate
• CooldownWindowMinutes — prevents continuous boosts (recommended when SamplingRateBoost is configured)

---

ADOT collector configuration

Architecture

[Receivers] → [Processors] → [Exporters]

CloudWatch + X-Ray pipeline

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 30s
    send_batch_size: 8192

exporters:
  awsxray:
    region: us-east-1
  awsemf:
    namespace: MyApplication
    region: us-east-1

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [awsxray]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [awsemf]

EKS DaemonSet deployment

resources:
  limits:
    memory: 200Mi
  requests:
    cpu: 250m
    memory: 100Mi

Cardinality prevention (three-layer defense)

1. OTel SDK level: Don't emit high-cardinality attributes (ContainerID, CustomerID, RequestID) 2. ADOT Collector level: Use Filter Processor to drop metrics by name/attribute 3. Backend level: Use backend-specific dimension filtering (CloudWatch: dimension_rollup_option + metric_declarations; Prometheus: metric_relabel_configs)

Filter as early as possible in the pipeline to reduce cost and cardinality.

---

Instrumentation patterns

Lambda: enable active tracing (CDK)

import { Tracing } from 'aws-cdk-lib/aws-lambda';

const fn = new lambda.Function(this, 'MyFunction', {
  runtime: lambda.Runtime.NODEJS_20_X,
  handler: 'index.handler',
  code: lambda.Code.fromAsset('lambda'),
  tracing: Tracing.ACTIVE,
});

API Gateway: enable tracing

const api = new apigateway.RestApi(this, 'MyApi', {
  deployOptions: {
    tracingEnabled: true,
  },
});

Or via CLI: aws apigateway update-stage --rest-api-id <id> --stage-name prod --patch-operations op=replace,path=/tracingEnabled,value=true

Trace-log correlation

Inject trace ID into application logs for cross-pillar correlation:

import logging
from opentelemetry import trace

ctx = trace.get_current_span().get_span_context()
trace_id = format(ctx.trace_id, '032x')
logging.info("Processing request", extra={"trace_id": trace_id})

---

Migration constraints (X-Ray SDK → OTel)

Annotations require explicit opt-in

In OTel, all span attributes become X-Ray metadata by default. To make an attribute a searchable X-Ray annotation, add its key to the aws.xray.annotations list:

span.set_attribute("aws.xray.annotations", ["order_id", "customer_tier"])
span.set_attribute("order_id", "12345")

Without this, you lose all annotation-based filtering after migration.

Centralized sampling requires a proxy

The ADOT collector config must include the awsproxy extension (or use the CloudWatch agent as a proxy) for X-Ray centralized sampling rules to work. Without a proxy, the SDK falls back to a default local rule (1 req/sec + 5%):

extensions:
  awsproxy:
    endpoint: 127.0.0.1:2000
service:
  extensions: [awsproxy]

SDK env vars: OTEL_TRACES_SAMPLER=xray and OTEL_TRACES_SAMPLER_ARG=endpoint=http://localhost:2000

Centralized sampling language support: Java, .NET, Python, Node.js (ADOT). Vanilla OTel SDK: Java, .NET, Go.

Mixed propagation during incremental migration

OTel defaults to W3C Trace Context; X-Ray SDK uses X-Ray trace header. During migration, configure both:

OTEL_PROPAGATORS=xray,tracecontext

Without this, traces break at service boundaries between old and new instrumentation.

Port conflict: stop X-Ray daemon before starting ADOT

Both use port 2000. Running both simultaneously causes silent data loss.

Lambda ADOT layer adds cold start latency

ADOT Lambda layers increase memory usage and cold start time. For latency-sensitive functions where you don't need OTel's multi-backend capabilities, X-Ray SDK may still be preferable.

W3C trace ID version requirement

ADOT Collector 0.34.0+ (X-Ray Exporter 0.86.0+) is required to accept W3C-format trace IDs. Older versions silently reject them.

---

Common mistakes

1. Using X-Ray SDK for new projects — Maintenance mode. Use ADOT/OpenTelemetry.

2. Storing searchable data as metadata — Metadata is NOT indexed. Use annotations for data you need to filter by.

3. Exceeding 50 annotations per trace — Hard limit. Plan your annotation schema.

4. Not stripping X-Amzn-Trace-Id from untrusted requests — Users can inject trace IDs or sampling decisions.

5. Default sampling for all services — 1 req/sec + 5% is too conservative for low-traffic services (may miss issues) and too aggressive for high-traffic (unnecessary cost). Tune per service.

6. StepFunctions tracing overrides Lambda — When StepFunction tracing is enabled, downstream Lambda tracing is always enabled regardless of Lambda's own config.

7. Cross-account tracing — Trace IDs propagate naturally across accounts, but unified cross-account viewing requires CloudWatch Observability Access Manager (OAM) setup with monitoring/source account links.

Observability Troubleshooting

Error → cause → fix for CloudWatch, X-Ray, and CloudTrail issues. Start with the 5 most common fixes.

Top 5 Fixes

1. Alarm stuck in INSUFFICIENT_DATA → Check namespace/dimensions match exactly, verify metric is being published, check missing data treatment setting 2. Alarm not triggering → Check Evaluation Range (wider than configured), verify M-of-N settings, check metric delay 3. Missing logs → Check log group exists, verify IAM permissions, check log retention hasn't expired (takes up to 72 hours after expiry) 4. X-Ray traces missing → Check sampling rules (default: 1/sec + 5%), verify tracing is enabled on all services in the path, check IAM permissions 5. High CloudWatch bill → Check log retention (default: never expire), audit GetMetricData callers, check custom metric dimension cardinality

---

Alarm Issues

INSUFFICIENT_DATA state

Alarm not triggering

Alarm flapping (OK → ALARM → OK rapidly)

---

Log Issues

Missing logs

Log Insights query issues

---

Metric Issues

Custom metrics not appearing

High metric costs

---

Tracing Issues

Missing traces

Annotation/metadata issues

---

CloudTrail Issues

Can't find events