Hyperpod Cluster Debugger

HyperPod Cluster Debugger

Operating policy. Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a Suggested command (run this yourself) block and wait for the customer to run it. Destructive order: investigate → reboot → replace (replace destroys root + secondary volumes; not supported on Slurm controller nodes).

Before any state-changing CLI: ask if it's IaC-managed. HyperPod clusters, SGs, EKS access entries, and IAM are usually provisioned via CloudFormation / CDK / Terraform. If yes, the fix belongs in IaC — running the CLI will drift and the next deploy reverts it. Use the CLI only when IaC is unavailable (locked out, predates IaC, mid-review).

scripts/diagnose-cluster.sh is read-only: it collects state via AWS APIs (and SSM for Slurm controller health) and prints each issue as [FAIL] ... → references/<file>.md § <section>.

---

Workflow

1. Collect HyperPod cluster name (not EKS name), region, exact error string. 2. Run scripts/diagnose-cluster.sh (or --validate for pre-create). 3. For every [FAIL] line, Read the referenced section. 4. Present finding, root cause, and the Suggested-command block verbatim. Wait for customer approval. 5. Re-run the diagnostic to confirm.

---

Step 1: Run diagnostics

# Diagnose an existing cluster:
bash scripts/diagnose-cluster.sh --cluster <CLUSTER_NAME_OR_ARN> --region <REGION>

# Pre-flight (no cluster needed) — validates SGs, subnets, IAM, VPC endpoints,
# optionally S3 lifecycle scripts and per-AZ capacity:
bash scripts/diagnose-cluster.sh --validate --region <REGION> \
  --sg-ids <sg-1,sg-2> --subnet-ids <sub-1,sub-2> [--iam-role <role-arn>] \
  [--s3-uri s3://<BUCKET>/path/] [--instance-type ml.p5.48xlarge]

Pass --instance-type when the target instance type is known — enables the per-AZ capacity check (warns if none of the provided subnets are in an AZ that offers that type, which causes insufficient-capacity failures at creation time).

Tags: [PASS] · [FAIL] (counted, has → references/... pointer) · [WARN] · [INFO]. Priorities: P0 blocks operation · P1 degraded · P2 informational.

---

Step 2: Match signal → section

Error messages / events:

---

A: EFA Health Checks

SG missing self-reference. Add inbound + outbound self-ref to every SG on the cluster, plus least-privilege egress for the AWS APIs the node needs (HTTPS 443 to S3 / ECR / SageMaker / SSM / STS / CloudWatch Logs — via VPC-endpoint prefix-lists when possible). Full procedure: cluster-diagnostics-detail.md § A.

B: Capacity & AZ

Instance type unavailable in the requested AZ. Verify with describe-instance-type-offerings, then change AZ, use Flexible Training Plans, or request ODCR. Full: § B · strategy: capacity-planning.md.

C: Lifecycle Scripts

Script failed or timed out during provisioning. Read CloudWatch under /aws/sagemaker/Clusters/<name>/<id> — common causes: missing S3 VPC endpoint, IAM gap, CRLF line endings, instance-group name mismatch. Full: § C · layout: lifecycle-scripts.md.

D: EKS Access / kubectl

IAM identity not in EKS access entries. Verify with sts get-caller-identity, create an access entry with admin policy, update kubeconfig. Full: § D.

E: Cluster Provisioning

InService without all instances is expected under Continuous Provisioning — failures surface as events, not cluster errors. For stuck Creating/Updating/Deleting: check CFN nested stacks (§ H), IAM, capacity, events; if stuck Deleting check VPC ENI dependencies. Full: § E.

F: SSM Connectivity

Target is not connected: use sagemaker-cluster:<CLUSTER_ID>_<GROUP>-<INSTANCE_ID> format (not raw EC2 ID), install session-manager-plugin, confirm node Running. Check IAM + VPC endpoints on timeouts. Full: § F.

G: Node Replacement

Auto-repair: confirm NodeRecovery=Automatic, check Health Monitoring Agent (HMA) logs + node labels / Slurm reason, confirm capacity. Manual: reboot first, replace only if reboot fails. Replace requires the cluster to have been patched via UpdateClusterSoftware at least once and cannot target a Slurm controller node. Full: § G.

H: CloudFormation Errors

Embedded stack failed hides the real error. Drill into nested stacks via Events tab (filter Failed) until you reach a non-stack resource. CLI: describe-stack-events --query 'StackEvents[?ResourceStatus==\CREATE_FAILED\]'. Also covers SLR creation failures and permission-boundary denials. Full: § H · catalog: cloudformation-errors.md.

I: Utilities

Map Slurm node names (ip-10-x-y-z) to HyperPod instance IDs via list-cluster-nodes or on-node /opt/ml/config/resource_config.json. Full: § I.

J: AMI & Cluster Updates

UpdateClusterSoftware fails and rolls back, or the cluster stays in a post-maintenance rollback state. Common causes: lifecycle script incompatible with new AMI, HMA version too old, insufficient rolling-update capacity. If the cluster has active nodes, collect diagnostics and escalate rather than delete-and-recreate. Full: § J.

K: Dangling Nodes & Cleanup

Nodes in kubectl get nodes but not in list-cluster-nodes (ghost EKS nodes), or the inverse (HyperPod nodes that never registered kubelet). Script flags both. Full: § K.

L: Autoscaler Compatibility

Cluster Autoscaler errors on HyperPod provider IDs and breaks autoscaling for all node groups. No officially endorsed workaround — escalate to AWS Support. Karpenter does not conflict with HyperPod nodes by default. Full: § L.

---

Prerequisites

• aws CLI v2.13+ authenticated to the cluster's account
• jq, python3, bash 4.2+
• kubectl authenticated to the EKS cluster (EKS checks skipped if absent)
• session-manager-plugin (Slurm controller health checks only)

IAM policy: references/iam-permissions.md.

Defaults

• Region — required: pass --region or set $AWS_DEFAULT_REGION.
• Mode — --cluster <NAME> (diagnose) or --validate (pre-create).
• Event window — up to 500 most recent events (5 × 100, paginated).
• Colors — auto-disabled on non-TTY; --no-color to force off.

Error handling

Exit codes: 0 no critical failures · 1 one or more critical failures (cluster not found, fatal prerequisite missing, or any [FAIL] in diagnose or --validate mode). [WARN] lines do not affect the exit code.

Skill delegation

Escalate to AWS Support

Escalate when:

1. EFA health checks fail despite correct SG rules. 2. Capacity errors persist despite a valid Flexible Training Plan / ODCR. 3. Node replacement fails repeatedly without clear events / log signal. 4. Cluster stuck in a non-terminal state (Creating, Updating, or a post-maintenance rollback state) for an extended period. 5. CloudFormation root-cause is an internal service error.

Before opening the case

Run these commands and attach the output. Goal: AWS Support has everything at case open.

# 1. Cluster identity + status (confirms region, ARN, orchestrator, instance groups)
aws sagemaker describe-cluster --cluster-name <CLUSTER> --region <REGION>

# 2. Full cluster-level diagnostic bundle
bash scripts/diagnose-cluster.sh --cluster <CLUSTER> --region <REGION> > diag.txt

# 3. Per-node log/config bundle to S3 (delegates to hyperpod-issue-report skill)
#    See skills/hyperpod-issue-report/SKILL.md for the exact invocation.

Include in the case

• Cluster name + ARN (or ClusterId suffix) and AWS region
• ClusterStatus + FailureMessage from describe-cluster
• Timestamp window (UTC start / end) of the failure
• Exact error strings observed (copy verbatim from events / logs / console)
• Affected instance IDs / NodeLogicalIds / instance group names
• diag.txt from step 2 above
• S3 URI of the hyperpod-issue-report bundle from step 3

Capacity Planning

Companion to SKILL.md § B and --validate. Capacity errors are one of the most common creation failures.

---

Capacity options

On-demand

Fine for small instance types and short experiments. Not guaranteed for large GPU types (p4d, p5, p5e, trn1, trn2). No physical-proximity guarantees — sub-optimal for distributed training.

# Which AZs have this instance type. The EC2 API uses bare instance-type
# names, so strip the SageMaker `ml.` prefix before filtering.
aws ec2 describe-instance-type-offerings \
  --location-type availability-zone \
  --filters "Name=instance-type,Values=p5.48xlarge" \
  --region us-west-2 \
  --query 'InstanceTypeOfferings[*].Location' --output table

Flexible Training Plans

Guaranteed capacity for a reserved period, discounted pricing, co-located instances. Requires advance planning.

aws sagemaker list-training-plans \
  --filters Name=Status,Value=Active \
  --region <REGION> \
  --query 'TrainingPlanSummaries[*].{Name:TrainingPlanName,Type:InstanceType,Count:TotalInstanceCount,AZ:AvailabilityZone,Status:Status,Start:StartTime,End:EndTime}' \
  --output table

Use in cluster config:

aws sagemaker create-cluster \
  --cluster-name my-cluster \
  --instance-groups '[{
    "InstanceGroupName": "gpu-workers",
    "InstanceType": "ml.p5.48xlarge",
    "InstanceCount": 4,
    "ExecutionRole": "arn:aws:iam::<ACCT>:role/HyperPodRole",
    "TrainingPlanArn": "arn:aws:sagemaker:<REGION>:<ACCT>:training-plan/<PLAN_NAME>",
    "LifeCycleConfig": {"SourceS3Uri": "s3://sagemaker-lifecycle-<guid>/", "OnCreate": "on_create.sh"}
  }]' \
  --vpc-config '{"SecurityGroupIds":["sg-xxx"],"Subnets":["subnet-xxx"]}' \
  --region <REGION>

Critical: the subnet must be in the same AZ as the training plan's AvailabilityZone.

Reserved capacity (via account team)

For large or long-term capacity. Contact the AWS account team — customized placement and pricing, longer lead time.

---

AZ selection

Instance-type availability varies by AZ, and AZ names (us-west-2a) map to different physical zones per account. When coordinating with AWS Support or the account team about reserved capacity, use AZ IDs (usw2-az1), not AZ names — they're consistent across accounts.

# AZ name → ID:
aws ec2 describe-availability-zones --region <REGION> \
  --query 'AvailabilityZones[*].{Name:ZoneName,ID:ZoneId,State:State}' --output table

# Your subnet's AZ:
aws ec2 describe-subnets --subnet-ids <SUBNET> --region <REGION> \
  --query 'Subnets[0].{AZ:AvailabilityZone,AZ_ID:AvailabilityZoneId}'

# Instance-type offerings by AZ-ID:
aws ec2 describe-instance-type-offerings \
  --location-type availability-zone-id \
  --filters "Name=instance-type,Values=<TYPE>" \
  --region <REGION> \
  --query 'InstanceTypeOfferings[*].Location'

If your subnet's AZ doesn't appear in the offerings list, create a new subnet in an AZ that does.

---

Service quotas

Check ml.<type> for cluster usage quotas before creating a cluster. EKS on HyperPod also consumes ENIs and subnet IPs — size subnets generously; CIDRs cannot be changed after creation.

# SageMaker HyperPod quotas:
aws service-quotas list-service-quotas \
  --service-code sagemaker --region <REGION> \
  --query 'Quotas[?contains(QuotaName,`cluster`) || contains(QuotaName,`HyperPod`)].{Name:QuotaName,Value:Value,Code:QuotaCode}' \
  --output table

# Subnet free IPs:
aws ec2 describe-subnets --subnet-ids <SUBNET> --region <REGION> \
  --query 'Subnets[0].{CIDR:CidrBlock,FreeIPs:AvailableIpAddressCount}'

Request quota increases proactively — processing time varies by quota and region.

---

Troubleshooting

Insufficient capacity

1. Check which AZs have the instance type (commands above) 2. Verify your subnet is in one of those AZs 3. If no AZ has capacity: try a different region/type or contact account team 4. Using a Training Plan: verify TrainingPlanArn and that the subnet AZ matches the plan AZ

No subnets in the capacity AZ

Cluster specifies subnets, but none are in the AZ where AWS has capacity. Create a subnet in that AZ and add it to the cluster config.

Stuck in Creating with no events

Likely waiting for capacity. Check list-cluster-events; if no events after >1 hour, contact AWS Support.

Partial provisioning

Capacity was available for some instances but not all. With NodeProvisioningMode=Continuous the cluster keeps retrying. Check events for the failing instance group; consider reducing InstanceCount or using MinInstanceCount for elastic scaling.

CloudFormation Error Reference

Deep-dive companion to SKILL.md § H. HyperPod console deployments create nested CloudFormation stacks; the root-cause error is typically in a nested stack's leaf resource.

---

Navigate to the real failure

1. CloudFormation console → correct region → find the failed HyperPod stack (CREATE_FAILED or ROLLBACK_COMPLETE) 2. Events tab → filter by CREATE_FAILED → note the earliest failure 3. Resources tab → find AWS::CloudFormation::Stack entries with CREATE_FAILED 4. Click the Physical ID → opens the nested stack 5. Repeat until you reach a stack with only leaf resources 6. The Status reason on the failed leaf resource is the root cause

CLI alternative (per stack — nested stacks need to be iterated):

aws cloudformation describe-stack-events --stack-name <STACK> --region <REGION> \
  --query 'StackEvents[?ResourceStatus==`CREATE_FAILED`].{Time:Timestamp,Resource:LogicalResourceId,Type:ResourceType,Reason:ResourceStatusReason}' \
  --output table

---

Resource error catalog

AWS::SageMaker::Cluster

AWS::IAM::Role

AWS::EC2::VPC / Subnet / SecurityGroup

AWS::FSx::FileSystem

Custom::Resource / AWS::Lambda::Function

Lambda-backed custom resources fail with the underlying Lambda error. Find the function name in the Resources tab, then:

aws logs tail /aws/lambda/<FUNCTION_NAME> --region <REGION> --since 1h

---

Rolled-back stacks

When a stack rolls back, CloudFormation deletes what it created. List them:

aws cloudformation list-stacks \
  --stack-status-filter ROLLBACK_COMPLETE DELETE_COMPLETE \
  --region <REGION> \
  --query 'StackSummaries[?contains(StackName,`HyperPod`) || contains(StackName,`hyperpod`)].{Name:StackName,Status:StackStatus,Time:CreationTime}' \
  --output table

Cluster Diagnostics — Detailed Procedures

Full diagnostic and fix procedures for each section referenced from SKILL.md.

---

A: EFA Health Checks

Signals: "EFA health checks did not run successfully. Ensure that your VPC and security groups are properly configured before attempting to create a new cluster."

Root cause: Security group missing self-referencing rules — a common cluster-creation failure.

Diagnose

bash scripts/diagnose-cluster.sh --cluster <CLUSTER> --region <REGION>

# Or directly:
SG=$(aws sagemaker describe-cluster --cluster-name <CLUSTER> --region <REGION> \
  --query 'VpcConfig.SecurityGroupIds[0]' --output text)
aws ec2 describe-security-groups --group-ids $SG --region <REGION> \
  --query 'SecurityGroups[0].{Inbound:IpPermissions,Outbound:IpPermissionsEgress}' \
  --output json

Look for self-referencing rules where source/destination is the SG itself.

Fix — apply to every SG on the cluster

Customer-run. Apply the two self-ref rules to each SG in describe-cluster → VpcConfig.SecurityGroupIds, then add least-privilege egress for the AWS APIs the node needs to reach. Idempotent: InvalidPermission.Duplicate = already exists, treat as success.

SG=<security-group-id>
REGION=<region>

# Inbound self-ref (inter-node communication, EFA)
aws ec2 authorize-security-group-ingress --group-id $SG --region $REGION \
  --ip-permissions '[{"IpProtocol":"-1","UserIdGroupPairs":[{"GroupId":"'"$SG"'"}]}]'

# Outbound self-ref (EFA RDMA)
aws ec2 authorize-security-group-egress --group-id $SG --region $REGION \
  --ip-permissions '[{"IpProtocol":"-1","UserIdGroupPairs":[{"GroupId":"'"$SG"'"}]}]'

Egress for AWS APIs. The node needs HTTPS (443) outbound to reach the AWS services HyperPod uses: S3 (lifecycle scripts), ECR (container images), SageMaker (HyperPod control plane), SSM / SSMMessages / EC2Messages (Session Manager), STS, and CloudWatch Logs. The narrowest practical rule is TCP 443 to the VPC-endpoint prefix-lists for those services (com.amazonaws.<region>.<service> resolves to a pl-XXXXXXXX ID via aws ec2 describe-prefix-lists), referenced in authorize-security-group-egress --ip-permissions as PrefixListIds. See the AWS docs on VPC endpoint prefix lists for the exact CLI shape. aws ec2 describe-vpc-endpoints lists which services the cluster VPC already has endpoints for.

Self-ref opens all ports between instances in this SG (intended for intra-cluster EFA). For multi-SG clusters see cluster-operations.md § 1.

---

B: Capacity & AZ

Signals: "We currently do not have sufficient capacity in the Availability Zone you requested" (public doc); also seen: subnets not in the AZ where capacity is available.

aws ec2 describe-instance-type-offerings \
  --location-type availability-zone \
  --filters "Name=instance-type,Values=<INSTANCE_TYPE>" \
  --region <REGION> \
  --query 'InstanceTypeOfferings[*].Location' --output table

Fix: add subnet in an AZ where the type is available, or use Flexible Training Plans / ODCR. Full strategy: capacity-planning.md.

---

C: Lifecycle Scripts

Signals: cluster-creation event indicates lifecycle script execution error or timeout; creation fails during provisioning.

CLUSTER_ID=$(aws sagemaker describe-cluster --cluster-name <CLUSTER> --region <REGION> \
  --query 'ClusterArn' --output text | cut -d/ -f2)
LOG_GROUP="/aws/sagemaker/Clusters/<CLUSTER_NAME>/${CLUSTER_ID}"

aws logs describe-log-streams --log-group-name "$LOG_GROUP" --region <REGION> \
  --query 'logStreams[?starts_with(logStreamName,`LifecycleConfig`)].logStreamName' --output table

aws logs get-log-events --log-group-name "$LOG_GROUP" \
  --log-stream-name "LifecycleConfig/<group-name>/<instance-id>" \
  --region <REGION> --query 'events[*].message' --output text

Full S3 layout, node-type detection, and on-node debug: lifecycle-scripts.md.

---

D: EKS Access / kubectl

Signals: "couldn't get current server API group list: the server has asked for the client to provide credentials", kubectl get nodes fails or returns nothing.

# Your identity
aws sts get-caller-identity

# EKS cluster behind the HyperPod cluster
EKS_ARN=$(aws sagemaker describe-cluster --cluster-name <HYPERPOD> --region <REGION> \
  --query 'Orchestrator.Eks.ClusterArn' --output text)
EKS_NAME=$(echo $EKS_ARN | awk -F'/' '{print $NF}')

# Existing access entries
aws eks list-access-entries --cluster-name $EKS_NAME --region <REGION>

# Auth mode
aws eks describe-cluster --name $EKS_NAME --region <REGION> \
  --query 'cluster.accessConfig.authenticationMode' --output text

Suggested command — grant yourself EKS access (run this yourself)

Preconditions: $MY_ARN is the IAM role ARN, not the assumed-role session ARN. EKS auth mode is API or API_AND_CONFIG_MAP.

Command:

MY_ARN=$(aws sts get-caller-identity --query 'Arn' --output text)

aws eks create-access-entry \
  --cluster-name $EKS_NAME --region <REGION> --principal-arn $MY_ARN

aws eks associate-access-policy \
  --cluster-name $EKS_NAME --region <REGION> --principal-arn $MY_ARN \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
  --access-scope '{"type": "cluster"}'

aws eks update-kubeconfig --name $EKS_NAME --region <REGION>
kubectl get nodes

Blast radius: AmazonEKSClusterAdminPolicy grants cluster-wide admin on the EKS cluster — use a narrower policy (AmazonEKSEditPolicy / AmazonEKSViewPolicy + namespace scope) for day-to-day operators. update-kubeconfig overwrites the current kubectl context.

If the EKS cluster's auth mode is CONFIG_MAP only, access entries are not available. Switching auth mode is a cluster-level, administrator-level change — review the EKS access-entries documentation before proceeding and coordinate with anyone who depends on the existing aws-auth ConfigMap.

---

E: Cluster Provisioning

Signals: Cluster InService but instances not visible, kubectl get nodes returns nothing, list-cluster-nodes shows fewer nodes than expected.

With Continuous Provisioning, the cluster goes InService before all instances are created. Instance creation is asynchronous; failures appear as events.

aws sagemaker describe-cluster --cluster-name <CLUSTER> --region <REGION> \
  --query '{Status:ClusterStatus,Groups:InstanceGroups[*].{Name:InstanceGroupName,Count:CurrentCount,Target:InstanceCount,Status:InstanceGroupStatus}}' \
  --output table

aws sagemaker list-cluster-events --cluster-name <CLUSTER> --region <REGION> \
  --query 'ClusterEventSummaries[*].{Time:EventTime,Type:EventType,Message:Message}' \
  --output table

aws sagemaker list-cluster-nodes --cluster-name <CLUSTER> --region <REGION> \
  --query 'ClusterNodeSummaries[*].{ID:InstanceId,Group:InstanceGroupName,Status:InstanceStatus.Status}' \
  --output table

See cluster-operations.md § 5.

---

F: SSM Connectivity

Signals: "Target is not connected", SSM session fails.

For interactive shell or repeated SSM access, use the [`hyperpod-ssm`](../../hyperpod-ssm/SKILL.md) skill — it wraps the cluster-ID derivation, target-format construction, and session start shown below. The block here is for one-off connectivity diagnosis; hyperpod-ssm is the right tool for actually working on nodes.

---

G: Node Replacement

G.1: Auto-replacement not triggering

Diagnose (read-only):

# Is NodeRecovery enabled?
aws sagemaker describe-cluster --cluster-name <CLUSTER> --region <REGION> \
  --query 'InstanceGroups[*].{Group:InstanceGroupName,Recovery:NodeRecovery}' --output table

# Replacement activity
aws sagemaker list-cluster-events --cluster-name <CLUSTER> --region <REGION> \
  --query 'ClusterEventSummaries[?contains(Message,`replace`) || contains(Message,`reboot`) || contains(Message,`hardware`) || contains(Message,`recovery`)]' \
  --output table

# Health-monitoring-agent logs (pattern: SagemakerHealthMonitoringAgent/<group>/<instance>)
CLUSTER_ID=$(aws sagemaker describe-cluster --cluster-name <CLUSTER> --region <REGION> \
  --query 'ClusterArn' --output text | cut -d/ -f2)
aws logs describe-log-streams \
  --log-group-name "/aws/sagemaker/Clusters/<CLUSTER>/${CLUSTER_ID}" \
  --region <REGION> \
  --query 'logStreams[?starts_with(logStreamName,`SagemakerHealthMonitoringAgent`)].logStreamName' \
  --output table

# EKS node health labels — the sagemaker.amazonaws.com/node-health-status
# label on each node indicates the action HyperPod has decided on.
kubectl get nodes --show-labels
kubectl describe node <NODE>

sinfo -o "%N %T %30E"

Common blockers: NodeRecovery=None, health agent hasn't detected (wait for next cycle), lifecycle script failing on new instance (same log group, LifecycleConfig/... stream), no capacity (see B), cluster not InService.

Suggested command — enable NodeRecovery (run this yourself)

Destructive — replaces the whole `InstanceGroups` list. Any group omitted from the payload is deleted; any field drift (instance type, count, lifecycle config) is applied as-is. Re-run describe-cluster first and copy every existing field into the payload below before adding NodeRecovery=Automatic. If unsure, use the SageMaker console — it preserves existing fields by default. Never run this command yourself; present it to the customer.

Preconditions: NodeRecovery=None confirmed above. Derive every field for every instance group from the current `describe-cluster` output — update-cluster replaces the whole InstanceGroups list; any field drift is applied as-is.

Command:

aws sagemaker update-cluster --cluster-name <CLUSTER> --region <REGION> \
  --instance-groups '[{"InstanceGroupName":"<G>","InstanceType":"ml.p5.48xlarge",
    "InstanceCount":<N>,
    "LifeCycleConfig":{"SourceS3Uri":"<URI>","OnCreate":"<SCRIPT>"},
    "ExecutionRole":"<ROLE>",
    "OnStartDeepHealthChecks":["InstanceStress","InstanceConnectivity"],
    "NodeRecovery":"Automatic"}]'

Blast radius: any instance group omitted from the list is deleted; any field drift (instance type, count, lifecycle config) is applied as-is. If unsure, use the console, which preserves existing fields by default.

G.2: Manual replacement

Diagnose (read-only):

aws sagemaker list-cluster-nodes --cluster-name <CLUSTER> --region <REGION> \
  --query 'ClusterNodeSummaries[*].{ID:InstanceId,Group:InstanceGroupName,Status:InstanceStatus.Status}' \
  --output table

aws sagemaker describe-cluster --cluster-name <CLUSTER> --region <REGION> \
  --query 'ClusterStatus' --output text

Suggested command — reboot (run this yourself)

Preconditions: <INSTANCE_ID> belongs to the cluster (confirmed from list-cluster-nodes above); workload can tolerate a restart; on Slurm clusters, rebooting will not disrupt critical cluster operations (per the API doc). NodeIds batch size: 1-25 per call.

Command:

aws sagemaker batch-reboot-cluster-nodes --cluster-name <CLUSTER> --region <REGION> \
  --node-ids '["<INSTANCE_ID>"]'

aws sagemaker list-cluster-events --cluster-name <CLUSTER> --region <REGION> \
  --query 'ClusterEventSummaries[0:5].{Time:EventTime,Message:Message}' --output table

Blast radius: soft recovery via EC2 RebootInstances — preserves instance identity, root volume, and secondary volumes. Training processes on the node are interrupted.

Suggested command — replace (run this yourself, only if reboot did not clear the fault)

Preconditions:

• Reboot attempted first and did not clear the fault.
• Hardware fault confirmed (uncorrectable ECC, GPU-bus errors, EFA hardware failure); not a software / config issue.
• Data on root + secondary volumes is backed up — per the API doc: "Replacing nodes destroys all instance volumes, including both root and secondary volumes. All data stored on these volumes will be permanently lost and cannot be recovered."
• Cluster has been patched via UpdateClusterSoftware — per the API doc: "If you want to invoke this API on an existing cluster, you'll first need to patch the cluster by running the UpdateClusterSoftware API."
• Target is NOT a Slurm controller — per the API doc: "For SageMaker HyperPod clusters using the Slurm workload manager, you cannot replace instances that are configured as Slurm controller nodes."
• NodeIds batch size: 1-25 per call (API limit).

Command:

aws sagemaker batch-replace-cluster-nodes --cluster-name <CLUSTER> --region <REGION> \
  --node-ids '["<INSTANCE_ID>"]'

Blast radius: destroys root + secondary volumes on the replaced instance (permanent data loss). New hardware is provisioned with the same AMI and instance configuration.

Karpenter note (per the HyperPod EKS manual-recovery doc): on Karpenter-managed clusters, BatchReplaceClusterNodes terminates the node but does not guarantee a replacement — Karpenter only creates a new node if pending pods cannot be rescheduled onto remaining capacity. Per-workload configuration (pod anti-affinity, resource requests) can force a new node.

---

H: CloudFormation Errors

Signals: "Embedded stack failed", CREATE_FAILED / ROLLBACK_COMPLETE, generic console error.

Navigate to root cause

1. CloudFormation console → correct region 2. Find the failed HyperPod stack 3. Events tab → filter by CREATE_FAILED (earliest failure is the real one; later ones are cascades) 4. If error is "Embedded stack failed", open Resources → find AWS::CloudFormation::Stack with CREATE_FAILED 5. Click Physical ID → opens the nested stack 6. Repeat until you reach a non-stack leaf resource 7. The Status reason on the leaf is the actionable error

CLI alternative:

aws cloudformation describe-stack-events --stack-name <STACK> --region <REGION> \
  --query 'StackEvents[?ResourceStatus==`CREATE_FAILED`]'

For Custom::Resource failures, find the Lambda function name and check its logs.

Full resource-by-resource catalog: cloudformation-errors.md.

Service-linked role (SLR)

SageMaker HyperPod uses the SLR AWSServiceRoleForSageMakerHyperPod (attached to the AmazonSageMakerHyperPodServiceRolePolicy managed policy). It is created automatically on first cluster creation — you do not need to pre-create it. If cluster creation fails with an SLR error, the cause is almost always an SCP or permission boundary blocking iam:CreateServiceLinkedRole for the caller.

# Verify the SLR exists in the account
aws iam get-role --role-name AWSServiceRoleForSageMakerHyperPod

If iam:CreateServiceLinkedRole is denied by an SCP, have an account admin either:

• Grant the permission to the caller and retry cluster creation, or
• Request the SCP be adjusted to allow the specific SLR creation.

Permission boundary denials

Even when a role's inline policy grants a permission, an attached permission boundary can deny it.

ROLE_NAME=$(aws sagemaker describe-cluster --cluster-name <C> --region <R> \
  --query 'Orchestrator.Eks.ExecutionRoleArn' --output text | awk -F/ '{print $NF}')
aws iam get-role --role-name "$ROLE_NAME" --query 'Role.PermissionsBoundary'

If PermissionsBoundary is non-null, inspect the boundary policy — any denial there overrides all grants.

Cluster in Failed terminal state

ClusterStatus=Failed cannot be updated. Options:

1. Collect diagnostics (diagnose-cluster.sh + CFN events above) 2. Fix root cause (usually IAM / VPC / SG) 3. aws sagemaker delete-cluster and recreate

Deletion is destructive — migrate active workloads first.

Multi-AZ and EFA

EFA is intra-AZ only. Cross-AZ collectives fall back to TCP. For EFA-accelerated training, keep all training instance groups in a single AZ. describe-instance-type-offerings to pick one.

Service quotas

Check SageMaker HyperPod, EC2 EFA, and VPC quotas before creation — see capacity-planning.md § service quotas. Quota increases take 1-3 business days.

---

I: Utilities

Slurm node name → instance ID

Slurm nodes use IP-named hostnames (ip-10-1-123-45). Quick lookup:

# Works from anywhere
aws sagemaker list-cluster-nodes --cluster-name <CLUSTER> --region <REGION> \
  --query 'ClusterNodeSummaries[*].{ID:InstanceId,DNS:PrivateDnsHostname,Group:InstanceGroupName}' \
  --output table

# On head node
IP=$(echo "ip-10-1-123-45" | sed 's/ip-//; s/-/./g')
sudo cat /opt/ml/config/resource_config.json | jq | grep -A 3 "$IP"

For bulk lookups, list-cluster-nodes output can be piped to jq to produce a CSV of node → instance ID (there are also community scripts in public AWS sample repositories).

---

J: AMI & Cluster Updates

UpdateClusterSoftware fails and rolls back, or the cluster remains in a post-maintenance rollback state. Common causes: lifecycle script incompatible with new AMI, insufficient capacity during rolling update, IAM gaps.

aws sagemaker list-cluster-events --cluster-name <NAME> --region <REGION> \
  --query 'ClusterEventSummaries[?contains(Message, `Update`) || contains(Message, `Rollback`)]'

aws sagemaker describe-cluster --cluster-name <NAME> --region <REGION> \
  --query '{Status:ClusterStatus,FailureMsg:FailureMessage}'

# Per-instance-group lifecycle logs on the nodes that were rolled over:
aws logs describe-log-streams \
  --log-group-name "/aws/sagemaker/Clusters/<NAME>/<CLUSTER_ID>" \
  --region <REGION>

Decisions

---

K: Dangling Nodes & Cleanup

After a failed scale-up or rollback, EKS may show nodes that HyperPod no longer manages ("dangling"). The inverse — HyperPod nodes not registered in EKS — usually means kubelet or bootstrap failed.

kubectl get nodes -l sagemaker.amazonaws.com/compute-type=hyperpod \
  -o jsonpath='{range .items[*]}{.spec.providerID}{"\n"}{end}' \
  | sed 's|.*/||' | sort > /tmp/eks-nodes.txt

aws sagemaker list-cluster-nodes --cluster-name <NAME> --region <REGION> \
  --query 'ClusterNodeSummaries[*].InstanceId' --output text \
  | tr '\t' '\n' | sort > /tmp/hp-nodes.txt

# EKS-only (dangling) — registered in EKS but not in HyperPod
comm -23 /tmp/eks-nodes.txt /tmp/hp-nodes.txt

# HyperPod-only (kubelet never registered) — in HyperPod but not in EKS
comm -13 /tmp/eks-nodes.txt /tmp/hp-nodes.txt

Remediation

Fix — delete a dangling EKS node

Customer-run. Only delete when the EKS node has no matching HyperPod instance (confirmed by comm above) AND the EC2 instance is terminated — confirm with the first command below.

aws ec2 describe-instances --instance-ids <IID> --region <REGION> \
  --query 'Reservations[0].Instances[0].State.Name'
kubectl delete node <NODE_NAME>

If the EC2 instance is still running and registered, kubelet re-registers the node — the delete is a no-op with transient scheduling churn.

Orphaned HyperPod node (not in EKS): kubelet never registered. Triage with hyperpod-node-debugger — common causes are instance IAM role misconfigured, VPC endpoints missing, or lifecycle script failure.

---

L: Autoscaler Compatibility

Cluster Autoscaler (CAS) in the same EKS cluster can fail to parse HyperPod node provider IDs, which can break autoscaling for every node group in the cluster — not only HyperPod. Diagnose via CAS logs: look for node-info parse errors tied to HyperPod-managed nodes. If hit, escalate to AWS Support; do not apply untested CAS flags.

Karpenter does not manage HyperPod nodes directly and should not conflict. If Karpenter is attempting to disrupt HyperPod training pods, the standard Karpenter annotation karpenter.sh/do-not-disrupt: "true" on the pod prevents disruption (see the Karpenter upstream documentation for current annotation syntax).

Cluster Operations Reference

Operational deep-dives for the hyperpod-cluster-debugger skill. See SKILL.md for the workflow entry points.

---

1. EFA Security Group (multi-SG clusters)

The EFA health check runs during instance provisioning, before lifecycle scripts execute. If it fails, lifecycle scripts never run and CloudWatch lifecycle logs are empty — the cluster event will say "EFA health checks did not run successfully".

When a cluster uses multiple security groups, all SGs must have the self-referencing rules. Check each:

for SG in $(aws sagemaker describe-cluster --cluster-name <C> --region <R> \
  --query 'VpcConfig.SecurityGroupIds[]' --output text); do
  echo "=== $SG ==="
  aws ec2 describe-security-groups --group-ids $SG --region <R> \
    --query 'SecurityGroups[0].{In:IpPermissions,Out:IpPermissionsEgress}'
done

Fix commands are in cluster-diagnostics-detail.md § A.

---

2. Capacity

See capacity-planning.md.

---

3. Lifecycle scripts

See lifecycle-scripts.md.

---

4. EKS access control

Authentication modes

Access entries require API or API_AND_CONFIG_MAP. If the cluster is on CONFIG_MAP only, aws eks list-access-entries returns nothing useful; verify the mode with describe-cluster --query 'cluster.accessConfig.authenticationMode' and consult the EKS access-entries documentation for the switching procedure.

Access policies (EKS-native)

Troubleshooting kubectl auth

aws sts get-caller-identity            # your identity
kubectl config current-context         # which cluster kubeconfig points at
kubectl cluster-info                   # API server reachable?

If using an assumed role: access entries reference the IAM role ARN, not the assumed-role session ARN.

• Role ARN: arn:aws:iam::123456789012:role/MyRole
• Session ARN: arn:aws:sts::123456789012:assumed-role/MyRole/session-name

---

5. Continuous Provisioning (EKS only)

The cluster transitions to InService once the control plane is ready; instances are created asynchronously and failures are reported as events, not cluster failures. Failed instances can be individually replaced.

# Poll instance creation:
watch -n 30 "aws sagemaker describe-cluster --cluster-name <C> --region <R> \
  --query 'InstanceGroups[*].{Name:InstanceGroupName,Current:CurrentCount,Target:InstanceCount}' --output table"

# Poll cluster events:
watch -n 30 "aws sagemaker list-cluster-events --cluster-name <C> --region <R> \
  --query 'ClusterEventSummaries[0:5].{Time:EventTime,Msg:Message}' --output table"

Nodes in list-cluster-nodes but not in kubectl get nodes

1. Check lifecycle script logs — it registers the node with EKS 2. Verify the EKS endpoint is reachable from worker subnets 3. Check kubelet on the node via SSM 4. Verify the node's IAM role has AmazonEKSWorkerNodePolicy

Cluster events are emitted for HyperPod EKS. For HyperPod Slurm, events are not yet surfaced — use CloudWatch logs and list-cluster-nodes instead.

---

6. SSM target format

See the hyperpod-ssm skill's SKILL.md for the target format (sagemaker-cluster:<CLUSTER_ID>_<GROUP>-<INSTANCE_ID>), prerequisites, and manual-command examples. HyperPod requires start-session — not send-command against raw instance IDs.

---

7. Node replacement (batch APIs)

Full Suggested-command blocks with preconditions + blast radius are in cluster-diagnostics-detail.md § G.2. Summary:

• Cluster must be InService
• Batch limit: 1-25 node IDs per call for both APIs
• batch-replace-cluster-nodes destroys root + secondary volumes and is not supported on Slurm controller nodes — back up first
• Monitor with list-cluster-events after the call
• Prefer batch APIs over legacy paths (Slurm reason fields, K8s labels)

---

8. Slurm — controller operations

The per-node Slurm operations (resuming a single node, fixing a single Slurm state) live in the hyperpod-node-debugger skill. This section is controller-level only.

Diagnose controller health (via SSM on the controller)

scontrol ping                                     # slurmctld responsive?
systemctl status slurmctld                        # service state
systemctl is-active munge && systemctl status munge   # auth daemon (required)
systemctl is-active slurmdbd                      # accounting DB (if used)

slurmctld down

journalctl -u slurmctld --since "1 hour ago" --no-pager | tail -100
tail -200 /var/log/slurm/slurmctld.log

Common causes and fixes:

• OOM on controller: restart the service; investigate the job scale that triggered it.
• Munge auth failure (Invalid authentication credential): munge key mismatch. Re-sync /etc/munge/munge.key to every node, restart munge + slurmctld.
• Accounting DB unreachable (slurmdbd + MariaDB / RDS): check network path and credentials. slurmctld won't start if accounting is required but unreachable.
• Config error in `slurm.conf`: slurmctld -D -vvv (foreground) prints the parse error. Roll back to the last known-good config.

Fix — restart slurmctld

Customer-run on the Slurm controller (via SSM) after the root cause is diagnosed. Running jobs, pending queue, and node states are preserved; caches and resource calculations reset. Brief scheduler pause.

sudo systemctl restart slurmctld
scontrol ping   # expect "Slurmctld(primary) is UP"

If slurm.conf is broken the service will not return — roll back the config first.

munge inactive

Diagnose:

systemctl status munge
ls -l /etc/munge/munge.key   # expect munge:munge, mode 0400
sudo md5sum /etc/munge/munge.key   # must match on controller + every compute node

Fix — start munge

Customer-run. Safe when munge is inactive and the key file is present and matches other nodes.

sudo systemctl start munge

If md5 mismatches another node, jobs will still fail auth — re-distribute the controller's key cluster-wide and restart munge on every node.

Stuck jobs (PENDING / COMPLETING / CONFIGURING)

squeue -o "%i %j %T %R %N" --noheader | grep -iE "COMPLETING|CONFIGURING|PENDING"
scontrol show job <JOBID>
scancel <JOBID>               # if safe to cancel

Common reason codes:

• (Resources) — waiting for free nodes. Check sinfo -o "%P %a %l %D %T".
• (AssocGrpNodeLimit) / (QOSMaxJobsPerUserLimit) — quota-related. sacctmgr show assoc.
• (NodeDown) — partition has no healthy nodes. Use the hyperpod-node-debugger skill.
• (BeginTime) — scheduled for a future start time.

Restarting slurmctld to clear stuck-job symptoms uses the same Suggested-command block as above (§ slurmctld down).

Verify after remediation

scontrol ping                                   # "Slurmctld(primary) is UP"
sinfo                                            # no "down*" or "drain" states
systemctl is-active slurmctld munge
scontrol show config | grep StateSaveLocation   # must be persistent + writable

---

9. Filesystem performance

Symptom: training bottlenecked by data loading, checkpoint save / load, or slow executable / script loading.

Diagnose on the node

mount | grep -E "fsx|nfs|lustre|ebs|nvme"
df -hT
iostat -x 1 5                 # per-device throughput / IOPS / utilization

# FSx for Lustre:
lfs df -h                     # per-OST utilization (uneven = hotspot)
lfs getstripe <path>          # striping config; wider = more parallelism

# FSx for OpenZFS / NFS:
nfsstat -m                    # per-mount retransmissions / wait times
nfsiostat 5                   # ops/s, throughput, RTT

# EBS:
lsblk -o NAME,TYPE,SIZE,MOUNTPOINT

CloudWatch (from your workstation)

# FSx for Lustre throughput saturation:
aws cloudwatch get-metric-statistics \
  --namespace AWS/FSx --metric-name DataReadBytes \
  --dimensions Name=FileSystemId,Value=<FSxId> \
  --statistics Sum --period 300 \
  --start-time "$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --end-time   "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --region <REGION>
# Also: DataWriteBytes, FreeDataStorageCapacity, MetadataOperations

# EBS throughput / IOPS:
aws cloudwatch get-metric-statistics \
  --namespace AWS/EBS --metric-name VolumeReadOps \
  --dimensions Name=VolumeId,Value=<vol-id> \
  --statistics Sum --period 60 \
  --start-time "$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --end-time   "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --region <REGION>
# Also: VolumeWriteOps, VolumeReadBytes, VolumeWriteBytes, BurstBalance

Interpret

Choose the right filesystem

For HyperPod Slurm, the default lifecycle script supports FSx for OpenZFS for /home — evaluate it if home is on Lustre and you see metadata-op saturation.

Verify after remediation

• CloudWatch: throughput / IOPS climbs past the old flat-line
• Training step time drops; data-loading fraction of step time drops
• iostat %util stays below 80% under sustained load

IAM Permissions Required

Read-only diagnostic:

{
  "Action": [
    "sagemaker:DescribeCluster",
    "sagemaker:ListClusterNodes",
    "sagemaker:ListClusterEvents",
    "sagemaker:ListClusters",
    "ec2:DescribeSecurityGroups",
    "ec2:DescribeSubnets",
    "ec2:DescribeVpcs",
    "ec2:DescribeVpcEndpoints",
    "ec2:DescribeInstances",
    "ec2:DescribeInstanceTypeOfferings",
    "eks:DescribeCluster",
    "eks:ListAccessEntries",
    "eks:ListAddons",
    "eks:DescribeAddon",
    "iam:GetRole",
    "iam:ListAttachedRolePolicies",
    "s3:ListBucket",
    "s3:GetObject",
    "logs:DescribeLogGroups",
    "logs:DescribeLogStreams",
    "logs:GetLogEvents",
    "cloudformation:DescribeStackEvents",
    "cloudformation:DescribeStacks",
    "servicequotas:ListServiceQuotas",
    "ssm:StartSession",
    "ssm:TerminateSession"
  ]
}

SSM on HyperPod uses start-session with sagemaker-cluster:<cluster-id>_<group>-<iid> targets — not send-command against plain instance IDs. Grant ssm:StartSession / ssm:TerminateSession.

For remediations the operator runs, add the matching write permission (e.g. ec2:AuthorizeSecurityGroupIngress, eks:CreateAccessEntry).

Lifecycle Script Reference

Companion to SKILL.md § C and cluster-operations.md § 3. Lifecycle scripts run on each node during provisioning. A failure here blocks the node — and often the entire cluster — from reaching InService.

---

Layout

Default AWS-published lifecycle scripts (commonly called "base-config") handle provisioning for Slurm and EKS. Before deep debugging, compare the customer's in-use scripts against the latest published version — upstream fixes often resolve the failure.

Slurm entry point (typical base-config layout)

on_create.sh → lifecycle_script.py for orchestration (detects node type from /opt/ml/config/resource_config.json and runs per-type steps). Controller nodes provision first; compute / login nodes wait for the controller to write slurm.conf to shared storage. Customer-forked pipelines may differ — read on_create.sh on the affected node to confirm.

Controller failure cascades to all compute nodes — if the controller's lifecycle script fails, compute nodes cannot find slurm.conf and also fail.

EKS entry point

on_create.sh → on_create_main.sh (configures containerd storage, kubelet, FSx client, EFA).

S3 URI validation

• SourceS3Uri starts with s3://
• OnCreate filename matches an S3 key in that prefix
• Execution role has s3:GetObject and s3:ListBucket on the bucket

---

Common errors

S3 access

Timeout reaching S3 from the lifecycle script (e.g. Connect timeout on endpoint URL: s3://...) → no S3 VPC endpoint; node cannot reach S3 from a private subnet.

Fix — add an S3 Gateway endpoint

Customer-run. Gateway endpoint type is free; Interface endpoints are billed per-hour.

aws ec2 create-vpc-endpoint \
  --vpc-id <VPC_ID> \
  --service-name com.amazonaws.<REGION>.s3 \
  --route-table-ids <ROUTE_TABLE_ID> \
  --vpc-endpoint-type Gateway

Caution: routes S3 traffic for every resource using the listed route tables through the VPC endpoint. Can break workloads that rely on going to S3 via public DNS + NAT with custom endpoint policies. Review the VPC's default endpoint policy (or set --policy-document) before creating.

AccessDenied / 403 Forbidden on GetObject — add s3:GetObject + s3:ListBucket on the lifecycle bucket to the execution role.

Script execution

Slurm

Compute nodes fail because slurm.conf not found — controller's lifecycle failed. Fix the controller first.

slurmctld: error ... — check /var/log/slurmctld.log on controller via SSM. Common causes: wrong SlurmctldHost, partition/node definition errors, missing MUNGE key.

FSx

mount.lustre: ... Connection timed out — FSx in different VPC/AZ, or SG doesn't allow Lustre traffic. FSx and HyperPod nodes must share a VPC; SG must allow TCP 988 and 1018-1023 between nodes and FSx. Verify FSx is AVAILABLE.

---

Reading logs

CloudWatch (from workstation)

CLUSTER_ID=$(aws sagemaker describe-cluster --cluster-name <NAME> --region <R> \
  --query 'ClusterArn' --output text | cut -d/ -f2)
LOG_GROUP="/aws/sagemaker/Clusters/<CLUSTER_NAME>/${CLUSTER_ID}"

# List lifecycle log streams:
aws logs describe-log-streams \
  --log-group-name "$LOG_GROUP" --region <R> \
  --query 'logStreams[?starts_with(logStreamName,`LifecycleConfig`)].{Stream:logStreamName,LastEvent:lastEventTimestamp}' \
  --output table

# Read a specific stream:
aws logs get-log-events \
  --log-group-name "$LOG_GROUP" \
  --log-stream-name "LifecycleConfig/<GROUP>/<INSTANCE_ID>" \
  --region <R> --limit 100 \
  --query 'events[*].message' --output text

On-node (via SSM)

cat /var/log/provision/provisioning.log      # full provisioning log
cat /opt/ml/config/resource_config.json      # node topology
cat /opt/slurm/etc/slurm.conf                # Slurm config (if generated)
cat /opt/ml/metadata/resource-metadata.json  # node metadata

Test locally

file on_create.sh         # must not say "with CRLF line terminators"
head -1 on_create.sh      # must start with #!/bin/bash
bash -n on_create.sh      # syntax check
shellcheck on_create.sh   # optional lint