Hyperpod Nccl

HyperPod NCCL Debugger

Operating policy. Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a Suggested command (run this yourself) block and wait for the customer. Destructive order: investigate → reboot → replace (replace destroys root + secondary volumes; not supported on Slurm controller nodes). Never discard training state on speculation.

Diagnose NCCL failures on SageMaker HyperPod (EKS and Slurm). scripts/nccl-diagnose.sh reads state via AWS APIs, kubectl, and SSM, then prints each issue as [FAIL] ... → references/<file>.md § <section>. Read-only.

Signal sourcing: list-cluster-events carries infrastructure-level state only (lifecycle, bootstrap, EFA health check, capacity, replacement, reboot, AMI rollback). It does not carry NCCL timeouts, GPU XID/ECC, or per-pod training signals — those come from pod logs, CloudWatch training streams, on-node SSM probes, and NCCL env audit. "No events" on a training-time NCCL issue is expected, not a clean bill of health.

---

Workflow

1. Collect cluster name, region, namespace/job (EKS), exact NCCL error string. 2. Run the diagnostic (always — the output drives everything else). 3. For every [FAIL] line, Read the referenced section. 4. Present finding, root cause, and the Suggested-command block with concrete values (instance IDs, SG IDs, namespaces) filled in from the script output. Wait for customer approval. 5. Re-run the diagnostic to confirm.

If a finding has no matching section, report it as a bug — do not invent a fix.

Step 1: Authenticate kubectl (EKS)

EKS_ARN=$(aws sagemaker describe-cluster --cluster-name <HYPERPOD-NAME> --region <REGION> \
  --query 'Orchestrator.Eks.ClusterArn' --output text)
EKS_NAME=$(echo "$EKS_ARN" | awk -F'/' '{print $NF}')
aws eks update-kubeconfig --name "$EKS_NAME" --region <REGION>
kubectl get nodes

Step 2: Run the diagnostic

# Basic:
bash scripts/nccl-diagnose.sh --cluster <HYPERPOD-NAME> --region <REGION>

# Scope to an EKS job/namespace:
bash scripts/nccl-diagnose.sh --cluster <NAME> --region <REGION> --namespace <NS> --job <JOB>

# Force orchestrator:
bash scripts/nccl-diagnose.sh --cluster <NAME> --region <REGION> --orchestrator slurm

# Larger hardware sample (default 3):
bash scripts/nccl-diagnose.sh --cluster <NAME> --region <REGION> --sample-nodes 10

# Specific node only:
bash scripts/nccl-diagnose.sh --cluster <NAME> --region <REGION> --node i-0abc123def456

Tags: [PASS] · [FAIL] (counted in Issues Found, has reference pointer) · [WARN] · [INFO]. Priorities: P0 blocks training · P1 degraded · P2 informational.

---

Remediation index

Each [FAIL] line in the script already points directly at the right section. This table is a lookup for manual triage.

---

Prerequisites

• aws CLI v2.13+ authenticated (aws sts get-caller-identity)
• jq, python3, bash 4.2+
• unbuffer (from the expect package: yum install expect / apt install expect)
• kubectl authenticated to the EKS cluster (K8s checks skipped if absent)
• session-manager-plugin for on-node hardware checks

Defaults

• Region — required: pass --region or set $AWS_DEFAULT_REGION.
• Orchestrator — auto-detected; override with --orchestrator eks|slurm.
• Namespace / job (EKS) — all namespaces; scope with --namespace <NS> --job <JOB>.
• Hardware sampling — 3 nodes over SSM (capped at 50). --node <ID> for a specific node. Node probes run serially (180 s per node): --sample-nodes 10 can take ~30 min.
• CloudWatch window — last 2 hours.
• Colors — auto-disabled on non-TTY or TERM=dumb.

Error handling

Exit codes: 0 diagnostic complete · 1 fatal prerequisite missing or cluster unreachable.

IAM permissions

Full policy + RBAC in operations.md § 2. SSM on HyperPod uses start-session against sagemaker-cluster:<cluster-id>_<group>-<iid> targets — grant ssm:StartSession / ssm:TerminateSession, not ssm:SendCommand.

Scale strategy

Large clusters: the PyTorch NCCL backend defaults to a 10-minute collective-op timeout (per the PyTorch distributed docs). Large clusters routinely exceed that on first rendezvous; raise it via torch.distributed.init_process_group(timeout=timedelta(seconds=<N>)). HyperPod support has also observed NCCL topology-graph-search hangs on 256+ node clusters when memlock is unlimited; using a large fixed memlock (e.g. 8388608) in pod securityContext or /etc/security/limits.conf has cleared these in field cases. This memlock pattern is a field observation, not AWS- or NCCL-documented behavior.

For FSDP, DeepSpeed, or Megatron-LM tuning: debugging-guide.md § 18.

Skill delegation

Escalate to AWS Support

Escalate when:

1. All SG rules correct, EFA verified on-node, but NCCL still times out. 2. Hardware checks pass on all nodes but AllReduce still hangs. 3. Issues Found: 0 but training still fails. 4. GPU XID errors persist after node replacement. 5. Collective-op timeout raised and memlock workaround applied but large-cluster rendezvous still hangs.

Before opening the case

# 1. Cluster identity + status
aws sagemaker describe-cluster --cluster-name <C> --region <R>

# 2. Full NCCL diagnostic (sample more nodes for escalation)
bash scripts/nccl-diagnose.sh --cluster <C> --region <R> --sample-nodes 10 > nccl-diag.txt

# 3. Per-node log/config bundle to S3 (delegates to hyperpod-issue-report)
#    See skills/hyperpod-issue-report/SKILL.md for the exact invocation.

Include in the case

• Cluster name + ARN and AWS region
• Orchestrator (EKS or Slurm) and EKS cluster name / Slurm controller node
• Timestamp window (UTC start / end) of the failure
• Exact NCCL / libfabric error strings (copy verbatim from pod logs or journalctl)
• Affected instance IDs / node names / pod names / namespace / job name
• nccl-diag.txt from step 2 above
• S3 URI of the hyperpod-issue-report bundle from step 3
• NCCL env vars in effect (printenv | grep -E '^NCCL|^FI_|^TORCH_' from one pod)

References

• error-patterns-quick-ref.md — log pattern → code → fix table
• debugging-guide.md — per-scenario procedures (21 sections incl. NVLS/PXN/topology)
• performance-testing.md — nccl-tests, bandwidth thresholds, straggler detection
• operations.md — IAM, SSM format, CloudWatch, env-var reference, node labels, Slurm ops, remediations

NCCL HyperPod — Detailed Debugging Guide

Detailed procedures for each failure type. See SKILL.md for the quick reference.

Table of Contents

---

1. NCCL Timeout / Rendezvous Hang

Always start minimal: Reproduce with 2 ranks and torch.ones(100) before debugging full training.

import os, torch, torch.distributed as dist, datetime
rank = int(os.environ.get('RANK', 0))
world_size = int(os.environ.get('WORLD_SIZE', 2))
master = os.environ.get('MASTER_ADDR', 'localhost')
port  = os.environ.get('MASTER_PORT', '29500')
dist.init_process_group('gloo',
    init_method=f'tcp://{master}:{port}',
    world_size=world_size, rank=rank,
    timeout=datetime.timedelta(seconds=120))
t = torch.ones(100) * rank
dist.all_reduce(t, op=dist.ReduceOp.SUM)
expected = sum(range(world_size))
assert t[0].item() == expected, f"Got {t[0].item()}, expected {expected}"
print(f"[Rank {rank}] [PASS] AllReduce PASSED", flush=True)
dist.destroy_process_group()

Debug env vars:

export NCCL_DEBUG=INFO              # verbose NCCL output
export NCCL_DEBUG_SUBSYS=ALL        # all subsystems
export TORCH_DISTRIBUTED_DEBUG=DETAIL
export TORCH_NCCL_ASYNC_ERROR_HANDLING=1    # surface NCCL timeouts as exceptions
export NCCL_DEBUG_FILE=/tmp/nccl_rank${RANK}.log
# Extend PyTorch collective timeout in training code:
#   dist.init_process_group("nccl", timeout=timedelta(seconds=1800))

Dump call stack of hung process:

# Inside the pod (EKS):
kubectl exec -n <ns> <pod> -- pip install py-spy -q
kubectl exec -n <ns> <pod> -- py-spy dump --pid $(pgrep -f python | head -1)

# On the node via SSM (both orchestrators):
aws ssm start-session --target sagemaker-cluster:<CLUSTER_ID>_<GROUP>-<INSTANCE_ID>
# On node:
py-spy dump --pid $(pgrep -f python | head -1)
py-spy record -o /tmp/profile.svg --pid <PID> --duration 30

Root cause matrix:

Slurm MASTER_ADDR setup (no headless service needed — Slurm resolves hostnames natively):

# In your sbatch script:
export MASTER_ADDR=$(scontrol show hostnames $SLURM_JOB_NODELIST | head -1)
export MASTER_PORT=29500
# Verify DNS works from all nodes:
srun --overlap bash -c "nslookup $MASTER_ADDR"

For 100+ node clusters — prioritized fix order:

1. Extend the PyTorch collective timeout (default: 10 min for NCCL, per the PyTorch distributed docs). Example starting value: init_process_group(timeout=timedelta(seconds=<N>)) where N is tuned from your observed step time. nodes*5+600 is a starting heuristic only. 2. Check memlock — see Section 17 (field-observed workaround for topology-search hangs on 256+ node clusters). 3. Run straggler detection — see references/performance-testing.md pairwise bandwidth test. 4. Check for NCCL version drift after rolling node replacements — see Section 10

---

2. Security Group Self-Reference Rules

Commands and verification are in operations.md § 8. Without inbound + outbound self-reference on the cluster SG, NCCL rendezvous and EFA RDMA traffic are dropped.

---

3. NCCL_SOCKET_IFNAME — Interface Selection

On EFA nodes (p4d/p5), always set explicitly:

# Correct for EFA nodes — exclude non-VPC interfaces:
export NCCL_SOCKET_IFNAME=^lo,docker,efa,veth,virbr

# Find the correct VPC interface name:
ip -br addr show | grep -vE "^lo|docker|br-|virbr|veth|efa" | grep UP | awk '{print $1}'

Validate the setting works (leaves at least one interface):

# After setting NCCL_SOCKET_IFNAME, verify it leaves interfaces:
PATTERN="${NCCL_SOCKET_IFNAME#^}"
ip -br addr show | grep UP | awk '{print $1}' | \
  grep -vE "$(echo "$PATTERN" | tr ',' '|')"
# Must show at least one interface (e.g., ens5)

Also set matching MPI variable:

export OMPI_MCA_btl_tcp_if_include=ens5   # match your VPC ENI
# OR:
export OMPI_MCA_btl_tcp_if_exclude=lo,docker0,virbr0

---

4. Container OOM — Pod Killed Mid-Training (exit code 137)

Symptom: Pod status = OOMKilled, exit code 137. The Linux kernel killed the process due to cgroup memory limit. This is different from GPU OOM (see section 11).

Detect:

# EKS: check container termination reason
kubectl describe pod <POD> -n <NS> | grep -A5 "Last State:"
# Shows: Reason: OOMKilled, Exit Code: 137

# On node via SSM:
dmesg | grep -i "oom\|killed process" | tail -10
free -h

Fix options (in order of impact):

# 1. Gradient checkpointing (most impact, slower backward pass)
model.gradient_checkpointing_enable()

# 2. FSDP (shard model across all GPUs in job)
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
model = FSDP(model, device_id=torch.cuda.current_device())

# 3. Mixed precision (halve activation memory)
from torch.cuda.amp import autocast, GradScaler
scaler = GradScaler()
with autocast():
    loss = model(inputs)

# 4. Reduce batch size
batch_size = batch_size // 2  # halve until OOM resolves

# Increase K8s memory limits:
resources:
  limits:
    memory: "64Gi"   # increase as needed
    nvidia.com/gpu: "8"

---

5. Wrong Results — Gradient Sync Issues

Verify AllReduce is actually happening:

def check_allreduce_consistency(tensor, name, rank, world_size):
    """Verify all ranks have same values after AllReduce."""
    dist.all_reduce(tensor, op=dist.ReduceOp.SUM)
    results = [None] * world_size
    dist.all_gather_object(results, tensor.sum().item())
    if rank == 0:
        if len(set(round(r, 4) for r in results)) > 1:
            print(f"[FAIL] INCONSISTENT '{name}': {results}", flush=True)
        else:
            print(f"[PASS] CONSISTENT '{name}': {results[0]:.4f}", flush=True)

Check FSDP/DTensor placements:

from torch.distributed.tensor import DTensor
for name, param in model.named_parameters():
    if isinstance(param, DTensor):
        print(f"[Rank {dist.get_rank()}] {name}: placements={param.placements}")
    else:
        print(f"[Rank {dist.get_rank()}] {name}: NOT sharded (unexpected for FSDP)")

Print from all ranks in order (debugging):

def print_all_ranks(msg):
    for r in range(dist.get_world_size()):
        if dist.get_rank() == r:
            print(f"[Rank {r}] {msg}", flush=True)
        dist.barrier()

---

6. EFA Configuration

Required for full performance on p4d/p5:

export FI_PROVIDER=efa
export FI_EFA_USE_DEVICE_RDMA=1     # GPU Direct RDMA
export NCCL_SOCKET_IFNAME=^lo,docker,efa,veth
export NCCL_PROTO=Simple            # large-message protocol (valid: LL, LL128, Simple)
# Collective timeout is a PyTorch arg — set via init_process_group(timeout=timedelta(seconds=1800))

K8s pod spec for EFA:

resources:
  limits:
    vpc.amazonaws.com/efa: <N>   # match EFA device count for the instance type
  requests:
    vpc.amazonaws.com/efa: <N>

Suggested command — install EFA K8s device plugin (run this yourself)

Preconditions: EKS orchestrator with GPU nodes (p4d / p5 / p5e / p5en / p6); node AMI already has EFA kernel modules (verify fi_info -p efa returns endpoints on one node); cluster admin has approved installing a daemonset into kube-system. If EFA is already allocated to pods (pod limits.vpc.amazonaws.com/efa > 0), the plugin is already installed — skip.

Command:

helm repo add eks <aws-eks-charts-helm-repo>
helm install aws-efa-k8s-device-plugin --namespace kube-system \
  eks/aws-efa-k8s-device-plugin

Blast radius: installs a daemonset on every node in kube-system (one pod per node) that advertises vpc.amazonaws.com/efa as a schedulable resource. Cannot be removed by a single command — requires helm uninstall. Interacts with every GPU-scheduling pod; misconfiguration can starve pods of EFA resources.

Verify EFA on node:

fi_info -p efa                              # lists EFA endpoints
cat /opt/amazon/efa_installed_packages      # EFA installer version
lsmod | grep efa                            # kernel module loaded
ls /dev/infiniband/uverbs*                  # device files exist
nvidia-smi nvlink --status                  # NVLink (p4d/p5)

---

7. Node Hardware Failures

NCCL errors caused by GPU / EFA hardware faults (Xid errors, ECC, NVLink, off-bus) are diagnosed and remediated in the node-debugger skill: hyperpod-node-debugger § G (GPU/Accelerator) and § F (Hardware / Auto-Repair).

Get the instance ID from a K8s node name:

kubectl get node <NODE_NAME> -o jsonpath='{.spec.providerID}' | cut -d'/' -f5

Suggested command — drain before reboot/replace (EKS) (run this yourself)

Preconditions: hardware fault confirmed on <NODE_NAME> (XID/ECC/NVLink/off-bus — see hyperpod-node-debugger § G); customer accepts that pods using emptyDir volumes on this node will lose that data when evicted; drain is preparation for batch-reboot-cluster-nodes (try first) or batch-replace-cluster-nodes — not a fix on its own. See hyperpod-cluster-debugger § G.2.

Command:

kubectl cordon <NODE_NAME>
kubectl drain <NODE_NAME> --ignore-daemonsets --delete-emptydir-data

Blast radius: --delete-emptydir-data discards emptyDir scratch on this node (training caches, ephemeral checkpoints not persisted to PVC//opt/sagemaker); pods are rescheduled elsewhere if capacity exists, otherwise stay Pending. Drain is reversible (kubectl uncordon) only if you decide not to proceed with reboot/replace.

---

8. Slurm-Specific Procedures

NCCL batch script template:

#!/bin/bash
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-node=8
#SBATCH --job-name=nccl-training

# EFA settings (p4d/p5):
export FI_PROVIDER=efa
export FI_EFA_USE_DEVICE_RDMA=1
export NCCL_SOCKET_IFNAME=^lo,docker,efa,veth
export NCCL_DEBUG=WARN
# Set the PyTorch collective timeout in training code, not via env:
#   dist.init_process_group("nccl", timeout=timedelta(seconds=1800))

# Rendezvous (torchrun manages RANK/WORLD_SIZE automatically):
export MASTER_ADDR=$(scontrol show hostnames $SLURM_JOB_NODELIST | head -1)
export MASTER_PORT=29500

srun torchrun \
  --nnodes=$SLURM_NNODES \
  --nproc_per_node=8 \
  --rdzv_backend=c10d \
  --rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT \
  train.py

Slurm node management and the RemoveIPC=no requirement are in operations.md § 7.

---

9. NCCL RAS — Live Job Health

NCCL's RAS (Reliability, Availability, Serviceability) subsystem lets you query the state of a running NCCL job without attaching a debugger. Per the NCCL env-var reference, RAS is available since NCCL 2.24 and is enabled by default (NCCL_RAS_ENABLE=1); the listen address is configured via NCCL_RAS_ADDR. Confirm the actual port your build uses (it can be overridden by env or NCCL config) before assuming the example port number below.

# Find the RAS port for the running NCCL process (configurable via NCCL_RAS_ADDR):
#   - Check the env of the training process:
#       cat /proc/$(pgrep -f python | head -1)/environ | tr '\0' '\n' | grep NCCL_RAS_ADDR
#   - Or check what's listening locally:
#       ss -ltnp | grep -i nccl

# Example (replace <PORT> with the actual RAS port for your build):
echo "verbose status" | nc -w 3 localhost <PORT>

# With the ncclras binary :
ncclras -v
ncclras -f json | python3 -m json.tool   
ncclras -m                              

# Inside a K8s pod:
kubectl exec -n <NS> <POD> -- sh -c "echo 'verbose status' | nc -w 3 localhost <PORT>"

Interpret status:

• RUNNING OK — all ranks alive, progressing normally
• MISMATCH — some ranks behind → possible straggler
• INCOMPLETE — missing rank data → one rank unresponsive
• DEAD / PEER_DEAD — a rank process is confirmed dead → this is the rank that hung the collective

---

10. NCCL Version Mismatch (NCCL function not found)

Symptom: NCCL function not found or Incompatible NCCL version at job startup. Cause: Different NCCL builds across nodes — mixed container images or manual installs.

Diagnose:

# Check NCCL version per running pod:
for pod in $(kubectl get pods -n <NS> -l job-name=<JOB> --no-headers | awk '{print $1}'); do
    echo -n "$pod: "
    kubectl exec -n <NS> "$pod" -- \
        python3 -c "import torch; print(torch.cuda.nccl.version())" 2>/dev/null \
        || echo "unavailable"
done

# Check via library file:
kubectl exec -n <NS> <POD> -- \
    find /usr/local/cuda/lib64 /usr/lib -name "libnccl.so*" 2>/dev/null | head -3

# Check CUDA driver version per node:
kubectl get nodes -o custom-columns=\
'NAME:.metadata.name,DRIVER:.metadata.labels.nvidia\.com/cuda\.driver-version' \
2>/dev/null || kubectl get nodes -o wide

Fix:

# All pods in a job MUST use identical container images.
# Verify your job spec uses the same image for all replicas:
kubectl get pod -n <NS> -l job-name=<JOB> \
    -o jsonpath='{range .items[*]}{.metadata.name}: {.spec.containers[0].image}{"\n"}{end}'
# Every line must show the same image:tag

# If different, update your job spec to pin every replica to the same image:
# spec.template.spec.containers[0].image: <AWS DLC image URI from your region's DLC account>
# e.g. an AWS Deep Learning Container pytorch-training image tagged for your CUDA + Python + OS combo

Common cause on HyperPod: Rolling node replacement installs a new AMI with a different NCCL version while old nodes are still in the cluster. Use lifecycle scripts to pin NCCL versions.

---

11. GPU OOM — CUDA out of memory / cudaMalloc failed

Symptom: CUDA out of memory, cudaMalloc failed, or RuntimeError: CUDA error: out of memory. This is GPU VRAM exhaustion — distinct from container OOMKill (section 4). The process does NOT get killed by the kernel; PyTorch raises a Python exception.

Diagnose:

# Check GPU memory usage on all GPUs:
kubectl exec -n <NS> <POD> -- \
    nvidia-smi --query-gpu=index,name,memory.used,memory.total,utilization.gpu \
    --format=csv,noheader

# In training script — add before suspected OOM:
import torch
for i in range(torch.cuda.device_count()):
    used = torch.cuda.memory_allocated(i) / 1e9
    reserved = torch.cuda.memory_reserved(i) / 1e9
    total = torch.cuda.get_device_properties(i).total_memory / 1e9
    print(f"GPU {i}: allocated={used:.1f}GB reserved={reserved:.1f}GB total={total:.1f}GB")
    print(torch.cuda.memory_summary(i))

Fix options (in order of impact):

# 1. Gradient checkpointing — trade compute for memory (most impactful)
model.gradient_checkpointing_enable()

# 2. ZeRO optimizer — shard optimizer states across ranks (DeepSpeed)
# In deepspeed config:
# "zero_optimization": {"stage": 3}   # ZeRO-3: shards params, grads, optimizer states

# 3. FSDP — shard model weights across all GPUs
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
model = FSDP(model)

# 4. Mixed precision — halve activation memory
from torch.cuda.amp import autocast
with autocast(dtype=torch.bfloat16):
    loss = model(inputs)

# 5. Reduce batch size — simplest fix
batch_size = batch_size // 2

# 6. Clear cache between steps (if fragmentation is the issue)
torch.cuda.empty_cache()

Memory fragmentation fix:

# If OOM happens after many steps (fragmentation):
import gc
gc.collect()
torch.cuda.empty_cache()
# Or: set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

---

12. DNS Resolution Failure (Name or service not known)

Symptom: Name or service not known, getaddrinfo failed, or rendezvous hangs forever. Cause: MASTER_ADDR hostname cannot be resolved. Common on EKS when no headless Service is in place to give pods a stable DNS name.

Diagnose:

# Check DNS from inside a pod:
kubectl exec -n <NS> <POD> -- nslookup $MASTER_ADDR
kubectl exec -n <NS> <POD> -- getent hosts $MASTER_ADDR

# Check if headless service exists:
kubectl get svc -n <NS> -o wide | grep None
# Should show: ClusterIP: None with selector matching training pods

# Check CoreDNS is healthy:
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=20

Fix:

# Create headless service for training job DNS:
apiVersion: v1
kind: Service
metadata:
  name: my-training-svc
  namespace: <NS>
spec:
  clusterIP: None
  selector:
    app: my-training-job   # must match training pod labels
  ports:
  - port: 29500
    name: nccl-rendezvous

# Set MASTER_ADDR using the service DNS:
export MASTER_ADDR="<job-name>-0.<service-name>.<namespace>.svc.cluster.local"

---

13. EFA TCP Fallback (NET/OFI Using TCP)

Symptom: In NCCL_DEBUG=INFO output, you see NET/OFI Using TCP instead of NET/OFI Using EFA. Training runs but at 10-100x lower bandwidth than expected.

Diagnose:

# Check if EFA device plugin is installed:
kubectl get daemonset -A | grep -i efa

# Check if pod requests EFA:
kubectl get pod <POD> -n <NS> -o jsonpath='{.spec.containers[0].resources.limits}'
# Must include: vpc.amazonaws.com/efa

# Check EFA env vars:
kubectl exec -n <NS> <POD> -- env | grep FI_

# Check on node via SSM:
fi_info -p efa  # Must list EFA endpoints

Fix checklist:

1. Install the EFA K8s device plugin — see the Suggested-command block earlier in this file (§ EFA device plugin). 2. Request EFA in pod spec:

   resources:
     limits:
       vpc.amazonaws.com/efa: <N>   # match EFA device count for the instance type

3. Set EFA env vars in the pod:

   export FI_PROVIDER=efa
   export FI_EFA_USE_DEVICE_RDMA=1
   export NCCL_SOCKET_IFNAME=^lo,docker,efa,veth

4. Ensure the aws-ofi-nccl plugin is in the container image (find /opt/amazon -name "libnccl-net.so" 2>/dev/null).

---

14. GPU P2P Access Blocked (ACS/IOMMU)

Symptom: NCCL WARN P2P not supported between dev X and dev Y or peer access is not supported. Intra-node AllReduce is 10-50x slower because GPU Direct P2P transfers are blocked by PCI ACS.

Diagnose:

# Check ACS on node via SSM:
lspci -vvv 2>/dev/null | grep -A20 "PCI bridge" | grep "ACSCtl:"
# If "SrcValid+" appears → ACS is enabled → P2P blocked

# Check IOMMU:
dmesg | grep -i iommu
grep -oE "intel_iommu=[^ ]+" /proc/cmdline

# Check P2P topology:
nvidia-smi topo -m
# NV# = NVLink (fast), PIX/PXB/PHB = PCIe (slow)

Suggested command — disable ACS on NVIDIA GPU bridges (last resort; run this yourself)

Preconditions: P2P GPU traffic confirmed to fall back to CPU hops via nvidia-smi topo -m; GPU peer-to-peer blocked by PCIe ACS (ACSCtl: SrcValid+ observed via lspci -vvv); confirmed the node is single-tenant (training workload only); you have reviewed that this weakens IOMMU isolation for the affected PCI bridges. Do NOT apply to multi-tenant or security-sensitive hosts.

Command:

# Disable ACS on NVIDIA GPU upstream bridges only — scoping to 10de: avoids
# weakening IOMMU isolation on unrelated PCI devices.
for BDF in $(lspci -D -d 10de: | awk '{print $1}'); do
  sudo setpci -s "$BDF" ECAP_ACS+0x6.w=0000 2>/dev/null
done

# For persistence, add the same NVIDIA-only scope to the lifecycle script:
echo 'for BDF in $(lspci -D -d 10de: | awk "{print \$1}"); do setpci -s $BDF ECAP_ACS+0x6.w=0000 2>/dev/null; done' \
  >> /opt/ml/scripts/on_create.sh

Blast radius: host-wide PCIe change for every NVIDIA GPU bridge on the node — takes effect immediately and persists for the life of the OS (or until the lifecycle script is re-run after a reboot). IOMMU isolation for those bridges is reduced, which is acceptable on a dedicated training host but NOT acceptable on multi-tenant hosts. If applied incorrectly, reboot restores the default ACS state unless the lifecycle-script change was made.

---

15. Stale Shared Memory (unlink shared memory)

Symptom: unlink shared memory /dev/shm/nccl-* failed: No such file or new training job fails with File exists on /dev/shm/nccl-* files left by a previous crash.

Cause: Either systemd RemoveIPC=yes (default on RHEL/Amazon Linux) deletes NCCL shm mid-training, or a crashed training process left orphaned shm files.

Diagnose:

# Check on node:
ls -la /dev/shm/nccl-*
grep RemoveIPC /etc/systemd/logind.conf

Suggested command — clean stale shm and disable RemoveIPC (run this yourself)

Preconditions: no NCCL training job is currently running on this node (ps aux | grep -E 'python.*torchrun|mpirun' returns empty); RemoveIPC=yes confirmed in /etc/systemd/logind.conf; brief systemd-logind restart is acceptable on this node.

Command:

# 1. Clean up stale files
rm -f /dev/shm/nccl-*

# 2. Prevent systemd from deleting shm mid-training
echo "RemoveIPC=no" >> /etc/systemd/logind.conf
sudo systemctl restart systemd-logind

# 3. For persistence across replacements, add to the lifecycle script:
echo 'echo "RemoveIPC=no" >> /etc/systemd/logind.conf && systemctl restart systemd-logind' \
  >> /opt/ml/scripts/on_create.sh

Blast radius: rm -f /dev/shm/nccl-* silently destroys any active NCCL shared-memory segments — running a collective at the same time will fail. RemoveIPC=no is a persistent systemd change; the systemctl restart logs out anyone in a systemd user session. Lifecycle-script edit persists across node replacements.

---

16. Host Firewall Blocking NCCL (iptables/nftables)

Symptom: NCCL timeout even though SG rules and NetworkPolicy are correct. Root cause: host-level iptables or nftables DROP/REJECT rules blocking NCCL ports.

Diagnose:

# On node via SSM:
iptables -L -n | grep -E "DROP|REJECT"
nft list ruleset 2>/dev/null | grep -E "drop|reject"

Suggested command — adjust host firewall to allow NCCL traffic (run this yourself)

Preconditions: identified a specific iptables/nftables rule blocking NCCL traffic via iptables -L -n --line-numbers; confirmed the rule is not managed by kube-proxy (those typically appear in the KUBE-* chains — never delete those) or the VPC CNI; customer has approved either deleting the specific rule or adding an explicit ACCEPT rule for NCCL ports.

Command (preferred — add explicit allow rather than touch existing rules):

# Allow NCCL rendezvous port range:
iptables -I INPUT -p tcp --dport 29400:29500 -j ACCEPT
# Allow the NCCL RAS port if RAS is enabled and used (read your NCCL_RAS_ADDR setting):
# iptables -I INPUT -p tcp --dport <NCCL_RAS_PORT> -j ACCEPT

Command (alternative — delete a specific custom rule by line number):

iptables -L -n --line-numbers   # confirm the line number first
iptables -D INPUT <rule_number>

Blast radius: iptables -I INPUT ... -j ACCEPT adds a rule at the top of the INPUT chain — host-wide effect, cleared on reboot unless persisted via iptables-save. Deleting a rule by line number is precise but irreversible without the original rule definition; capture iptables-save first if you may need to roll back. Never run iptables -F on an EKS worker — it flushes kube-proxy's service rules and VPC CNI NetworkPolicy enforcement, breaking pod networking cluster-wide.

---

17. RDMA Memory Registration Failure (ibv_reg_mr failed)

Symptom: NCCL WARN Call to ibv_reg_mr failed followed by EFA falling back to TCP — training continues but at 10-100x lower bandwidth.

Cause: The Linux memlock limit prevents the EFA driver from pinning memory for RDMA DMA transfers. With memlock=0 or very low values, EFA cannot register any memory buffers.

Diagnose:

# Check current memlock limit:
ulimit -l
# Should be: unlimited or ≥8388608 (8GB in KB)
# If 0 or 64 → FAIL

# Check on the actual node via SSM:
aws ssm start-session --target sagemaker-cluster:<CLUSTER_ID>_<GROUP>-<INSTANCE_ID>
# On node:
ulimit -l
cat /proc/$(pgrep -f python | head -1)/limits | grep "Max locked"

# In NCCL debug output (NCCL_DEBUG=INFO):
# "NCCL WARN Call to ibv_reg_mr failed, got error (12)" → errno 12 = ENOMEM (memlock)

Suggested command — raise memlock for EFA RDMA (run this yourself)

Preconditions: ulimit -l confirmed at 0 / 64 / very low on the affected node; Call to ibv_reg_mr failed confirmed in NCCL/EFA logs; customer accepts a session/login change (immediate path) or a persistent change to /etc/security/limits.conf (permanent path); for K8s pods the change must be applied in the pod spec, not on the node.

Command — immediate (session only, lost on logout):

ulimit -l 8388608       # 8 GB in KB

Command — permanent (system-wide):

echo "* soft memlock 8388608" >> /etc/security/limits.conf
echo "* hard memlock 8388608" >> /etc/security/limits.conf
# Requires re-login to take effect.

# For Slurm:
echo "ulimit -l 8388608" >> /etc/slurm/prolog.sh

Pod spec (K8s) — required for containerized training:

securityContext:
  capabilities:
    add: ["IPC_LOCK"]
# A high memlock limit on the host is not visible inside the container without
# IPC_LOCK; without this capability, the pod still hits memlock=0 / very low.

Blast radius: session ulimit affects only the current login shell. /etc/security/limits.conf change persists across reboots and applies to every user who logs in afterwards. Slurm prolog change applies to every job step launched after the edit. K8s pod-spec change is per-pod. For HyperPod, replication across replacement nodes requires baking the limits.conf change into the lifecycle script.

Note — field observation on large clusters (not NCCL- or AWS-documented): HyperPod support has seen NCCL topology-graph-search failures on 256+ node clusters when memlock is set to unlimited. Using a large fixed value (e.g. 8388608) instead of unlimited has cleared these in field cases. If you hit this, engage AWS Support with the NCCL topology-search failure output.

Verify fix worked:

# After fix, NCCL_DEBUG=INFO should show:
# "NCCL INFO NET/OFI Using EFA RDMA" (not TCP fallback)
# No more "ibv_reg_mr failed" warnings

# Check effective bandwidth after fix:
/opt/nccl-tests/build/all_reduce_perf -b 1G -e 8G -f 2 -g 1
# Should match expected algbw for your instance type

---

18. Distributed Training Frameworks — NCCL Tuning

NCCL issues often surface differently depending on the distributed training framework. Framework-specific guidance:

FSDP (Fully Sharded Data Parallel — PyTorch native)

Common NCCL issues with FSDP:

Recommended NCCL env vars for FSDP on HyperPod:

export NCCL_SOCKET_IFNAME=^lo,docker
export FI_PROVIDER=efa
export FI_EFA_USE_DEVICE_RDMA=1
export NCCL_ALGO=Ring           # Ring is generally better for FSDP all-gather patterns
export NCCL_PROTO=Simple        # Simple protocol for large-message FSDP comms
# FSDP checkpoint can be slow at scale — extend the PyTorch collective timeout:
#   dist.init_process_group("nccl", timeout=timedelta(seconds=1800))

DeepSpeed

Common NCCL issues with DeepSpeed:

DeepSpeed config tuning for HyperPod:

{
  "comms_config": {
    "comms_backend": "nccl",
    "timeout": 1800
  },
  "zero_optimization": {
    "stage": 3,
    "stage3_max_live_parameters": 1e8,
    "stage3_prefetch_bucket_size": 5e7,
    "reduce_bucket_size": 5e8
  }
}

Megatron-LM

Common NCCL issues with Megatron-LM:

Megatron-LM parallelism mapping for HyperPod:

Rule of thumb:
  TP (tensor parallel) = within a single node (8 GPUs on p5)
  PP (pipeline parallel) = across nodes (minimizes cross-node comms volume)
  DP (data parallel) = remaining nodes

  World size = TP × PP × DP
  Example: 32 p5.48xlarge (256 GPUs)
    TP=8, PP=4, DP=8 → 8×4×8 = 256

---

19. Advanced NCCL Tuning (NVLS, PXN, Topology, Cross-NIC)

NVLS — NVLink SHARP (GPU-to-GPU hardware offload)

NVLS is NVIDIA's in-network aggregation over NVLink. Per the NCCL env-var reference, NCCL_NVLS_ENABLE defaults to 2 (since NCCL 2.17), meaning NVLS is enabled when supported. It speeds up small-message AllReduce on H100/H200 nodes but requires matching driver and container versions — driver/container mismatch is a common cause of NVLS-related hangs in field cases.

Symptoms:

• Hang inside ncclAllReduce on p5/p5e/p5en
• NCCL INFO ... NVLS ... failed
• Fine on 1 node, hang on 2+ nodes

Diagnosis:

# Check NCCL version (container side)
python3 -c "import torch; print(torch.cuda.nccl.version())"
# Check driver version (node side, via SSM)
nvidia-smi --query-gpu=driver_version --format=csv

Mitigations:

1. Disable NVLS temporarily to isolate:

   export NCCL_NVLS_ENABLE=0

2. Pin NCCL version across all pods/jobs (match container image digest, not tag). 3. Upgrade the NVIDIA driver on the AMI via UpdateClusterSoftware if the container expects a newer driver.

PXN — P2P Cross-NUMA (p5.48xlarge optimal config)

PXN lets NCCL route inter-node traffic via an intermediary GPU on a different NUMA node to maximize NIC utilization. The documented PXN env var is NCCL_P2P_PXN_LEVEL (since NCCL 2.12), which controls PXN usage for send/receive — default is 2 (always use PXN); set 0 to disable. There are also NCCL_PXN_DISABLE and NCCL_PXN_C2C knobs; consult the NCCL env-var reference for the version in use.

NCCL_CROSS_NIC defaults to 2 (per the NCCL docs: "Try to use the same NIC for the same ring/tree, but still allow for the use of different NICs if it would result in a better performance") — leave at default unless you've measured a regression.

# Tuning knobs — measure before/after with nccl-tests:
export NCCL_P2P_PXN_LEVEL=2     # default; 0 disables PXN

# Channel count: NCCL_MIN_NCHANNELS / NCCL_MAX_NCHANNELS were deprecated in
# NCCL 2.17 in favor of NCCL_MIN_CTAS / NCCL_MAX_CTAS (per NCCL env-var docs).
# Both names still work on recent versions.
export NCCL_MIN_CTAS=4

If these cause regressions on smaller jobs (< 16 nodes), unset and re-measure with the defaults.

NCCL_TOPO_FILE — Custom Topology

NCCL auto-discovers topology on p-family instances and usually picks the right plan. Use a custom topology file only when:

• Running in containers that hide the PCIe topology from NCCL
• Using an instance type NCCL doesn't recognize
• Debugging suboptimal ring/tree selection

To export the topology NCCL sees for manual inspection:

export NCCL_TOPO_DUMP_FILE=/tmp/nccl-topo.xml
# Run any NCCL op (e.g., all_reduce_perf), then inspect /tmp/nccl-topo.xml

Do not ship a hand-edited topology file unless you've confirmed the default is wrong — this is an advanced-user escape hatch.

NCCL_SOCKET_FAMILY — IPv4 Forcing

Dual-stack environments (IPv6 enabled on the VPC but IPv4 intended for NCCL) can cause silent TCP fallback. Force IPv4:

export NCCL_SOCKET_FAMILY=AF_INET

Mixed instance families

Mixing different P-family generations in a single NCCL communicator (e.g. p4d + p5) is risky — the topology and EFA adapter counts differ, which can cause NCCL algorithm-selection issues. If you need to do this, measure carefully with nccl-tests first; otherwise launch separate jobs per instance family.

NCCL_COLLNET_ENABLE on EFA

NCCL_COLLNET_ENABLE=1 enables NVIDIA's Collective Network (CollNet) protocol, used with SHARP on InfiniBand fabrics. EFA is not InfiniBand and does not provide a SHARP-compatible CollNet provider, so leaving CollNet enabled on EFA can lead to wasted init time or fallback. If a job script sets NCCL_COLLNET_ENABLE=1, set it to 0 for HyperPod EFA clusters:

export NCCL_COLLNET_ENABLE=0

Instance family EFA counts (reference)

Counts from authoritative AWS sources where available. Always confirm live with ls /dev/infiniband/uverbs* | wc -l on the node — instance counts vary across firmware revisions.

For other EFA-supported types (p4de, p5.4xlarge, trn1, trn1n, trn2, etc.), check the current EC2 instance-types doc rather than hard-coding a value here. Mismatch with the live count → EFA driver not loaded, or a subset of NICs didn't attach at boot. Reboot via batch-reboot-cluster-nodes first; replace if reboot doesn't recover.

---

20. Pending / CrashLoopBackOff / Init-Container Failures

Pod lifecycle failures surface as Pending, CrashLoopBackOff, or stuck in an init container. These are NOT NCCL bugs per se — they block the NCCL job from starting. Diagnose in this order:

Pending pods

# Why is it pending?
kubectl describe pod <POD> -n <NS> | sed -n '/Events:/,$p' | head -40

Common reasons and where to fix:

CrashLoopBackOff

kubectl logs <POD> -n <NS> --previous | tail -100   # logs from the crashed container
kubectl describe pod <POD> -n <NS>                   # last termination state + exit code

Map the exit code to the guide section:

Stuck in init container

kubectl get pod <POD> -n <NS> -o jsonpath='{.status.initContainerStatuses}' | python3 -m json.tool
kubectl logs <POD> -n <NS> -c <INIT_CONTAINER_NAME>

Common init-container failures:

• Fetching model weights from S3 — check IAM, VPC endpoint, bucket policy.
• Downloading dataset — DNS / network / auth.
• Running a chown/chmod on a large volume — timeout.
• Waiting for another pod (headless service / init-container-as-gate pattern) — the dependency pod never became Ready.

Remediation is always customer-driven

None of these states have a one-command fix. Walk the customer through the diagnosis above, identify the specific cause, then apply the targeted fix. Do not kubectl delete pods without understanding why.

---

21. GPU Row-Remap / DCGM Health (Marginal Memory Silent Degrader)

When NCCL aborts or training accuracy regresses without matching Xid/ECC counts — sporadic NaNs, intermittent AllReduce hangs, DCGM default medium,memtest passes but a GPU is silently returning bad data — the cause is usually a pending row-remap or a marginal GPU that DCGM's combined-run is masking.

Diagnosis procedure, remap state table, DCGM split-run workaround, and escalation bundle (nvidia-bug-report.sh + /var/log/nvidia-dcgm/) are in the node-debugger skill: hyperpod-node-debugger § G.1.a/b.

NCCL Error Pattern Reference

Quick-lookup table of NCCL log patterns → code → root cause → fix. Used by the diagnostic script to map log lines to a remediation section in debugging-guide.md.

NCCL Operations Reference

Operational procedures and lookup tables for the NCCL skill.

---

1. Getting cluster names

The HyperPod cluster name ≠ the EKS cluster name.

# List HyperPod clusters:
aws sagemaker list-clusters --region <REGION> \
  --query 'ClusterSummaries[*].[ClusterName,ClusterStatus,CreationTime]' --output table

# EKS cluster behind a HyperPod cluster:
EKS_ARN=$(aws sagemaker describe-cluster \
  --cluster-name <HYPERPOD-NAME> --region <REGION> \
  --query 'Orchestrator.Eks.ClusterArn' --output text)
EKS_NAME=$(echo $EKS_ARN | awk -F'/' '{print $NF}')

aws eks update-kubeconfig --name $EKS_NAME --region <REGION>

---

2. IAM

Read-only diagnostic

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "NCCLSkillReadOnly",
    "Effect": "Allow",
    "Action": [
      "sagemaker:DescribeCluster",
      "sagemaker:ListClusters",
      "sagemaker:ListClusterNodes",
      "sagemaker:ListClusterEvents",
      "ec2:DescribeSecurityGroups",
      "ec2:DescribeVpcs",
      "ec2:DescribeSubnets",
      "ec2:DescribeInstances",
      "logs:DescribeLogGroups",
      "logs:DescribeLogStreams",
      "logs:FilterLogEvents",
      "logs:GetLogEvents",
      "ssm:StartSession",
      "ssm:DescribeSessions",
      "ssm:TerminateSession"
    ],
    "Resource": "*"
  }]
}

Per-remediation permissions

Granted only if the operator applies the suggested fix:

kubectl RBAC (EKS read — write only if operator applies a fix)

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: nccl-skill-read
rules:
- apiGroups: [""]
  resources: ["nodes", "pods", "pods/log", "namespaces", "services"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"]
- apiGroups: ["networking.k8s.io"]
  resources: ["networkpolicies"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["daemonsets"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["get", "list"]

If the operator deletes/creates a NetworkPolicy, grant delete/create on networkpolicies scoped to the training namespace.

---

3. SSM target format (HyperPod)

sagemaker-cluster:<CLUSTER_ID>_<INSTANCE_GROUP>-<INSTANCE_ID>

CLUSTER_ID is the ARN suffix — not the cluster name. Full connect procedure is in the node-debugger skill (references/node-diagnostics-detail.md § K). send-command against a bare instance ID will fail with ValidationException — HyperPod's managed fleet requires start-session with the prefixed target.

---

4. CloudWatch — NCCL log collection

NCCL logs are not collected by HyperPod by default. Add this to the lifecycle script so logs ship to the same log group as lifecycle/health-monitoring logs:

# Amazon Linux: yum install -y amazon-cloudwatch-agent
# Ubuntu:       apt-get install -y amazon-cloudwatch-agent

cat > /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json <<'EOF'
{
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {"file_path": "/var/log/nccl.log",
           "log_group_name": "/aws/sagemaker/Clusters/${CLUSTER_NAME}/${CLUSTER_ID}",
           "log_stream_name": "{instance_id}/nccl"},
          {"file_path": "/var/log/training/*.log",
           "log_group_name": "/aws/sagemaker/Clusters/${CLUSTER_NAME}/${CLUSTER_ID}",
           "log_stream_name": "{instance_id}/training"}
        ]
      }
    }
  }
}
EOF

/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a fetch-config -m ec2 \
  -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -s

Query NCCL errors

CLUSTER_ID=$(aws sagemaker describe-cluster --cluster-name <NAME> --region <R> \
  --query 'ClusterArn' --output text | awk -F'/' '{print $NF}')

aws logs filter-log-events \
  --log-group-name "/aws/sagemaker/Clusters/<NAME>/${CLUSTER_ID}" \
  --filter-pattern '"NCCL WARN"' \
  --start-time $(($(date +%s) - 7200))000 \
  --region <R> \
  --query 'events[*].[timestamp,logStreamName,message]' --output table

---

5. NCCL environment variable reference

Required

EFA (p4d / p5 / p3dn)

Collective-op timeout (PyTorch)

NCCL_TIMEOUT is not a standard NCCL or PyTorch env var — some launchers (DeepSpeed, AWS samples) wrap it, but setting it alone has no effect in pure PyTorch. Control the collective timeout via init_process_group and the TORCH_* env vars:

# In training code — replaces any NCCL_TIMEOUT env var:
import datetime, torch.distributed as dist
dist.init_process_group("nccl", timeout=datetime.timedelta(seconds=1800))

# Surfaces hangs as Python exceptions instead of silent waits:
export TORCH_NCCL_ASYNC_ERROR_HANDLING=1
export TORCH_NCCL_BLOCKING_WAIT=1   # debug only — has perf cost at scale

Performance tuning

EFA network-card counts per instance type

Used to populate vpc.amazonaws.com/efa requests in K8s pod specs. The canonical EC2 EFA doc enumerates which types support EFA but doesn't always state the per-instance card count; counts below are taken from authoritative AWS sources where available. Always count with ls /dev/infiniband/uverbs* | wc -l on a live node and adjust if your build differs.

For other types in the EFA-supported list (e.g. p4de.24xlarge, p5.4xlarge, trn1.32xlarge, trn1n.32xlarge, trn2.48xlarge) — check the current EC2 instance-types doc and confirm with ls /dev/infiniband/uverbs* | wc -l on the node before pinning a value.

K8s pod spec (EFA-enabled)

env:
- { name: MASTER_ADDR,            value: "my-job-svc.my-ns.svc.cluster.local" }
- { name: MASTER_PORT,            value: "29500" }
- { name: WORLD_SIZE,             value: "16" }        # 2 nodes × 8 GPUs
- { name: NCCL_SOCKET_IFNAME,     value: "^lo,docker,efa,veth,virbr" }
- { name: FI_PROVIDER,            value: "efa" }
- { name: FI_EFA_USE_DEVICE_RDMA, value: "1" }
- { name: FI_EFA_FORK_SAFE,       value: "1" }
- { name: NCCL_DEBUG,             value: "WARN" }
# Set PyTorch collective timeout via init_process_group(timeout=1800s) in training code
# (NCCL_TIMEOUT env var is a non-standard convention — not read by NCCL or PyTorch directly)
resources:
  limits:
    nvidia.com/gpu: 8
    vpc.amazonaws.com/efa: <N>   # match the EFA-adapter count for the instance type (table above)
  requests:
    nvidia.com/gpu: 8
    vpc.amazonaws.com/efa: <N>
volumes:
- { name: dshm, emptyDir: { medium: Memory, sizeLimit: "10Gi" } }
volumeMounts:
- { name: dshm, mountPath: /dev/shm }

---

6. HyperPod node health labels (EKS)

HMA also writes a sagemaker.amazonaws.com/fault-details annotation on the node with the full JSON (timestamp, type, reason, message) — see the node-debugger skill § F.

NodeRecovery modes (per instance group): Automatic (replace failed nodes) or None (manual). Toggle via update-cluster — fetch the current instance-group spec first (describe-cluster), edit only NodeRecovery, push back.

---

7. Slurm — NCCL-specific operations

Diagnose (read-only):

sinfo -o "%10N %10T %10C %30E" --noheader
squeue -o "%10i %20j %8T %12R %N" --noheader
scontrol show node <NODE> | grep Reason

Suggested command — resume a DRAINING node (run this yourself)

Preconditions: the original drain reason no longer applies (the underlying issue — straggler bandwidth, hardware fault, RemoveIPC, etc. — has been investigated and resolved); the customer accepts that pending jobs may schedule onto this node immediately; you are running on the Slurm controller via SSM.

Command:

scontrol update nodename=<NODE> state=resume

Blast radius: node returns to the idle pool. Reversible by setting state=drain again. If the original cause is unfixed, the node will likely re-fail; resume only after a clean diagnostic.

Suggested command — disable RemoveIPC for NCCL persistence (run this yourself)

Preconditions: NCCL job is terminating with "unlink shared memory" or /dev/shm/nccl-* disappearing mid-training; confirmed that RemoveIPC=yes is set in /etc/systemd/logind.conf; node is quiescent or a brief systemd-logind restart is acceptable.

Command:

grep RemoveIPC /etc/systemd/logind.conf   # diagnose
echo "RemoveIPC=no" >> /etc/systemd/logind.conf
sudo systemctl restart systemd-logind

Blast radius: persistent change to the node's systemd configuration — logs out anyone in a systemd user session during the restart. Change survives reboot. For new nodes, add the same commands to the lifecycle script so the setting persists across replacements.

Slurm prolog for NCCL env

#!/bin/bash
# /etc/slurm/prolog.sh
export NCCL_SOCKET_IFNAME=^lo,docker
export FI_PROVIDER=efa
export FI_EFA_USE_DEVICE_RDMA=1
# Collective timeout is set in training code: init_process_group(timeout=timedelta(seconds=1800))
mount -o remount,size=10G /dev/shm 2>/dev/null || true

---

8. NCCL-specific remediations

Security group self-reference

Detected when: [FAIL] SG sg-xxx missing inbound/outbound self-reference — NCCL rendezvous or EFA RDMA blocked.

Root cause: EFA requires the SG to reference itself with AllTraffic (-1) on both ingress and egress. Without this, NCCL packets between nodes are dropped.

Suggested command — apply self-ref to every cluster SG (run this yourself)

Preconditions: the rule check (e.g. nccl-diagnose.sh Check 4 or hyperpod-node-debugger's check-efa-sg.sh) reports [FAIL] on inbound or outbound self-ref for <SG>; <SG> is one of the security groups attached to the HyperPod cluster (describe-cluster → VpcConfig.SecurityGroupIds); apply once per SG if multiple are attached; for IaC-managed SGs, see the operating-policy IaC note before running directly. Per the HyperPod prerequisites doc, do not add a 0.0.0.0/0 outbound rule on the EFA SG.

Command:

# Inbound self-ref (NCCL rendezvous)
aws ec2 authorize-security-group-ingress --group-id <SG> --region <R> \
  --ip-permissions '[{"IpProtocol":"-1","UserIdGroupPairs":[{"GroupId":"<SG>"}]}]'

# Outbound self-ref (EFA RDMA)
aws ec2 authorize-security-group-egress --group-id <SG> --region <R> \
  --ip-permissions '[{"IpProtocol":"-1","UserIdGroupPairs":[{"GroupId":"<SG>"}]}]'

Blast radius: opens all protocols between instances that share this SG (intended scope for intra-cluster EFA / NCCL). Idempotent: InvalidPermission.Duplicate = the rule already exists. Reversible with revoke-security-group-ingress/revoke-security-group-egress using the same --ip-permissions payload.

NetworkPolicy blocking NCCL

Detected when: [WARN] NetworkPolicies found in <ns> + a [FAIL] indicating blocked inter-pod NCCL traffic.

Before deleting any NetworkPolicy, read it — it may be intentional tenant isolation or compliance-required. Confirm with the customer.

kubectl get networkpolicy -n <NS> -o yaml

Allow-all intra-namespace policy for NCCL training namespaces:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-nccl-intranamespace
  namespace: <NS>
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: <NS> }
  egress:
    - to:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: <NS> }
    - ports:
        - { port: 53, protocol: UDP }

Suggested command — delete a blocking NetworkPolicy (run this yourself)

Preconditions: the policy has been read (kubectl get networkpolicy <NAME> -n <NS> -o yaml) and confirmed not to be intentional tenant isolation or compliance-required; customer has explicitly approved removal; a replacement allow-list policy (if needed) is already applied.

Command:

kubectl delete networkpolicy <NAME> -n <NS>

Blast radius: changes default-deny traffic rules for every pod matched by the policy's podSelector in namespace <NS>. Cannot be reverted by a single command — the original YAML must be re-applied. Misdiagnosis can expose production traffic.

Node reboot / replacement for GPU faults

Ordering and commands are in node-debugger: references/node-diagnostics-detail.md § F. Reboot first (clears transient GPU/EFA faults, preserves data); replace only if reboot doesn't clear the fault.

NCCL Performance Testing & Straggler Detection

Measure NCCL bandwidth and identify slow nodes.

---

Install nccl-tests (once per cluster)

# On each compute node (add to lifecycle script for persistence). Source: NVIDIA nccl-tests.
cd /opt && git clone <nccl-tests-source> nccl-tests
cd /opt/nccl-tests
make MPI=1 MPI_HOME=/usr/local/mpi NCCL_HOME=/usr/local/nccl CUDA_HOME=/usr/local/cuda
# Binary: /opt/nccl-tests/build/all_reduce_perf

---

Single-Node Baseline Test

Run first to confirm the node itself is healthy before multi-node tests.

# Single-GPU test (quick sanity check):
/opt/nccl-tests/build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

# All-GPU test (p4d: 8 GPUs, p5: 8 GPUs):
/opt/nccl-tests/build/all_reduce_perf -b 8 -e 8G -f 2 -g 8

# Expected output column headers:
# size  count  type  redop  root  time  algbw  busbw  error  time  algbw  busbw

How to identify stragglers: there is no single published GB/s threshold that applies across EFA generations, NCCL versions, and test message sizes. Run all_reduce_perf on every node against a known-good peer and compare the busbw (bus bandwidth) column. The outliers in the bottom quartile at the same message size are the stragglers. For reference workflow and exact test command, see the AWS EC2 EFA + NCCL getting-started doc. Also compare against the results of a recent known-good run on the same instance type and NCCL version — hardware generations differ widely and a static table rots quickly.

---

Multi-Node AllReduce Test

# With MPI (from head node):
mpirun -np <TOTAL_RANKS> \
  --hostfile /etc/hosts \
  -N <RANKS_PER_NODE> \
  -x FI_PROVIDER=efa \
  -x FI_EFA_USE_DEVICE_RDMA=1 \
  -x NCCL_SOCKET_IFNAME=^lo,docker,efa,veth \
  -x NCCL_DEBUG=WARN \
  /opt/nccl-tests/build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

# With Slurm:
srun --nodes=4 --ntasks-per-node=8 \
  /opt/nccl-tests/build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

# With kubectl (EKS, 2 nodes, 8 GPUs each):
# Deploy as a K8s Job with 2 pods, each requesting 8 GPUs.
# Use mpirun inside the container, or the Kubeflow MPI Operator.
kubectl exec -n <NS> <POD> -- mpirun -np 16 -N 8 \
  --hostfile /etc/hosts \
  -x FI_PROVIDER=efa -x FI_EFA_USE_DEVICE_RDMA=1 \
  /opt/nccl-tests/build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

---

Pairwise Bandwidth Test (identify slow pairs)

# Test each node pair individually to find the outlier:
# From node A → node B:
fi_ping -p efa -I 100 <NODE_B_IP>

# From node B → node A:
fi_ping -p efa -I 100 <NODE_A_IP>

# Automate across all pairs (run on head node):
for node in $(scontrol show hostnames $SLURM_JOB_NODELIST); do
    echo -n "Testing $node: "
    fi_ping -p efa -I 10 "$node" 2>/dev/null | tail -1 || echo "FAILED"
done

Interpreting fi_ping output:

• Normal: < 5 microseconds latency, consistent
• Straggler: > 50 microseconds, or high variance across runs

---

NCCL_DEBUG_FILE Analysis

# Enable per-rank debug files:
export NCCL_DEBUG=INFO
export NCCL_DEBUG_FILE=/tmp/nccl_rank${RANK}.log

# After training (or timeout), check which rank was slow:
# Look for the last "AllReduce" timestamp before the timeout:
grep -h "AllReduce\|ring\|timeout" /tmp/nccl_rank*.log | sort -k1,1 | tail -30

# Compare timestamps across ranks — the one furthest behind is the straggler:
for f in /tmp/nccl_rank*.log; do
    echo -n "$f: last line timestamp = "
    tail -1 "$f" | awk '{print $1, $2}'
done

---

Collective-op timeout scaling

PyTorch's init_process_group default timeout for NCCL is 10 minutes (600 s). Too low for large clusters — a slow rank or straggler can blow past 10 min during warm-up or a large all-gather.

Scale up via the timeout argument (NOT via a NCCL_TIMEOUT env var — that is not a standard NCCL or PyTorch variable):

import datetime
import torch.distributed as dist

# nodes * 5 + 600 is a simple heuristic — tune against your actual step time:
nodes = int(os.environ.get("WORLD_SIZE", "1")) // 8   # GPUs per node
timeout_s = nodes * 5 + 600

dist.init_process_group(
    backend="nccl",
    timeout=datetime.timedelta(seconds=timeout_s),
)

Field-observed starting points (not AWS- or PyTorch-prescribed; tune from your actual step time and slowest collective):

To surface hangs as Python exceptions instead of silently waiting, also set:

export TORCH_NCCL_ASYNC_ERROR_HANDLING=1
export TORCH_NCCL_BLOCKING_WAIT=1   # for debugging; has a perf cost at scale

---

NCCL_DEBUG=INFO Performance Impact

Never leave `NCCL_DEBUG=INFO` in production. The NCCL env-var reference describes TRACE as printing "replayable trace information on every call" but does not publish overhead percentages. Field experience on HyperPod is:

Use INFO / TRACE only for debugging, then set back to WARN. Measure your own overhead before and after if it matters for the workload.

---

EFA Performance Settings

# Full EFA performance configuration:
export FI_PROVIDER=efa
export FI_EFA_USE_DEVICE_RDMA=1    # GPU Direct RDMA
export NCCL_PROTO=Simple           # large-message protocol (valid: LL, LL128, Simple)
export NCCL_SOCKET_IFNAME=^lo,docker,efa,veth
# Collective timeout goes in training code: init_process_group(timeout=timedelta(seconds=1800))

# Optional tuning for very large jobs:
export FI_EFA_FORK_SAFE=1            # safe for multiprocessing
export FI_EFA_ENABLE_SHM_TRANSFER=1  # intra-node shared memory

# Do NOT set in production:
# NCCL_DEBUG=INFO  (verbose; runtime overhead — disable in production)
# CUDA_LAUNCH_BLOCKING=1  (disables GPU/CPU overlap, very slow)

---

Straggler Node — Detection and Replacement

Detection workflow

1. Run nccl-tests across all nodes — compare algbw values 2. Check nvidia-smi nvlink -e for NVLink error counters 3. Check dmesg for XID errors, hardware failures 4. Compare fi_ping latency pairwise — outlier has degraded EFA port

Replacement workflow

Diagnose (read-only):

# Identify the bad node's instance ID:
kubectl get node <NODE_NAME> -o jsonpath='{.spec.providerID}' | cut -d'/' -f5
# OR for Slurm — list-cluster-nodes does NOT return PrivateDnsHostname (only describe-cluster-node does).
# Two-step: list candidate IDs, then describe each one until DNS matches the Slurm name.
SLURM_NODE="<SLURM_NODE_NAME>"
for IID in $(aws sagemaker list-cluster-nodes --cluster-name <C> --region <R> \
               --query 'ClusterNodeSummaries[?InstanceStatus.Status==`Running`].InstanceId' --output text); do
  DNS=$(aws sagemaker describe-cluster-node --cluster-name <C> --region <R> --node-id "$IID" \
          --query 'NodeDetails.PrivateDnsHostname' --output text 2>/dev/null)
  case "$DNS" in "$SLURM_NODE."*) echo "$SLURM_NODE → $IID"; break ;; esac
done

Suggested command — drain the straggler node before reboot/replace (run this yourself)

Preconditions: straggler behavior confirmed across multiple nccl-tests runs (single-run outliers can be transient — don't drain on one bad sample); customer accepts that pods using emptyDir volumes on this node will lose that data when evicted (EKS path); on Slurm, customer accepts that no new jobs will be scheduled to the node until state=resume runs after recovery; drain is preparation for reboot/replace, not a fix on its own.

Command:

# EKS — cordon prevents new pods; drain evicts existing pods (emptyDir data lost).
kubectl cordon <NODE_NAME>
kubectl drain <NODE_NAME> --ignore-daemonsets --delete-emptydir-data

# Slurm — on the controller via SSM; running jobs continue until they finish.
scontrol update nodename=<NODE> state=drain reason="low-bandwidth-$(date +%Y%m%d)"

Blast radius: EKS — --delete-emptydir-data discards emptyDir scratch on this node; pods are rescheduled elsewhere if capacity exists, otherwise stay Pending. Slurm — running jobs finish on the node; pending jobs route around it. Drain is reversible (kubectl uncordon / scontrol update state=resume) only if you decide not to proceed with reboot/replace.

See hyperpod-cluster-debugger § G.2 for the reboot-before-replace ordering.

Suggested command — replace the node (run this yourself, only after reboot did not clear the fault)

Preconditions: reboot was tried first and did not clear the fault (see hyperpod-cluster-debugger § G.2). Data on root + secondary volumes is backed up. Not supported on Slurm controller nodes. NodeIds batch: 1-25 per call.

Command:

aws sagemaker batch-replace-cluster-nodes \
  --cluster-name <C> --region <R> \
  --node-ids '["<INSTANCE_ID>"]'

# Monitor replacement completion (read-only):
watch -n 10 "aws sagemaker list-cluster-nodes --cluster-name <C> --region <R> \
  --query 'ClusterNodeSummaries[*].{ID:InstanceId,State:InstanceStatus.Status}' \
  --output table"

Blast radius: destroys root + secondary volumes on the replaced instance — all data permanently lost. New hardware is provisioned with the same AMI.