Large training jobs can fail even when every node looks healthy. A GPU with a subtle fault can pass its stress tests and still return wrong values. When a job stalls on an NCCL timeout, it keeps its GPUs, and Slurm still shows it as running. If a device plugin crashes on one node, the job fails with an error that never mentions the plugin. Teams usually find these problems only after the loss diverges or the job fails, and by then the compute is spent. Each of these failures lowers goodput, the share of your cluster’s time that goes to useful training. Today, we’re introducing workload recovery and infrastructure checks on Amazon SageMaker HyperPod that catch these failures sooner and raise the goodput of your training workloads.
SageMaker HyperPod already handles nodes that report hardware faults. The health monitoring agent (HMA) watches each node for GPU errors, deep health checks test nodes during provisioning, and automatic node recovery reboots or replaces faulty hardware. These features look at one node’s hardware at a time. They can’t tell whether the GPUs train correctly, whether a job is still making progress, or whether the Kubernetes components a job depends on are working.
In this post, we walk through the three new capabilities that catch these failures. For each one, we show how to set it up and what it reports when it finds a problem:
MultiNodeTrainingValidationis a new deep health check for HyperPod clusters orchestrated by Amazon Elastic Kubernetes Service (Amazon EKS). When HyperPod provisions an instance, or when you run the check on demand, it trains a small model on the instance and then across instances of the same type. The instance fails if the loss becomes NaN or infinite, or training doesn’t finish in time.- The HyperPod Kubernetes Resource Monitoring Agent watches the Kubernetes layer of an EKS cluster for problems such as crash-looping device plugins, missing DaemonSets, and tainted nodes. When a job fails, it reports the pods and nodes involved as Kubernetes events and Amazon CloudWatch logs.
- Hung job detection finds Slurm jobs that have stopped making progress. An agent on each node watches the job’s output and GPU activity, or log patterns that you define. It writes an event to CloudWatch and can cancel the job so that Slurm requeues it.
Why node health alone isn’t enough
The existing HyperPod resiliency features check a single node at a time. The health monitoring agent (HMA) watches each node for GPU errors such as Xid and NVLink faults, Elastic Fabric Adapter (EFA) errors, and kernel errors. The InstanceStress deep health check runs DCGM level 4 diagnostics and tests each instance’s GPUs, GPU memory, NVLink, and EFA devices. The InstanceConnectivity check runs NCCL tests across GPU instances, or NCCOM tests on AWS Trainium. Together, these confirm that each node’s hardware works. They don’t show whether the GPUs compute correctly, whether a job is still making progress, or whether the cluster’s Kubernetes components are healthy. Three kinds of failure get past them:
- Numerical faults – A GPU can pass every hardware test and still compute wrong results. The first sign is often NaN or infinite loss in a production run. Meta attributed 6 of the 419 unexpected interruptions in its 54-day Llama 3 405B pre-training run to silent data corruption (The Llama 3 Herd of Models).
- Hung jobs – An NCCL timeout, a stalled data pipeline, or a deadlock can stop a job without ending it. The job keeps its GPUs and their memory, and Slurm reports it as running until someone checks the loss curve or the log.
- Cluster-level failures – Some failures happen in Kubernetes, not in a node’s hardware. A device plugin crash-loops, a required DaemonSet never starts on a new node, or ranks on different nodes fall out of step in an NCCL collective. HMA checks each node on its own, so it doesn’t see these problems. When the job fails, the error often doesn’t say which pod or node caused it.
Solution overview
Each capability covers a different point in a training run. MultiNodeTrainingValidation runs when instances are provisioned, or on demand, and confirms that their GPUs train correctly. Hung job detection runs alongside each Slurm job and catches a job that has stopped making progress. The HyperPod Kubernetes Resource Monitoring Agent watches the cluster continuously and, when a job fails, reports the pods and nodes that were involved. The following table summarizes the three.
| Capability | What it catches | When it runs | Available on |
MultiNodeTrainingValidation deep health check |
NaN or infinite loss, or training that doesn’t finish in time, on GPUs that pass hardware checks | When instances are created or updated, or on demand | EKS clusters, on supported EFA-enabled NVIDIA GPU instances |
| HyperPod Kubernetes Resource Monitoring Agent | Node condition changes, pod and DaemonSet failures, and failed training jobs, with the pods and nodes involved. NCCL communication state for jobs that opt in | Continuously | EKS clusters |
| Hung job detection | Jobs that stop making progress but don’t exit | While each job runs | Slurm clusters |
Hung job detection is new for Slurm clusters. On EKS clusters, HyperPod already detects hung jobs that run through the HyperPod training operator, and Ray Train jobs get hang detection by default. We describe both options in the hung job detection section.
Each capability targets a different source of lost goodput. Scheduling goodput is the share of time that all the resources a job needs are available. Runtime goodput is the share of that time in which the job makes forward progress. Program goodput is the share of the hardware’s peak performance that the job gets. The following table shows the effect of each capability.
| Capability | Scheduling goodput | Runtime goodput | Program goodput |
MultiNodeTrainingValidation |
Instances that fail the training test are labeled and, with automatic node recovery, replaced, so jobs don’t start on hardware that produces invalid output | Fewer production runs lost to NaN or infinite loss from a faulty GPU | Flags instances whose training doesn’t finish within the time limit |
| HyperPod Kubernetes Resource Monitoring Agent | Reports nodes that lose GPUs or EFA devices, missing device plugins, and taints, so you can restore capacity sooner | Names the pods and nodes behind a failed job, so you find the cause and restart sooner | No direct effect |
| Hung job detection | No direct effect | Flags a hung job in minutes instead of hours, and can cancel it so Slurm requeues it | No direct effect |
Consider a pre-training job running on 32 ml.p5.48xlarge instances (256 NVIDIA H100 GPUs). If the job hangs at 1:00 AM and no one notices until 9:00 AM, it wastes 256 instance-hours or 2,048 GPU-hours. At the SageMaker HyperPod on-demand price of approximately $66 per instance-hour in US East (N. Virginia), the hang costs about $16,900. With a 15-minute hung job detection rule, HyperPod cancels the stalled job and Slurm requeues it. The hang costs about $530 instead, saving approximately $16,400.
Validate training correctness with the MultiNodeTrainingValidation deep health check
The MultiNodeTrainingValidation deep health check trains a fixed, known-good workload, first on each instance and then across instances. It catches GPUs that pass InstanceStress and InstanceConnectivity but produce NaN or infinite loss once they train together. If an instance fails, HyperPod labels it as failed and, with automatic node recovery turned on, reboots or replaces it. When the check runs during provisioning, this happens before your jobs can use the instance.
How it works
MultiNodeTrainingValidation adds two tests to the deep health check run:
- Tiny Megatron test: A short Megatron-LM training run on all GPUs of each instance.
- Megatron test: The same workload across instances of the same type, communicating over EFA.
Either test fails an instance if the training loss becomes NaN or infinite, or if training doesn’t finish within the time limit. The workload is fixed, so a failure points to the hardware rather than to a model or code change.
The following table shows how it fits with the other deep health checks.
| Check | What it runs | What it catches |
InstanceStress |
GPU and NVLink count, DCGM level 4 diagnostics, and EFA latency and bandwidth tests (Neuron tests on Trainium) | Hardware that’s missing, failing, or unstable |
InstanceConnectivity |
NCCL tests across GPU instances (NCCOM on Trainium) | Slow or broken communication between instances |
MultiNodeTrainingValidation |
Tiny Megatron test on each instance, Megatron test across instances | GPUs that run but train incorrectly: NaN or infinite loss, or training that doesn’t finish in time |
Prerequisites
You need the following:
- An Amazon SageMaker HyperPod cluster orchestrated by Amazon EKS.
- An instance group of a supported EFA-enabled NVIDIA GPU instance type, such as ml.p5.48xlarge. The Megatron test runs across instances of the same type.
Turn on MultiNodeTrainingValidation for an instance group
HyperPod runs deep health checks when it creates or updates instances. To include MultiNodeTrainingValidation, add it to OnStartDeepHealthChecks in the instance group configuration that you pass to create-cluster or update-cluster. We turned on all three checks:
{
"InstanceGroupName": "<instance-group>",
"InstanceType": "ml.p5.48xlarge",
"InstanceCount": 2,
"LifeCycleConfig": {
"SourceS3Uri": "s3://<bucket>/<prefix>",
"OnCreate": "on_create.sh"
},
"ExecutionRole": "arn:aws:iam::<account-id>:role/<hyperpod-execution-role>",
"OnStartDeepHealthChecks": [
"InstanceStress",
"InstanceConnectivity",
"MultiNodeTrainingValidation"
]
}
To confirm the setting, run the following command:
aws sagemaker describe-cluster \
--cluster-name <cluster-name> \
--query "InstanceGroups[?InstanceGroupName=='<instance-group>'].OnStartDeepHealthChecks"
The output lists the three checks:
[
[
"InstanceStress",
"InstanceConnectivity",
"MultiNodeTrainingValidation"
]
]
Run MultiNodeTrainingValidation on demand
You can also check instances that are already in the cluster, for example before a long pre-training run. Use the StartClusterHealthCheck API:
aws sagemaker start-cluster-health-check \
--cluster-name <cluster-name> \
--deep-health-check-configurations \
'[{"InstanceGroupName": "<instance-group>", "DeepHealthChecks": ["MultiNodeTrainingValidation"]}]'
To check specific instances instead of the whole group, add "InstanceIds": [...] to the configuration.
Check the results
HyperPod writes one line per test to the DeepHealthCheckResults/<id> log stream in your cluster’s CloudWatch log group, /aws/sagemaker/Clusters/<cluster-name>/<cluster-id>. Here’s the stream from our run on two ml.p5.48xlarge instances. TINY_MEGATRON and MEGATRON are the two MultiNodeTrainingValidation tests:
2026-11-11T10:58:33Z Deep health check configuration validation passed
2026-11-11T11:01:41Z 2 out of 2 nodes succeeded HARDWARE_CHECK test. Instance IDs: [i-0a1b2c3d4e5f60001 i-0a1b2c3d4e5f60002]
2026-11-11T12:16:07Z 2 out of 2 nodes succeeded DCGM test. Instance IDs: [i-0a1b2c3d4e5f60001 i-0a1b2c3d4e5f60002]
2026-11-11T12:17:14Z 2 out of 2 nodes succeeded EFA test. Instance IDs: [i-0a1b2c3d4e5f60001 i-0a1b2c3d4e5f60002]
2026-11-11T12:21:16Z 2 out of 2 nodes succeeded TINY_MEGATRON test. Instance IDs: [i-0a1b2c3d4e5f60001 i-0a1b2c3d4e5f60002]
2026-11-11T12:22:25Z 0 nodes failed NCCL test at batch size 2. Instance IDs: []
2026-11-11T12:22:29Z 2 out of 2 nodes succeeded NCCL test. Instance IDs: [i-0a1b2c3d4e5f60001 i-0a1b2c3d4e5f60002]
2026-11-11T12:26:31Z 0 nodes failed MEGATRON test at batch size 2. Instance IDs: []
2026-11-11T12:26:31Z 2 out of 2 nodes succeeded MEGATRON test. Instance IDs: [i-0a1b2c3d4e5f60001 i-0a1b2c3d4e5f60002]
When the run finishes, each instance carries the result as a node label:
kubectl get nodes \
-l sagemaker.amazonaws.com/instance-group-name=<instance-group> \
-L sagemaker.amazonaws.com/deep-health-check-status
NAME STATUS ROLES AGE VERSION DEEP-HEALTH-CHECK-STATUS
hyperpod-i-0a1b2c3d4e5f60001 Ready <none> 120m v1.34.2-eks-ecaa3a6 Passed
hyperpod-i-0a1b2c3d4e5f60002 Ready <none> 120m v1.34.2-eks-ecaa3a6 Passed
For the training details, read /var/log/aws/clusters/sagemaker-deep-health-check.log on the instance, for example through an AWS Systems Manager session. On the instance that ran rank 0, the log ends with a summary of the Megatron test:
CONVERGENCE TEST PASSED
Completed 300 training iterations in 161.5s
Training forward steps: 300
Evaluation forward steps: 20
Total forward steps: 320
Final training loss: 3.6213
Initial training loss: 11.5671
Loss reduction: 7.9458
Training iterations per second: 1.86
The other instance’s log has a line for every iteration, with the loss and a running count of NaN iterations:
iteration 300/ 300 | consumed samples: 9600 | elapsed time per iteration (ms): 509.5 |
learning rate: 1.000E-04 | global batch size: 32 | lm loss: 3.397480E+00 | loss scale: 1.0 |
grad norm: 1.386 | number of skipped iterations: 0 | number of nan iterations: 0 |
In our run, the loss fell from 11.57 to 3.62 over 300 iterations, with no NaN or skipped iterations, at about 507 milliseconds per iteration on 16 H100 GPUs.
How long MultiNodeTrainingValidation takes
On our two-node cluster, the full deep health check took 1 hour 28 minutes after the instances finished their lifecycle scripts. DCGM level 4 diagnostics took most of that. The two MultiNodeTrainingValidation tests added about 8 minutes. Instances can’t take workloads until the checks finish, so plan for this additional time when you add capacity.
| Test | Check | Time since previous result |
HARDWARE_CHECK |
InstanceStress |
3 min |
| DCGM (level 4) | InstanceStress |
74 min |
| EFA | InstanceStress |
1 min |
TINY_MEGATRON |
MultiNodeTrainingValidation |
4 min |
| NCCL | InstanceConnectivity |
1 min |
MEGATRON (2 instances) |
MultiNodeTrainingValidation |
4 min |
Keep in mind that MultiNodeTrainingValidation tests the hardware with a fixed reference workload. A pass means the instances trained that workload correctly. It doesn’t guarantee that your own job will converge. We recommend running it on new capacity, on replacement instances, and before long pre-training runs.
Catch cluster-level failures with the HyperPod Kubernetes Resource Monitoring Agent
HMA watches the hardware on each node, but many failures that stop a training job happen in Kubernetes. A device plugin can crash-loop until its node stops offering GPUs to the scheduler, or a taint can evict a job’s pods. The job error rarely names the pod or node involved, so engineers piece together what happened from kubectl output, per-rank logs, and timestamps.
The HyperPod Kubernetes Resource Monitoring Agent watches these resources from one place in the cluster. When something breaks, it tells you what changed and which nodes, pods, and jobs are affected. It reports through Kubernetes events and structured CloudWatch logs. The agent only reports. It doesn’t reboot nodes or restart jobs, because the cause might be in the workload. HMA and HyperPod node recovery still handle hardware faults, and the HyperPod training operator still handles job restarts.
How the agent works
The agent runs as a single pod per cluster in the aws-hyperpod namespace and prefers CPU nodes, so it doesn’t use GPU capacity. It watches the foll