In large AI clusters, GPU hardware faults are among the most common reasons training runs stop (e.g., 58.7% of unexpected interruptions in Meta’s Llama 3 training), and denser, hotter GPUs in ever-larger clusters only raise the risk. A single failing GPU can halt a job that spans thousands of other GPUs, forcing it to roll back to its last checkpoint and losing hours of compute. Most monitoring tools raise an alert only after the GPU has already failed, when the damage is done.
GPU Failure Prediction is part of Federator.ai Cortex, ProphetStor’s full-stack AIOps platform for AI factories. It watches the early signs that a GPU is degrading, predicts which GPUs are likely to fail, and acts before they do: moving their workloads to healthy hardware and keeping them out of new allocations until they are repaired.
How It Works
Federator.ai Cortex runs a three-stage pipeline on telemetry collected from every GPU through NVIDIA DCGM:
Detect
Continuously analyzes 19 telemetry signals per GPU for early signs of degradation, such as rising temperatures, accelerating memory errors, and erratic power draw.
Assess
Rates each GPU on thermal, memory, power, compute, and interconnect health, combines them into one health score, then predicts, 12 to 72 hours in advance, its failure probability, likely failure type, and time to failure.
Act
Martin FSD acts on the prediction with no person in the loop: it moves workloads off GPUs that cross the risk threshold and flags those GPUs so the scheduler no longer assigns new work to them.
Failure Modes Covered
Federator.ai Cortex tracks the failure modes that most often interrupt large-scale AI training, including:
- Thermal degradation: abnormal temperature rise and higher temperatures at the same power draw
- ECC memory errors: accelerating single-bit errors and double-bit error counts
- Power instability: power draw spikes and power-related throttling outside the normal envelope
- Clock degradation: a gradual decline in boost frequency
- NVLink degradation: rising link error rates and reduced bandwidth
- PCIe link errors: increasing replay counts and reduced link width
- XID and firmware errors: error codes reported by the NVIDIA GPU driver and faults in GPU firmware
What Teams Gain
Early Warning, Not Late Alarms
Federator.ai Cortex predicts GPU failures 12 to 72 hours before they happen, estimating each GPU’s likelihood of failure and time to failure, so operations teams can act while a GPU is still running instead of after it has taken a job down.
Cascading Failures Caught Early
GPUs in a rack share heat, so one hot GPU stresses its neighbors. Federator.ai Cortex follows how heat couples GPUs within a rack, so it predicts cascading failures across the rack, not just the failure of a single GPU.
Workloads Moved Before Failure
When a GPU crosses the risk threshold you set, Martin FSD acts on its own, with no operator needed, and kMotion live-migrates the GPU’s workloads to healthy hardware. Training continues on new GPUs instead of rolling back to the last checkpoint.
At-Risk GPUs Kept Out of Rotation
A GPU flagged as at risk is removed from allocatable capacity until it is repaired, replaced, or cleared by an operator. Adaptive Distributed Parallelism recomputes the training split across the remaining GPUs, so the run continues without manual re-planning.
Fleet Health at a Glance
A health score for every GPU, a heatmap by zone, and a fleet-wide view across power, thermal, memory, compute, and interconnect show where risk is building before it becomes downtime.
Predictions That Learn From Your Fleet
When an operator confirms what happened to a flagged GPU, that confirmation becomes ground truth for the scorer. Prediction precision and recall are measured against confirmed outcomes, so predictions are tuned to your own fleet over time.
Frequently Asked Questions
How does Federator.ai Cortex predict GPU failures?
Federator.ai Cortex analyzes 19 telemetry signals per GPU, collected through NVIDIA DCGM, looking for early signs of degradation in temperature, memory errors, power delivery, clock speed, and interconnect health. It estimates each GPU’s probability of failure, likely failure type, and expected time to failure, and follows how heat spreads between neighboring GPUs to predict cascading failures.
How far in advance can Federator.ai Cortex predict a GPU failure?
Federator.ai Cortex predicts GPU failures 12 to 72 hours before they happen, giving operations teams time to move workloads to healthy GPUs before a failure disrupts training.
Which GPU failure types can Federator.ai Cortex detect?
Federator.ai Cortex covers the failure modes that most often interrupt AI training, including thermal degradation, ECC memory errors, power instability, clock degradation, NVLink and PCIe link errors, and XID and firmware errors.
What happens to running workloads when a GPU is predicted to fail?
When a GPU crosses a risk threshold set in advance, Martin FSD acts automatically, with no person in the loop. Federator.ai Cortex uses kMotion to live-migrate its workloads to healthy GPUs, so training continues without rolling back to the last checkpoint.
Will new jobs be scheduled on a GPU that is predicted to fail?
No. Federator.ai Cortex flags at-risk GPUs and removes them from allocatable capacity, so new workloads are placed only on healthy GPUs until the flagged GPU is repaired or cleared.