GPU fleets are now heterogeneous, spanning several hardware generations, and older GPUs keep earning long after newer ones arrive. Training jobs have not caught up. Most are launched with a fixed mix of data, tensor, and pipeline parallelism, often paired with a sharded optimizer, chosen once by hand and tuned for a single GPU type. Give every GPU the same share of work, and the slowest one becomes a straggler that the fastest must wait for at every step. When the hardware, the network, or the facility changes, the job keeps running the same way and leaves performance on the table.
Adaptive Distributed Parallelism (ADP) is part of Federator.ai Cortex, ProphetStor’s full-stack AIOps platform for AI factories. It works with NVIDIA’s open-source KAI Scheduler to choose the right parallelism strategy for each job, place it on the right mix of GPUs, and reconfigure it as conditions change. The Megatron-LM integration executes that plan within the training framework, making heterogeneous GPU training practical, with every GPU generation in the fleet contributing to the same run.
How ADP Works with KAI Scheduler
KAI Scheduler, originally developed as part of NVIDIA Run:ai, is a Kubernetes-native scheduler for AI workloads. It manages queues, enforces fair sharing and quotas between teams, handles preemption, and gang-schedules the pods of a distributed job so they start together. ADP adds the layer above it. Federator.ai Cortex sizes each job’s resource envelope and decides where it should land based on GPU type, thermal margin, and power. ADP matches the job’s split to the GPUs it lands on, even when they span more than one generation. KAI Scheduler places the pods and keeps fairness inside that envelope. Neither overrides the other, and no NVIDIA tool needs to be removed.
What ADP Delivers
Native KAI Scheduler Integration
ADP works alongside NVIDIA’s open-source KAI Scheduler instead of replacing it. KAI Scheduler keeps handling queues, fairness, and gang scheduling inside the envelope Cortex sets, while ADP adds GPU-aware placement and parallelism decisions.
Parallelism Matched to Each Job
ADP picks a parallelism plan that fits each job and the GPUs it runs on, dividing the work across data-parallel ranks and pipeline stages, with sharded optimizers such as DeepSpeed ZeRO where needed. Launch scripts no longer need hand tuning.
Reconfiguration During Training
When conditions on the fleet change, such as a GPU being drained ahead of a predicted failure, ADP recomputes the split across the remaining GPUs. Failure-protected checkpointing in the Megatron-LM integration lets the run continue instead of stopping.If a workload must move, kMotion handles the live migration.
Training Across GPU Generations
One job can span H100, B200, GB300, and Vera Rubin GPUs. ADP weights each GPU by its capability, gives each data-parallel rank a proportional micro-batch, and splits pipeline stages unevenly so every stage finishes together. Faster GPUs are not held back, and older ones are not left idle.
Power- and Cooling-Aware Placement
Placement accounts for each rack’s power budget and thermal margin, drawn from the IT and OT telemetry in Federator.ai Cortex. Jobs land where the facility can sustain them at full speed, not just where GPUs are free.
A Plan You Can Read
Submit a job once, through the Intent Compiler or the Megatron-LM integration, without picking a GPU generation. ADP shows the micro-batch and pipeline split for each GPU, along with the capability it assumed for each device.
Put the Right GPU on the Right Workload
A GPU fleet is a long-term investment, and each generation should keep paying off as new hardware arrives. ADP draws on a catalog of 19 GPU generations and their variants, so a mixed fleet is split correctly, and builds on ProphetStor’s patented technologies, including Spatial-Temporal GPU Optimization (U.S. Patent No. 12,596,580 B2), to keep every generation productive.
Frequently Asked Questions
What is Adaptive Distributed Parallelism (ADP)?
ADP is a Federator.ai Cortex solution that selects and reconfigures the parallelism plan for each AI training job. It considers the GPU types available, each GPU’s health, and the power and cooling limits of the racks to decide how a job divides its work and where each part runs.
How does ADP work with NVIDIA KAI Scheduler?
KAI Scheduler is NVIDIA’s open-source, Kubernetes-native scheduler for AI workloads. It manages queues, fair sharing, quotas, and gang scheduling. ADP works above it: Federator.ai Cortex sets each job’s resource envelope and placement across GPU types, and KAI Scheduler keeps fairness inside that envelope. Jobs are not limited to a single GPU type, and no NVIDIA tool needs to be removed.
Can ADP run one training job across different GPU generations?
Yes. ADP is built for heterogeneous GPU training. It draws on a catalog of 19 GPU generations and their variants, so one job can span H100, B200, GB300, and Vera Rubin GPUs. Users submit the job without picking a GPU generation, and ADP shows the micro-batch and pipeline split it assigned to each GPU.
Which training frameworks does ADP support?
ADP works with NVIDIA NeMo Megatron, DeepSpeed (including ZeRO), Ray, and Alpa.
Can you train a single LLM on H100 and B200 GPUs together without the H100s bottlenecking the job?
Yes. In heterogeneous GPU training, giving every GPU the same share of work turns the H100s into stragglers: each training step waits for them while the B200s sit idle. ADP weights each GPU by its capability, giving H100 ranks smaller micro-batches and lighter pipeline stages than B200 ranks, so both finish each step together. The Megatron-LM integration applies this uneven split during training.