In most AI data centers, the teams that run GPUs and the systems that cool them never talk to each other. The IT side schedules training and inference jobs on Kubernetes or Slurm. The OT side runs the liquid cooling plant: coolant distribution units (CDUs), pumps, and valves. Because they share no data, cooling reacts only after temperatures rise. Operators run pumps near maximum to stay safe, and GPUs still throttle when a large job ramps up faster than the cooling can follow.
The Unified IT and OT Closed-Loop System is part of Federator.ai Cortex, ProphetStor’s full-stack AIOps platform for AI factories. It turns IT/OT convergence into a single control loop, bringing workload data and liquid cooling data together. Federator.ai Cortex knows which jobs are about to run and how much heat they will produce, so it can adjust cooling and workload placement together instead of treating them as separate problems. It is software only and works with the CDUs already installed, through the equipment vendor’s own management interfaces.
How the Loop Closes
Sense
Collects IT telemetry, such as GPU temperature, clock speed and power, rack load, the scheduler’s job forecast, and Kubernetes metadata, alongside OT telemetry, such as CDU pump and valve states, coolant flow, supply and return temperatures, and ambient conditions, with readback on every command.
Predict
A physics model of how fast each chip heats for a given load and coolant flow, combined with the job forecast, predicts rack heat minutes ahead. Model Predictive Control (MPC) uses that forecast to set coolant flow and set-points before temperatures begin to climb.
Actuate
Federator.ai Cortex earns control in four stages: shadow (read-only), advisory (set-points approved by operators), bounded autonomy (automatic within a defined blast radius), and full autonomy. In the later stages, coolant flow and set-points follow the job forecast through the CDU vendor’s own interface.
Safety is layered. The CDU hardware and its existing controls stay in place, a PID safety layer responds in under 100 milliseconds, model predictive control plans ahead, and workload scheduling sits on top. If the predictive layer fails, control falls back to PID, and hardware interlocks on GPU temperature, coolant temperature, and flow remain active. Every command stays within the equipment vendor’s limits for pump speed and valve travel, and is issued only when leak, flow, and pressure readings are within safe range. Operators keep full authority over the physical plant.
What the Closed Loop Delivers
Workload-Aware Cooling
Federator.ai Cortex reads Kubernetes/Slurm job schedules and live GPU power, not just utilization, so cooling ramps up before a training job starts and backs off when real heat drops. Each rack gets the coolant flow it will need next, not what it needed a minute ago.
Right-Sized Coolant Flow
Fixed-flow systems run pumps near maximum even when racks sit idle, wasting energy without adding thermal margin. Federator.ai Cortex holds coolant temperature rise within a target window and trims flow to match real heat at low load, cutting pump energy by 25 to 30 percent.
Steady Clocks Under Load
GPU power can swing by up to 50 percent within seconds. A plant run by hand reacts after temperatures rise, so clocks drop: the clock you paid for is not the clock you get. Predictive control acts ahead of the heat, so clocks and throughput hold steady.
Thermal-Aware Scheduling
The loop runs in both directions. When a rack runs short of thermal headroom, Federator.ai Cortex steers new jobs toward racks with cooling capacity to spare, avoiding hotspots and wasted cooling. Placement weighs time, space, thermal margin, and power together.
Causal Root-Cause Analysis
With IT and OT data on one timeline, Federator.ai Cortex tells whether a hot GPU comes from a heavy job, a weakening pump, or a blocked coolant line. A physics model confirms which, so operators see a cause, not an alarm, and Martin FSD can respond.
Early Warning for Failing GPUs
Because the loop knows the workload and coolant conditions, it can spot a GPU running hotter than it should under the same load, an early sign of failure. GPU Failure Prediction in Federator.ai Cortex flags the GPU and moves work off it before the fault occurs.
How to Get Started
- Connect to Existing CDUs. Federator.ai Cortex connects to the CDUs and controllers you already have through Redfish, Modbus-TCP, BACnet/IP, SNMP, or vendor APIs. No new hardware is required.
- Start in Shadow Mode. Federator.ai Cortex reads telemetry only, learns how your cooling plant responds, and shows what it would have done, without changing any set-points.
- Earn Autonomy Step by Step. Operators approve recommended set-points in advisory mode, then allow automatic control within a defined blast radius, and finally full closed-loop control.
Turn More Power into Compute
In many AI data centers, power is now a tighter constraint than GPUs. Every watt saved on cooling is a watt available for training and inference. Because cooling is planned ahead of the load, the plant uses only the cooling it needs instead of running pumps near maximum. By closing the loop between IT and OT, Federator.ai Cortex helps keep PUE between 1.10 and 1.20 and puts more of each facility’s power budget to productive work.
Frequently Asked Questions
What is a unified IT and OT closed-loop system?
It is a control system that combines workload data from the IT side, such as GPU utilization and job schedules, with facility data from the OT side, such as coolant flow and temperatures. Federator.ai Cortex uses this combined view to adjust cooling and workload placement together in real time.
How is this different from AI cooling optimization that only monitors the facility?
How far ahead does Federator.ai Cortex predict heat load?
Federator.ai Cortex forecasts rack heat minutes ahead, using high-frequency GPU power telemetry, Kubernetes/Slurm job schedules, and ambient conditions. This lead time lets coolant pumps and valves ramp up before a workload spike arrives, so GPUs are already being cooled when the heat hits. Clock speeds stay clear of thermal throttling, and throughput remains steady even through sharp load changes.
Does Federator.ai Cortex replace the existing BMS or CDU controller?
No. Federator.ai Cortex works through existing controllers and building management systems. It earns control in four stages: shadow (read-only), advisory (set-points approved by operators), bounded autonomy, and full autonomy. At every stage, commands respect vendor safety limits, and operators retain full control of the physical plant.
What role does Multi-Layer Correlation play in the IT and OT closed loop?
Multi-Layer Correlation (U.S. Patent No. 11,579,933 B2) maps how changes in application workload cascade into infrastructure resource demand. This gives Federator.ai Cortex its workload awareness. Combined with IT and OT convergence, it makes Federator.ai Cortex both workload-aware and thermal-aware, so compute and cooling can be tuned together to boost AI factory performance.
Does Federator.ai Cortex require new hardware?
No. Federator.ai Cortex is software only. It runs on the CDUs and cooling plant already installed and connects through Redfish, Modbus-TCP, BACnet/IP, SNMP, or vendor APIs.
How does this relate to Federator.ai Smart Liquid Cooling™?
Federator.ai Smart Liquid Cooling™ provides predictive, workload-aware control of the liquid cooling plant. Within Federator.ai Cortex, it becomes part of the unified IT and OT closed loop, so cooling, scheduling, and operations share one data model.