← Back to feed

NVIDIA’s GB200 NVL72 puts all 72 GPUs in a rack on a single NVLink fabric, making cross-rack communication the bottleneck and rack-local jobs the only workloads that get full interconnect bandwidth. Standard Kubernetes schedulers were built for 4- or 8-GPU nodes with PCIe. On NVL72 hardware, they produce systematic fragmentation that renders roughly 27% of the cluster’s GPUs unallocatable.

SkyPilot, which manages tens of thousands of customer GPUs on production NVL72 clusters, published the arithmetic on October 29.

The reliability problem

The fault blast radius scales with slice size. A single 4-GPU node has a mean time between interruptions (MTBI) of roughly 490 days. A full 18-node NVL72 rack, where any node failure can pull the entire rack out of service, has an MTBI of 490 ÷ 18 ≈ 27 days. On a 128-rack cluster, that produces a downed rack every 5 hours if full-rack job allocation is used.

SkyPilot measured 2% node unavailability on production NVL72 clusters. At that rate, NVIDIA’s own capacity model shows how different slice sizes compare on a 128-rack (2,304-node) cluster:

Slice size	Usable nodes (at 2% unavailability)
18 nodes (full rack)	1,602
16 nodes	2,037
9 nodes	1,963
8 nodes	~2,040

The 16-node slice beats 18-node by 435 nodes — 27% more usable capacity from the same physical hardware. It also beats 9-node slices by 4%, while giving each job twice as many GPUs per NVLink domain. The break-even point with 8-node slices occurs around 3.3% unavailability.

Sixteen GPUs also maps cleanly to power-of-two parallelism groups, the default sizing for tensor and pipeline parallel AI workloads.

What breaks without topology-awareness

When a naive scheduler places a multi-node job across racks, the job loses NVLink for its inter-node communication and falls back to InfiniBand or Ethernet. For tensor-parallel all-reduce operations, this means the slowest link in the cluster sits on the critical path. The speedup from NVLink disappears entirely.

Fragmentation compounds over time: as jobs of different sizes complete, available nodes scatter across racks in patterns no large job can use.

SkyPilot’s three-part response

Topology-Aware Scheduling keeps multi-node jobs within the same NVLink domain by enforcing rack affinity at placement time.

Frontier Trainer — SkyPilot’s hot-spare mechanism — replaces a failed node within a running job from a warm spare in the same rack. A single node failure no longer terminates the job.

Automatic Defragmentation consolidates scattered free nodes back into usable contiguous slices without requiring manual intervention.

The result is that two spare nodes per rack absorb most single-node failures at the cost of those two nodes sitting idle when everything is healthy — an exchange SkyPilot says nets positive at production unavailability rates.