Skip to the yard
Waxsealai

Lambda Deploys NVIDIA Photonics Switches Across a 10,000-GPU AI Cluster

Shipped
Shipper
Brian Sparks
Readers
941

Lambda reports lower network latency and cooler optics with NVIDIA Photonics switches, while keeping throughput in line with conventional optics.

Lambda Deploys NVIDIA Photonics Switches Across a 10,000-GPU AI Cluster

Lambda is connecting more than 10,000 GPUs in NVIDIA GB300 NVL72 racks with switches that place optical engines alongside the switch chip. The deployment puts NVIDIA Photonics co-packaged optics into a large AI cluster, where the network’s power use, reliability and latency can affect how much work the GPUs complete.

The reported results point to targeted gains rather than a change in network throughput. Lambda says its tests found latency about 4% lower at rack scale and up to about 10% lower at pod scale. Across rack, scalable unit and pod configurations, the network maintained the same throughput as conventional pluggable optics. That distinction matters: the approach aims to move data with less delay and lower network overhead while preserving the data rate.

Why the network matters to cluster performance

Large training jobs divide work across many GPUs, which exchange data through the network. If a link is interrupted, other parts of the job may have to wait for communication to resume. Lambda describes a single interruption as capable of stalling thousands of GPUs in a large training run. In agentic inference, where multiple agents exchange information, a disruption can spread latency across their interactions.

Those examples explain why network reliability is a measure of useful compute capacity, not just an equipment specification. Time spent waiting on communication is time GPUs cannot spend on the task. Power has a similar tradeoff: electricity used by networking equipment is unavailable to the compute systems it connects. The practical value of a faster or more efficient switch depends on how it performs across a full cluster, not only on the headline rate of one port.

Lambda says the deployment is intended to improve power efficiency and resiliency at scale. Its president of cloud services, Dave Ward, said networking must keep pace with larger AI clusters and described the use of co-packaged optics at this scale as a step toward infrastructure for future AI workloads.

What co-packaged optics changes

Conventional pluggable optics use transceiver modules that connect to the switch. NVIDIA Photonics instead places optical engines on the same package as the switch chip. The engines convert electrical signals into light, which carries data over fiber between parts of the cluster. Bringing that conversion closer to the chip shortens the electrical path from inches to millimeters, according to the deployment details.

Shorter electrical paths reduce signal loss. The design also removes the need for retimer chips, which otherwise help restore signals along the path. Fewer retimers mean lower power draw and fewer components in the signal chain. NVIDIA says Photonics offers up to five times better power efficiency and up to 15 times higher reliability than conventional pluggable optics. These are stated maximum comparisons; the deployment details do not specify the test conditions behind those figures.

The architecture also uses four times fewer lasers, reducing the number of light sources that could fail. Each Lambda switch has 18 removable external light sources that feed the optical engines. Technicians can replace those sources in the field. With conventional pluggable optics, a failed laser requires replacing the entire module, according to the deployment description. That service difference may affect repair work: replacing a light source can leave the optical engine and other parts of the module in place.

Cooling, temperature and reported results

Each switch in Lambda’s deployment has 144 ports operating at 800 Gb/s. Liquid cooling removes heat from the switch chip and optical engines. Lambda reports a median optics temperature of 36°C, compared with 51°C for conventional pluggable optics, and says its tests found more stable signal integrity.

Temperature and signal stability are useful operating measures, but the reported figures do not establish how the system behaves under every workload or over a particular service period. The available details also don’t provide absolute latency values, total power consumption, the size of the test runs or a breakdown of reliability results. Those measurements would help operators compare the design against their own network requirements and estimate the effect on facility power and maintenance.

For now, the clearest evidence is the combination Lambda reports: throughput matching conventional pluggable optics across several cluster configurations, with lower measured latency and cooler optics. The deployment shows how co-packaged optics can change the physical signal path and field servicing model inside a large AI network. Its wider operational value will depend on how those gains hold up across sustained workloads, repairs and the full cost of running the system.

Source: Brian Sparks · blogs.nvidia.com

Same vesselRelated cargo