The Network is
the Computer
Why interconnect is the hidden bottleneck of AI
agent37.club
“Future data centers of all kinds will be built like high-performance computers.”
— Jensen Huang, March 2019 (announcing the $6.9B Mellanox acquisition)
In 2019, most saw NVIDIA as a GPU company. Huang saw that a GPU is a component — the network is what turns thousands of GPUs into one computer. This is the most important insight in AI infrastructure.
The 36× Bandwidth Cliff
Scale-up vs scale-out — the fundamental tension
NVLink 5
1,800 GB/s per GPU
Scale-up (NVLink)
GPU-to-GPU within a rack. 1,800 GB/s on Blackwell. Fast enough for tensor parallelism — splitting individual layers across GPUs.
Scale-out (InfiniBand/Ethernet)
Node-to-node across the data center. 50 GB/s per link. Every parallelism strategy is designed to minimize crossing this boundary.
Scale-Up:
NVLink & NVSwitch
Turning GPUs into one giant processor
NVLink — GPU-to-GPU at Warp Speed
| Generation |
Platform |
Bandwidth/GPU |
Max GPU Domain |
Year |
| NVLink 3.0 |
Ampere (A100) |
600 GB/s |
8 |
2020 |
| NVLink 4.0 |
Hopper (H100) |
900 GB/s |
8 |
2022 |
| NVLink 5.0 |
Blackwell (B200) |
1,800 GB/s |
72 |
2024 |
| NVLink 6.0 |
Rubin (R200) |
3,600 GB/s |
72+ |
H2 2026 |
The leap to Blackwell: Not just 2× bandwidth — the NVLink domain grew from 8 GPUs to 72 GPUs. An entire rack communicates as if it were one massive processor.
GB200 NVL72 — The AI Superchip Rack
What’s in the rack
- 72 Blackwell GPUs + 36 Grace CPUs
- 13.5 TB HBM3e total memory
- 130 TB/s aggregate NVLink bandwidth
- 9 NVSwitch trays, 18 NVLink Switch chips
- Every GPU ↔ every GPU at full 1.8 TB/s — non-blocking
- Liquid-cooled: ~120 kW per rack
A single NVL72 rack: ~1,440 PFLOPS FP4 inference. 25× more performance vs H100 at the same power.
Why 72 matters
- Trillion-parameter models fit within one NVLink domain
- No need to cross the slow scale-out network for model parallelism
- Each B200 has 18 NVLink ports × 100 GB/s each
Next gen: Vera Rubin NVL72
260 TB/s aggregate bandwidth. 3.6 TB/s per GPU. HBM4 at 22 TB/s. Up to 50 PFLOPS per rack. Arriving H2 2026.
The Interconnect Hierarchy
Bandwidth drops at every boundary
HBM3e (on-chip memory)
8 TB/s
NVLink 5 (intra-rack)
1.8 TB/s per GPU
PCIe Gen5 x16
128 GB/s
InfiniBand NDR (per NIC)
50 GB/s
8× IB NDR (per node)
400 GB/s total
Every parallelism strategy in AI training is designed to keep the heaviest communication as high in this stack as possible.
Scale-Out:
The Data Center Fabric
InfiniBand, Ethernet, and the battle for the backbone
InfiniBand — The Gold Standard
| Standard |
Per-Link Speed |
Era |
| HDR |
200 Gbps |
2019 |
| NDR |
400 Gbps |
2022 (H100) |
| XDR |
800 Gbps |
2024 (Blackwell) |
- Sub-2 microsecond latency
- Zero packet loss (credit-based flow control)
- RDMA: GPUs talk to remote GPUs directly, bypassing CPUs
- SHARP: collective ops computed inside the switch
Per-node bandwidth
8-GPU server with 8 NICs:
8 × 400 Gbps = 3.2 Tbps per node (NDR)
At XDR speeds:
8 × 800 Gbps = 6.4 Tbps per node
Microsoft Eagle
14,400 H100s, InfiniBand NDR, three-tier fat-tree. 200,000+ km of cabling — enough to wrap around Earth 5×. #3 on Top500 at 561 PFLOPS.
InfiniBand vs Ethernet
The great debate
InfiniBand
- Purpose-built for HPC/AI — zero packet loss
- Sub-2 µs latency, credit-based flow control
- SHARP in-network computing (All-Reduce in switches)
- Single vendor (NVIDIA) — fully optimized stack
- 2.3× more expensive than Ethernet
- ~15% raw performance advantage
Ethernet (RoCE / Spectrum-X)
- Ubiquitous — familiar ops, multi-vendor
- Standard RoCE: only 60% throughput at scale (flow collisions)
- NVIDIA Spectrum-X: 95% throughput with adaptive routing
- Lower cost, existing operational expertise
- Meta proved Ethernet can match IB with tuning
The market shift: Late 2023, InfiniBand held >80% of AI back-end networking. By mid-2025, Ethernet overtook InfiniBand in total AI deployments (Dell’Oro Group). Projected: 91% Ethernet by 2029.
Spectrum-X — NVIDIA’s Ethernet Play
- NVIDIA’s answer to “what if customers want Ethernet?”
- ConnectX-7/8 SuperNICs + Spectrum SN5600 switches
- SN5600: 51.2 Tbps, 64 × 800GbE ports
- Adaptive routing eliminates congestion hotspots
- Direct Data Placement — data straight to GPU memory
$10B+ revenue
Spectrum-X surpassed $10 billion annualized revenue by late 2025. NVIDIA wins whether the market picks InfiniBand or Ethernet.
xAI Colossus validation
100,000 H100s connected via Spectrum-X. World’s largest Ethernet AI cluster: 95% data throughput, zero packet loss.
Playing both sides
Quantum InfiniBand for maximum performance. Spectrum-X Ethernet for broader market. The Mellanox acquisition gave NVIDIA a monopoly on both paradigms.
Ultra Ethernet Consortium
The industry’s answer to NVIDIA’s networking lock-in
- Launched 2023 under the Linux Foundation
- 9 founders: AMD, Arista, Broadcom, Cisco, Eviden, HPE, Intel, Meta, Microsoft
- Notably absent: NVIDIA
- 55+ members by 2026
UEC 1.0 Spec (June 2025)
- Ultra Ethernet Transport (UET) protocol
- Programmable congestion management
- Standardized AI/HPC telemetry
First products shipping
AMD Pensando Pollara 400 AI NIC deployed at Oracle Cloud. Full-stack UEC hardware shipping late 2025 / early 2026.
Threat assessment
Long-term: Real threat. Open ecosystem + multi-vendor = structural advantage.
Short-term: InfiniBand still dominates for >4K GPU training. UEC interop still maturing.
The $6.9B Bet — Mellanox
NVIDIA’s most important acquisition
March 2019 — NVIDIA announces $6.9B acquisition of Mellanox — largest deal ever (previous max: ~$400M)
April 2020 — Deal closes. NVIDIA acquires InfiniBand, ConnectX NICs, 4,000+ engineers (mostly Israel)
2024 — Networking generates more revenue per quarter than the entire $6.9B purchase price
FY2026 — Networking projected at ~$29 billion annual revenue
The full-stack moat
NVIDIA now controls:
- GPU (compute)
- NVLink (scale-up)
- InfiniBand / Ethernet (scale-out)
- DPU (infrastructure offload)
Competitors assemble multi-vendor stacks. NVIDIA offers one optimized solution. This is why the UEC exists.
NVIDIA’s Networking Arsenal
| Product |
Type |
Speed |
Role |
| ConnectX-7 |
SuperNIC |
400 Gbps |
RDMA NIC in every GPU server. SHARP collective offload. |
| ConnectX-8 |
SuperNIC |
800 Gbps |
Integrated PCIe Gen6. Doubles bandwidth per GPU. (2025) |
| BlueField-3 |
DPU |
400 Gbps |
Offloads networking/storage/security. Used in xAI Colossus. |
| BlueField-4 |
DPU |
800 Gbps |
64 Grace Arm cores. 6× compute vs BF-3. (Vera Rubin era) |
| Quantum-2 |
IB Switch |
51.2 Tbps |
64× NDR 400G. 57B transistors. SHARP in-network compute. |
| Spectrum SN5600 |
Eth Switch |
51.2 Tbps |
64× 800GbE. Adaptive routing + congestion control. |
Vera Rubin platform (H2 2026) ships six new chips: Vera CPU, Rubin GPU, NVLink 6 Switch, ConnectX-9, BlueField-4, Spectrum-6.
Co-Packaged Optics — The Next Leap
When copper hits the wall, bring light onto the chip
- The problem: Pluggable optics consume ~45% of networking power — and 200G/lane signals can’t travel reliably over copper
- The solution: Integrate photonic engines directly on the switch ASIC package — convert electrical to light at the chip edge
- Power savings: 65-73% reduction per port (~15W → ~5W at 800G)
- Density: 10× shoreline bandwidth (1-2 Tbps/mm vs 0.1 copper)
- The wall: 204.8 Tbps switches are physically impossible without silicon photonics
| Company | Product | Speed |
|---|
| Broadcom | Davisson (TH6) | 102.4 Tbps |
| NVIDIA | Spectrum-X Photonics | 409.6 Tbps |
| Marvell | Nova + Teralynx 10 | 51.2 Tbps |
| Intel | OCI Chiplet | 4 Tbps |
Market trajectory
~$470M (2025) → $20B+ by 2036 at 37% CAGR. Broadcom shipping today; NVIDIA entering 2026. Pluggable transceiver makers face structural disruption.
Real Clusters
How the world’s largest AI systems are actually networked
Meta & xAI — Ethernet at Scale
Meta: 24,576 H100s
- Two identical clusters, two different fabrics
- Cluster 1: RoCE Ethernet
- Cluster 2: InfiniBand NDR
- Both multi-level fat-tree topology
- Tuned both to “equivalent performance”
- Llama 3 405B trained on the RoCE cluster
xAI Colossus: 100K H100s
- Built in 122 days
- NVIDIA Spectrum-X + BlueField-3 SuperNICs
- One 400GbE NIC per GPU + one extra per server
- 3.6 Tbps per HGX server
- 95% throughput, zero packet loss
- Plans to double to 200K GPUs
Both Meta and xAI proved that Ethernet can work for the largest training clusters. This was the inflection point that shifted the market away from InfiniBand.
Google — Custom Everything
TPU v5p Pod (2023)
- 8,960 TPU chips per superpod
- 3D torus: 16 × 20 × 28
- 4,800 Gbps ICI per chip
- 48 Optical Circuit Switches
- 4.45 ExaFLOPS per superpod
Google uses neither InfiniBand nor Ethernet for TPU interconnect. Entirely proprietary ICI.
Ironwood / TPU v7 (2025)
- 9,216 chips per superpod
- 9.6 Tbps (1.2 TB/s) ICI per chip
- 144 cubes of 4×4×4
- OCS inter-cube, DAC cables intra-cube
- Sub-10 ns OCS switching
- 50× less downtime via ICI resiliency
Google’s bet: own the full vertical stack. Custom chips, custom interconnect, custom topology. Zero NVIDIA dependency.
Parallelism & The Network
Which strategies stress the network most?
| Parallelism |
Splits |
Communication |
Network Need |
Runs Over |
| Tensor Parallel |
Layers/matrices |
All-Reduce every layer |
Highest |
NVLink only |
| Pipeline Parallel |
Model stages |
Point-to-point activations |
Medium |
Within/across nodes |
| Data Parallel |
Training batches |
All-Reduce for gradients |
High (bulk) |
IB/Ethernet |
| Expert Parallel |
MoE experts |
All-to-All token dispatch |
Highest cross-node |
IB/Ethernet |
The MoE network problem: Expert parallelism uses All-to-All — every GPU sends tokens to every other. DeepSeek-V3 spent >50% of training time on communication without optimization. They built DeepEP, a custom library, to fix it.
Network Topologies at Scale
Fat-Tree (CLOS)
- Classic HPC topology — full bisection bandwidth
- 3-tier: leaf → spine → core switches
- Any GPU can talk to any other at full speed
- Expensive at scale: 30K GPUs = ~$200M in networking alone, 4.7 MW power
- Used by: Microsoft Eagle (IB), Meta (both clusters)
Rail-Optimized
- Each of 8 GPUs in a node connects to a different “rail” (leaf switch)
- Same-rank GPUs across nodes share a rail
- Most LLM training traffic stays within rails
- Eliminates one switch tier — saves cost and power
- Benchmark: zero performance impact on 3D parallelism; only 11% overhead on All-to-All (MoE)
- Used by: xAI Colossus, many new builds
Rail-optimized works because LLM training is structured: tensor parallel within nodes, data parallel across rails. The topology matches the communication pattern.
The Hidden Cost Multiplier
Networking as % of total cluster cost
The real cost of bad networking
Networking is 10-13% of cost but determines 100% of multi-node GPU utilization.
At $30K+ per H100, even a 5% utilization drop across 10,000 GPUs = $15 million in wasted hardware.
Concrete example: A full CLOS fabric for 32K GPUs at 400 Gbps costs ~$153 million in networking alone, consuming 4.7 MW of power.
Failure is the Norm
What changes at 100K+ GPUs
Mean Time Between Failures
Meta’s data from Llama 3 training (16,384 H100s)
| Cluster Size |
MTBF |
Implication |
| 8 GPUs |
~47 days |
Rarely notice failures |
| 64 GPUs |
~6 days |
Weekly disruptions |
| 1,024 GPUs |
~8 hours |
Multiple failures per day |
| 16,384 GPUs |
~1.8 hours |
Something breaks every 3 hours |
| 100K GPUs |
~minutes |
Constant failure stream |
Despite ~3-hour MTBF, Meta achieved >90% effective training time through automated recovery. Only 3 manual interventions during the entire Llama 3 training run.
What Breaks — Llama 3 Failure Breakdown
GPU hardware
30.1% (148 events)
GPU HBM3
17.2% (72 events)
GPUs are the #1 failure source
GPU hardware + HBM3 memory account for 47% of all failures. Network switches and cables cause ~8%. The network is actually more reliable than the GPUs.
Resilience strategies
Frequent checkpointing, automated node replacement, health monitoring, and — at Google — optical circuit switches that route around faults for 50× less downtime.
NVIDIA Networking Revenue Explosion
| Period |
Quarterly Revenue |
YoY Growth |
| Q1 FY2025 (Apr 2024) |
~$3.2B |
2.4× |
| Q2 FY2026 (Jul 2025) |
~$7.3B |
~2× |
| Q3 FY2026 (Oct 2025) |
~$8.2B |
162% |
| FY2026 Full Year (est.) |
~$29B |
~126% |
The Mellanox acquisition for $6.9B now generates more than $6.9B per quarter. NVIDIA’s networking segment alone is larger than most semiconductor companies. Together, compute + networking power over 250 of the world’s TOP500 supercomputers.
3D Parallelism — Mapping to the Network
The practical hierarchy
Tensor Parallel (TP)
within node — NVLink
Pipeline Parallel (PP)
nearby nodes — NVLink / fast IB
Data Parallel (DP / FSDP)
across cluster — IB / Ethernet
Expert Parallel (EP)
across cluster — All-to-All (hardest)
The design principle: Put the most communication-intensive parallelism on the fastest interconnect. TP runs over NVLink (1.8 TB/s). Running TP over InfiniBand causes 20-40% performance degradation. Everything is engineered to respect the bandwidth hierarchy.
Key Takeaways
- The network is the computer — GPUs are components, the interconnect makes them one system
- 36× bandwidth cliff between NVLink and InfiniBand defines every design choice
- Ethernet is winning the market (91% by 2029), but NVIDIA wins either way via Spectrum-X
- Mellanox was a $6.9B bet that now generates $29B/year — the most important acquisition in semiconductor history
- Failure is the norm at scale — something breaks every 3 hours at 16K GPUs
- Google built their own everything — the only hyperscaler fully independent of NVIDIA networking
- MoE models are the new stress test — All-to-All communication is the hardest pattern to network
- UEC is the long-term threat to NVIDIA’s networking moat, but early
1,800 GB/s within.
50 GB/s between.
Everything follows
from this gap.
The network is the computer.
agent37.club