Every level, every part, in text

345–500 kV backbone

High-voltage lines tie every campus to power plants and the wider grid. A gigawatt campus needs a new substation and often new lines, which is why builders spread clusters across regions where power is available.

Interconnection

230–500 kV

Assumed

Example

Amazon Project Rainier: existing Olive 345 kV station, new interconnection (AWS New Carlisle Data Center Campus: Project Rainier — Where Anthropic Trains Claude)

Reported

Map symbols

substations, towers and supply lines are representative

Assumed

Gas, coal, nuclear, wind, solar

Plants inject power into the grid far from the campus; the grid delivers it with about 5% lost on the way.

US grid losses

≈5% (EIA) (How much electricity is lost in electricity transmission and distribution)

Spec

Plant symbols

representative types, heights exaggerated

Assumed

100 MW at the meter

The campus this page follows, 100 MW at the meter, placed in southwest Ohio for the map. Pick a real campus in the scenario bar to move it. Go in to the substation and follow the power down.

Meter

100 MW

Assumed

IT load

86 MW at PUE 1.16

Calc.

Ohio: 456 g CO₂/kWh

Shaded states have EIA figures: teal for hydro-heavy grids, amber and red for coal and gas. The same campus emits three to four times more in Wisconsin than in Washington.

Ohio, 2024

1,005 lb/MWh, 456 g/kWh (Ohio Electricity Profile 2024)

Spec

US average, eGRID 2022

373 g/kWh (eGRID Summary Tables 2022)

Spec

Lowest shown, Washington

113 g/kWh (Washington Electricity Profile 2024)

Spec

Highest shown, Wisconsin

494 g/kWh (Wisconsin Electricity Profile 2024)

Spec

Abilene, TX · partly live

OpenAI, Oracle, Crusoe, SoftBank. Buildings 1–4 of 8 are live, about 421 MW of IT power. Buildings 5–8 are roofed and being fitted out; the full campus is due between Q4 2026 and Q1 2027. Choose it under Real campuses in the scenario bar to rebuild this page around it.

Status, 09/24/2026

Partly live · ≈421 MW IT (OpenAI Stargate Abilene, Frontier Data Centers (updated 09/24/2026))

Reported

IT power operating, buildings 1–4 of 8

≈421 MW (OpenAI Stargate Abilene, Frontier Data Centers (updated 09/24/2026))

Reported

Planned

≈1.2 GW (Epoch, 04/2026 estimate) (OpenAI Stargate: where the US sites stand)

Reported

Grid

double 345 kV corridor (Midland–Graham), Mulberry Creek substation (What Does a Hyperscale Data Center Really Look Like?)

Reported

On-site generation

natural gas reported by Epoch (04/2026); not itemized in the current tracker (OpenAI Stargate: where the US sites stand)

Reported

Memphis, TN · operating, partly live

SpaceXAI (formerly xAI). Colossus 1: Fully built, about 200,000 GPUs, and leased in full to Anthropic since 05/06/2026. The confirmed grid supply is still 150 MW; TVA approved 300 MW in 02/2026. Colossus 2: Live and still growing: about 946 MW of IT power and 440,000 Nvidia chips by satellite count on 09/24/2026. A day later Elon Musk put it at 550,000 GPUs, 110k GB200 and 440k GB300, with 660,000 more GB300s planned by the end of the year. The most powerful AI data center operating today, by IT power and by compute (Epoch AI, 09/24/2026). Amazon and Anthropic’s New Carlisle campus is next at about 910 MW, with more chips but less compute. Choose it under Real campuses in the scenario bar to rebuild this page around it.

Colossus 1: status, 09/23/2026

Operating · 150 MW from the grid, plus turbines (xAI Colossus, Memphis, TN (checked 09/23/2026))

Reported

Colossus 2: status, 09/24/2026

Partly live · ≈946 MW IT (xAI Colossus 2, Frontier Data Centers (updated 09/24/2026))

Reported

Colossus 2: ranking, epoch ai

Most powerful operating today (Largest AI data centers by power capacity (updated 09/24/2026))

Reported

GPUs, per Elon Musk, 09/25/2026

230k: 150k H100, 50k H200, 30k GB200 (Post on X: “Colossus 1 is 150k H100, 50k H200 and 30k GB200. Colossus 2 is 110k GB200 and 440k GB300…”)

Reported

GPUs, reported earlier

≈200,000 (H100, H200, some GB200) (Musk's Colossus is fully operational with 200,000 GPUs)

Reported

Phase 2 power

≈300 MW (Musk's Colossus is fully operational with 200,000 GPUs)

Reported

Grid supply

MLGW/TVA, ≈150 MW (xAI installs Tesla Megapack batteries at supercomputer site)

Reported

On-site generation

≈35 gas turbines, ≈422 MW (xAI Colossus, Memphis, TN (checked 09/23/2026))

Reported

Batteries

Tesla Megapacks, ≈150 MW (xAI installs Tesla Megapack batteries at supercomputer site)

Reported

Backup, per SpaceXAI

more than 240 batteries, enough for the site to come fully off the grid (Mid-South: Colossus I and II, Memphis, TN / Southaven, MS)

Spec

Cooling water, per SpaceXAI

≈820,000 gal a day, hybrid system (Mid-South: Colossus I and II, Memphis, TN / Southaven, MS)

Spec

Fayetteville, GA · partly live

Microsoft. 4 of 9 main-campus buildings are live, about 636 MW of IT power. The rest of the 13 planned buildings, across the main, east and Fairwater campuses, are under construction toward about 1.5 GW. Choose it under Real campuses in the scenario bar to rebuild this page around it.

Status, 09/24/2026

Partly live · ≈636 MW IT (Microsoft Fairwater Atlanta, Frontier Data Centers (updated 09/24/2026))

Reported

IT power running, satellite estimate

≈636 MW, 4 of 9 Main Campus buildings (Microsoft Fairwater Atlanta, Frontier Data Centers (updated 09/24/2026); Microsoft's Fairwater Atlanta: A Grid-Only Gigawatt AI Data Center)

Reported

Planned

≈1.5 GW across all 13 buildings (Fairwater + Main + East Campus) (Microsoft's Fairwater Atlanta: A Grid-Only Gigawatt AI Data Center)

Reported

Accelerators

NVIDIA Blackwell NVL72 (GB200/GB300 family), ≈140 kW per rack (Infinite scale: The architecture behind the Azure AI superfactory; Microsoft's Fairwater Atlanta: A Grid-Only Gigawatt AI Data Center)

Spec

Cooling

closed-loop liquid, no evaporation (Infinite scale: The architecture behind the Azure AI superfactory; Microsoft's Fairwater Atlanta: A Grid-Only Gigawatt AI Data Center)

Spec

Backup power

no on-site generation, UPS or dual-corded distribution (grid-only design) (Infinite scale: The architecture behind the Azure AI superfactory; Microsoft's Fairwater Atlanta: A Grid-Only Gigawatt AI Data Center)

Spec

Mount Pleasant, WI · partly live

Microsoft. Building 1 has run since 04/16/2026, about 369 MW of IT power. Building 2 is under construction, due in 2028. Choose it under Real campuses in the scenario bar to rebuild this page around it.

Status, 09/24/2026

Partly live · ≈369 MW IT (Microsoft Fairwater Wisconsin, Frontier Data Centers (updated 09/24/2026))

Reported

Power

Building 1 ≈369 MW live; $3.3B initial investment (Microsoft Fairwater Wisconsin, Frontier Data Centers (updated 09/24/2026); Made in Wisconsin: The world’s most powerful AI datacenter)

Reported

Accelerators

GB200 NVL72 racks (NVLink 5.0 inside), racks linked by 800G Ethernet (Microsoft Opens Fairwater: Wisconsin AI Campus Runs as One Supercomputer via 800G Ethernet)

Reported

Cooling

closed-loop liquid, filled once (Made in Wisconsin: The world’s most powerful AI datacenter)

Spec

Opened

06/23/2026 (Microsoft completes construction on first datacenter facility in Mount Pleasant, Wisconsin)

Reported

Richland Parish, LA · under construction

Meta. Under construction, with nothing serving yet in the latest satellite imagery (Epoch AI, updated 09/24/2026). The first phase, about 2 GW, is due by 2030; Meta’s full build is 5 GW, expected around 2032. Choose it under Real campuses in the scenario bar to rebuild this page around it.

Status, 09/27/2026

Under construction · nothing live yet (Meta Hyperion, Frontier Data Centers)

Reported

Phase 1

≈2 GW by 2030 (5GW Data Center Buildout Requires Novel Engineering)

Reported

Planned

5 GW at full build (~2032), more than $50B (The largest Meta data center yet brings big impact to Louisiana; Hyperion (data center))

Spec

GPUs, planned

more than 3 million at full build (reporter estimate) (5GW Data Center Buildout Requires Novel Engineering)

Reported

New gas generation

2.26 GW combined, three turbine plants (5GW Data Center Buildout Requires Novel Engineering; Hyperion (data center))

Reported

New Carlisle, IN · partly live

Amazon Web Services (built for Anthropic). About 16 of roughly 30 planned buildings are live: near 910 MW of IT power and about a million Trainium2 chips by Epoch AI’s count. The rest are under construction toward about 1.9 GW, due around Q1 2028. Choose it under Real campuses in the scenario bar to rebuild this page around it.

Status, 09/24/2026

Partly live · ≈910 MW IT (Anthropic-Amazon New Carlisle, Frontier Data Centers (updated 09/24/2026))

Reported

IT power operating, ~16 of ~30 buildings live

≈910 MW (Anthropic-Amazon New Carlisle, Frontier Data Centers (updated 09/24/2026))

Reported

Planned full campus, due ~Q1 2028

≈1,925 MW (1.9 GW) (Anthropic-Amazon New Carlisle, Frontier Data Centers (updated 09/24/2026))

Reported

Chips, Oct. 2025 launch vs. 09/2026

≈500,000 Trainium2 at launch (64-chip UltraServers); ≈1,045,000 by Epoch's 09/24/2026 count (Amazon Project Rainier — Amazon Web Services data center in New Carlisle, IN; Anthropic-Amazon New Carlisle, Frontier Data Centers (updated 09/24/2026))

Reported

Grid interconnection

Indiana Michigan Power (AEP), existing Olive 345 kV station (AWS New Carlisle Data Center Campus: Project Rainier — Where Anthropic Trains Claude; Amazon Project Rainier — Amazon Web Services data center in New Carlisle, IN)

Reported

On-site generation

none for operating power; diesel gensets for backup only (AWS New Carlisle Data Center Campus: Project Rainier — Where Anthropic Trains Claude)

Reported

New Albany, OH · partly live

Meta. About 496 MW of IT power runs across a patchwork of tents, colocation space, and traditional buildings, roughly half of the ≈1,022 MW Epoch expects by Q3 2028. Two off-grid gas plants, Socrates North and South, feed the campus directly, outside the regional grid. Choose it under Real campuses in the scenario bar to rebuild this page around it.

Status, 09/24/2026

Partly live · ≈496 MW IT (Meta Prometheus)

Reported

IT power operating, Epoch satellite estimate

≈496 MW (Meta Prometheus)

Reported

Compute, satellite estimate

≈600k H100-eq compute; ≈237k Nvidia B200 chips now, B300 to follow (Meta Prometheus)

Reported

Planned, by Q3 2028

≈1,022 MW IT power, ≈$38.7B total capital (Meta Prometheus)

Reported

On-site generation

Socrates North + South gas plants, ≈400 MW, fully off-grid (Meta Deploys Tent-Based Data Centers in Ohio and Tennessee, Backed by 400MW of Off-Grid Gas Power)

Reported

Construction

5 tent structures (~125,000 sq ft each) built Apr–Jun 2026, plus traditional colocation buildings (Meta Deploys Tent-Based Data Centers in Ohio and Tennessee, Backed by 400MW of Off-Grid Gas Power)

Reported

One model, several sites

Builders now train single models across campuses, splitting the work so the slow links carry the least traffic. Google trains its largest models across campuses and metros; Microsoft links Fairwater sites about 700 miles apart.

Microsoft AI WAN fiber added

120,000 miles (From Wisconsin to Atlanta: Microsoft connects datacenters to build its first AI superfactory)

Spec

NVIDIA Spectrum-XGS

nearly 2× NCCL across sites (NVIDIA Introduces Spectrum-XGS Ethernet to Connect Distributed Data Centers)

Vendor

DeepMind Decoupled DiLoCo

4 US regions over 2–5 Gb/s (Decoupled DiLoCo)

Reported

≈5 milliseconds per 1,000 km, one way

Illustrative route: real campuses, an invented path between them. Light in glass covers about 200 km per millisecond one way, before any switching, routing or queueing delay. An illustrative 1,000 km route adds about 5 ms of propagation each way, roughly 10 ms round trip—fine for inference, hard for tightly synchronized training.

Speed in fiber

≈4.9 µs per km, one way

Calc.

1,000 km, propagation only

≈5 ms one way, ≈10 ms round trip

Calc.

Microsoft hollow-core fiber

≈33% lower latency; over 1,200 km carrying live traffic (Microsoft Scales Up Hollow Core Fiber Production; Microsoft's hollow core fiber delivers the lowest signal loss ever)

Reported

Per fiber pair, C-band 800ZR

32 × 800G = 25.6 Tb/s (ECOC: Coherent Extends Pluggable Line System Across Full C-Band)

Reported

Every 60–100 km

Small buildings along the route boost the light directly, in the optical domain, without converting it back to electricity or reading the data.

Spacing

≈80–100 km, rule of thumb

Assumed

Coherent DWDM

At each campus, coherent transceivers—in routers or dedicated transponder shelves—each turn one signal into one wavelength, 800 Gb/s to 1.6 Tb/s. A multiplexer combines many of those wavelengths onto a single fiber pair; amplifier huts (next) carry the combined light between campuses, and a demultiplexer splits it back into wavelengths at the far end.

Per wavelength, WaveLogic 6

up to 1.6 Tb/s (Ciena Unveils WaveLogic 6)

Spec

800G pluggable, e.g. Marvell COLORZ 800

800 Gb/s to ≈500 km (Marvell launches the first 800G ZR/ZR+ modules for data center interconnects)

Spec

Field trial

1.6 Tb/s over 1,100 km (Telstra) (Telstra Extends 1.6 Tbps Wavelength to 1,100 km With Ciena)

Reported

C+L band

about 2× capacity per fiber (Telstra Extends 1.6 Tbps Wavelength to 1,100 km With Ciena)

Reported

Inside a line terminal, e.g. Ciena 6500 RLS

mux/demux, ROADM and amplifier modules in compact shelves (6500 Reconfigurable Line System data sheet (DS365))

Spec

Mux/demux, e.g. Ciena CMD64

up to 64 channels on a 75 GHz grid (6500 Reconfigurable Line System data sheet (DS365))

Spec

Go in

Go into the campus and follow the data in.

GPUs

≈44,640

Calc.

Heat stays local

Power travels between sites; heat is rejected locally. Builders favor places where outside air is cool enough to reject heat most of the year, and where water is not scarce. Rising pulses show local heat rejection schematically; their height, count and speed are not measurements.

Free cooling

most hours in cool climates

Assumed

Go in

Go into the campus and follow the heat out.

Heat out

100 MW

Calc.

345 kV AC · 3 phases × 2 circuits

Lattice towers carry two three-phase circuits of bundled aluminum conductor, with a shield wire on top to take lightning. At 345 kV the whole 100 MW campus rides on ≈90 A per phase on each circuit, which is why power travels far at high voltage.

Voltage

345 kV line-to-line

Assumed

Current, 100 MW

≈90 A per phase, on each circuit

Calc.

US grid losses, 2018–2022

≈5% (EIA) (How much electricity is lost in electricity transmission and distribution)

Spec

Example

Stargate Abilene: double 345 kV corridor (What Does a Hyperscale Data Center Really Look Like?)

Reported

Utility interconnect

The line dead-ends on steel gantries and lands on a ring of SF₆ circuit breakers and disconnect switches. Instrument transformers measure it, surge arresters clip lightning, and tall masts shield the yard.

Breakers

6 dead-tank SF₆, ring bus

Assumed

Equipment drawn

typical forms, representative

Assumed

Yard

≈200 × 150 m gravel pad

Assumed

Interconnection study

1–3 years alone (Data Center Substation Construction in 2026: What Developers and Hyperscalers Need to Know)

Reported

345 kV → 34.5 kV

Oil-filled transformers, each the weight of a loaded freight car, step the line down to the campus distribution voltage. Radiators and fans shed their heat; concrete fire walls keep one fire from taking the others.

Rating

3 × 75 MVA, N+1

Calc.

Efficiency, 345 kV class

>99.6% at all loading levels (345 kV Power Transformer Projects)

Spec

Loss at 100 MW

≈0.4 MW

Calc.

Fittings drawn

radiator banks, fans, bushings: a typical layout

Assumed

Lead time, 2026

128–144 weeks (Power Transformer Lead Times Hit 128 Weeks in 2026)

Reported

Campus distribution

Prefabricated switchgear buildings split the transformer output into feeders, each breaker-protected, that run in concrete duct banks under the roads to the data halls.

Feeders

≈10, each ≈10 MW

Calc.

Voltage

34.5 kV (some campuses use 13.8 kV) (Medium-Voltage Distribution for Data Center Campuses in 2026)

Reported

Loss, cables + gear

≈0.3 MW

Calc.

Buildings drawn

a typical prefab kit, representative

Assumed

Smooths GPU load swings

Thousands of GPUs stepping in lockstep during training can swing campus load by tens of megawatts in seconds. Grid-side batteries absorb the swings the utility would otherwise see, and can sell grid services.

Size here

≈20 MW / 40 MWh

Calc.

Training load swings

tens to hundreds of MW, seconds (Power Stabilization for AI Training Datacenters)

Reported

Example

SpaceXAI Colossus 1: ≈150 MW of Megapacks (xAI installs Tesla Megapack batteries at supercomputer site)

Reported

Diesel, 480 V stepped up to 34.5 kV

Containerized diesel sets, such as Cummins’ QSK78 or Caterpillar’s C175-16 in the 2.5-3 MW class, start within about ten seconds of a grid failure. The UPS batteries carry the load until they take over. They run a few hours a year, mostly for testing.

Unit size

2.5–3 MW class (DQKAN 2.5 MW generator data sheet; C175-16 diesel generator set)

Spec

Units here

≈40, N+20%

Calc.

Fuel, 2.5 MW at full load

173 US gal/h (≈0.26 L/kWh) (DQKAN 2.5 MW generator data sheet)

Spec

Start to load

≈10 s (Ask the Experts: NFPA 110 for Emergency Power Systems; NFPA 110 Generator Requirements for Data Centers and Hospitals)

Spec

48 hours at full load

Horizontal steel tanks hold enough diesel to run the whole campus for two days, with polishing skids that keep stored fuel clean.

Volume, 48 h at 100 MW

≈1.2 million L

Calc.

Tanker deliveries to refill

≈42

Calc.

Tanks drawn

double-wall horizontal, representative fittings

Assumed

34.5 kV → 480 V

A line of pad-mounted transformers along each hall drops the feeders to 480 V right outside the electrical rooms, keeping the high-current low-voltage runs short.

Count

≈45 × 2.5 MVA

Calc.

Efficiency, 2500 kVA class

≈99.5% (10 CFR § 431.196 – Energy conservation standards and their compliance dates (distribution transformers))

Spec

2 halls, ≈620 racks, ≈45k GPUs

The halls hold the IT load, about 45 MW each. Each floor is a slab with no raised floor: racks are too heavy and the cooling is water, not air under the floor.

IT load

86 MW at PUE 1.16

Calc.

Racks

≈620

Calc.

Halls

2

Calc.

Floor load, one rack

≈1.4 t on 0.6 × 1.1 m (Is Your Data Center Ready for the NVIDIA GB200 NVL72?)

Reported

Heat out, no water used

Rooftop coils, such as EVAPCO’s Apex or Baltimore Aircoil’s TrilliumSeries dry coolers, reject the heat carried out of the GPUs by warm water with big fans. Water at 30–40 °C is warm enough to dump heat to outside air most of the year without chillers.

Heat rejected

≈91 MW

Calc.

Units, ≈0.8 MW each

≈119

Calc.

Water classes

ASHRAE W32–W45: 32–45 °C max supply (What You Need to Know About ASHRAE's Fifth Edition of Thermal Guidelines; Emergence and Expansion of Liquid Cooling in Mainstream Data Centers)

Spec

Water use, dry + adiabatic

≈0.15–0.17 L/kWh (Environmental and Economic Implications of Artificial Intelligence Data Centers in the United States)

Reported

For the hottest days

Evaporative towers trim water temperature on hot afternoons, and the tanks hold treated makeup water and fire water.

Towers

≈6

Calc.

WUE, on site

≈0.16 L/kWh IT

Assumed

Use

peak days only

Calc.

Cell drawn

counterflow, representative

Assumed

Two diverse routes

Long-haul fiber enters at two vaults on opposite sides of the site, so one backhoe cannot cut the campus off. Tokens leave the same way the questions arrive.

Routes

2 or more, physically separate (The Importance of Diverse Fiber Routes in Data Centers)

Reported

The outer layer of several

Physical security comes in layers, from the fence and a staffed gate inward to the building, the hall and the racks. The fence and gatehouse drawn here are the outermost one.

Microsoft’s layers

facility perimeter, building perimeter, inside the building, the hall floor (Azure facilities, premises, and physical security)

Reported

Google’s data centers

six layers of physical security (One percent of Googlers get to visit a data center, but I did)

Reported

Watching power, cooling and security

Operators watch the site from software rather than by walking it. A power monitoring system gathers readings from the switchgear and distribution points; infrastructure management software tracks equipment, temperatures and security. It is drawn here as the site’s office building.

Power monitoring (EPMS)

readings from key distribution points (Power management systems for data centers: enhancing control and efficiency)

Reported

Infrastructure management (DCIM)

assets, environment, security (EcoStruxure IT DCIM software)

Reported

Two diverse routes

Long-haul fiber enters at vaults on opposite sides of the site, so one backhoe cannot cut the campus off. Questions arrive and tokens leave the same way.

Routes

2 or more, physically separate (The Importance of Diverse Fiber Routes in Data Centers)

Reported

Where outside networks connect

Carriers’ fiber ends in a meet-me room, where outside networks cross-connect to the campus. Border switches then peer with those networks over BGP. NVIDIA calls this the edge network: it links the cluster’s services to networks outside it, while the GPU-to-GPU fabric stays inside.

NVIDIA’s name for it

the edge network (Networking Planning and Design, NVIDIA Mission Control North-South Network Configuration Guide 2.0.0)

Spec

Meet-me room

where carriers physically interconnect (Meet-me room)

Reported

Coherent DWDM

Coherent transceivers here each produce or receive one wavelength, hundreds of gigabits to over a terabit; a multiplexer combines dozens of them onto each fiber pair bound for other campuses, and amplifiers along the route keep the combined signal alive without converting it back to electricity.

Per wavelength, Ciena WaveLogic 6

up to 1.6 Tb/s (Ciena Unveils WaveLogic 6; Ciena introduces single-carrier 1.6-Tbps WaveLogic 6 coherent optical engines)

Spec

400ZR reach, amplified

up to ≈120 km (What 400ZR means for next-gen DCI and metro networks)

Reported

Module power, 400ZR / 800ZR

≈18–20 W / ≈23–25 W (Coherent Optical Module Power Consumption: Complete Guide for Network Engineers)

Reported

Cut away beside the hut

Opened up, the hut splits the work three ways. Coherent pluggables sit in the router’s own ports, each sending one wavelength. A passive mux/demux lays those wavelengths side by side on one fiber, each in its own slot of a fixed grid. A ROADM shelf with built-in amplifiers can add, drop or pass single wavelengths without turning the rest back into electricity, and amplifies the combined light before it leaves through the wall. The cutaway is drawn beside the hut so it reads on its own; its colors only tell the wavelengths apart, all of them infrared.

Coherent pluggables

one wavelength each, straight from a router port (QSFP-DD and OSFP 800G ZR/ZR+ Coherent Optics Modules Data Sheet)

Spec

Mux/demux, e.g. Ciena CMD64

2RU, up to 64 channels on a 75 GHz C-band grid (6500 Reconfigurable Line System data sheet (DS365))

Spec

A 64 × 75 GHz grid, in frequency

191.3625–196.0875 THz, ≈1,529–1,567 nm (64 Channels DWDM Mux Demux, Super C-band 75GHz)

Reported

ROADM shelf drawn, Ciena 6500 RLS R8-300

8 slots; 330 × 440 × 281 mm, 7.5U (6500 Reconfigurable Line System data sheet (DS365))

Spec

Modules in that family

ROADM with line amplifier (WSS and EDFAs), dual line amplifier, Raman amplifier, colorless mux/demux (6500 Reconfigurable Line System data sheet (DS365))

Spec

What a ROADM does

switches single wavelengths without converting the rest to electricity (Reconfigurable optical add-drop multiplexer)

Reported

Racks, router, slot layout, port counts and fiber thickness as drawn

representative

Assumed

One fabric, 2 buildings

Thousands of strands in the duct banks join the spines of every hall, so a single training job can span every GPU on the campus. Each link needs optics rated for its length. The two halls drawn here sit about 140 m apart at their spine ends, inside the 500 m of the same parallel-fiber DR optics used within a hall. Longer runs between buildings step up to FR4 (2 km) or LR4 (10 km), which put four wavelengths on one fiber pair, each carrying four-level PAM4 signals rather than the coherent ones used between campuses; DR and FR4 modules cannot be linked to each other.

Strands

tens of thousands per hall pair

Calc.

Drawn spine-to-spine route

≈140 m between the halls, inside DR reach

Assumed

DR, parallel fiber

500 m, eight fibers on MPO-12/APC (LinkX: Transceivers and Cables for AI Factory Networks)

Spec

FR4, e.g. NVIDIA MMS4X50-NM (2 × FR4)

2 km, four wavelengths per 400G port, duplex LC (MMS4X50-NM 800Gbps Twin-port OSFP 2xFR4, 2x400Gb/s Single Mode, 2km: specifications; LinkX: Transceivers and Cables for AI Factory Networks; 400GBASE-FR4: single-mode Ethernet standards update)

Spec

LR4, 400GBASE-LR4

10 km, four wavelengths on one fiber pair (400GBASE-LR4 Baseline Proposal (lewis_3cu_02a_0519))

Spec

Fiber between the halls, cut away

Between buildings, fiber runs in 4-inch conduits cast in concrete, one high-count ribbon cable per conduit, with a spare row. In this layout half the spine-to-core links cross between halls: about 179k strands, or roughly 26 cables of 6,912 fibers each.

Strands crossing

≈179k

Calc.

Cable

6,912-fiber ribbon fits a 2-inch duct (FlexRibbon whitepaper)

Spec

Duct-bank layout

general telecom practice

Assumed

Scale-out fabric inside

Inside, every GPU has its own optical port into a leaf-and-spine fabric.

GPUs

≈44,640

Calc.

Scale across

The fiber leaving the site runs to other campuses hundreds of kilometers away; the route drawn is illustrative, not a real carrier path.

Light in fiber

≈4.9 µs per km, one way

Calc.

Heat into air, no water

Warm facility water runs through finned coils on the roofs while big fans pull outside air across them. With water at 30–45 °C, outside air can take the heat most of the year without chillers.

Heat rejected

≈91 MW

Calc.

NVIDIA warm-water spec

45 °C in, ≈55 °C out (Hotter Than a Hot Tub: The 45°C Breakthrough to Cool AI's Biggest Machines)

Spec

Water classes

ASHRAE W32–W45 (What You Need to Know About ASHRAE's Fifth Edition of Thermal Guidelines; Emergence and Expansion of Liquid Cooling in Mainstream Data Centers)

Spec

Hot days cost water

Evaporating water carries heat away far better than air, so towers trim the loop on the hottest afternoons. Every kilowatt-hour moved this way costs water.

WUE, this design

≈0.16 L/kWh IT

Assumed

Water treatment

anti-scale, anti-corrosion and disinfectant dosing, plus blowdown (Controlling Legionella in Cooling Towers)

Reported

Water per day

≈332 m³

Calc.

At the power plant, typical thermal

≈1.8 L/kWh (NREL) (A review of operational water consumption and withdrawal factors for electricity generating technologies)

Spec

All of it, as heat

Every watt that came in on the 345 kV line leaves as warm air above the roofs. The campus is, physically, a 100 MW heater that happens to make tokens on the way.

Heat out

100 MW

Calc.

Warm water is still worth something

In cold climates the return water can feed a district heating network, with heat pumps lifting it to 70–75 °C. This campus exports none; these do.

Meta Odense, Denmark

≈165,000 MWh a year, ≈11,000 homes (Harnessing heat from Odense data center for reuse in district heating)

Reported

Microsoft + Fortum, Finland

up to 180 MW of district heat (Fortum has started heat production at two large data centre sites in Finland)

Reported

Stockholm Data Parks

30+ data centers selling heat (Stockholm Data Parks; Stockholm Exergi: Data Parks shows the potential for scaling the reuse of waste heat when it becomes a tradable product)

Reported

Tie-in drawn here

an illustration, not built

Assumed

34.5 kV → 480 V, 2.5 MVA

Outside the wall, a pad-mounted transformer takes one campus feeder and makes 480 V three-phase. Its secondary runs a few meters through the wall into the switchgear.

Rating

2.5 MVA

Assumed

Secondary current

≈3,000 A at full load

Calc.

Efficiency

≈99% (DOE Finalizes Energy Efficiency Standards for Distribution Transformers That Protect Domestic Supply Chains and Jobs, Strengthen Grid Reliability, and Deliver Billions in Energy Savings)

Assumed

Enclosure as drawn

representative

Assumed

Breakers and transfer

A lineup of drawout breakers protects every outgoing circuit and switches the room between utility and generator when the grid drops.

Transfer

automatic, utility ↔ generator

Assumed

Markings

nameplate and arc-flash signs, representative

Assumed

Double conversion

The UPS turns AC into DC and back to clean AC, with batteries on the DC link. It rides through the seconds between a grid failure and the generators taking load.

Module

1.25–1.5 MW

Assumed

Modules here

≈73

Calc.

Efficiency, Eaton 9395XR

up to 97.5% online, 99% eco (EATON MW 9395XR UPS is designed for intelligent computing)

Reported

Loss at 100 MW

≈3.1 MW

Calc.

Lithium-ion, ≈5 minutes

Racks of lithium-ion modules on the UPS DC link. Five minutes is plenty: the generators are carrying the load within a minute.

Runtime

set by string count, often ≈5 min

Assumed

Chemistry

Li-ion (LFP or NMC)

Assumed

415 V to every rack

A transformer steps 480 V to 415 V, the voltage OCP rack power shelves take. Copper bars in an aluminum housing then run over each row, and plug-in tap-off boxes drop a short cable into each rack, so moving a rack means moving a plug.

Rack voltage

415 V three-phase (LV Distribution: Busway, PDUs, RPPs & Rack Power)

Reported

Per rack

≈190 A per phase at 131 kW

Calc.

Why busway

tap-offs move without rewiring

Assumed

Hardware as drawn

representative

Assumed

131 kW each

Each rack draws what a whole row of racks drew ten years ago. About 87% of its heat leaves in water, the rest in air.

Power, this model

≈131 kW

Calc.

Published range

120–132 kW (This is the NVIDIA DGX GB200 NVL72)

Reported

GPUs

72 Blackwell (GB200 NVL72)

Spec

Liquid / air

114 kW / 17 kW

Calc.

For the heat water misses

Glass roofs and doors close the aisle between rack backs, so the warm air goes straight back to the coolers instead of mixing into the room.

Air share of heat

≈13%

Calc.

Doors and roof as drawn

representative

Assumed

Two loops, one heat exchanger

The CDU keeps the rack loop, filtered water with glycol running through cold plates, separate from facility water. Units such as Vertiv’s CoolChip or Motivair’s CDU line pack the pumps, plate heat exchanger and controls into one cabinet at the row end.

Capacity range

70 kW – 2.3 MW (Vertiv CoolChip CDU; CDU 2025 product brochure)

Spec

Units here, ≈1.25 MW

≈61

Calc.

Rule

rack loop stays above dew point (CDU 2025 product brochure)

Spec

Cabinet drawn

representative

Assumed

Supply and return headers

Insulated steel headers carry water between the CDUs and the rooftop dry coolers. Blue carries cooler supply, red carries warm return.

Supply → return

≈42 → 52 °C

Assumed

Temperature rise

≈10 °C across the racks

Assumed

Pipework as drawn

representative

Assumed

Pipe markers

green with white letters and a flow arrow, ASME A13.1 style (Pipe Color Codes: ASME A13.1, ISO 14726 & BS 1710)

Assumed

Air side

A wall of fans and coils cools the air that carries the remaining heat from power shelves, switches, optics and memory.

Share of rack heat

≈13%

Calc.

Cells as drawn

representative

Assumed

Powered through the host switch

Hall distribution feeds the network racks. Inside each switch, its power supply and board regulators provide 3.3 V DC to the pluggable modules through their electrical connectors. The fibers carry light, not the module’s electrical supply. The hall shows representative rack feeds; go inside to follow power from the host connector through a representative 1.6T module. Installed module rates depend on the scenario.

Supply

3.3 V on 4 power contacts, up to 2.5 A each (3.25 A on OSFP1600) (OSFP Module Specification, rev 5.22 (MSA text checked through the fluxlight.com mirror))

Spec

Edge connector

60 contacts: 32 high-speed (8 TX + 8 RX differential pairs), 4 control, 4 power, 20 ground (OSFP Module Specification, rev 5.22 (MSA text checked through the fluxlight.com mirror))

Spec

A comparison, not deployed here

This scenario’s switches use pluggable modules. The CPO switch set apart at the end of the spine row is a comparison: NVIDIA puts co-packaged optics at as low as 9 W a port, against 30 W with pluggables.

Per port, NVIDIA figure

as low as 9 W (Scaling AI Factories with Co-Packaged Optics for Better Power Efficiency)

Vendor

Q3450 switch, NVIDIA figure

3.95 kW (Unbox one of NVIDIA's first co-packaged optics samples with Lambda)

Vendor

Where tokens leave

Spine switches tie every rack to every other and to the fiber out of the building. Dense yellow trays carry thousands of fibers overhead.

Per GPU

400 Gb/s scale-out (NVIDIA GB200 NVL72 on CoreWeave: Quantum-2 InfiniBand at 400 Gb/s per GPU)

Spec

Where every link is patched

Fabric links do not run switch to switch in one piece. Trunk cables land on patch frames, and short jumpers make the actual connections, so a link can be moved without pulling cable through the ceiling.

Fabric strands, whole campus

≈1.1 million

Calc.

Housing density, Corning EDGE8

144 fibers per 1U (EDGE8 Housings)

Spec

4U housings for this campus

≈1,860

Calc.

Bay layout as drawn

representative

Assumed

Through the floor

Cables for the links that cross buildings drop through a floor sleeve into the duct bank outside.

Strands

≈179k

Calc.

One replica, four racks

The tinted rack tops show one way to lay a model out: its layers split into four stages, one rack each, passing activations down the line like an assembly line. 4 racks × 72 GPUs = one copy of the model.

Traffic

point to point, per micro-batch (The Llama 3 Herd of Models)

Reported

Llama 3 405B

pipeline parallel 16 (The Llama 3 Herd of Models)

Spec

Layout drawn here

illustrative

Assumed

Many copies, one model

Every group of four racks holds another full copy. Each copy trains on different data, and all of them average their gradients across the fabric once per step.

Traffic

large all-reduce, once per step (The Llama 3 Herd of Models)

Reported

Llama 3 405B

TP 8 × CP 16 × PP 16 × DP 8 = 16,384 GPUs (The Llama 3 Herd of Models)

Spec

DeepSeek-V3

no tensor parallel; EP 64, PP 16, ZeRO-1 DP (DeepSeek-V3 Technical Report)

Spec

Where scale-out starts

Each rack sends one optical link per GPU up into the fiber runway overhead: 72 ports per rack before the first switch, 400G each.

Per GPU

400 Gb/s (NVIDIA GB200 NVL72 on CoreWeave: Quantum-2 InfiniBand at 400 Gb/s per GPU)

Spec

Ports per rack

72 (GB200 NVL72)

Spec

Rail-optimized

Network racks at the row ends hold leaf switches, such as NVIDIA’s Quantum-2 QM9700 (InfiniBand). In a rail-optimized layout, GPU number n in every rack plugs into the same leaf, so most traffic crosses only one switch.

Leaf switches, campus

≈1,395

Calc.

Hops, same rail

1

Assumed

Switch hop, InfiniBand

under ≈100 ns; NVIDIA publishes none

Assumed

Switch drawn, QM9700

1U, 32 OSFP cages (QM97xx Switch Specifications (32 OSFP, 25.6 Tbps))

Spec

Face arrangement

representative

Assumed

Any GPU to any GPU

The spine connects every leaf to every other. Two tiers of 64-port switches reach about 2,048 GPUs; this campus uses 3.

Switch

Quantum-2, 64 × 400G (QM97xx Switch Specifications (32 OSFP, 25.6 Tbps))

Spec

Merchant switch chips, same role

Broadcom Tomahawk 6 (102.4 Tb/s), Marvell Teralynx 10 (51.2 Tb/s) (Broadcom now shipping the first 102.4 Tb/s switch in production volume; Marvell Teralynx 10 51.2T Ethernet Switch Enters Volume Production for Global AI Cloud Deployments)

Spec

Spine + core switches

≈2,093

Calc.

Switch drawn, QM9700

1U, 32 OSFP cages (QM97xx Switch Specifications (32 OSFP, 25.6 Tbps))

Spec

Face arrangement

representative

Assumed

A separate network

The GPU servers reach shared storage over a network of its own, apart from the GPU fabric. NVIDIA’s reference design takes storage from certified partners, reached over Ethernet with RDMA. A few storage racks stand in for it here; this model’s IT load counts the GPU racks and their fabric only.

Separate networks, NVIDIA’s design

compute, storage, in-band and out-of-band management, besides NVLink (Network Fabrics, NVIDIA DGX SuperPOD reference architecture (DGX GB200))

Spec

Storage attach

RDMA over Ethernet (RoCEv2), a partner’s shared file system (Key Components of the DGX SuperPOD (DGX GB200))

Spec

Where the cluster is run

A small set of ordinary CPU servers runs the cluster rather than the model: installing and monitoring every node, scheduling jobs, and giving users somewhere to log in. They sit on the in-band management network.

In NVIDIA’s B300 design

2 for Base Command Manager, 3 for Kubernetes, 2 Slurm login nodes (Management Servers, NVIDIA DGX SuperPOD reference architecture (DGX B300))

Spec

Yellow means fiber

Overhead yellow trays carry thousands of fiber strands. A parallel optical link runs each lane on its own fiber, one each way, so strand counts climb fast.

Fibers per link

8

Assumed

Multimode fiber jacket

aqua (MMA4Z00-NS400 400Gb/s Single-port OSFP 400Gb/s Multimode SR4 50m: product specifications (PDF); MFP7E20-Nxxx Optical Multimode Splitter Fiber Cable: product specifications (PDF))

Spec

Single-mode fiber jacket

yellow (MFP7E40-Nxxx Single mode 1:2 Fiber Splitter Cable: product specifications (PDF))

Spec

Splitter cable reach

up to 50 m, through two patch panels (MFP7E20-Nxxx Optical Multimode Splitter Fiber Cable: product specifications (PDF); MFP7E40-Nxxx Single mode 1:2 Fiber Splitter Cable: product specifications (PDF))

Spec

Aqua cables in this runway

representative

Assumed

Several per GPU

Every link is lit at both ends by a pluggable module, from merchant suppliers such as InnoLight and Coherent as well as NVIDIA’s own LinkX line. Here they fill the faces of the leaf switches at the row ends and of the spine switches, with a link light on each and fiber rising to the runway. 1 physical scale-out link leaves the rack per GPU, and every tier above adds more modules: about 3.5 per GPU, 2.3 MW for this campus. Runs this short can also be multimode: short-reach optics on aqua fiber, rated to 50 m on OM4. A splitter (breakout) cable can also share one switch port between two adapters, each at half rate. One leaf switch here shows both: a tan-tabbed module sends two straight aqua cables to two racks, and the module beside it sends two 1:2 splitter cables to four. They are representative examples, not counted in the totals.

At the switch: twin-port 800G OSFP, 2 × 400G

17 W max (800Gb/s Twin-Port OSFP 2x400Gb/s Single-Mode DR8 datasheet; MMA4Z00-NS 800Gb/s Twin-port OSFP, 2x400Gb/s Multimode 2xSR4, 50m)

Spec

Short runs, multimode option

2 × SR4, OM4 up to 50 m (OM3 30 m) (MMA4Z00-NS 800Gb/s Twin-port OSFP, 2x400Gb/s Multimode 2xSR4, 50m: Specifications)

Spec

Multimode light source

850 nm VCSEL (MMA4Z00-NS400 400Gb/s Single-port OSFP 400Gb/s Multimode SR4 50m: product specifications (PDF); MMA4Z00-NS 800Gb/s Twin-port OSFP, 2x400Gb/s Multimode 2xSR4, 50m: Specifications)

Spec

How multimode is marked

tan pull tab, aqua fiber (MMA4Z00-NS 800Gb/s Twin-port OSFP, 2x400Gb/s Multimode 2xSR4, 50m: Specifications; MMA4Z00-NS400 400Gb/s Single-port OSFP 400Gb/s Multimode SR4 50m: product specifications (PDF))

Spec

1:2 splitter cables

one 4-channel MPO-12 port to two 2-channel ends, 200G each (MFP7E20-Nxxx Optical Multimode Splitter Fiber Cable: product specifications (PDF); MFP7E40-Nxxx Single mode 1:2 Fiber Splitter Cable: product specifications (PDF))

Spec

Twin-port module with two splitters

one switch cage to four 200G adapters (MFP7E20-Nxxx Optical Multimode Splitter Fiber Cable: product specifications (PDF); MMA4Z00-NS400 400Gb/s Single-port OSFP 400Gb/s Multimode SR4 50m: product specifications (PDF))

Spec

Adapter module on a splitter end

2 lanes lit, 200G; ≈5.5 W typical instead of 8 W (MMA4Z00-NS400 400Gb/s Single-port OSFP 400Gb/s Multimode SR4 50m: product specifications (PDF))

Spec

Straight and split on one module

not mixed: both ports straight, or both split (MMA4Z00-NS400 400Gb/s Single-port OSFP 400Gb/s Multimode SR4 50m: product specifications (PDF); MFP7E20-Nxxx Optical Multimode Splitter Fiber Cable: product specifications (PDF))

Spec

Aqua straight links and splitters at one leaf, as drawn

representative

Assumed

The DSP inside each module

e.g. Marvell Ara, Broadcom Sian, Credo Bluebird (Marvell ushers in the 1.6T era with an expanded optical DSP portfolio; Broadcom Extends 200G-Lane DSP PHY Leadership for Next-Generation AI Infrastructure; Bluebird 1.6T optical DSP)

Spec

Linear-drive (LPO), Semtech target

about 10 W versus 23–25 W retimed (200G LPO Power, Reach and Loss: Real Numbers)

Vendor

A comparison, not deployed here

This scenario does not deploy CPO: none of its switches, power or fiber counts change because of this card. One extra switch stands apart at the end of the spine row as a schematic stand-in for the alternative, NVIDIA Spectrum-X/Quantum-X Photonics-style CPO, for comparison only. CPO is a kind of switch, not an add-on: a fabric that adopts it uses CPO switches in place of pluggable ones, which is why this one is set apart rather than drawn in the row. Real CPO switches put the optical engines on (or beside) the switch package itself, shortening the electrical path to the optical engines and cutting out the pluggable modules, which changes both signal-processing needs and electrical losses; it is not simply "every removed block is saved power." Fewer lasers, from sharing external laser sources across ports, is also not the same claim as fewer traffic fibers: CPO does not by itself reduce how many fibers carry data. Every switch actually counted in this hall still takes pluggables, as most fabrics do today.

NVIDIA Quantum-X / Spectrum-X Photonics

5× power efficiency, 4× fewer lasers, not fewer fibers (Aug. 2026 reporting; NVIDIA’s own March 2025 launch claimed 3.5×) (NVIDIA Spectrum-X Ethernet Photonics Enters Full Production; NVIDIA Spectrum-X co-packaged optics networking switches for AI factories; How Industry Collaboration Fosters NVIDIA Co-Packaged Optics)

Vendor

Broadcom Davisson

102.4 Tb/s, ≈3.5 W per 800G port (Broadcom now shipping the first 102.4 Tb/s switch in production volume; The Third Time Will Be the Charm for Broadcom Switch Co-Packaged Optics)

Vendor

Stand-in faceplate, Q3450 counts

144 MPO, 18 laser modules, 4 capped UDQ4 ports (Unbox one of NVIDIA's first co-packaged optics samples with Lambda)

Reported

Faceplate layout

representative

Assumed

Scale-up stays inside

Inside each rack, 72 GPUs talk over copper NVLink, 18 times faster each way than the fabric outside.

NVLink per GPU

1.8 TB/s (NVIDIA Blackwell Platform Arrives to Power a New Era of Computing)

Spec

Domain

72 GPUs (GB200 NVL72)

Spec

Where the two loops meet

A plate heat exchanger, in units such as Vertiv’s CoolChip or Motivair’s CDU line, passes heat from the rack loop into facility water without mixing them. The rack side stays above the dew point so nothing condenses.

Capacity range

70 kW – 2.3 MW (Vertiv CoolChip CDU; CDU 2025 product brochure)

Spec

Approach, facility to rack loop

a few °C (≈3 °C here)

Assumed

Rack loop, supply → return

≈45 → 55 °C

Assumed

The air-side heat

Rack backs face each other across a sealed aisle, so hot air rises and flows to the coolers instead of warming the room.

Air share of rack heat

≈13%

Calc.

Doors and roof as drawn

representative

Assumed

Air back to cool

Fans pull hot-aisle air through water coils and blow it back into the room cool, closing the air loop.

Moves

the ≈13% air share

Calc.

Cells as drawn

representative

Assumed

Supply blue, return red

Insulated headers carry warm return water up to the roof and cooler supply water back. The temperature difference sets how much water has to move.

Rise

≈10 °C

Assumed

Supply → return

≈42 → 52 °C

Assumed

Pipework as drawn

representative

Assumed

Pipe markers

green with white letters and a flow arrow, ASME A13.1 style (Pipe Color Codes: ASME A13.1, ISO 14726 & BS 1710)

Assumed

Liquid-cooled

The comparison CPO switch cools its switch chips and their optics with liquid, rather than blowing air across front-panel modules. Inside, the package drawn at this door has one cold plate, as a representative layout.

Cooling, Q3450 switch

liquid; 4 UDQ4 connections, dual internal loops (Unbox one of NVIDIA's first co-packaged optics samples with Lambda)

Reported

Smoke found early

Air-sampling detectors pull air from the room through a pipe network and test it continuously, so they can find smoke before a fire grows. Sprinkler lines run above the aisles. NFPA 75 sets the minimum fire protection for rooms of IT equipment.

Air-sampling detection

continuous, the earliest warning (VESDA Aspirating Smoke Detection)

Reported

Devices as drawn

representative

Assumed

NFPA 75 covers

fire, smoke, corrosion, heat and water damage (Applying NFPA 75 in data centers)

Reported

Heat leaves the building

The headers turn up and out to the dry coolers.

Carries

nearly all of the hall’s heat

Assumed

Pipework as drawn

representative

Assumed

Pipe markers

green with white letters and a flow arrow, ASME A13.1 style (Pipe Color Codes: ASME A13.1, ISO 14726 & BS 1710)

Assumed

415 V AC in

Two tap-off cables from the overhead busway plug into the top of the rack: A and B feeds for redundancy.

Feeds

A + B

Assumed

Tap-off hardware as drawn

representative

Assumed

415 V AC → ≈50 V DC

Each 1U shelf, such as LITEON’s power shelf for NVL72 racks, holds six hot-swap rectifiers in a 3+3 arrangement that turn AC into about 50 V DC.

Shelf

≈33 kW, 6 × 5.5 kW (Custom power shelves for NVIDIA GB200 (ORv3, 33 kW, six 5.5 kW PSUs, up to 97.5% peak efficiency))

Spec

Shelves per rack

6 (Custom power shelves for NVIDIA GB200 (ORv3, 33 kW, six 5.5 kW PSUs, up to 97.5% peak efficiency))

Assumed

Efficiency

≈97.5% peak, half load (Custom power shelves for NVIDIA GB200 (ORv3, 33 kW, six 5.5 kW PSUs, up to 97.5% peak efficiency))

Spec

≈2,600 A down the back

A vertical copper busbar runs the full height of the rack. Every tray has a clip on its back that grabs the bar when it slides in, so there are no power cables to trays.

Voltage

≈50 V DC (OCP ORv3) (DGX GB Rack Scale Systems User Guide, Hardware)

Spec

Busbar rating

1,400 A per section (NVIDIA Contributes GB200 NVL72 Designs to Open Compute Project)

Spec

Next: NVIDIA Kyber, 2027

800 V DC, 45% less copper (NVIDIA 800 V HVDC Architecture Will Power the Next Generation of AI Factories)

Vendor

18 trays, 4 GPUs each

Each tray holds two superchips: two Grace CPUs and four Blackwell GPUs. GB200 and GB300 share this broad layout, while their GPUs and network interfaces differ. Enclosure details are representative.

Trays

18 (DGX GB Rack Scale Systems User Guide, Hardware; NVL72 AI Factory reference architecture, System Hardware & Components)

Spec

GPUs per tray

4 (DGX GB Rack Scale Systems User Guide, Hardware; NVL72 AI Factory reference architecture, System Hardware & Components)

Spec

CPUs per tray

2 (DGX GB Rack Scale Systems User Guide, Hardware; NVL72 AI Factory reference architecture, System Hardware & Components)

Spec

Tray power

≈6.5 kW

Calc.

Front panels as drawn

representative

Assumed

9 trays in the middle

Nine switch trays connect all 72 GPUs as one NVLink domain. One tray is pulled out for inspection; connections across that display gap are schematic. Remote access is limited by the interconnect and workload, not the GPU’s local HBM bandwidth.

Trays

9 (DGX GB Rack Scale Systems User Guide, Hardware; NVL72 AI Factory reference architecture, System Hardware & Components)

Spec

Switch chips per tray

2 (DGX GB Rack Scale Systems User Guide, Hardware; NVL72 AI Factory reference architecture, System Hardware & Components)

Spec

Bandwidth per GPU

1.8 TB/s (NVIDIA GB200 NVL72 Delivers Trillion-Parameter LLM Training and Real-Time Inference)

Spec

Domain total

130 TB/s (GB200 NVL72)

Spec

Front handles

gold, for removing the tray (This is the NVIDIA DGX GB200 NVL72)

Reported

≈5,000 copper cables

The rear cable cartridges join compute and switch trays. The GB200 user guide explicitly describes its backplane as passive copper; an older NVIDIA OCP article calls the cables active, so that wording alone does not establish their electronics. This view represents connectivity rather than a lane-by-lane wiring drawing.

Links

more than 5,000 passive copper (NVIDIA Contributes GB200 NVL72 Designs to Open Compute Project; DGX GB Rack Scale Systems User Guide, Hardware)

Spec

Total length

≈2 miles (A closer look at Nvidia's 120kW DGX GB200 NVL72 rack system)

Reported

Signaling

224G PAM4 (NVIDIA GB200 Interconnect Architecture Analysis: NVLink, InfiniBand and Future Trends)

Reported

Tray connection

blind-mate backplane, no loose cables (A closer look at Nvidia's 120kW DGX GB200 NVL72 rack system)

Reported

Cartridge hardware as drawn

representative

Assumed

Blue in, red out

Two vertical manifolds with dripless quick disconnects feed every tray. The disconnects limit leakage when a tray is serviced.

Liquid-cooled parts

GPUs, CPUs, switch chips (DGX GB Rack Scale Systems User Guide, Hardware)

Spec

Tray connection

blind-mate nozzles (This is the NVIDIA DGX GB200 NVL72)

Reported

Manifold hardware as drawn

representative

Assumed

Pipe markers

TCS supply and return, ASME A13.1 style

Assumed

The chattiest work lives here

Inside one rack a model layer’s math is split across GPUs, or its experts are spread across as many as its 72 GPUs. The GPUs trade partial results inside every layer, which only NVLink is fast enough for.

Traffic

every layer, many times per token (The Llama 3 Herd of Models)

Reported

Llama 3 405B, H100

tensor parallel 8, inside each server (The Llama 3 Herd of Models)

Spec

NVL72 wide expert parallel

up to 72 GPUs, the NVLink domain (How NVIDIA GB200 NVL72 and NVIDIA Dynamo Boost Inference Performance for MoE Models)

Reported

Scale-up: one domain

Nine switch trays connect all 72 GPUs. One of the nine is pulled out for inspection; highlighted connections across that display gap are schematic, not exposed physical cables.

Trays

9, 2 switch chips each (18) (DGX GB Rack Scale Systems User Guide, Hardware; NVL72 AI Factory reference architecture, System Hardware & Components)

Spec

Per GPU

NVLink 5, 1.8 TB/s (NVIDIA GB200 NVL72 Delivers Trillion-Parameter LLM Training and Real-Time Inference)

Spec

Copper, not light

Cable cartridges down the back carry more than 5,000 copper connections between compute and switch trays. Their physical routing is represented schematically; this is separate from the optical scale-out fabric.

Links

more than 5,000 (NVIDIA Contributes GB200 NVL72 Designs to Open Compute Project)

Spec

Power saved vs optics, NVIDIA

≈20 kW per rack (NVIDIA's Optical Boogeyman: NVL72, InfiniBand)

Vendor

Passive copper reach at 224G

≈1 m (IEEE 802.3dj electrical ad hoc, copper objectives (Ran, Cisco, Apr 2023))

Reported

Scale-up versus scale-out

The NVIDIA hardware shown uses copper for its NVLink domain and separate pluggable optics for the compute network. For comparison, Google scales up differently, not purely with light: within each 64-chip cube, most ICI links are copper, wired directly in a 3D torus, with optical transceivers only at the cube’s outer edges. Mirror-based optical circuit switches (OCS) then join whole cubes together. Google’s TPU v4 paper describes its switch: 3D MEMS mirrors that re-aim the light in milliseconds, with circulators sending both directions down one fiber, which halves the ports and cables. Google’s Jupiter data center network uses OCSes a layer up, where they take the place of the spine switches.

Ironwood superpod

9,216 chips, 144 cubes of 64 (Inside the Ironwood TPU codesigned AI stack)

Spec

ICI per chip, inside a cube

1.2 TB/s; mostly copper (Ironwood: the first Google TPU for the age of inference; TPUv7: Google Takes a Swing at the Merchant Silicon Market)

Reported

TPU v4: 4,096 chips in 64 racks

48 OCSes join them (TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings)

Spec

Packet switches for the same job, Google’s estimate

568 InfiniBand switches (TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings)

Vendor

Palomar OCS ports

136 × 136: 128 in use + 8 spares (TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings)

Spec

How it switches

3D MEMS mirrors, in milliseconds (TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings)

Spec

OCS power, reported

≈108 W vs ≈3,000 W for a 136-port packet switch (Google's Apollo: The $3 Billion Optical Circuit Switching Game)

Reported

Jupiter data center network

an OCS layer replaces the spine (Jupiter Evolving: Transforming Google’s Datacenter Network via Optical Circuit Switches and Software-Defined Networking)

Spec

Beyond the NVLink domain

Four compute-fabric OSFP cages sit on the front of each tray, one per GPU. The narrower QSFP cages serve storage and in-band networking. A subset of OSFP modules and patch leads is populated here for visibility; this is not a complete cabling plan. The NVLink switch trays connect through rear copper, not these optics.

Ports

72 (NVIDIA's Optical Boogeyman: NVL72, InfiniBand)

Reported

Aggregate per GPU

400 Gb/s (DGX GB Rack Scale Systems User Guide, Hardware)

Spec

4 GPUs each

Each tray is where the networks meet: copper NVLink at the back, pluggable scale-out and storage optics at the front, and CPU links inside. One tray is extended in an illustrative service position so its interior is visible; it is not an operating configuration.

GPUs

4 (DGX GB Rack Scale Systems User Guide, Hardware; NVL72 AI Factory reference architecture, System Hardware & Components)

Spec

Out-of-band

A small copper switch at the top runs the rack’s management network: firmware, sensors and power control, separate from the fabrics that move model data.

Rate

1–10 GbE class

Assumed

TOR switches

2 (DGX GB Rack Scale Systems User Guide, Hardware)

Spec

Out-of-band network, GB200 reference

every compute and switch tray’s management port, on SN2201 switches (Network Fabrics, NVIDIA DGX SuperPOD reference architecture (DGX GB200))

Spec

Leads as drawn

one RJ45 copper lead per tray, to the 1 GbE ports (NVIDIA SN2201 Switch Systems User Manual: Specifications)

Assumed

Ports as drawn

48 × 1 GbE + 4 × 100 GbE (NVIDIA SN2201 Switch Systems User Manual: Specifications)

Assumed

Free units above

cable manager + blanking panels

Assumed

Cool in, warm out

Supply comes up one side, fans out to every tray through dripless quick disconnects, and returns warmer down the other.

To liquid, this model

≈114 kW

Calc.

Rise across the rack

≈10 °C (GPU Thermal Density & Coolant Flow Specs: B200, GB200, MI300)

Reported

Supply → return, this design

≈45 → 55 °C (Hotter Than a Hot Tub: The 45°C Breakthrough to Cool AI's Biggest Machines)

Spec

Flow rate

sources disagree ≈5× (GPU Thermal Density & Coolant Flow Specs: B200, GB200, MI300)

Assumed

The last 13%

Power shelves, switch trays, optics and drives still shed heat into air, which leaves the back of the rack into the hot aisle.

To air, this model

≈17 kW

Calc.

Where the heat starts

Each tray carries its heat into its cold plates.

Per tray

≈6.5 kW

Calc.

≈50 V DC in

Spring copper fingers at the back of the tray grab the rack busbar. More than a hundred amps flows through this clip when the tray is working hard.

Tray power

≈6.5 kW

Calc.

Current at 50 V

≈130 A

Calc.

Finger count and housing

representative

Assumed

50 V → 12 V

Fixed-ratio converter bricks cut the voltage by about four and hand 12 V to the board. They are very efficient because they do not regulate. Vendors do not publish figures for this board, so the loss here is an estimate.

Efficiency

≈97–98%

Assumed

Loss, campus-wide

≈1.4 MW

Calc.

12 V → ≈0.8 V

Dozens of switching phases ring each GPU, each an inductor and a power stage switching at around a megahertz. They sit as close to the chip as they can, because every millimeter at a thousand amps costs power.

Phases per GPU

≈20–30

Assumed

Efficiency

≈91%

Assumed

Loss, campus-wide

≈5.3 MW

Calc.

Core current

≈1,300 A

Calc.

Board layout, traces and passives

representative

Assumed

4 per tray, 1,200 W each

Each GPU package is two large dies and 8 stacks of HBM. It is where most of the power in the building finally goes.

Power

≈1,200 W (NVIDIA GB200 NVL72 Specs, Pricing & AI Inference Benchmarks)

Reported

Transistors

208 billion (Blackwell Architecture)

Spec

Memory, from rounded rack total

≈186 GB HBM3e (GB200 NVL72)

Calc.

2 per tray

Each Arm CPU feeds two GPUs over a 900 GB/s coherent link and keeps its own LPDDR5X memory beside it.

Cores

72 Arm Neoverse V2 cores (Grace CPU)

Spec

CPU–GPU link

900 GB/s NVLink-C2C (Grace CPU; Grace Hopper Superchip)

Assumed

CPU memory

Low-power LPDDR5X is the Grace CPU’s system memory. The illustrated placement is representative; it is separate from the GPU’s HBM.

Capacity

480 GB LPDDR5X per CPU (17 TB per rack) (GB200 NVL72)

Calc.

Water on every hot chip

Copper plates with fine internal fins sit on each GPU and CPU. Coolant enters cool, picks up over a kilowatt per GPU, and leaves warm.

Heat per GPU

≈1.2 kW (NVIDIA GB200 NVL72 Specs, Pricing & AI Inference Benchmarks)

Reported

Plate, fitting and screw shapes

representative

Assumed

The front of the tray

ConnectX-7 SuperNIC cards carry scale-out traffic to the spine, 2 BlueField DPUs handle storage and security, and E1.S drives hold local data.

Scale-out

400 Gb/s per GPU (DGX GB Rack Scale Systems User Guide, Hardware)

Spec

NIC mounting

mezzanine boards on Mirror Mezz connectors (GB200 Hardware Architecture and Component Supply Chain & BOM)

Reported

To the front cages

DensiLink flyover cables (GB200 Hardware Architecture and Component Supply Chain & BOM)

Reported

Cable path and cage board

representative

Assumed

Power at the front edge

Each scale-out port’s module takes its power from the host it plugs into. Inside, the DSP is a major share of it.

1.6T transceiver, Broadcom figure

under 23 W (Broadcom Extends 200G-Lane DSP PHY Leadership for Next-Generation AI Infrastructure)

Vendor

Marvell Ara, Marvell figure

over 20% less module power (Marvell Unveils Industry's First 3nm 1.6 Tbps PAM4 Interconnect Platform)

Vendor

To the spine

High-density connectors at the rear mate with the copper spine when the tray is pushed home.

Per GPU

NVLink 5, 1.8 TB/s (NVIDIA GB200 NVL72 Delivers Trillion-Parameter LLM Training and Real-Time Inference)

Spec

NVLink 5

Each GPU’s NVLink links leave the back of the tray and mate with the copper spine when the tray is pushed home.

Per GPU

1.8 TB/s (NVIDIA GB200 NVL72 Delivers Trillion-Parameter LLM Training and Real-Time Inference)

Spec

CPU to GPU

Each Grace CPU talks to its GPUs over a coherent chip-to-chip link, so the GPUs can use CPU memory as a slower extension of their own.

Bandwidth

900 GB/s (Grace CPU; Grace Hopper Superchip)

Assumed

Four per tray

Four ConnectX-7 SuperNIC interfaces carry the four GPUs’ scale-out traffic, allowing communication with other racks without passing through the CPU.

NIC

ConnectX-7 SuperNIC (DGX GB Rack Scale Systems User Guide, Hardware)

Spec

Per GPU

400 Gb/s per GPU, NVIDIA’s reference design; ConnectX-8 upgrades to 800 Gb/s (DGX GB Rack Scale Systems User Guide, Hardware)

Spec

PCIe retimers, CPU to GPU

none on the reference board (GB200 Hardware Architecture and Component Supply Chain & BOM)

Reported

Electrons become light

Pluggable modules at the front turn the NIC’s electrical signal into light. Each 400G port takes a 400G module; at the switch end, one twin-port module carries two of these links.

NIC to cage

DensiLink flyover cables (GB200 Hardware Architecture and Component Supply Chain & BOM)

Reported

Cable path

representative

Assumed

A 400G module, e.g.

NVIDIA single-port 400G OSFP, DR4 (MMS4X00-NS400 400Gb/s single-port OSFP DR4 transceiver: specifications)

Spec

Its power

9 W max (MMS4X00-NS400 400Gb/s single-port OSFP DR4 transceiver: specifications)

Spec

Front-end network

A separate network carries user requests, storage and management. This tray has 2 BlueField-3 DPUs to offload network, storage and security work.

DPUs per tray

2 (DGX GB Rack Scale Systems User Guide, Hardware)

Spec

BlueField-3

up to 400 Gb/s (Introduction, NVIDIA BlueField-3 Networking Platform User Guide)

Spec

Role

storage, security, tenant networking (Introduction, NVIDIA BlueField-3 Networking Platform User Guide)

Spec

Where the links begin

Every one of these links starts at the edge of the GPU dies.

Links per GPU

NVLink, C2C, PCIe to the NIC (NVL72 AI Factory reference architecture, System Hardware & Components)

Reported

Water on every hot chip

Copper plates with fine internal fins sit on each GPU and CPU, lifted here to show the chips. The illustrated loop links them in series; exact production plumbing can differ.

Heat per GPU

≈1.2 kW (NVIDIA GB200 NVL72 Specs, Pricing & AI Inference Benchmarks)

Reported

Four GPUs, two CPUs

Almost all of the tray’s power ends up here, in a few square centimeters of silicon under each plate.

Tray heat

≈6.5 kW

Calc.

For what water misses

Small fans push air past the peripheral electronics that are not served by CPU/GPU cold plates.

Air-cooled parts

peripheral electronics (DGX GB Rack Scale Systems User Guide, Hardware)

Reported

Heat at the front edge

Heat leaves each module through its shell into the host cage and cooling hardware. The switch-side example has a finned top; server-side flat-top modules use the host’s cooling arrangement.

1.6T transceiver, Broadcom figure

under 23 W (Broadcom Extends 200G-Lane DSP PHY Leadership for Next-Generation AI Infrastructure)

Vendor

Marvell Ara, Marvell figure

over 20% less module power (Marvell Unveils Industry's First 3nm 1.6 Tbps PAM4 Interconnect Platform)

Vendor

OSFP body

107.8 × 22.58 × 13.0 mm (OSFP Form Factor: Complete Guide to 400G/800G/1.6T)

Reported

Dripless

Couplings at the back seal as the tray is pulled, so a tray comes out dry.

Per tray

one supply, one return per board (DGX GB Rack Scale Systems User Guide, Hardware)

Reported

A thousand-plus amps comes up here

Thousands of solder balls carry power and signals from the board into a many-layer organic substrate. Most of the balls are power and ground: at 0.8 V it takes many parallel paths to carry a thousand amps.

Core voltage

≈0.7–0.9 V

Assumed

Core current, P ÷ V

≈1,300 A over several rails

Calc.

Stiffener ring and capacitors, as drawn

representative

Assumed

Ball and bump pitch, as drawn

coarser than real

Assumed

CoWoS-L

Instead of one large silicon interposer, small silicon bridges embedded in the interposer carry the finest wiring: under the seam between the dies and under each die-to-HBM edge.

Packaging

TSMC CoWoS-L (Nvidia shifts to CoWoS-L packaging for Blackwell GPU production ramp-up)

Reported

Structure

bridge-based, not a monolithic silicon interposer (NVIDIA Blackwell B200: High-Performance Interconnect and Packaging Analysis)

Reported

Bridges and microbumps, as drawn

representative

Assumed

208 billion transistors

Two reticle-limit dies act as one GPU, joined by a 10 TB/s die-to-die link. Nearly every watt that reaches them, whether it runs computation, on-chip memory, communication or leakage, ends as heat.

Transistors

208 billion (NVIDIA Blackwell Platform Arrives to Power a New Era of Computing)

Spec

Die-to-die link

10 TB/s NV-HBI (NVIDIA Blackwell Platform Arrives to Power a New Era of Computing)

Spec

Process

TSMC 4NP (NVIDIA Blackwell Platform Arrives to Power a New Era of Computing)

Spec

Floorplan shown

illustrative x-ray

Assumed

8 stacks, 192 GB

Each stack, from suppliers such as SK hynix, Micron and Samsung, is 8 DRAM dies thinned and stacked with through-silicon vias. Moving model weights out of HBM for every token is a large share of inference energy.

Capacity

192 GB nominal; rack total implies ≈186 GB (GB200 NVL72; HBM3E product brief)

Spec

Bandwidth

8 TB/s (GB200 NVL72; HBM3E product brief)

Spec

Layers per stack

8 (HBM3E product brief)

Spec

Share of GPU power

≈8–15%

Assumed

Stack height, as drawn

about 3× real

Assumed

What leaves

Every token a model writes is a pass through billions of weights. Run the numbers below to see how many a kilowatt-hour buys.

Google, median Gemini text prompt

0.24 Wh, all-in (Measuring the environmental impact of AI inference)

Spec

LLaMA-65B on V100, 2023

≈3–4 J per token (From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference)

Spec

GB200 vs H200

≈8× tokens per MW (InferenceMAX: Open Source Inference Benchmarking)

Reported

8 TB/s, millimeters away

The fastest link in the building is the shortest: thousands of wires through the interposer between each HBM stack and the dies.

Bandwidth

8 TB/s (GB200 NVL72; HBM3E product brief)

Spec

10 TB/s die to die

The two dies join across their seam fast enough that software sees one GPU.

Bandwidth

10 TB/s (NVIDIA Blackwell Platform Arrives to Power a New Era of Computing)

Spec

NVLink 5 leaves here

Serializer circuits along the die edge push NVLink out through the package, 1.8 TB/s per GPU.

Per GPU

1.8 TB/s (GB200 NVL72; NVIDIA Blackwell Platform Arrives to Power a New Era of Computing)

Spec

What comes next

Today the GPU speaks copper and a module turns it into light. Switches already carry optical engines on the package; bringing them to the GPU would let scale-up reach beyond one rack.

GB200

electrical I/O only (DGX GB200 User Guide, hardware overview)

Reported

What leaves

After all those links, the output is small: a few bytes per token, sent back out the front-end network to whoever asked.

Per token of text

a few bytes

Assumed

Hottest point in the building

Transistors switching billions of times a second turn nearly every watt into heat right at the surface of the silicon.

Package power

≈1,200 W (NVIDIA GB200 NVL72 Specs, Pricing & AI Inference Benchmarks)

Reported

Compute dies, this model

≈1,042 W; the rest is HBM

Calc.

At full load, this operating point

≈75 °C

Assumed

Throttle point

near ≈85 °C; NVIDIA publishes none

Assumed

Like a stovetop, but denser

About 1,042 W through two reticle-size dies averages about 65 watts per square centimeter, several times a stove burner. Hot spots on the die run far higher, and those set the cold plate design.

Die area, two dies

≈16 cm² (NVIDIA Deep-Dives Into Blackwell Infrastructure: NV-HBI Used to Fuse Two AI GPUs Together)

Reported

Average flux

≈65 W/cm²

Calc.

Hot spots, cooling trade press

500+ W/cm² (GPU Thermal Density & Coolant Flow Specs: B200, GB200, MI300)

Reported

The first hop out

A thin thermal interface material carries heat from the dies and the HBM into a metal lid, or heat spreader, and on into the cold plate. The lid is drawn lifted and see-through so the heat shows through it; whether this package ships with one is not public. Each layer costs a few degrees.

Lid, as drawn

representative; not confirmed for this package

Assumed

Layers to coolant

die, interface, lid, interface, cold plate

Assumed

Heat in layers

Stacked DRAM traps heat between its 8 layers, and DRAM leaks more as it warms, so memory often sets the temperature limit before the GPU does.

Share of package power

≈8–15%

Assumed

HBM3e limit, Micron

105 °C (HBM3E product brief)

Spec

Back out to the cage

Gold fingers on both faces of the board carry power, control signals, and eight high-speed lanes each way. Ground contacts engage first, then power, then signals. Each signal pair connects into the module through copper traces and vias. The data view reveals the inner board layers near the connector so you can follow the complete path; routing is representative. The cage it plugs into is on the level you came from.

Edge connector

60 contacts: 32 high-speed (8 TX + 8 RX differential pairs), 4 control, 4 power, 20 ground (OSFP Module Specification, rev 5.22 (MSA text checked through the fluxlight.com mirror))

Spec

Supply

3.3 V on 4 power contacts, up to 2.5 A each (3.25 A on OSFP1600) (OSFP Module Specification, rev 5.22 (MSA text checked through the fluxlight.com mirror))

Spec

OSFP body

107.8 × 22.58 × 13.0 mm (OSFP Form Factor: Complete Guide to 400G/800G/1.6T)

Reported

Rails for every chip

The host supplies one voltage. Small converters on the module make the separate rails the DSP, the driver, the TIA and the lasers need.

Layout

representative, not one product

Assumed

A major draw

In the design drawn, one DSP handles eight 200G lanes in each direction, 1.6T each way, for both ports. DSP makers sell single chips for exactly this; NVIDIA does not publish whether its twin-port module uses one DSP or one per port. Its line-side lanes run in separate eight-channel TX and RX banks, divided into two four-lane, 800G optical ports at the fibers. Transmit and receive are not added to name the module rate. This is a representative architecture, not a claim about a specific module’s chip count. The DSP is a major power draw. The LRO comparison keeps it for transmit only; the LPO comparison removes this chip and shows its empty footprint.

Example: Marvell Ara DSP

one DSP, 8 × 200G in each direction (Ara 1.6T PAM4 DSP product brief)

Spec

One DSP for both ports, e.g.

Credo Cardinal 1650: 1.6T DR8 or 2 × 800G DR4 (Optical DSP product family)

Spec

DSPs in NVIDIA’s module

not published: one for both ports or one per port

Assumed

1.6T transceiver, Broadcom figure

under 23 W (Broadcom Extends 200G-Lane DSP PHY Leadership for Next-Generation AI Infrastructure)

Vendor

Marvell Ara, Marvell figure

over 20% less module power (Marvell Unveils Industry's First 3nm 1.6 Tbps PAM4 Interconnect Platform)

Vendor

Inside a twin-port module

two independent 800G ports (MMS4A00 1600Gbps, 2xDR4, Twin-port OSFP: Specifications)

Spec

Layout

representative, not one product

Assumed

Module power at 200G/lane, Semtech figures

LRO ≈16 W · full DSP 23–25 W · LPO target ≈10 W (200G LPO Power, Reach and Loss: Real Numbers)

Vendor

Credo Dove 850 (800G LRO DSP), Credo figure

up to 50% less DSP power (Credo Introduces World’s First 800G DSP for Linear Receive Optics, Targeting Hyperscale and AI Data Centers)

Vendor

Transmit-only DSP, Marvell

Ara T: 8 × 200G TRO; sampling from Q1 2026 (Marvell ushers in the 1.6T era with an expanded optical DSP portfolio)

Spec

LRO as drawn

same package, receive half hatched; representative

Assumed

Transmit amplifier

The driver amplifies each outgoing lane enough to swing a modulator’s electrodes. The converters feed it a separate rail, drawn as its own branch. A linear-drive (LPO) module keeps the driver and drops the DSP.

LPO keeps

a linear driver (TX), the TIA (RX) (Introducing Linear Pluggable Optics (LPO))

Reported

Layout

representative, not one product

Assumed

Light only

In a silicon photonics module, as drawn, separate continuous-wave lasers make steady light, because silicon cannot make light efficiently; the data goes onto it in the modulators. Other modules put the laser and modulator in one chip instead: NVIDIA’s 800G twin-port uses EMLs.

Design drawn

silicon photonics, separate lasers

Assumed

Short-reach DR8

directly modulated lasers are also used (Know Your 1.6T Transceiver)

Spec

NVIDIA’s 800G twin-port (MPO-16)

1310 nm EML lasers (MMS4X00-NM16 NVIDIA Ethernet 800Gbps OSFP Twin-port Finned Transceiver, 1xMPO16, 1310nm SMF up to 500m)

Spec

Back out to the cage

Gold fingers on both faces of the board carry power, control signals, and eight high-speed lanes each way. Ground contacts engage first, then power, then signals. Each signal pair connects into the module through copper traces and vias. The data view reveals the inner board layers near the connector so you can follow the complete path; routing is representative. The cage it plugs into is on the level you came from.

Host lanes

8 × 200G electrical, each way (Know Your 1.6T Transceiver)

Spec

Edge connector

60 contacts: 32 high-speed (8 TX + 8 RX differential pairs), 4 control, 4 power, 20 ground (OSFP Module Specification, rev 5.22 (MSA text checked through the fluxlight.com mirror))

Spec

Cleans up every lane, both ways

In the design drawn, one DSP handles all eight transmit lanes and all eight receive lanes, at 200G per lane: 1.6T each way, split into two independent 800G ports at the fibers. On transmit, host traces enter the DSP, then line-side traces fan out to the eight-channel driver and transmit bank. On receive, the eight-channel TIA feeds this same DSP, which recovers the signals and sends the data to the host. The LRO comparison keeps the DSP on transmit only and runs receive straight from the TIA to the host; the LPO comparison removes the DSP and connects the host directly to the linear driver and TIA.

One DSP for both ports, e.g.

Credo Cardinal 1650: 1.6T DR8 or 2 × 800G DR4 (Optical DSP product family)

Spec

DSPs in NVIDIA’s module

not published: one for both ports or one per port

Assumed

Example: Marvell Ara DSP

one DSP, 8 × 200G in each direction (Ara 1.6T PAM4 DSP product brief)

Spec

What it does

retiming, equalization, error correction (Know Your 1.6T Transceiver)

Spec

Example 1.6T DSP families

Marvell Ara (1.6T), Broadcom Sian3 (800G / 1.6T) (Marvell Unveils Industry's First 3nm 1.6 Tbps PAM4 Interconnect Platform; Broadcom Extends 200G-Lane DSP PHY Leadership for Next-Generation AI Infrastructure)

Spec

Half-retimed (LRO, RTLR)

DSP on transmit only; the host’s processing replaces a receive DSP (OIF Publishes Implementation Agreement for 112 Gb/s Retimed Transmitter Linear Receiver (RTLR) Electrical and Optical Interface Advancing Energy Efficiency; FAQs)

Spec

OIF RTLR interface

OIF-EEI-112G-RTLR 1.0, 112 Gb/s lanes, published 11/18/2025 (OIF Publishes Implementation Agreement for 112 Gb/s Retimed Transmitter Linear Receiver (RTLR) Electrical and Optical Interface Advancing Energy Efficiency)

Spec

Transmit-only DSP, Marvell

Ara T: 8 × 200G TRO; sampling from Q1 2026 (Marvell ushers in the 1.6T era with an expanded optical DSP portfolio)

Spec

LRO as drawn

same package, receive half hatched; representative

Assumed

Transmit only

The driver takes each outgoing lane from the DSP and swings a modulator’s electrodes with it, through bond wires to the photonic chip.

LPO keeps

a linear driver (TX), the TIA (RX) (Introducing Linear Pluggable Optics (LPO))

Reported

Layout

representative, not one product

Assumed

Light for the transmit side

Continuous-wave laser sources supply steady light to the transmit modulators. Their placement and coupling are representative; they do not feed the receive path.

Design drawn

silicon photonics, separate lasers

Assumed

Writing data onto light

Eight Mach-Zehnder modulators, one per lane. Each splits the lasers’ light into two arms, shifts one arm with the lane’s signal and recombines them, so the light brightens and dims with the data. Waveguides carry it to the fiber edge.

Kind

Mach-Zehnder, a common silicon-photonics modulator design (OFC Preview: The Race to 400G per Lane)

Reported

Design drawn

silicon photonics, separate lasers

Assumed

One fiber per lane, each way

Two MPO-12 connectors, one per DR4 half: positions 1–4 transmit, 9–12 receive, 5–8 unused. Half of each direction’s fibers go to each connector, which is why the transmit and receive fibers cross on the way.

Connectors

dual MPO-12 (Know Your 1.6T Transceiver; 800Gb/s Twin-Port OSFP 2x400Gb/s Single-Mode DR8 datasheet)

Spec

Fibers lit

16: 4 out and 4 in per connector (800Gb/s Twin-Port OSFP 2x400Gb/s Single-Mode DR8 datasheet)

Spec

Receptacles

two, ferrules vertical, 10.0 mm apart (OSFP Module Specification, rev 5.22 (MSA text checked through the fluxlight.com mirror); NVIDIA MMS4A00 1600Gbps, 2xDR4, Twin-port OSFP, 2xMPO, 1310nm Single Mode Transceiver: datasheet (PDF))

Spec

Splitter cables, 800G twin-port generation

one MPO-12 port split into two 2-channel ends (MFP7E40-Nxxx Single mode 1:2 Fiber Splitter Cable: product specifications (PDF); MFP7E20-Nxxx Optical Multimode Splitter Fiber Cable: product specifications (PDF))

Spec

Splitter fibers

8 lit at the 4-channel end, shared between the two ends (MFP7E20-Nxxx Optical Multimode Splitter Fiber Cable: product specifications (PDF); MFP7E40-Nxxx Single mode 1:2 Fiber Splitter Cable: product specifications (PDF))

Spec

Nose, saddle and receptacle bodies

shape representative

Assumed

Receive only

In a silicon photonics design, as drawn, light from each receive fiber runs along a waveguide to a germanium photodiode on the same chip, which turns it into a small current. Other designs use separate photodiode chips.

Receive, per module

8 photodiodes, one per lane (Know Your 1.6T Transceiver)

Spec

Layout

representative, not one product

Assumed

Receive only

The transimpedance amplifier turns each photodiode’s current into a voltage and sends it to the DSP, or straight to the host in an LPO or half-retimed (LRO) module.

What it does

photodiode current in, voltage out (Know Your 1.6T Transceiver)

Spec

LPO keeps

a linear driver (TX), the TIA (RX) (Introducing Linear Pluggable Optics (LPO))

Reported

In an LRO module

linear receive: the TIA output goes to the host (OIF Publishes Implementation Agreement for 112 Gb/s Retimed Transmitter Linear Receiver (RTLR) Electrical and Optical Interface Advancing Energy Efficiency; FAQs)

Spec

Linear receive, Semtech on LPO

the host SerDes is calibrated to supply the equalization (AI Data Center Basics: What Is Linear Pluggable Optics?)

Reported

A major heat source

The shared DSP is a major heat source. One gap pad carries its heat up into the shell. The LRO comparison keeps a transmit-only DSP under the pad; the LPO comparison removes both the DSP and its thermal pad, and an outline marks the absent chip’s footprint.

1.6T transceiver, Broadcom figure

under 23 W (Broadcom Extends 200G-Lane DSP PHY Leadership for Next-Generation AI Infrastructure)

Vendor

Marvell Ara, Marvell figure

over 20% less module power (Marvell Unveils Industry's First 3nm 1.6 Tbps PAM4 Interconnect Platform)

Vendor

Module power at 200G/lane, Semtech figures

LRO ≈16 W · full DSP 23–25 W · LPO target ≈10 W (200G LPO Power, Reach and Loss: Real Numbers)

Vendor

Credo Dove 850 (800G LRO DSP), Credo figure

up to 50% less DSP power (Credo Introduces World’s First 800G DSP for Linear Receive Optics, Targeting Hyperscale and AI Data Centers)

Vendor

Cooled by the host’s air

The module has no fan of its own. Its finned top sits in the air the switch or server moves past the cages.

OSFP body

107.8 × 22.58 × 13.0 mm (OSFP Form Factor: Complete Guide to 400G/800G/1.6T)

Reported

Lid print

OSFP 1.6T 2xDR4; the LPO view adds LPO

Assumed

Layout

representative, not one product

Assumed

Likely the package’s largest power draw

The switch chip does the switching and drives every lane a few millimeters to the optical engines around it. One of these packages is drawn; a Quantum-X Photonics switch holds four.

Q3450 switch, NVIDIA figure

3.95 kW (Unbox one of NVIDIA's first co-packaged optics samples with Lambda)

Vendor

Layout

representative; counts are NVIDIA’s

Assumed

Powered from the package

Each engine is an electronic chip bonded on top of a photonic chip, fed from the package substrate like the switch chip beside it. One engine does the work of one 1.6T module; this package holds 18.

Engine stack, as reported

electronic die above photonic die, TSMC SoIC-X (TSMC Celebrates 30th North America Technology Symposium with Innovations Powering AI with Silicon Leadership; Co-Packaged Silicon Photonics Switches for Gigawatt AI Factories)

Reported

Per switch chip

6 subassemblies × 3 engines = 18 (How Industry Collaboration Fosters NVIDIA Co-Packaged Optics)

Spec

Per engine

1.6 Tb/s each way, 8 × 200G (How Industry Collaboration Fosters NVIDIA Co-Packaged Optics)

Spec

Swappable, at the front

The lasers are kept out of the hot package, in modules at the front panel that can be replaced without opening the switch. Five are shown here; their allocation to this package is illustrative. The switch’s 18 serve its four packages.

Laser modules

18 per Q3450 switch, 8 lasers each (How Industry Collaboration Fosters NVIDIA Co-Packaged Optics; Unbox one of NVIDIA's first co-packaged optics samples with Lambda)

Reported

Transmit lanes per module

32 (How Industry Collaboration Fosters NVIDIA Co-Packaged Optics)

Spec

The lanes start here

The switch chip’s SerDes drive its transmit lanes to the optical engines around it and recover its receive lanes from them.

Per switch chip

28.8 Tb/s each way (How Industry Collaboration Fosters NVIDIA Co-Packaged Optics)

Spec

Per port, NVIDIA figure

as low as 9 W (Scaling AI Factories with Co-Packaged Optics for Better Power Efficiency)

Vendor

Electrical, millimeters

Copper traces in the package substrate carry each lane from the switch chip to an engine’s electronic chip: a few millimeters, where a pluggable module sits centimeters away across a board. That shorter path is where CPO saves its power.

Electrical loss, NVIDIA figures

≈4 dB, from 20–22 dB (Scaling AI Factories with Co-Packaged Optics for Better Power Efficiency; Unbox one of NVIDIA's first co-packaged optics samples with Lambda)

Vendor

Layout

representative; counts are NVIDIA’s

Assumed

Driver and TIA circuit blocks

The electronic die is shown with transmit-driver and receive-TIA circuit blocks. Short electrical bonds connect these circuits to modulators and photodiodes on the photonic die below. These blocks are functions within the illustrated EIC, not separate board-mounted chips. Their placement is representative; NVIDIA documents the stacked engine, not this internal floorplan.

Engine stack, as reported

electronic die above photonic die, TSMC SoIC-X (TSMC Celebrates 30th North America Technology Symposium with Innovations Powering AI with Silicon Leadership; Co-Packaged Silicon Photonics Switches for Gigawatt AI Factories)

Reported

Circuit partition drawn

driver/TIA functions on EIC; modulators/photodiodes on PIC

Assumed

Transmit

Tiny rings sit beside waveguides carrying laser light. Electrical driver signals shift each ring’s resonance, changing the transmitted light intensity to encode data. Slower thermal control keeps the operating point stable. The ring arrangement and waveguide routing are representative.

Modulators, NVIDIA figure

micro-rings, 200G PAM4 per wavelength (How Industry Collaboration Fosters NVIDIA Co-Packaged Optics)

Vendor

Per engine

1.6 Tb/s each way, 8 × 200G (How Industry Collaboration Fosters NVIDIA Co-Packaged Optics)

Spec

Receive

In the illustrated partition, incoming light reaches photodiodes on the photonic die. Their electrical currents pass through short bonds to receive-TIA circuits on the electronic die above. The TIAs amplify these currents into voltage signals; light does not enter the electronic die. Circuit placement is representative.

Per engine

1.6 Tb/s each way, 8 × 200G (How Industry Collaboration Fosters NVIDIA Co-Packaged Optics)

Spec

Layout

representative; counts are NVIDIA’s

Assumed

Circuit partition drawn

driver/TIA functions on EIC; modulators/photodiodes on PIC

Assumed

Light in, no data

Laser light reaches each engine by fiber from modules at the front panel, two fibers per engine, drawn here running beside its data fibers. It carries no data until an engine’s rings put some on it.

Laser modules

18 per Q3450 switch, 8 lasers each (How Industry Collaboration Fosters NVIDIA Co-Packaged Optics; Unbox one of NVIDIA's first co-packaged optics samples with Lambda)

Reported

Transmit lanes per module

32 (How Industry Collaboration Fosters NVIDIA Co-Packaged Optics)

Spec

Fibers per engine

8 transmit, 8 receive, 2 laser in (How Industry Collaboration Fosters NVIDIA Co-Packaged Optics)

Spec

A sealed fiber interface

Each engine has eight transmit fibers and eight receive fibers. They continue outward through a sealed interface toward front-panel ports outside this diagram. The separate lower amber fibers supply laser light; they do not join the data fibers or form an engine-to-engine loop. Quantum-X uses socketed electrical connections for its three-engine optical subassemblies. Spectrum-X uses a detachable optical connector. The drawn blocks and fiber bends are representative; NVIDIA shows microlens surface coupling at the photonic die.

Interface, Quantum-X

socketed subassemblies, sealed fiber interface (How Industry Collaboration Fosters NVIDIA Co-Packaged Optics)

Reported

Spectrum-X, for comparison

a detachable optical connector (How Industry Collaboration Fosters NVIDIA Co-Packaged Optics)

Spec

Fibers per engine

8 transmit, 8 receive, 2 laser in (How Industry Collaboration Fosters NVIDIA Co-Packaged Optics)

Spec

Front panel, Q3450

144 MPO connectors (Unbox one of NVIDIA's first co-packaged optics samples with Lambda)

Reported

One package, one plate

The switch chip and the optical engines around it share one package, and their heat goes up into one cold plate.

Cooling, Q3450 switch

liquid; 4 UDQ4 connections, dual internal loops (Unbox one of NVIDIA's first co-packaged optics samples with Lambda)

Reported

Cold plate drawn

one per package, representative

Assumed

Q3450 switch, NVIDIA figure

3.95 kW (Unbox one of NVIDIA's first co-packaged optics samples with Lambda)

Vendor

Water, not air

Water through the plate carries the package’s heat away. The lasers, at the front panel, stay out of it.

Cooling, Q3450 switch

liquid; 4 UDQ4 connections, dual internal loops (Unbox one of NVIDIA's first co-packaged optics samples with Lambda)

Reported

Cold plate drawn

one per package, representative

Assumed

Electrical signal processing

The host supplies DC power to the coherent DSP. Its transmit and receive processing both consume energy. The cited wattage is for the complete module, not the DSP alone.

800ZR module, FiberMall ZR page

24–25 W (800G ZR & ZR+ Coherent Modules)

Reported

Layout

OSFP envelope and cited nano-ITLA case to scale; other parts representative

Assumed

Electrical power to laser light

The laser’s electrical supply powers its light source and tuning controls. Optical carrier and local-oscillator light split from the same source; the two light paths are not electrical supply wires.

Shared laser, same-wavelength link

CW light splits to the modulator and receiver local oscillator (Coherent for Service Provider Edge and Access Network Applications)

Reported

Nano-ITLA, JLT 2023 research example

25.0 × 15.6 × 6.5 mm, under 3 W (Nano-ITLA research paper, Journal of Lightwave Technology vol. 41, no. 16 (2023))

Reported

EFFECT Photonics nITLA17

2.9 W (17 dBm nano ITLA for 400/800G coherent transceivers)

Spec

Layout

OSFP envelope and cited nano-ITLA case to scale; other parts representative

Assumed

Separate powered electronics

The discrete driver package has its own electrical supply and sends the modulation drive to the separate optical assembly. The supply feed and the high-speed signal links serve different purposes. No individual chip wattage is assigned.

800G-class research example

separate 128 GBd driver IC (IC Design Technologies for Enabling High-Speed Coherent Optical Communication)

Reported

Drawn as

discrete driver and TIA packages; separate optical assemblies, representative

Assumed

Layout

OSFP envelope and cited nano-ITLA case to scale; other parts representative

Assumed

Electrical drive and optical carrier

The modulator receives high-speed electrical drive from the separate driver and light from the laser. Bias and control connections are qualitative; the drawing does not assign the modulator its own share of module power.

Power and heat shown

qualitative; no component wattage assigned

Assumed

Drawn as

discrete driver and TIA packages; separate optical assemblies, representative

Assumed

Layout

OSFP envelope and cited nano-ITLA case to scale; other parts representative

Assumed

Photodetector bias

The photodiodes convert light to electrical current. Their illustrated bias feed is distinct from the incoming optical paths and the small detector signals going to the TIA. No receiver-optics wattage is specified.

Power and heat shown

qualitative; no component wattage assigned

Assumed

Drawn as

discrete driver and TIA packages; separate optical assemblies, representative

Assumed

Layout

OSFP envelope and cited nano-ITLA case to scale; other parts representative

Assumed

Separate receiver electronics

The discrete TIA package receives its own electrical supply to amplify detector currents for the DSP. Its supply wire carries power; the nearby photodiode connections carry the received electrical signal.

800G-class research example

separate 128 GBd TIA IC (IC Design Technologies for Enabling High-Speed Coherent Optical Communication)

Reported

Drawn as

discrete driver and TIA packages; separate optical assemblies, representative

Assumed

Layout

OSFP envelope and cited nano-ITLA case to scale; other parts representative

Assumed

Electrical signal processing

Host digital signals and optical-side analog signals land at distinct banks on the DSP package. Transmit processing and digital-to-analog conversion produce the driver signals; analog-to-digital conversion and receive processing recover data from the TIA outputs. These are processing paths, not straight wires through the chip. The DSP sits nearest the host connector, with the driver and TIA right beside its line-side edge. The four drawn host-path groups are schematic, not a lane count or pinout. Power figures cover the complete module.

Skew, OIF IC-TROSA

high-speed contacts grouped to suit common coherent DSP ball maps (Implementation Agreement for Integrated Coherent Transmit-Receive Optical Sub Assembly (OIF-IC-TROSA-01.0))

Spec

Makers, per one 2026 survey

Cisco (Acacia), Marvell, Ciena, Nokia (Infinera) (Tracking the Coherent DSP Supply Chain - 2026)

Reported

Electrical routes drawn

four host-path groups; schematic, not a lane count or pinout

Assumed

800ZR module, FiberMall ZR page

24–25 W (800G ZR & ZR+ Coherent Modules)

Reported

800G ZR tier, FiberMall guide

20–30 W (Coherent Optical Modules: The Complete Guide (100ZR to 800ZR))

Reported

Layout

OSFP envelope and cited nano-ITLA case to scale; other parts representative

Assumed

Electrical signal in, electrical signal out

The driver sits in its own board-mounted electronic package beside the DSP’s line-side edge, separate from the optical modulator assembly. It amplifies the DSP’s analog transmit signals and sends electrical drive across short board connections to the modulator’s RF end; the modulator’s fibers leave from its other end. These high-speed paths are kept short and direct, so the laser sits elsewhere. This package arrangement is representative.

800G-class research example

separate 128 GBd driver IC (IC Design Technologies for Enabling High-Speed Coherent Optical Communication)

Reported

Drawn as

discrete driver and TIA packages; separate optical assemblies, representative

Assumed

Optical packages, OIF agreements

RF pads at one end, fibers at the opposite end (Implementation Agreement for Micro Intradyne Coherent Receivers (OIF-DPC-MRX-02.0); Implementation Agreement for High Bandwidth Coherent Driver Modulator (OIF-HB-CDM-01.0); Implementation Agreement for Integrated Coherent Transmit-Receive Optical Sub Assembly (OIF-IC-TROSA-01.0))

Spec

Order as drawn

DSP → driver/TIA → optics → laser, short direct RF paths

Assumed

Common alternative, NeoPhotonics COSA

modulator co-packaged with drivers, receiver with TIAs (NeoPhotonics to Demonstrate at OFC a 64 Gbaud Coherent Optical Sub-Assembly (COSA) for Coherent 400G-1.2T Applications)

Vendor

Commercial example, 400G ZR/ZR+

CHR2094 driver paired with CHR1094 TIA (Coherent Unveils a Family of Integrated Circuits for Next-Generation Optical Transceivers)

Spec

Layout

OSFP envelope and cited nano-ITLA case to scale; other parts representative

Assumed

Electrical drive changes the light

Laser light enters the optical modulator. Electrical signals from the separate driver control nested Mach–Zehnder modulators, changing amplitude and phase for each polarization. The modulated light then leaves for the transmit fiber. Its RF pads face the driver and the DSP; both fibers, laser light in and modulated light out, use the far end, as in the OIF HB-CDM. This standalone optical assembly excludes the separately packaged driver.

Optical function

nested modulators for two polarizations (High-Bandwidth Coherent Driver Modulator (HB-CDM))

Vendor

Modulation

DP-16QAM at ≈118 GBd (800ZR and 800ZR+: The Coherent Pluggable Wave; 800G ZR & ZR+ Coherent Modules)

Reported

Drawn as

discrete driver and TIA packages; separate optical assemblies, representative

Assumed

Layout

OSFP envelope and cited nano-ITLA case to scale; other parts representative

Assumed

One precise color, used twice

One continuous-wave laser feeds an optical splitter. One branch supplies the modulator’s transmit carrier; the other supplies the receiver’s local oscillator, the light reference mixed with the incoming signal. This shared-laser design assumes transmit and receive use the same nominal wavelength. The laser tunes across the C-band. It is drawn toward the fiber end, with its pigtail running back to the splitter, so it never sits between the DSP and the analog chips.

Shared laser, same-wavelength link

CW light splits to the modulator and receiver local oscillator (Coherent for Service Provider Edge and Access Network Applications)

Reported

Nano-ITLA, JLT 2023 research example

25.0 × 15.6 × 6.5 mm, under 3 W (Nano-ITLA research paper, Journal of Lightwave Technology vol. 41, no. 16 (2023))

Reported

EFFECT Photonics nITLA17

2.9 W (17 dBm nano ITLA for 400/800G coherent transceivers)

Spec

Pigtail, vendor nano-ITLA example

PANDA fiber, bend radius 5 mm or more; Molex board connector (Nano Integrable Tunable Laser Assembly of C band (NITLA-C-17))

Spec

Tunes across

C-band, 1528.58–1567.34 nm (QSFP-DD and OSFP 800G ZR/ZR+ Coherent Optics Modules Data Sheet)

Spec

Integrated example, Furukawa IC-TROSA

laser at the LC end; driver and TIA at the RF flex end (Integrated Coherent Transmit-Receive Optical Sub-Assembly (IC-TROSA) for Data Center Interconnects, Furukawa Electric Review No. 54)

Reported

Order as drawn

DSP → driver/TIA → optics → laser, short direct RF paths

Assumed

Light becomes electrical current

Incoming light and local-oscillator light meet in 90-degree optical hybrids. Balanced photodiode pairs convert their outputs into four electrical signals: in-phase and quadrature for each polarization. Those currents travel over short electrical connections to the separately packaged TIA. The signal and local-oscillator fibers enter the far end, opposite the electrical outputs, as in the OIF micro-ICR. This standalone optical assembly excludes the TIA electronics.

Optical packages, OIF agreements

RF pads at one end, fibers at the opposite end (Implementation Agreement for Micro Intradyne Coherent Receivers (OIF-DPC-MRX-02.0); Implementation Agreement for High Bandwidth Coherent Driver Modulator (OIF-HB-CDM-01.0); Implementation Agreement for Integrated Coherent Transmit-Receive Optical Sub Assembly (OIF-IC-TROSA-01.0))

Spec

Optical receiver function

two 90° hybrids, four balanced photodiode pairs (OIF-DPC-RX-01.2: Integrated Dual Polarization Intradyne Coherent Receivers)

Spec

Drawn as

discrete driver and TIA packages; separate optical assemblies, representative

Assumed

Layout

OSFP envelope and cited nano-ITLA case to scale; other parts representative

Assumed

Detector currents become voltage signals

The TIA sits in its own board-mounted electronic package between the receiver optical assembly and the DSP, separate from both. It converts and amplifies photodiode currents into voltage signals and sends them a few millimeters to the DSP. Light stops at the photodiodes; no light enters this package. Its short connections are representative.

800G-class research example

separate 128 GBd TIA IC (IC Design Technologies for Enabling High-Speed Coherent Optical Communication)

Reported

Optical packages, OIF agreements

RF pads at one end, fibers at the opposite end (Implementation Agreement for Micro Intradyne Coherent Receivers (OIF-DPC-MRX-02.0); Implementation Agreement for High Bandwidth Coherent Driver Modulator (OIF-HB-CDM-01.0); Implementation Agreement for Integrated Coherent Transmit-Receive Optical Sub Assembly (OIF-IC-TROSA-01.0))

Spec

Order as drawn

DSP → driver/TIA → optics → laser, short direct RF paths

Assumed

Common alternative, NeoPhotonics COSA

modulator co-packaged with drivers, receiver with TIAs (NeoPhotonics to Demonstrate at OFC a 64 Gbaud Coherent Optical Sub-Assembly (COSA) for Coherent 400G-1.2T Applications)

Vendor

Commercial example, 400G ZR/ZR+

CHR1094: four channels, 64 GBd, wire-bonded die (Transimpedance Amplifiers (TIA))

Spec

Drawn as

discrete driver and TIA packages; separate optical assemblies, representative

Assumed

Layout

OSFP envelope and cited nano-ITLA case to scale; other parts representative

Assumed

800G on one wavelength

Where a DR8 module needs sixteen fibers, a coherent module sends everything on one wavelength over one fiber each way, so a DWDM system can stack dozens of them on a single pair across a region.

Connector

LC duplex (QSFP-DD and OSFP 800G ZR/ZR+ Coherent Optics Modules Data Sheet)

Spec

Receptacle as drawn

body, sleeves and port pitch representative

Assumed

Reach, Cisco modules

120 km amplified (800ZR) · over 1,000 km amplified (ZR+) · 75–80 km unamplified (QSFP-DD and OSFP 800G ZR/ZR+ Coherent Optics Modules Data Sheet)

Spec

Standard

OIF 800ZR, published 10/30/2024 (OIF Releases 800ZR Coherent Interface Implementation Agreement)

Spec

One module in one router port

The whole assembly is an OSFP pluggable: it slides into a router or switch port, and a pull on the tab at the fiber end releases the latch. The label sits where the OSFP specification recommends, on top at the fiber end. Here the finned top housing is lifted and drawn in x-ray so the parts inside stay visible.

Form factors, Cisco 800G ZR/ZR+

QSFP-DD and OSFP (QSFP-DD and OSFP 800G ZR/ZR+ Coherent Optics Modules Data Sheet)

Spec

OSFP module, MSA

22.58 mm wide, 13.0 mm tall (OSFP Module Specification, rev 5.22 (MSA text checked through the fluxlight.com mirror))

Spec

Label location, MSA

top face at the fiber end, about 15 × 20 mm (OSFP Module Specification, rev 5.22 (MSA text checked through the fluxlight.com mirror))

Spec

Length with pull tab, Cisco OSFP 800G

116 mm max (Cisco OSFP 800G Transceiver Modules Data Sheet)

Spec

Pull tab color as drawn

white, the MSA color for 1550 nm modules; no coherent row

Assumed

Label and tab as drawn

wording, stock and tab shape representative

Assumed

Conduction toward the shell

Signal processing turns electrical power into heat. The marks show a qualitative path toward the shell; the exploded display gap is not a physical thermal interface.

800ZR module, FiberMall ZR page

24–25 W (800G ZR & ZR+ Coherent Modules)

Reported

Drawn above the DSP

finned top housing, pedestal and gap pad; representative

Assumed

Layout

OSFP envelope and cited nano-ITLA case to scale; other parts representative

Assumed

Light generation and temperature control

Electrical losses in the tunable laser and its controls must leave through the module’s thermal structure. The shell is shown in x-ray for inspection. Pulse count is not wattage.

Nano-ITLA, JLT 2023 research example

25.0 × 15.6 × 6.5 mm, under 3 W (Nano-ITLA research paper, Journal of Lightwave Technology vol. 41, no. 16 (2023))

Reported

EFFECT Photonics nITLA17

2.9 W (17 dBm nano ITLA for 400/800G coherent transceivers)

Spec

Layout

OSFP envelope and cited nano-ITLA case to scale; other parts representative

Assumed

Transmit amplifier losses

Driving the modulators dissipates heat in the discrete driver package. The animation marks heat leaving this package toward the shell without specifying watts or a particular thermal interface.

800G-class research example

separate 128 GBd driver IC (IC Design Technologies for Enabling High-Speed Coherent Optical Communication)

Reported

Power and heat shown

qualitative; no component wattage assigned

Assumed

Layout

OSFP envelope and cited nano-ITLA case to scale; other parts representative

Assumed

Qualitative optical-component heat

Optical absorption and electrical drive or bias losses can produce heat around the modulator. These marks are qualitative; they do not imply that the modulator dissipates as much heat as the separate driver.

Power and heat shown

qualitative; no component wattage assigned

Assumed

Layout

OSFP envelope and cited nano-ITLA case to scale; other parts representative

Assumed

Distinct from TIA heat

Photodetection absorbs light, and detector bias can contribute heat. The illustrated path is qualitative. The TIA’s electronic losses are shown at its separate chip, not folded into this optical block.

Power and heat shown

qualitative; no component wattage assigned

Assumed

Layout

OSFP envelope and cited nano-ITLA case to scale; other parts representative

Assumed

Receive amplifier losses

The powered TIA dissipates heat while amplifying detector currents. Its distinct heat path identifies the separate electronic package; pulse count does not establish a power ratio against the optics, driver or DSP.

800G-class research example

separate 128 GBd TIA IC (IC Design Technologies for Enabling High-Speed Coherent Optical Communication)

Reported

Power and heat shown

qualitative; no component wattage assigned

Assumed

Layout

OSFP envelope and cited nano-ITLA case to scale; other parts representative

Assumed

About 0.1 W per end

No redriver or retimer to power; the plug’s small ID memory draws a little. NVIDIA’s documented GB200 NVL72 spine is passive copper too, as fixed cable cartridges rather than pluggable cables.

Power, NVIDIA DAC

≈0.1 W per end (Copper DAC and LACC Cables Overview)

Spec

In the signal path

no redriver, retimer or DSP (DAC, ACC, or AEC?)

Reported

GB200 NVL72’s NVLink spine

passive copper (DGX GB200 User Guide, hardware overview)

Spec

A couple of watts

One small analog chip in the plug, powered from the port.

Power and reach

a couple of watts per end, ≈3 m at 200G/lane (How Rack Power Density Is Opening the ACC Market)

Reported

Inside

a redriver on the receive direction; NVIDIA’s LACC has one in each end (Active Copper Cables: A New Class of Rack Interconnects for Further Optimizing AI; Copper DAC and LACC Cables Overview)

Reported

A DSP in each end

Two DSP retimers per cable, one in each plug, each drawing power from its port.

Power

≈20 W per end at 200G/lane; 2.5–3.5 W in one guide, lane rate unstated (How Rack Power Density Is Opening the ACC Market; Understanding High-Speed Copper Cables: DAC, ACC, and AEC)

Reported

Inside

a DSP retimer in each end: CTLE, DFE, CDR, FIR (Active Copper Cables: A New Class of Rack Interconnects for Further Optimizing AI; Understanding High-Speed Copper Cables: DAC, ACC, and AEC)

Reported

No active signal conditioning

Twinax pairs run straight from the plug’s card into the cable, so the copper’s own loss limits it to a meter or two at today’s rates, and it draws only a little power, for its ID memory. The documented GB200 NVL72 spine is passive copper too, though built as fixed cable cartridges the trays plug into, not cables like this one.

In the signal path

no redriver, retimer or DSP (DAC, ACC, or AEC?)

Reported

Power, NVIDIA DAC

≈0.1 W per end (Copper DAC and LACC Cables Overview)

Spec

Reach, 800G-class

1–2 m (DAC, ACC, or AEC?; Copper DAC and LACC Cables Overview)

Reported

GB200 NVL72’s NVLink spine

passive copper (DGX GB200 User Guide, hardware overview)

Spec

Layout

representative, not one product

Assumed

One redriver

A small analog chip boosts and equalizes the signal arriving at the plug, the receive direction, but does not recover its clock. NVIDIA’s version puts one in each end. It buys a little more reach for a couple of watts.

Inside

a redriver on the receive direction; NVIDIA’s LACC has one in each end (Active Copper Cables: A New Class of Rack Interconnects for Further Optimizing AI; Copper DAC and LACC Cables Overview)

Reported

Power and reach

a couple of watts per end, ≈3 m at 200G/lane (How Rack Power Density Is Opening the ACC Market)

Reported

Layout

representative, not one product

Assumed

A retimer in each end

Each end holds a DSP retimer, the same general kind of chip an optical module uses on its electrical side, for both directions: it recovers the clock and rebuilds the signal. That reaches several meters, at several times the power.

Inside

a DSP retimer in each end: CTLE, DFE, CDR, FIR (Active Copper Cables: A New Class of Rack Interconnects for Further Optimizing AI; Understanding High-Speed Copper Cables: DAC, ACC, and AEC)

Reported

Power

≈20 W per end at 200G/lane; 2.5–3.5 W in one guide, lane rate unstated (How Rack Power Density Is Opening the ACC Market; Understanding High-Speed Copper Cables: DAC, ACC, and AEC)

Reported

Layout

representative, not one product

Assumed

Qualitative heat transfer

The analog redriver draws power from the port and warms the plug. Moving marks illustrate heat reaching the case and surroundings. The case is lifted for inspection: the display gap is not a real thermal interface, and pulse count does not represent watts.

Inside

a redriver on the receive direction; NVIDIA’s LACC has one in each end (Active Copper Cables: A New Class of Rack Interconnects for Further Optimizing AI; Copper DAC and LACC Cables Overview)

Reported

Layout

representative, not one product

Assumed

Qualitative heat transfer

The retimer handles both signal directions and draws power from the port. Its heat must leave through the plug and its surroundings. This representative animation shows that transfer across the exploded display gap; it does not specify cooling hardware, temperature or an ACC-to-AEC power ratio.

Inside

a DSP retimer in each end: CTLE, DFE, CDR, FIR (Active Copper Cables: A New Class of Rack Interconnects for Further Optimizing AI; Understanding High-Speed Copper Cables: DAC, ACC, and AEC)

Reported

Layout

representative, not one product

Assumed