NVDA, AMD, AVGO, ANET, SMCI, VRT, ORCL, MSFT, GOOGL, TSM, MU, CRDO, CLS · US
Squeezing Silicon: Keeping GPUs from Sitting Idle | AI Infra & the Hundred-Billion Market
GPUs buy silicon; returns hinge on utilization. Collectives, HBM, schedulers, power/thermal, and inference batching decide effective output from hundred-billion AI CapEx. Hub NVIDIA; complements AVGO/ANET; constraints TSM/MU; physical layer VRT. See CPU industry.
Cite a section with a deep link, e.g. /en/r/ai-infra-gpu-utilization-2026#thesis
Snapshot
- Training MFU (indicative)
- ~30%–55%
- Infra invest stack
- Si → net → power → cloud
- Five idle drivers
- Comm/HBM/sched/therm/SW
- Book skeleton
- NVDA+net+HBM+liquid
Data as of ~2026-07/08 · Drawn from public filings, management commentary, industry research (IDC / TrendForce / SemiAnalysis-style estimates), and on-site equity packages. Industry theme note — not a single-name rating.
Thesis
Buying GPUs only buys silicon. Unit returns are decided by whether that silicon is fed. Idle time in clusters rarely means “the chip cannot compute”; it usually means waiting on collectives, starving on memory bandwidth, scheduler fragmentation, power/thermal caps, or tiny inference batches.
Proposition: the next phase of AI infra competition is systems engineering to squeeze silicon — whoever lifts productive FLOPs per wall-clock hour captures more of a hundred-billion-dollar data-center CapEx stack, or delivers more tokens per CapEx dollar.
On-site map: accelerators NVIDIA / AMD; networking Broadcom / Arista / Credo; racks & cooling Super Micro / Vertiv / Celestica; cloud CapEx & scheduling Microsoft / Alphabet / Oracle; manufacturing & memory TSMC / Micron. CPU orchestration: CPU industry deep dive.
1. Why GPUs sit idle
1.1 Utilization is not one number
| Metric | Meaning | Trap |
|---|---|---|
| SM / device busy | Hardware busy/idle | Waiting on memory still looks “busy” |
| MFU | Useful FLOPs / peak | Common in training; often low in serving |
| Goodput / tokens per GPU-hour | Business output | What operators and investors should track |
Indicative ranges (workload-dependent): steady large-scale training MFU often ~30%–55%; inference with small batches / long context can be worse. That gap is the addressable infra opportunity — not always “buy 2× GPUs”, but “+20%–50% useful output from the same fleet”.
1.2 Five idle sources
- Collective bubbles — compute waits on all-reduce / all-to-all (worse with scale and MoE).
- HBM / memory hierarchy starvation — activations, gradients, KV cache.
- Scheduler fragmentation — topology-unaware placement, preemption holes; continuous batching and prefill/decode disaggregation rewrite serving curves.
- Power / thermal caps — rack limits force clocks/power down.
- Software friction — frameworks, compilers, kernels, multi-cluster orchestration. CUDA’s moat is compounding “less idle” software.
1.3 Why it hurts more now
Deeper parallelism raises communication share; inference is production-critical; hyperscaler CapEx is already in the hundreds of billions annually — each +10ppt utilization is like delaying a cluster build.
2. The hundred-billion market
AI infra is a layered cake. Accents differ by source; direction is stable: accelerators remain the largest slice, while networking, HBM/advanced packaging, liquid cooling, and power gain share of growth and bargaining power.
| Layer | Near-term scale (indicative) | 2030 direction | Utilization role |
|---|---|---|---|
| Accelerators (GPU/ASIC) | Already multi-$100B run-rate territory for leaders’ DC franchises | Still largest; growth may normalize from hyper to high | Silicon + software stack |
| AI networking | High-growth tens of $B | Path to larger tens / low hundreds of $B | Kills collective bubbles |
| HBM / advanced packaging | Supply-constrained pricing | Tied to CoWoS/HBM capacity | Feeds compute |
| AI servers / liquid / power | Scales with rack kW | Tied to GW campuses | Sustained utilization |
| Cloud & scheduling software | Inside CSP margins / platforms | Agent / serving platforms | Compiler + scheduler |
Implication: as “getting cards” eases, pricing power partly migrates to goodput levers — networks, HBM, cooling, schedulers, and workload-fit ASICs.
3. How to squeeze silicon
Utilization levers × bottlenecks × names
| Lever | Bottleneck | Utilization elasticity | Investable map |
|---|---|---|---|
| Scale-up / scale-out fabric | Collective wait, NCCL bubbles | High — cuts idle during all-reduce | NVDA NVLink/Spectrum-X; AVGO; ANET; CRDO |
| HBM + memory hierarchy | Activation/KV cache spill | High — keeps compute fed | MU; TSM CoWoS; NVDA HBM attach |
| Cluster scheduler / continuous batching | Fragmentation, queue holes | Medium–High — software ROI | Hyperscaler stacks; ORCL; MSFT Azure |
| Inference stack (spec decode, MoE routing) | Low batch, KV bandwidth | High for serving fleets | NVDA software; AMD ROCm; CSP ASICs |
| Power / liquid cooling / power train | Thermal throttle, rack power cap | Medium — raises sustained clocks | VRT; SMCI; CLS |
3.1 Scale-up
NVLink/NVSwitch, UALink, Ethernet scale-up — lower card-to-card tax. Map: NVIDIA; Ethernet Broadcom / Arista; connectivity Credo.
3.2 Scale-out
Spectrum-X / RoCE / InfiniBand compete on collective completion time — congestion control, topology, NIC offload, optics/cables.
3.3 Memory pyramid
Training activation checkpoints vs memory; inference KV cache, prefix cache, paged attention, PD disaggregation. Map: Micron; packaging TSMC.
3.4 Software
vLLM / TensorRT-LLM / SGLang-class engines; topology-aware multi-tenant schedulers. Map: Microsoft / Google / Amazon; GPU cloud Oracle.
3.5 Power & thermal
Liquid cooling, CDUs, busbars, UPS/PDU. Map: Vertiv; rack speed SMCI; systems Celestica.
3.6 Custom silicon
TPU / Trainium / Inferentia / MTIA / Broadcom XPU raise energy-efficiency and $/token on fit workloads — complement, not instant GPU obsolescence. See AVGO.
4. Value chain
Wafer / advanced packaging (TSM…)
→ HBM / substrate / CoWoS
→ GPU/ASIC design (NVDA/AMD/AVGO/CSP)
→ NIC/switch/optics (AVGO/ANET/CRDO…)
→ Servers & liquid racks (SMCI/OEM + VRT…)
→ Campus power & construction
→ Cloud schedulers & model serving
→ Apps & agents (token demand)
| Side | Counterparties | Power / dependency |
|---|---|---|
| Upstream | Process, HBM, CoWoS, tools | Allocation decides “silicon exists” |
| Midstream | GPU / net / systems | Decides “silicon becomes a cluster” |
| Downstream | CSPs / model labs / enterprises | Decides “cluster has useful load” |
Supply risks: CoWoS/HBM; optics/copper lead times; power/cooling equipment; export controls reshaping sellable SKUs and China utilization curves.
5. Competition map
Theme core names
| Company | Layer | Strength | Weakness | Research stance |
|---|---|---|---|---|
| NVIDIA | GPU + system + software | CUDA + NVLink + networking stack | Valuation / export / customer ASIC | Core overweight |
| AMD | GPU + CPU | #2 AI + EPYC attach | Software / training share gap | Satellite / #2 bet |
| Broadcom | Custom XPU + AI networking | CSP ASIC + Tomahawk/Jericho | Customer concentration | Core complementary |
| Arista | Ethernet AI fabric | EOS + hyperscaler share | Cycle / InfiniBand share fight | Networking core |
| Super Micro | AI servers / liquid | Speed to rack | Margin / governance beta | Tactical |
| Vertiv | Cooling / power | Liquid + UPS for AI densification | Capex cycle lag | Infra satellite |
Three contests: (1) NVIDIA defining the AI supercomputer stack; (2) CSPs integrating ASICs + schedulers while still buying GPUs; (3) Ethernet camp standardizing AI scale-out (Arista / Broadcom).
Winners show measurable goodput, sit on migrating bottlenecks, and survive customer concentration (discount AVGO-like risk).
6. Scorecard (research use)
| Dimension (1–5) | NVDA | AMD | AVGO | ANET | SMCI | VRT |
|---|---|---|---|---|---|---|
| Theme fit | 5 | 4 | 5 | 5 | 4 | 4 |
| Moat / pricing | 5 | 3 | 4 | 4 | 2 | 3 |
| 12–24m visibility | 5 | 4 | 5 | 4 | 3 | 4 |
| Valuation cushion | 2 | 2 | 2 | 2 | 3 | 3 |
| Research stance | Core | Satellite | Core complement | Net core | Tactical | Satellite |
7. Portfolio framing
| Role | Names | Logic |
|---|---|---|
| Core α | NVDA | Defines utilization stack; size the position |
| Core complement | AVGO + ANET | ASIC + Ethernet fabric |
| Manufacturing constraint | TSM + MU | Packaging + HBM feed |
| Physical rack | VRT ± SMCI | Cooling/power vs systems beta |
| Cloud leverage | MSFT / GOOGL / ORCL | CapEx + software |
| Second source | AMD | Share option; software discount |
Track: CapEx guides; 800G/1.6T & IB vs Eth; HBM/CoWoS; tokens/GPU-hour & PD disagg; kW/rack & liquid penetration; export-control SKU impact.
8. Scenarios
| Scenario | Conditions | Theme implication |
|---|---|---|
| Bull | CapEx stays high; Eth + liquid accelerate; serving goodput jumps | Net/cooling/software elasticity > pure silicon |
| Base | GPU tightness eases slowly; utilization grinds higher; ASICs take some inference | Silicon + net + cooling triangle |
| Bear | CapEx cuts; utilization story fails; price wars | Rich system names de-rate; focus on orders & FCF |
9. Risks
10. Summary
Squeezing silicon is a contest under network, memory, scheduler, and power constraints. The investable hundred-billion story is shifting from “can I get GPUs?” to “watts and CapEx dollars per useful token.”
Research framing: NVIDIA as hub; Broadcom/Arista (+connectivity) for bubbles; Micron/TSMC for feed; Vertiv/liquid systems for sustained peaks; clouds for internal squeeze via software and ASICs. Build the book as silicon–network–power–cloud, and recalibrate with goodput and CapEx — not shipment headlines alone.