the bottleneck is everything around the gpu: 56.9 GB/s, 600 W, 2.00x vs 1.35x, 5.7 days

Fig. 1: Conceptual illustration of a silver GPU connected through a narrow interconnect to surrounding infrastructure on a charcoal drafting grid.

By An (@0xaqt), staff engineer at B3IQ.

Fig. 2: the bottleneck is everything around the gpu: 56.9 GB/s, 600 W, 2.00x vs 1.35x, 5.7 days

the gpu is never the bottleneck. every time we measured a slow job, the wall was somewhere else: the pipe that feeds the card, the power cap the card enforces on itself, the cache that misses by a few hundred tokens, the checkpoint that takes a week to leave the building. four walls, all measured where the owner runs, and the card in the middle was fast the whole time.

the industry talks about gpus the way it talks about the last mile: as if the card were the hard part. it is not. the card is the part that already works, and everything around it is the product.

Fig. 3: Figure 1 from the bottleneck is everything around the gpu: 56.9 GB/s, 600 W, 2.00x vs 1.35x, 5.7 days

the pipe

the card's copy engine, measured four ways as a cross-check: host to device at 56.9 GB/s, device to host at 56.6, device-to-device reads at 38.2 to 39.0, and 37.5 to 37.8 each way when both directions run at once. those are the numbers a healthy card makes, and they are read in seconds.

we read them because a card that trained at x8 or gen3 shows up here first, as a low host-to-device number. the industry's acceptance wall says the same thing. nvidia's dcgmi levels give every check a time budget: r1 under 2.5 s, r2 under 2.5 min on 4 gpus or 10.5 on 8, r3 under 10 / 35 min, r4 under 45 min / 2.25 h, and the targeted stress test passes at 0.95 of a per-gpu target rather than a perfect score. crusoe's rule is flatter: a gpu rated for gen5 x16 found running at gen3 x8 is flagged as critical.

the checks are cheap, and the wall is where bad silicon, bad slots and bad wiring all land first. the longest soak runs 24 to 72 hours; the pipe check is under 2.5 seconds.

when a renter asks whether a card is good, the answer is not one benchmark score. it is these four numbers plus the wall they imply: the failures all show up early, so the wall runs cheap checks first and long checks only to confirm.

Fig. 4: Figure 2 from the bottleneck is everything around the gpu: 56.9 GB/s, 600 W, 2.00x vs 1.35x, 5.7 days

the cap

a 12-minute bf16 burn on our h200 nvl sustained 538.6 tflop/s over 178 iterations with 0 mismatches. peak power was 601 W at 78 c. both cards burned at once: 548.0 and 562.2 tflop/s, a 2.6 percent spread, both near 597 W, and 76 to 80 c between them.

the interesting row is the throttle. swpowercap fired on 144 of 144 samples: the card is power-bound at its own 600 W part limit, not thermal. that is the opposite of a defect. it means the sustained number holds, because the limiter is the card's own design and not our cooling. the sm clock under load ran 855 to 1320 mhz, a 35 percent spread, which is what a power-bound card looks like: boost is not sustained, and the boost clock is exactly the number you do not promise a renter.

a card that throttles on its own spec is a card you can promise. the number to trust is the one that survived its own limits: 538.6 tflop/s, 178 iterations, 0 mismatches, on a card that never had to prove our cooling.

the cache

one sentence went to the model and the turn billed 25,267 uncached input tokens. the first call sent 42,036, the second call inside the same 15-second turn sent 50,937, and 8,183 of those tokens were byte-identical to the first. the cache floor for that model is 1,024 tokens; the mean classifyStructured prompt is 678, so it never qualified.

the arithmetic that follows is the whole case for the cache: sending something twice costs 2.00x uncached, versus 1.25 + 0.10 = 1.35x cached. by the fourth send it is 4.00x versus 1.55x. the rule of thumb is to cache anything you will send more than once, and nothing you will not. in the vendor's own testing on a 100-step task, automatic clearing of stale tool results cut token use by 84 percent.

the trace shows the same wall from inside: the double bill was a problem of layout, not of breakpoint count. the cache is the cheapest lever in the whole stack, and it misses by a few hundred tokens.

the lever is huge because the unit is small. a cache decision is made per prompt and pays on the second send. no hardware, no migration: just moving volatile content out of the cached prefix.

Fig. 5: Figure 3 from the bottleneck is everything around the gpu: 56.9 GB/s, 600 W, 2.00x vs 1.35x, 5.7 days

the checkpoint

a full fine-tune checkpoint costs 14 bytes per parameter: 16-bit weights plus 32-bit optimizer state, from the zero paper. a full fine-tune also saves optimizer state, which is bigger than the model itself, and the checkpoint carries both.

an 8b with optimizer state is about 112 GB out, which is about 15.6 hours on a 16 mbps uplink. a 70b is about 980 GB, about 5.7 days on the same link. for scale, the training dataset of 100 GB arrives in about 4.4 hours at 50 mbps, and a lora adapter for an 8b model is about 185 MB and leaves in about 1.5 minutes.

the asymmetry is the story: the same 8b checkpoint that leaves in 15.6 hours at 16 mbps leaves in 15 minutes at 1 gbps. the link is a dial with nearly two orders of magnitude on it, and the checkpoint is the thing that turns.

small adapters move fine; full checkpoints take hours over our link. lora renters are well served today. full fine-tunes of large models are not, until the link is faster or checkpoints can stay in the building on a volume. renters uploading from home go through the tunnel and are limited by their own upload speed. pull data onto the machine, push results to the renter's bucket, and the wall is the exit, not the training. our take: price the upload, not the gpu hours.

Fig. 6: Figure 4 from the bottleneck is everything around the gpu: 56.9 GB/s, 600 W, 2.00x vs 1.35x, 5.7 days

the waits

26 of 58 crowdsourced storage ideas are about waiting for weights, and 10 of the 11 strongest are. the whole catalog is 1.66 tb, so size is secondary. the storage pain that matters most is cold starts and weight loading, not disk space, and the model store is a speed problem before it is a space problem.

the posts say the same, louder: 102 posts from 88 people, 23 asking for a ready environment, 14 and 13 describing lost checkpoints or interrupted runs. the loudest need is a working environment on first boot, and the most painful loss is finished work.

what the number is not

none of these numbers is a benchmark. the copy-engine matrix is a cross-check on the python numbers, recorded and not judged. the burn is a verification run, not a score to chase. the cache arithmetic is plain multiplication. the checkpoint table is computed at line rate. each one states what is, and nothing here states anything else.

how it was measured

the pipe and the cap come from runs on our own hardware: a copy-engine matrix read on every burn, and a 12-minute bf16 battery run on the h200 nvl with both gpus idle. the cache comes from a single production turn, read from its own trace. the checkpoint sizes follow the zero paper's 14 bytes per parameter, with the times computed at line rate. the waits are a crowdsource: 58 storage ideas and 102 posts from 88 people.

the wall

four walls, and the card was never one of them. the pipe is read in seconds, the cap enforces itself, the cache misses by a few hundred tokens, and the checkpoint takes a week to leave. the gpu is the part that already works, and the product is everything around it.

the four walls share one property: each is cheaper to fix than the gpu, and each costs more than the gpu when ignored. the pipe is read in seconds, the cap enforces itself for free, the cache pays on the second send, and the checkpoint waits on a link that was already there.

the order that follows from the measurements: recipes and a container cache first, volumes and snapshots second. measure the wall, then remove it. whoever removes the wall wins, and the card in the middle was fast the whole time.

the numbers come from seven internal docs: the fleet copy-engine matrix, the host acceptance research, the fleet consolidation battery run, the prompt-cache trace, the cache arithmetic, the training-fit table and the crowdsourced training learnings. each figure in this article reproduces them verbatim.

---

Originally published by An (@0xaqt) on X on October 15, 2026.