six hours on an 8-gpu box: the speed, the heat, and what the dashboard got wrong

Fig. 1: Conceptual illustration of a silver GPU cooling assembly with a restrained blue diagnostic trace on a charcoal drafting grid.

By An (@0xaqt), staff engineer at B3IQ.

Fig. 2: six hours on an 8-gpu box: the speed, the heat, and what the dashboard got wrong

in short:

• one 8-card rtx pro 6000 box trained a 1.38B model from scratch in 6 h 23 min, 463k tokens/s at the median step, with no heat slowdown in any sample
• two slots ran about 14 °C hotter than the coolest card and dipped to −3 °C of thermal margin without slowing; the fans cant be read on this box, so the cause is still open
• the owner dashboard matched the cards within about 2 points on utilization, 1–2 °C and 0.2 GB/s; power read about 5% high, and the Healthy verdict missed the margin, which the code now reads

b3iq is a neocloud that belongs to its customers: owners run their own jobs on boxes we keep fast and boring. this is test 1 of the lab notebook: a full pretrain overnight on one box, with the owner dashboard compared against the node's own logs the whole way. no card slowed for heat, and the only limit the cards reported was the 600 W power cap. the run was nanochat's speedrun on 8 rtx pro 6000 blackwell server edition cards, bare metal, on the night of 2026-10-14. it trained a 1.38B-parameter model from scratch with a median step rate of 463k tokens/s over 8,352 steps, 8.76B tokens in 340.6 minutes. the whole recipe took 6 h 23 min from the start of pretraining, with no errors, no kernel messages and no heat slowdown in any sample. two cards still ran out of thermal margin: gpus 2 and 7 kept touching zero and dipped to −3 °C.

Fig. 3: one box, six hours: what the cards said

what ran

the recipe is public: pretrain a gpt-style model on web text, test it, fine-tune it into a chat model, test that. it keeps all 8 gpus busy talking to each other the whole time. the first try crashed, because the recipe's prebuilt attention kernel has no build for this gpu: the published kernel is built for compute capabilities 8.0 and 9.0a, and this card is 12.0, so nanochat's loader falls back to pytorch's built-in attention after a one-line switch. a test with the recipe's default attention pattern then did 280k tokens/s, and full attention did 464k, 1.66x faster, so we switched. pretraining started at 23:10 utc: 1.38B parameters, fp8 math, 8,352 steps of 1M tokens each. it finished at 04:50 after 340.6 minutes. the chat fine-tune started at 04:58 and the main pipeline was done at 05:33; the first 3 minutes of the fine-tune were a dataset download with the gpus idle.

Fig. 4: the run, station by station

463k tokens/s, and it never drifted

throughput held at 463k tokens/s, the median of all 8,352 steps, and 90% of steps ran between 462k and 464k. benchmark checks and checkpoint saves pause the run and do not show in the 5-minute medians. 8.76B tokens went through between 23:10 and 04:50, 340.6 minutes.

the 8 cards talk to each other the whole time, and the per-step rate did not drift, even while gpus 2 and 7 were at their thermal margin. one card slowing for heat or power would have bent the whole curve, and nothing did.

Fig. 5: 463k tokens per second, steady for 5.7 hours

how hard the cards worked

nvidia-smi said 98.5–99.2% busy, and that number only says a kernel was running almost all the time. the finer hardware counters (nvidia's gpm, read once a minute on all 8 cards from 01:55, so they cover the last 2 h 55 min of pretraining) split that time up: the compute units had work 82.5% of the time, the tensor cores (the matrix units, where the training math happens) ran 36.0%, each compute unit was 25.4% full with threads, and the memory moved 48.9% of what it can. pcie traffic was 3.9 GB/s out and 4.0 in per gpu, about 6% of a gen5 x16 link each way.

Fig. 6: how hard the cards worked, six counters

separately from the counters, the model math works out to 306 TFLOPS per gpu: 463k tokens/s × 5.28 GFLOPs per token ÷ 8 gpus. the 5.28 is nanochat's own formula, 6 FLOPs per matrix-multiply parameter plus the attention term, worked out by hand because the recipe has no peak figure for this card and prints 0% for it. which peak you divide by matters. at the 2.3 GHz the cards actually held, 188 SMs × 1,024 FLOPs per clock is 443 TFLOPS dense bf16, and the run sat at 69% of it. against the server edition's rated 2,430 MHz boost (468 TFLOPS) it is 65%. against the fp8 peak it is 35%, because that peak is twice the bf16 one (886 against 443 TFLOPS) and only the large matrix multiplies run in fp8; attention and the rest stay in bf16, and at this size the gpus spend a fair share of time waiting on memory and on each other. for scale, a plain bf16 matrix multiply benchmark sustains about 376 TFLOPS on this card at 600 W in aurora infra's bench, so 306 TFLOPS of model math is about four fifths of what the card gives a bare matmul.

power sat at 523–550 W per card, swinging within each 2.3-second step and peaking at the 600 W board limit. clocks held 2.31–2.35 GHz. the 600 W cap was the only limit the cards reported: 82% of samples show it, and none show a heat slowdown. the 600 W power cap is the normal limit for these cards, and it was the only clock limit they showed.

Fig. 7: power, clocks and the three peaks

two slots at the card's own limit

six cards ran at 67–77 °C on average during pretraining. gpus 2 and 7 ran at 81 °C, peaking at 86 and 87 °C. these are passively cooled server cards, so the chassis fans do all the cooling, and two slots running much hotter than the rest points at airflow. whether that is a weak fan or the position of those slots is still open, because the chassis fans cant be read on this box: it has no ipmitool.

the better measure is the card's own thermal margin (temperature.gpu.tlimit). nvidia defines it as the margin in degrees to the card's maximum operating temperature; at 0 or lower the card may start cutting its clock for heat. gpus 2 and 7 kept touching zero and dipped to −3, 16 and 22 samples at or below zero. gpu 0, the coolest card, never went below 13 °C of margin. core temperature is the wrong alarm for this: the two cards hit −2 at core readings anywhere from 77 to 87 °C, so no fixed celsius line catches it, and the dashboard's 90 °C line never would.

Fig. 8: eight cards, two hot slots, and the margin counter

no slowdown happened. even at −3 the clocks only showed the normal 600 W power cap. this was at night; a warmer room would likely push these two over. the dashboard's temperature chart shows the two hot cards plainly, 81 and 82 °C at midnight eastern against 67 for the coolest, under a 90 °C alert line that sits above every card's highest logged temperature, 87 °C on gpu 7.

Fig. 9: the dashboard's temperature chart at midnight eastern: GPU 2 at 81 °C and GPU 7 at 82 °C, alert line at 90 °C

is the dashboard right?

the dashboard here is the b3iq owner dashboard, the signed-in page the machine's owner sees, with its health tab and gpu cards. every 10 minutes a script compared what the dashboard stores (the data behind the health tab and the gpu cards) with the node's own logs. it averaged both sides over the same 10 minutes, because single readings swing within each training step. at each stage change I also reloaded the page and read it.

Fig. 10: the comparison: both sides of the dashboard check

utilization, temperature, memory and pcie traffic are right: within about 2 points, 1–2 °C and 0.2 GB/s of the cards' own readings, across seven steady windows. power reads 10–32 W high on the page in every one of them, about 5%. both sides read the same power.draw field, so this is not explained yet. it is small, but it is consistent.

Fig. 11: dashboard versus the cards' own logs, seven windows

two more things the check turned up. the page goes quiet in a background tab: it skips its 60-second refresh while hidden, on purpose, and catches up when shown; a visible page runs one heartbeat behind, about a minute, because the host software reports once a minute. and the gpu cards read Healthy for all 8 while gpus 2 and 7 sat at their margin. the page's utilization, temperature, memory and pcie numbers match the cards; its Healthy verdict did not match the margin counter. the dashboard code now reads the margin counter too: a card with 3 °C or less of margin is counted as at or near its heat limit, and at 0 it is a fault.

Fig. 12: the owner dashboard with all 8 gpus at 100%: 8 under load, none throttling

what a 6-hour model can and cant do

the pretrained model scored 0.2707 on CORE, an average of 22 standard tests (reading, common sense, trivia, simple reasoning). nanochat's reference is the original openai gpt-2 checkpoint, which it scores at 0.2565. the model kept improving to the last step: validation loss fell from 3.17 to 0.704 bits per byte, and the quick CORE checks read 0.201 at step 2,000, 0.215 at 4,000, 0.260 at 6,000 and 0.284 at 8,000. the quick check uses fewer questions per test, so it reads a little higher than the full run.

Fig. 13: validation loss and the quick CORE checks over 8,352 steps

after chat fine-tuning, the generic chat model scored 66.3% on ARC-Easy, 52.4% on ARC-Challenge, 38.3% on MMLU, 3.3% on GSM8K and 11.0% on HumanEval, for a ChatCORE of 0.247. asked why the sky is blue, it names rayleigh scattering correctly, then says the atmosphere absorbs red light and that the sky is blue "even on a cloudy day". fluent, half right and confident is typical for a model this size. this box trained a model that scores above nanochat's gpt-2 reference on CORE overnight; the chat model on top of it is fluent, half right and confident.

Fig. 14: what the chat model scores

what these numbers are not

none of this is a benchmark of the card. 463k tokens/s and 69% of peak belong to this model size, this batch, fp8 on the big matrix multiplies and full attention; a bigger model or a different recipe will land somewhere else, and the 35% fp8 figure is the share of the fp8 peak this run used. the time is not a speedrun time either. this run used nanochat's default of 12 tokens per parameter, 8.76B tokens; the recipe's own speedrun trains the same depth to 8 tokens per parameter in about 1.5 hours on 8 h100s, cards with about twice this one's bf16 peak and nvlink between them, and a c++ port of the recipe reached CORE 0.2592 on the same card class in 135 minutes. more tokens bought the higher score here, not a faster box. the dashboard check is seven ten-minute windows on one box, averaged; it says the numbers match, not that every one-second reading does. the two hot slots are one chassis at night, and the cause is still open between a weak fan and slot position. and the model is a 1.4B trained on 8.76B tokens: it scores above nanochat's gpt-2 reference on CORE, and its chat answers are fluent, half right and confident.

what this test settled, and what it left open

• settled: the box holds a median 463k tokens/s for 5.7 hours of pretraining at the 600 W cap, with no heat slowdown in any sample
• settled: the dashboard's utilization, temperature, memory and pcie numbers are the cards' numbers, within about 2 points, 1–2 °C and 0.2 GB/s
• settled: a 1.38B model trained to 12 tokens per parameter on this box scores above nanochat's gpt-2 reference on CORE
• open: why the page reads power 10–32 W high when both sides read the same power.draw field
• open: whether gpus 2 and 7 run hot because of a weak fan or the position of their slots; reading the chassis fans would settle it

next up is a serving run on the same box. tell us which test you want after that.

how it was measured

one box: 8 rtx pro 6000 blackwell server edition, bare metal, 2026-10-14 23:10 to 2026-10-15 07:02 utc. nanochat at commit 92d63d4: depth 24, 1,536 wide, 2,048-token context, fp8 on 145 of 158 linear layers, 1M-token batches, attention through pytorch sdpa with the full-attention pattern, pytorch 2.9.1 and nccl 2.27.5 in the recipe's own environment. per-gpu readings every 30 s from nvidia-smi (utilization, memory, temperature, power, clock and clock reasons); thermal margin every 60 s from 00:13; hardware counters every 60 s from 01:55 with nvidia-smi dmon --gpm-metrics; dmesg -W watched for gpu, pcie and memory errors and stayed empty. the dashboard check read the stored samples behind the health tab every 10 minutes and compared 10-minute averages with the node's log. owner-measured on b3iq, on an 8-card rtx pro 6000 box.

sources

• nanochat at the commit we ran, the recipe, its flops formula and its gpt-2 reference score — https://github.com/karpathy/nanochat/tree/92d63d4e8bb4df75c3b71618f31ddde2378b2bcd
• the c++ port's 135-minute run on 8 rtx pro 6000 — https://github.com/karpathy/nanochat/discussions/862
• nvidia rtx pro 6000 blackwell server edition — https://www.nvidia.com/en-us/data-center/rtx-pro-6000-blackwell-server-edition/
• nvidia rtx blackwell pro gpu architecture whitepaper, peak rates — https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/quadro-product-literature/NVIDIA-RTX-Blackwell-PRO-GPU-Architecture-v1.0.pdf
• nvidia-smi, the t.limit sensor and the gpm counters — https://docs.nvidia.com/deploy/nvidia-smi/index.html
• dcgm's definitions of sm activity and tensor activity — https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html
• aurora infra's 600 w vs 300 w bench — https://aurorainfra.ai/blog/rtx-pro-6000-blackwell-max-q-vs-600w-benchmarks

---

Originally published by An (@0xaqt) on X on October 17, 2026.