Two nights of training on an 8-GPU box: the detector, the tiny models, and the 8-bit step

Fig. 1: Conceptual illustration of an open server with eight silver accelerator modules on a charcoal drafting grid.

By An (@0xaqt), staff engineer at B3IQ.

Fig. 2: Two nights of training on an 8-GPU box: the detector, the tiny models, and the 8-bit step

In short:

• One 8-card RTX PRO 6000 box trained YOLO11x from scratch on COCO for the full 600 epochs in 19 h 10 min, every card at 86% and 415 W mean for all 19 hours with no heat slowdown; the best score was 0.527 mAP50-95 at epoch 412, against the 0.547 Ultralytics publishes
• Eight cards were 6.4× one card on YOLO11m, 409 s to 64 s per epoch, and the missing 1.6× shows in the watts, 344 to 220 W per card, not in the busy column
• Model size decided who was busy: 32 tiny models left the GPUs at 10–14% with the CPUs at 94%, and the same tiny model fell from 94.6% to 9–15% after INT8 because two early layers carry values about 6× wider than the rest

This is b3iq, a neocloud that belongs to its customers: owners run their own jobs on boxes we keep fast and boring. We build and host these boxes, and we had spent weeks tuning this one: the PCIe switch, the drivers, the qualification numbers. A spec sheet is not a model coming out the other end. So this is test 2 of the lab notebook, after the six-hour language-model pretrain of test 1, and the brief was simple: push eight cards as hard as they will go, for as long as it takes, and keep every number.

Two nights, three jobs. Night one was forty small vision models trained side by side, the workload shape we expect most from edge vision, then the step a camera board demands: shrinking each model to 8-bit numbers. Night two was the standard COCO benchmark with the standard YOLO detectors, one card against eight, and then the one the week was building toward: the largest YOLO11 trained from nothing for its full 600-epoch schedule, a whole day's job. The box is one 8-card RTX PRO 6000 Blackwell Server Edition machine, bare metal, 96 GB per card, 128 CPU cores and 756 GB of memory, and all the data is public: COCO 2017 (118,287 training photos, 80 kinds of everyday object) and Food-101 (101 dishes, 101,000 photos).

The headline is that the detector trained at full tilt for 19 hours and landed two points under the published number on an untuned recipe. The surprises were in the software around the box, not in the box: the recipe that overfit after hour 13, the 8-bit step that collapsed, and the checkpoints with nowhere to go.

Fig. 3: One box, two nights: what the runs said

What ran, and what the words mean

A few words first, since this is a training article. A model starts as millions of random numbers; you show it a photo and the right answer, nudge the numbers toward the answer, and repeat for every photo. One pass over the whole dataset is an epoch: 118,287 photos for COCO, 115 s on this box for YOLO11x. Accuracy is measured on photos the model never trained on; for detection the score is mAP50-95, 0 to 1, and the published YOLO11 models top out at 0.547. From scratch means random starting weights, no pretrained model. A checkpoint is the model's weights saved to disk mid-run: best.pt for the best so far, last.pt for the latest. INT8 means rounding a model's numbers to 8 bits so it fits a camera board's chip; it usually costs a few points of accuracy, sometimes everything.

The six runs, in order: eight tiny models at once, one per card; 32 at once, four per card; INT8 of every one of them, with six calibration variants on the one that broke; YOLO11m from scratch on eight cards and then on one; a segmentation model and a transformer detector; and YOLO11x for 600 epochs.

Fig. 4: Two nights, six runs

Night one: forty tiny models, and who was busy

Food-101 was cut into "stores" of 12 dishes, 9,000 photos per store, with one MobileNetV3-Small (1.5M weights) per store. Eight stores trained in 218 s and 32 stores in 204 s, so forty models in under eight minutes, at 86.9–94.8% accuracy on their 12 dishes. The first question to ask is "will my workload use the GPUs?", and for these models the answer was no, even with 32 of them running: the 128 CPU cores decoded jpegs and augmented photos at 94% busy, while the cards read 10–14% busy and 6.5 GB of 96 GB each on average.

The next night answered the same question the other way. For a 57M-weight detector on 640-pixel photos, 86% mean per card over 19 hours, 415 W mean, 34–43 GB, with the CPUs at 16%, at 1,030 photos/s. In these two runs the bottleneck followed the model: the tiny models starved on CPU decode, the big one kept the cards busy.

Fig. 5: Who was busy: 32 tiny models against one detector

The dashboard showed it plainly: during the 32 small jobs the health tab had the GPUs between 0% and 59% and uneven, with the host CPU bar at 94%. The lever for small-model work is the photo pipeline, not more cards: decode on the GPU (nvJPEG, DALI) or pre-decode once to tensors. The box's 128 cores and 756 GB are what make 32 jobs at once possible at all.

Fig. 6: The owner dashboard during the 32 small jobs: GPUs between 0% and 59% and uneven, 4 to 5 GB of 96 per card at this reading, host CPU 94%, load 51 on 128 cores

Fig. 7: Meme, a man in a suit explaining to another across a desk: no matter what CPU and GPU you have, there will always be a bottleneck

Night one: the 8-bit step is where the edge work hides

MobileNetV3-Small, 1.53M weights, scored 94.6% in float on its 12 dishes and 9–15% after INT8, near chance, since 1 in 12 is 8%. Every tool agreed: ONNX Runtime with six calibration variants, PyTorch's fx post-training quantization, and three epochs of quantization-aware training (training with the rounding simulated, 29%) all lost it. The prepared graph with the fake rounding switched off still scored 94.6%, so the fault was the numbers, not the tooling.

Fig. 8: Meme, the half-finished horse drawing: the carefully shaded rear is labelled in float, 94.6%, and the scribbled head and legs are labelled in INT8, 9 to 15%

The numbers: the first two layers' activations, the values passed between layers, span −141 to +143 over 512 calibration photos, while every later layer sits within ±25, about 6× narrower. Eight bits give 256 levels per tensor. What the data shows is that keeping those two layers in float restores 89.2%, so the damage sits in them; why, range or weights or both, we did not isolate.

Fig. 9: Why the 8-bit step collapsed: two early layers carry values about 6× wider than the rest

Two fixes worked. Leaving the stem and block 1 in float and quantizing the rest restored 89.2%, 5.9 MB to 1.8 MB. Picking a different architecture, MobileNetV2 at 2.24M weights, gave 97.1% in float and 96.4% in INT8 with per-channel scales (95.0% with one scale per tensor), 8.5 MB to 2.5 MB, with no tricks. All of it measured on the CPU, on store 7 of the small-model night. For a team shipping to a camera board this can surface late; on the box it took about eleven minutes, because six variants ran side by side.

Fig. 10: Three ways to shrink the same store model

Night two: one GPU to eight, 6.4×, and where the rest went

YOLO11m from scratch on COCO: 409 s per epoch on one card, 64 s on eight. 409 ÷ 64 = 6.4, against the 8× ideal of 51 s. The per-card watts say where the missing 1.6× went: 344 W in the one-card run and 220 W each in the eight-card run, same model, same batch of 32 per card. The lower watts mean less work per busy second. Our read is that the cards were waiting, on CPU-side augmentation (the random edits made to each photo before a card sees it) and on the sync between cards; we did not isolate the two with a run. The busy column never showed the gap: 71% on one card, 75% on eight. NVIDIA defines that number as the share of time one or more kernels was running, so it counts time, not effort.

Fig. 11: One GPU to eight: 6.4×, and where the rest went

The other two short runs that night: YOLO11m-seg from scratch reached 0.133 mask mAP after 10 epochs at 90 s each, and RT-DETR-l, a transformer detector started from pretrained weights, ran at about 530 photos/s with the cards at 49% and 186 W. Memory was 15 to 28 GB of 96 GB on each of the four short runs, so there was room for bigger batches.

Night two: nineteen hours flat out

YOLO11x started at 06:35 UTC on 2026-10-18 and finished at 01:45 the next day: 600 epochs at 115 s each, 69,018 s of wall-clock, no restarts. A sidecar read the cards every 30 s, 18,400 samples (2,300 reads of 8 cards): 86% busy and 415 W mean per card, 459 W peak, about 64 kWh for the GPUs over the run. The hottest card peaked at 76 °C, with per-card maxima from 63 to 76 °C, and the card's own thermal margin never read below 8 °C. Throttle flags appeared on 0.24% of samples: a power-cap flag on GPU 2 for 15 samples, and the rest in the first 90 s. GPUs 2 and 7 ran warmest on this chassis, as they did in test 1, this time with headroom.

Fig. 12: Nineteen hours flat out: the numbers

The health tab's GPU cards at one reading during the run: 8 under load, none throttling, 77 to 100% per card, 416 to 440 W of 600 W, every card healthy.

Fig. 13: The owner dashboard's GPU cards during the run: 8 under load, none throttling, 77 to 100% per card, 416 to 440 W of 600 W, every card marked healthy

The dashboard's 24-hour view at hour 15 is one flat block from the start of the run (06:35 UTC; the chart's own axis is local time): utilization, memory, temperature, heat margin and power all level.

Fig. 14: The owner dashboard's 24-hour health charts 15 hours into the run: utilization, memory, temperature, heat margin, power and PCIe width, one flat block from the start of the run

The 7-day charts, read on 2026-10-20 and cropped to the plot area from Oct 14 on, put the two tests side by side. The language-model pretrain of 2026-10-14/15 sits near 100% utilization, about 520 W per card and averages near 75 °C across the eight cards; this run sits at about 85%, about 420 W and a flat 65 °C, under the 90 °C alert line.

Fig. 15: The owner dashboard's 7-day utilization chart, read on 2026-10-20 and cropped to the plot from Oct 14 on: the Oct 14 to 15 pretrain near 100%, the Oct 18 to 19 YOLO11x run at about 85%

Fig. 16: The owner dashboard's 7-day power chart, read on 2026-10-20 and cropped to the plot from Oct 14 on: about 520 W per card in the pretrain, about 420 W in the YOLO11x run, the dashed 600 W limit line above

Fig. 17: The owner dashboard's 7-day temperature chart, read on 2026-10-20 and cropped to the plot from Oct 14 on: the average of 8 GPUs near 75 °C in the pretrain with the hottest card near 80, a flat 65 °C through the YOLO11x run, the dashed 90 °C alert line above

The model you ship is from hour 13

The best score came at epoch 412, hour 13: 0.527 mAP50-95. The run kept going for six more hours and ended at 0.490. The logged values give the shape: 0.272 at epoch 5, 0.374 at 10, 0.466 at 25, 0.483 at 50, a dip to 0.474 at 100, then 0.503 at 200, 0.517 at 300, 0.527 at 400, 0.522 at 500 and 0.490 at 600. Train losses kept falling to the end (mosaic augmentation switches off for the last ten epochs by default); validation losses turned up after about epoch 400, and mAP followed them down. That is overfitting, the model learning the training photos instead of the task, visible in one plot.

Fig. 18: The model you ship is from hour 13

Fig. 19: The run's results.png: ten panels of train and validation losses, precision, recall, mAP50 and mAP50-95 over 600 epochs

The best.pt file held the hour-13 model. The trainer rewrites best.pt whenever the validation score improves and last.pt every epoch, so the night produced a 0.527 model rather than the 0.490 it ended on. The run also saved periodic checkpoints with optimizer state: 20 GB on the box's NVMe by the end, 1.8 GB copied off. The copying was a hand-rolled puller on a laptop, and it died once. The box is not durable storage; a training box needs a home for its checkpoints off the box, written as they appear.

Fig. 20: Meme, the bell-curve format: the two ends say its just overfitting, the crying middle says classifier accuracy is a complex subject with many different interactions and failure modes

The two points to the published 0.547 look like a recipe gap. We ran an untuned recipe and it overfit after epoch 412; Ultralytics' 0.547 is from its own run of the same model at the same 640-pixel size, with a recipe we did not reproduce. Our read is that heavier augmentation closes the gap; that is a hyperparameter change, and it is untested here.

Fig. 21: What 0.527 looks like: predictions on held-out COCO photos; the confident boxes are trains, planes, zebras, pizza and vases at 0.9 and kites at 0.8, with low-confidence boxes around them

What these numbers are not

0.527 is one run of an untuned recipe, and 0.547 is Ultralytics' number from its own run; this is not a reproduction of their result, and nothing here says the gap is hardware. The 6.4× is YOLO11m at batch 32 per card, one run each, 10 epochs on eight cards against 2 on one; a bigger batch or a faster photo pipeline could move it. The tiny-model numbers are for 1.5M-weight models with JPEG decode on the CPU; a decoded-tensor pipeline might load the cards differently. The INT8 collapse is one architecture on one dataset, measured on the CPU; it says MobileNetV3-Small has this problem here, not that INT8 is unsafe. The thermals are one chassis over one 19-hour run, not a rating. And the 19-hour time is one box of this card class on this recipe, not a speed record for the model.

What we take from it

We set out to push eight cards to their limit and find out what one box is worth when you ask it for a real day's work. Two nights answered: it carried a benchmark-grade job through 19 hours with no restarts, it scaled to 6.4× on eight cards, it said plainly when the GPUs were not the bottleneck, and its own dashboard kept the health record. The surprises were in the software around the box.

• Ask what the model is before promising GPU utilization. 1.5M-weight models fed by CPU JPEG decode did not load eight cards; the honest story there is forty models in under eight minutes and a photo pipeline to fix.
• A standard detector trains at full tilt on one box: 6.4× from one card to eight on YOLO11m, 34 to 43 GB of 96 used on YOLO11x, 19 hours with no restarts, hottest card 76 °C with 8 °C to spare.
• Checkpoints need a home off the box. The model worth keeping came at hour 13, 20 GB of weights sat on the box's NVMe, and the puller that copied some of them off died once.
• The 8-bit step is a test, not a checkbox. Whether a model survives INT8 depends on its architecture, and a 128-core box tries six calibration variants side by side in minutes.
• The dashboard does not see the job. The health curves show the run and the activity log shows SSH events; nothing records what ran, when, or which checkpoints were written. This report had to assemble that by hand.

Two questions stay open: whether heavier augmentation closes the last two points to 0.547 on this box, and whether GPU-side JPEG decode gets a box full of 1.5M-weight models off the CPU ceiling.

Everything above was measured on the box itself, on public data (COCO 2017 and Food-101), conducted on b3iq: a sidecar read every card every 30 s, 18,400 samples over the long run, and the owner dashboard's health tab was captured at each stage.

That box is a b3iq node. Most GPUs live in someone else's building and you rent time on them; we think the people who use GPUs should own them, and still have someone run the boring parts. That is b3iq: a neocloud that belongs to its customers. You own a bare-metal box like this one, we keep it fast and boring, and the dashboard in these screenshots is the owner's view of it. If you want one, b3iq.org.

Tell us which test you want next.

---

Originally published by An (@0xaqt) on X on October 20, 2026.