Loading
Taking longer than expected.
Reload page

B3IQ Blog · Knowledge · 9 min read

Best GPUs for AI Inference: Choose by Model and Workload

How to choose an inference GPU by model fit, memory, context length, and concurrency—and what to test before buying.

B3IQ Team

Fig. 1: Choosing GPUs for AI Inference — white text over a blurred landscape.

TL;DR

Why a universal GPU ranking doesn't work for inference

A GPU can handle one inference workload and fail to fit another. A smaller quantized model serving short requests may leave ample memory, while a larger model or longer requests can exhaust the same card. Concurrent requests also need space for their own working data, so model size alone cannot determine the right GPU.

Training benchmarks cannot settle the choice. They measure performance under a particular training setup, not the time a different model takes to produce its first token or serve several requests at once. Even memory fit depends on the inference runtime. For example, vLLM sizes its KV cache using memory available after loading the model and warming up the runtime.

Start by checking whether the model and intended workload fit in memory. Then compare hardware that supports the required configuration, and test your model at the context length and concurrency you expect to use.

Step 1: Calculate the weight-only memory floor

Start with the model’s parameter count and its chosen precision. For a dense 70-billion-parameter model at a nominal 4 bits per parameter, each parameter takes 0.5 bytes. Multiplying 70 billion by 0.5 gives 35 billion bytes, or 35 decimal GB. That figure is a weight-only floor, not a prediction of the GPU memory needed to serve the model.

Actual memory use can exceed the floor before a request arrives. Quantized weights may need scales and other storage overhead, while the runtime needs its own buffers. During inference, the KV cache also holds information for active requests, so context length and concurrent requests affect how much memory remains available. vLLM sizes KV cache separately after loading weights and profiling memory. Adding a fixed percentage to the weight-only floor cannot guarantee that a workload will fit.

Step 2: Add KV cache, context length, and concurrency

KV cache memory grows as requests use more tokens and more requests run at once. The cache stores information the model needs to generate the next token, so a longer conversation can occupy more memory even when the model weights stay the same. Concurrent requests need cache space too. Size against the context lengths and request counts you expect, rather than treating the weight-only floor as a deployment size.

With its extensible KV cache enabled, vLLM profiles free GPU memory after warmup and CUDA graph capture, then uses the available space to size the cache. Its startup log reports both KV cache capacity in tokens and a maximum concurrency estimate for a specified per-request length. A longer configured context lowers that estimate because each request can require more cache space. The log gives you a starting estimate, not a guarantee that your workload will reach that concurrency.

Under representative load, watch for vLLM warnings that requests were preempted and must be recomputed. Those warnings indicate that the available KV cache cannot hold the requests being served together. vLLM documents gpu_memory_utilization, max_num_seqs, tensor_parallel_size, and pipeline_parallel_size as controls to examine when cache pressure appears. Test any change against your latency and throughput targets before choosing a machine.

Step 3: Account for runtime buffers and non-weight consumers

A startup memory profile cannot account for every allocation an inference service makes later. In vLLM, CUDA graph capture and warmup happen before the runtime measures free memory for its KV cache. That measurement reflects the memory already in use, but it cannot see allocations that happen afterward. For example, an optional KV connector may allocate device buffers after sizing, or another inference process may begin using the GPU. vLLM's memory guidance recommends reserving space for those consumers.

Before choosing a GPU, test your intended runtime configuration with optional features enabled and representative requests under load. Check peak GPU memory use after any processes you plan to run alongside it have started. If later allocations leave too little room, adjust the runtime's memory target or the machine configuration, then retest. A fixed margin added to model weights cannot establish fit for allocations that depend on the runtime and deployment.

Step 4: Decide whether one GPU is enough, or you need parallelism

Adding GPUs does not automatically combine their memory. If you want to fit a model and its working memory across several GPUs, you need a runtime that supports a deliberate way to split the work. Even when one GPU fits the model, adding another may help with concurrent requests without splitting each request across cards.

Tensor parallelism splits work within model layers across GPUs. It can make room for weights or more KV cache, but the GPUs must exchange data during inference. Pipeline parallelism assigns different layers to different GPUs, which can suit a model spread across nodes or GPUs without a fast direct link. For a compatible mixture-of-experts model, expert parallelism distributes its expert networks instead.

Choose a supported strategy for your model architecture and runtime, then test it on the intended GPU layout. NVLink or InfiniBand can reduce communication costs where the hardware supports them, but additional GPUs can still add latency. Compare measured latency and throughput before assuming a multi-GPU configuration helps your workload.

Compare the hardware: RTX 5090, RTX PRO 6000 Blackwell Server Edition, H200 NVL

The RTX 5090, RTX PRO 6000 Blackwell Server Edition, and H200 NVL offer different memory capacities, interconnect options, and power requirements. Compare those specifications against your model and deployment needs rather than treating the three GPUs as a ranked list.

RTX 5090: the desktop ceiling

The GeForce RTX 5090 has 32 GB of GDDR7 memory, no NVLink/SLI support, and a 575 W total graphics power rating. Those specifications make it a candidate when your model’s weights, KV cache, and runtime allocations fit on one card. Two RTX 5090s do not automatically provide a shared 64 GB memory pool.

Before choosing a machine, check the exact card’s dimensions and cooling requirements against its chassis. NVIDIA specifies a 1,000 W minimum system power supply for the RTX 5090, and the full configuration still needs a power check. Confirm service support and any deployment restrictions with the machine provider. If your workload requires hardware-isolated GPU partitions or a particular enterprise support package, verify those features rather than assuming a consumer card provides them. The RTX 5090 can serve concurrent or production requests when the configuration supports them, but its specifications alone cannot establish how many requests it will handle at your latency target.

RTX PRO 6000 Blackwell Server Edition: the single-card data center option

The RTX PRO 6000 Blackwell Server Edition gives you 96GB of GDDR7 memory with ECC, which can accommodate larger single-card workloads than the RTX 5090’s 32GB. You still need to check the model’s weights, KV cache, and runtime allocations against the memory available under your intended load. ECC helps detect and correct memory errors, but it does not establish inference quality or speed.

For separate tenants, NVIDIA’s Multi-Instance GPU feature can divide the card into up to four isolated instances, each with its own allocated memory and compute. An individual instance cannot use the card’s full 96GB, so check the chosen partition against each tenant’s workload. NVIDIA lists PCIe Gen 5 for this edition, rather than an NVLink bridge for scaling a model across cards.

Server fit also depends on the chassis. The card draws up to 600W at its configurable maximum, and NVIDIA offers a dual-slot air-cooled version or a single-slot liquid-cooled version. Check your server’s power and cooling support before treating either version as a deployment option.

H200 NVL: scaling memory and interconnect together

H200 NVL provides 141GB of HBM3e memory and 4.8TB/s of memory bandwidth per GPU. NVIDIA specifies a two- or four-way NVLink bridge with 900GB/s of interconnect bandwidth per GPU. That combination makes H200 NVL worth evaluating when model weights and the KV cache exceed one card’s capacity and your runtime supports splitting the workload across GPUs.

NVLink does not automatically pool memory. You still need a supported parallel configuration, and each GPU needs room for its assigned model data, KV cache, and runtime allocations. Test the intended model, precision, context length, and concurrency before treating the combined card capacity as usable. For deployment planning, NVIDIA lists H200 NVL as a dual-slot, air-cooled PCIe card with configurable power up to 600W. NVIDIA marks its specifications as preliminary.

Comparison table: VRAM, interconnect, power, and runtime support

  • Specification · RTX 5090 · RTX PRO 6000 Blackwell Server Edition · H200 NVL
  • Memory · 32 GB GDDR7 · 96 GB GDDR7 with ECC · 141 GB HBM3e
  • Bandwidth · Not specified on the cited page · 1,597 GB/s · 4.8 TB/s
  • Interconnect · PCIe Gen 5. No NVLink · PCIe Gen 5 · PCIe Gen 5. Two- or four-way NVLink bridge at 900 GB/s per GPU
  • Maximum power · 575 W graphics power · Up to 600 W, configurable · Up to 600 W, configurable
  • Form factor · Two-slot Founders Edition · Dual-slot air-cooled or single-slot liquid-cooled · Dual-slot, air-cooled PCIe
  • MIG partitioning · Not specified · Up to four instances · Up to seven instances

NVIDIA labels the H200 specifications preliminary. The RTX 5090 bandwidth and MIG cells reflect what the cited product page states, not a conclusion about hardware support.

Step 5: Validate before you buy — B3IQ's representative checks

A GPU that fits your model in memory still needs to pass a representative inference test before you buy it. Configure the exact model, quantization, runtime, context limit, and request pattern you expect to use. Treat those settings as configured evidence, not proof of performance. A sizing tool’s memory and capacity projections are estimated evidence, while results from a test run are measured evidence.

Start the test with prompts and reference outputs drawn from your intended use. Check whether the model’s answers meet your quality requirements at the chosen quantization, then measure time to first token and the delay between subsequent tokens. Record throughput at your target request rate, and increase simultaneous requests to see when latency rises or requests fail. Include long prompts and output lengths that reflect real use, since short test requests can leave too much KV cache unused to expose a memory limit.

Keep the test conditions with every result, including GPU count, runtime version, context length, and concurrency. For example, vLLM profiles available memory after warmup, so an estimate made before the run cannot replace observing memory use under load. These are evaluation steps, not published B3IQ benchmark results. Use B3IQ’s sizing methodology and hardware planning guide for the detailed procedure, then compare the tested configuration with the current machine catalog.

Choosing your GPU: a summary framework

Choose a GPU for the model and workload you plan to run, not for its position on a leaderboard. Start with the weight-only memory floor in decimal GB. Then account for quantization overhead, KV cache at your intended context length and concurrency, and runtime buffers. If the workload exceeds one GPU’s memory, check whether your runtime supports a suitable parallel configuration. Multiple GPUs do not pool memory automatically.

Compare hardware after you have sized the workload. Before buying, test representative requests for output quality, time to first token, inter-token latency, throughput, and concurrency. Use B3IQ’s sizing methodology, hardware planning guide, and current machine catalog to narrow your options. Buy against the results you measure, not specifications alone.