AI Server GPUs Compared: VRAM, Bandwidth, and Power per Accelerator
Server-grade NVIDIA and AMD accelerators with a computed matrix of how many GPUs each open model needs. Every spec comes from the vendor datasheet linked in the table.
Which server GPU has the most memory?
The AMD Instinct MI325X has the most accelerator memory available today: 256 GB of HBM3e per GPU with 6,000 GB/s of bandwidth, according to AMD's product datasheet. The NVIDIA B200 follows with 192 GB of HBM3e at 8,000 GB/s, and the AMD Instinct MI300X also provides 192 GB at 5,300 GB/s. For transformer inference, a model must fit its weights plus KV cache inside the memory of the GPUs allocated to it, so VRAM capacity — not raw FLOPS — usually decides how many GPUs a deployment needs.
Server GPU comparison table
| Accelerator | Year | VRAM | Type | Bandwidth | TDP | GB per kW | Interconnect |
|---|---|---|---|---|---|---|---|
| NVIDIA B300 (Blackwell Ultra) | 2025 | 288 GB | HBM3e | 8,000 GB/s | 1400 W | 205.7 GB | — |
| AMD Instinct MI355X | 2025 | 288 GB | HBM3E | 8,000 GB/s | 1400 W | 205.7 GB | — |
| AMD Instinct MI325X | 2024 | 256 GB | HBM3e | 6,000 GB/s | 1000 W | 256 GB | — |
| NVIDIA B200 | 2025 | 192 GB | HBM3e | 8,000 GB/s | 1000 W | 192 GB | NVLink 5 (1.8 TB/s) |
| AMD Instinct MI300X | 2023 | 192 GB | HBM3 | 5,300 GB/s | 750 W | 256 GB | — |
| H200 SXM | 2024 | 141 GB | HBM3e | 4,800 GB/s | 700 W | 201.4 GB | — |
| H100 SXM | 2023 | 80 GB | HBM3 | 3,350 GB/s | 700 W | 114.3 GB | — |
| NVIDIA A100 80GB SXM | 2021 | 80 GB | HBM2e | 2,039 GB/s | 400 W | 200 GB | — |
| NVIDIA H100 PCIe 80GB | 2023 | 80 GB | HBM2e | 2,039 GB/s | 350 W | 228.6 GB | PCIe 5.0 x16 |
| NVIDIA A100 40GB SXM (EOL) | 2020 | 40 GB | HBM2 | 1,555 GB/s | 400 W | 100 GB | — |
Specs from vendor datasheets (source link on each accelerator's detail page). GB per kW = VRAM ÷ board power, computed from the same datasheet values — a rough proxy for memory capacity per rack watt.
How many GPUs does each model need?
The matrix below computes GPUs needed = required VRAM ÷ accelerator VRAM, rounded up. Required VRAM is Q4_K_M weights (4.8 bits per weight) plus a 10% loading overhead and 4k-context KV cache, using the same ModelFitService math as our model pages. Green = fits on one GPU.
| Model (Q4_K_M, 4k ctx) | Required VRAM | B300 (Blackwell Ultra) 288 GB | MI355X 288 GB | MI325X 256 GB | B200 192 GB | MI300X 192 GB | H200 SXM 141 GB | H100 SXM 80 GB | A100 80GB SXM 80 GB | H100 PCIe 80GB 80 GB | A100 40GB SXM 40 GB |
|---|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek V4 & V4-Flash | 187.4 GB | 1 | 1 | 1 | 1 | 1 | 2 | 3 | 3 | 3 | 5 |
| Llama 3.1 70B | 47.9 GB | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 2 |
| Llama 3.3 70B | 47.9 GB | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 2 |
| Qwen 3 32B | 22.7 GB | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| Gemma 3 27B | 20.2 GB | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
Weights-only math. Tensor-parallel inference also needs headroom for activations and NCCL buffers; production clusters typically reserve ~20% more VRAM than this table shows.
What is the minimum server GPU for 70B-class models?
A single 80 GB accelerator runs a 70B-parameter model like Llama 3.3 70B at Q4_K_M: the weights are about 48 GB, which fits with room for KV cache on one A100 80GB, H100 80GB, or H200. The A100 40GB SXM cannot hold it and needs two GPUs. The cheapest realistic path to a 70B model today is a used dual-A100 80GB server; for new hardware, one H100 PCIe 80GB (350 W, 80 GB HBM2e) is the entry point.
What does it take to run DeepSeek V4 locally?
DeepSeek V4 is a 284B-parameter mixture-of-experts model, so at Q4_K_M its weights alone are roughly 170 GB and the full requirement is about 187.4 GB. A single B200 (192 GB), MI300X (192 GB), or MI325X (256 GB) holds it; H200 needs two GPUs and H100 needs three. Its 13B active-parameter count keeps token generation fast relative to dense models of the same total size, but all weights must still be resident in VRAM.
SXM versus PCIe for AI servers
SXM modules (B200, H100 SXM, H200, MI300X, MI325X) mount directly on the server board and exchange tensor-parallel traffic over NVLink 5 at up to 1.8 TB/s per GPU, which matters when a model's layers are split across GPUs. PCIe cards (H100 PCIe, A100 PCIe) cap GPU-to-GPU traffic at PCIe 5.0 x16 speeds and suit single-GPU or loosely coupled deployments. If your model fits on one GPU, PCIe is simpler and cheaper; if you split models across GPUs, SXM plus NVLink pays for itself in inter-GPU bandwidth.
Power and rack planning quick facts
- The NVIDIA B200 draws 1,000 W per GPU — an 8-GPU HGX B200 node needs roughly 10 kW of power and liquid cooling.
- Instinct leads on VRAM per watt among current accelerators: MI325X (256 GB at 1,000 W) and MI300X (192 GB at 750 W) both reach 256 GB per kW; B200 sits at 192 GB per kW.
- A100 40GB is end-of-life; new deployments should treat 80 GB as the floor for serving 70B models on one GPU.