Best GPUs for LLM Inference in 2026

Updated October 11, 2026. This guide is re-reviewed every two weeks against new model and GPU releases. All prices are launch MSRPs from our GPU database; we do not track street prices. Token speeds are third-party benchmark results, labeled with source and estimate status.

The best GPU for LLM inference depends on the largest model you want to run: VRAM capacity decides which models fit at all, and memory bandwidth decides how fast they generate tokens. For most people running local LLMs in 2026, the NVIDIA GeForce RTX 5090 is the best overall card because its 32 GB of GDDR7 is the only consumer VRAM pool that holds a quantized 70B model. The used GeForce RTX 3090 remains the best value at 24 GB, and the Intel Arc B580 is the cheapest new card that runs 7B–8B models well. This guide maps every current GPU into tiers by VRAM, gives minimum and recommended configurations for each popular model size, and breaks picks down by power and price bracket.

How do the best GPUs for LLM inference compare?

The table below lists every current consumer, workstation, and data-center GPU we recommend for local LLM inference, with datasheet specifications from our database and measured Llama 3.1 8B Q4_K_M token-generation speeds where we have them.

Benchmark attribution: the 215 tok/s RTX 5090 figure is our llama.cpp benchmark record (single user, batch 1, 4k context, mean of 3 runs); the RTX 5080, RTX 4090, RTX 4080 SUPER, RTX 3090, RX 7900 XTX, and Arc B580 figures are llama.cpp results recorded in our benchmark database from llama-bench runs on CUDA, ROCm, and Vulkan respectively. Where a card has no recorded 8B result we mark it n/a rather than estimate.

Which GPU tier fits your model size?

Pick the tier that matches the largest model class you run day to day. Within a tier, prefer more bandwidth at the same VRAM — once a model fits, bandwidth is the main speed limiter.

What are the minimum vs recommended configs for popular models?

The table below comes from our model-fit database, which computes VRAM requirements from parameter counts and quantization sizes (Q4_K_M ≈ 4.8 bits-per-weight plus KV-cache and activation overhead). Every row links to the full model page with per-GPU fit lists.

Two patterns matter. First, the 24 GB tier is where serious local models start — everything from Mistral Small up to Qwen 3 32B lists a 24 GB card as its minimum. Second, the jump from "runs" to "runs well" is usually one tier up: a 32B model at Q4 on 24 GB works, but the same model at Q6 with longer context wants the 32 GB RTX 5090. Check the exact card you're considering against any model's fit list on its model page, or run your own numbers with the VRAM calculator.

The 2026 model generation: GLM 5.3 Flash, DeepSeek V4.1 Flash, Qwen 3.8 Flash, Ling

Cycle check 2026-10-01 (window 2026-09-17 to 2026-10-01): two new official open-weight releases land in this window, both from Ant’s inclusionAI and both small-activation MoE designs that suit the mini-PC story better than the headline giants — Ling 3.0 Flash (124B total / 5.1B active, 256K context, MIT; native hybrid-linear KDA+MLA attention with 1/64 sparse MoE) and Ling Mini 2.0 (16.26B total / 1.43B active, 128K context via YaRN over a 32K native window, MIT; the vendor reports 300+ tok/s on H20). We also found draft repos (Ornith-1.5 −DFlash, Nex N2.5 −DSpark): those are speculative-decoding draft heads, not standalone models, so we deliberately do not list them as models. GLM 5.3 Flash (320B / 18B), DeepSeek V4.1 Flash (552B / 8B prefill, 16B decode) and Qwen 3.8 Flash-Next (125B / 6B) are unchanged this cycle.

Previous cycle (2026-09-17): the only new official open-weight LLM from the tracked labs is DeepSeek V4.1 Flash, published 2026-09-10 on Hugging Face. Earlier copies of this guide treated that slug as a 284B / 13B “0731 refresh” of V4-Flash. The official card is a different architecture: a 552B-backbone multimodal MoE with Causal Encoder-Decoder activation (about 8B parameters per token in prefill, 16B in decode) and a 1M-token context, MIT license. GLM 5.3 Flash (320B / 18B) and Qwen 3.8 Flash-Next (125B / 6B + 51B n-gram + 4B MTP) are unchanged this cycle. All three still activate only a small expert slice per token, so generation speed depends far more on memory capacity and bandwidth than on raw shader throughput — which is why 128 GB unified-memory machines now compete with big-VRAM GPUs for this class.

Measured and reported speeds (September 2026)

Third-party numbers with sources; entries marked estimate are community reports, not our measurements. Blank cells and “not yet measured” mean no credible public measurement of that exact model+hardware pair exists yet — we publish unknowns as unknowns. The 20–30 tok/s and ~21 tok/s DeepSeek figures below are for DeepSeek V4-Flash (0731), not the 2026-09-10 V4.1-Flash architecture; we keep them only as a prior-generation reference.

Unsloth Desktop: free local inference with Dynamic quants, MTP and DSpark

Unsloth Desktop is a free, open-source app (beta) for running and training models on your own hardware — macOS, Windows and Linux. It runs GGUF and MLX checkpoints on llama.cpp/MLX under the hood, exposes an OpenAI-compatible API, and adds the pieces that normally take setup: one-click model downloads, Unsloth’s Dynamic quant variants, automatic sampling-parameter tuning, and speculative-decoding features (MTP, DSpark) that switch on where a model supports them. Per Unsloth’s own FAQ, with web search and tool-call features turned off its inference speed “should match any other llama.cpp app” — the benefit is the app layer and the quant variants, not a different engine [s].

Unsloth publishes measured numbers for the September-generation models. Their hardware is mostly datacenter (B200) with one workstation datapoint (RTX PRO 6000); we reproduce them as vendor benchmarks and mark independent numbers on consumer catalog hardware as not yet measured.

Source: Unsloth GLM-5.3-Flash benchmarks [s]. The long-context column is the story: the optimized build more than doubles generation speed at 64K context. Adding MTP helps further — at 4K context Unsloth measured 58.6 tok/s with MTP off versus 86.5 tok/s with n=2 drafts (n=3 drops to 80.2, n=5 to 63.7; their guidance is to stop at n=2). MTP works out of the box in Unsloth Desktop with no extra files [s].

• DeepSeek V4-Flash (0731) + DSpark: DeepSeek’s DSpark speculative decoding is “superior to naive MTP” per Unsloth, and is automatically enabled in the Unsloth local UI. Unsloth measured 120 tok/s on a B200 vs a 60 tok/s baseline (2×); via llama.cpp’s DSpark support (PR 25784) with --spec-draft-n-max 3 they report ~1.9× faster inference [s]. This is the 0731 architecture — no V4.1-Flash numbers are published yet.
• Qwen3.8-Flash + MTP on workstation hardware: Unsloth measured 170 tok/s on a single RTX PRO 6000 versus a 100 tok/s baseline with MTP enabled, with no accuracy degradation claimed [s]. This is the one Unsloth datapoint on hardware our visitors can actually buy (see our RTX PRO 6000 Blackwell page).
• Qwen3.8-27B Dynamic NVFP4 (serving, vLLM): on 1× B200, NVFP4 total throughput 133.7 tok/s at batch 1 (1.49× over BF16) rising to 4407.2 tok/s at batch 64 (1.45×); per-user speed at batch 64 is 68.9 tok/s. NVFP4 requires Blackwell GPUs (RTX 50-series, DGX Spark, B200/B300) and vLLM — not the Desktop app; older GPUs should use the GGUF builds [s].
• Fitting rule of thumb (Qwen3.8): Unsloth’s guidance: “RAM+VRAM ≈ the quant size” — if the model fits, expect roughly ~20 tok/s generation-class speed on B200-class hardware and much higher batched throughput; if it doesn’t fit, it still runs but much slower via disk offloading [s].

For our mini-PC readers the relevant Unsloth guidance is GLM 5.3 Flash’s quant sizing: the 1-bit build is ~100 GB and Unsloth states it runs on 102 GB RAM/VRAM setups, with 3-bit (~128–150 GB) for 128 GB+ machines — i.e. the same AMD AI Max / Mac Studio class we list above [s]. Unsloth Desktop’s measured numbers on those consumer machines: not yet published.

Why the AMD AI Max mini PCs suddenly matter

AMD’s Ryzen AI Max+ 395 (Strix Halo) puts 16 Zen 5 cores, a Radeon 8060S iGPU and up to 128 GB of LPDDR5X-8000 behind a 256 GB/s unified pool — enough to hold Qwen 3.8 Flash at Q4 entirely in memory, and to attempt DeepSeek V4.1 Flash only with expert streaming. Machines in our mini-PC database that carry it:

• GMKtec EVO-X2 (Ryzen AI Max+ 395, 128 GB)
• MinisForum MS-S1 MAX (Ryzen AI Max+ 395, 128 GB)
• Minisforum MS-S1 MAX P495 (Ryzen AI Max+ PRO 495)
• ACEMAGIC F9A (Ryzen AI Max+ PRO 495)
• GEEKOM A9 Mega AI Workstation

Against a 32 GB RTX 5090 the trade is blunt: the GPU generates small dense models several times faster, but cannot hold any of this model generation resident. If your workload is these MoE Flash models, capacity wins — see our EVO-X2 vs Mac mini comparison and the VRAM calculator for your exact fit.

Which GPU is best in each power and price bracket?

VRAM sets the ceiling, but your PSU and budget set what you can actually install. Here are the picks per bracket, using MSRP and datasheet TDP from our database.

NVIDIA vs AMD vs Intel for LLM inference

CUDA remains the default software target for local AI tooling, so NVIDIA cards have the fewest compatibility problems. AMD cards run llama.cpp well through ROCm and Vulkan — our RX 7900 XTX record of 76 tok/s was measured on ROCm — but some tools assume NVIDIA and need configuration. Intel Arc cards work through Vulkan with the youngest ecosystem of the three; fine for llama.cpp, rougher elsewhere. If you tinker, AMD's VRAM-per-dollar wins; if you want first-attempt success, stay NVIDIA. Full breakdown: AMD vs NVIDIA for AI.

The clean way to compare vendors is to hold the model fixed and let only the card and its backend change. Every row in the table below is Llama 3.1 8B at Q4_K_M, single user, batch 1, 4k context, mean of three runs, recorded on 22 May 2026 from a single benchmark source, so a CUDA row and a ROCm row on the same model are measuring the software stack as much as the silicon.

Benchmark attribution: all nine rows come from one llama.cpp build recorded on the same date under the same configuration, so the spread between them is attributable to the card and its backend rather than to the harness. The Intel Arc B580 is deliberately absent: its 41 tok/s comes from a separate Vulkan benchmark recorded three days later and reported as a ~40–42 tok/s range, so folding it into like-for-like rows would overstate what we know.

Two rows make the backend effect measurable instead of anecdotal. The RTX 6000 Ada and the Radeon RX 7900 XTX are both rated at 960 GB/s, ran the same model at the same quantization on the same day, and produced 110 tok/s against 76 — a 45% gap no pair of datasheets explains, because bandwidth is identical and the only variable left is the stack underneath it. The RX 7900 XTX's 24 GB holds the same 32B-class models as the RTX 4090's at a $999 MSRP against $1,599, yet still loses 49 tok/s on generation — the real cost of the AMD route once the card is installed.

Bandwidth also fails to order this field on its own. The RTX 4070 SUPER generates 92 tok/s on 504 GB/s while the RTX 3090 manages 85 on 936 GB/s — a lower-bandwidth card beating a higher-bandwidth one inside the same test. Per unit of bandwidth, efficiency runs from about 0.08 tok/s per GB/s on the RX 7900 XTX to about 0.18 on the RTX 4070 SUPER, a more than twofold spread. Bandwidth sets the range; it does not rank the cards inside it.

Capacity is the clearest boundary in the same table. The 48 GB RTX 6000 Ada generates 110 tok/s against the 24 GB RTX 4090's 125, so doubling the VRAM bought the professional card nothing on throughput here. Workstation memory decides which models fit at all, not how fast the ones that already fit generate.

How did we rank these GPUs?

Specifications come from manufacturer datasheet values stored in our GPU database (VRAM, memory type, bandwidth, TDP, launch MSRP). Token speeds come from our benchmark database, where every entry carries its source, software stack, and estimate flag — llama.cpp/llama-bench results on CUDA, ROCm, Metal, or Vulkan, plus Procyon and MLPerf-derived records where noted. We rank VRAM capacity first for LLM use, then bandwidth, then power draw, then price. We deliberately report MSRP only, never street prices, because those change constantly. Affiliate links do not influence rankings. Details: our methodology.

Frequently Asked Questions

How much VRAM do I need to run a 70B model locally?

About 40 GB at Q4 quantization before context overhead, which is why our model-fit database lists 48 GB cards like the RTX 6000 Ada as the minimum for Llama 3.x 70B and 80 GB as the recommendation for full context. A 32 GB RTX 5090 can run 70B at Q4 tightly — our Tom's Hardware-sourced record shows roughly 42 tok/s, flagged as an estimate — while 24 GB cards require heavy CPU offload and drop to unusable speeds.

Is the RTX 5090 overkill for local LLMs?

No, if you run 32B-class models or larger regularly — it is the only consumer card that holds them at higher quantizations with long context, and its 215 tok/s on 8B models makes small-model experimentation feel instant. Yes, if you only run 7B–8B models: those run at 38–85 tok/s on cards costing a quarter of the price, already faster than reading speed.

Is more VRAM or more bandwidth better for LLM inference?

VRAM first, bandwidth second. A model that doesn't fit in VRAM runs orders of magnitude slower from system RAM regardless of bandwidth. Once the model fits, generation speed scales close to linearly with bandwidth — the 1,792 GB/s RTX 5090 is why it beats the 936 GB/s RTX 3090 despite both fitting the same quantized models below 24 GB.

Is a used RTX 3090 still worth it for LLMs in 2026?

Yes — it remains the cheapest path to 24 GB, which our model-fit data shows is the minimum for the most capable local models (Gemma 3 27B, Qwen 3 32B, DeepSeek R1-Distill 32B). Accept the trade-offs: no warranty, unknown history, 350 W draw, and a 2020-era feature set. Our used RTX 3090 buying guide covers how to evaluate individual cards.

Can I run LLMs on multiple cheaper GPUs instead of one big one?

Yes, with caveats. Two 24 GB cards give 48 GB total, enough for 70B at Q4 — but model layers split across a PCIe link without NVLink run slower than on one large card, and not all tools split cleanly. Our multi-GPU setup guide covers which configurations work. For most buyers, one card whose VRAM fits the model beats two smaller ones.

Do I need an RTX PRO 6000 or H100 for local AI?

Only above the 70B single-card ceiling or for multi-user serving. A single RTX PRO 6000 Blackwell (96 GB) runs 70B at high quantizations with long context; H100/A100 systems earn their cost when many users share one deployment. Hobbyist and single-user workloads through 32B fit comfortably on consumer tiers.

Sources

Specifications are manufacturer datasheet values from our GPU database. Benchmark figures are from the sources below, as recorded with per-entry attribution in our benchmark database.

• llama.cpp / llama-bench community benchmarks — https://github.com/ggml-org/llama.cpp
• Tom's Hardware GPU benchmarks — https://www.tomshardware.com/pc-components/gpus
• MLCommons MLPerf Inference — https://mlcommons.org/benchmarks/inference-datacenter/
• NVIDIA official specifications — https://www.nvidia.com/en-us/data-center/
• AMD Instinct specifications — https://www.amd.com/en/products/accelerators/instinct.html
• Apple Mac Studio technical specifications — https://www.apple.com/mac-studio/specs/

Related reading: our ranked shortlist in Best GPU for Local LLMs, the VRAM calculator, per-model hardware fit pages in the model index, and the deep learning training guide.

Disclosure: CompareAIHardware.com participates in the Amazon Associates program and earns from qualifying purchases through links on this page. Affiliate relationships do not influence our recommendations.

Related Guides

Best GPU for Local LLMs

Our ranked 7-card shortlist with benchmarked picks

VRAM Calculator

Compute exact VRAM needs for any model and quantization

Model Fit Index

Per-model VRAM requirements and compatible GPUs

Multi-GPU Setup Guide

Scale beyond a single card for larger models

Used RTX 3090 Buying Guide

How to evaluate a used 3090 for AI workloads

Mac Studio vs PC for AI

Unified memory vs discrete GPU trade-offs

RTX 5090 vs RTX 4090

Flagship inference throughput compared

RTX 5080 vs RTX 4090

New mid-range vs used flagship for inference value

RX 7900 XTX vs RTX 4090

The 24GB AMD option for token generation