Multi-GPU Guide for AI: When Two GPUs Beat One

The question of whether to buy one big GPU or two smaller ones has defined AI hardware discussions since the first crypto miners repurposed their rigs for machine learning. The answer, as always, is "it depends" — but the decision factors have shifted significantly in 2026. With the RTX 5090 offering 32GB at $1,999 and used RTX 3090s at $600 each, the multi-GPU equation has new variables. This guide covers everything from hardware requirements to software configuration.

When Does a Second GPU Actually Help?

This is the most misunderstood aspect of multi-GPU setups. The answer depends entirely on your workload:

The key insight: inference benefits from VRAM pooling, training benefits from compute scaling. If you just want to run a 40GB model that doesn't fit on one card, multi-GPU inference works — but a single bigger GPU would be faster if the model fits.

NVLink vs PCIe: Does the Bridge Matter?

NVLink is NVIDIA's proprietary high-bandwidth interconnect between GPUs. It provides dramatically more bandwidth than PCIe:

Critically, NVIDIA removed NVLink from the RTX 4090 and 5090. Only the RTX 3090 and 3090 Ti have NVLink bridges among consumer cards. For all other consumer GPUs, multi-GPU communication goes over PCIe.

When NVLink Matters:

• Training: Gradient synchronization between GPUs happens every step. NVLink's 112 GB/s dramatically reduces sync overhead, improving scaling from ~1.7x to ~1.9x.
• Tensor parallelism for inference: Splitting individual transformer layers across GPUs requires constant inter-GPU communication. NVLink makes this practical; PCIe makes it slow.

When PCIe Is Fine:

• Pipeline parallelism: Each GPU runs different layers sequentially. Communication is only the intermediate activations between layer boundaries — a few megabytes per token.
• Batch processing: Each GPU processes independent requests. No inter-GPU communication needed at all.
• Whisper, image generation: Parallel file/batch processing with no inter-GPU dependency.

Tensor Parallelism vs Pipeline Parallelism

When distributing a single model across multiple GPUs, there are two main strategies:

For most home practitioners, pipeline parallelism is the way to go. It's what llama.cpp uses by default for multi-GPU, and it works over standard PCIe without NVLink.

llama.cpp Multi-GPU Configuration

llama.cpp provides excellent multi-GPU support through its --tensor-split and --gpu-layers options:

Basic Multi-GPU Inference

1. Launch with -sm layer for pipeline parallelism (splits by layers): ./main -m model.gguf -p "Prompt" -sm layer -ngl 999
2. Use -sm row for tensor parallelism (requires similar GPUs): ./main -m model.gguf -p "Prompt" -sm row -ngl 999
3. Use -ts 1,1 to specify GPU allocation ratio (e.g., -ts 3,1 for a 3:1 split favoring GPU 0)

Example: 70B Model on Dual 3090 (48GB total)

• Model: Llama 3.1 70B Q4 (~38GB)
• VRAM: ~19GB on GPU 0, ~19GB on GPU 1
• Command: ./main -m llama-3.1-70b-q4_k_m.gguf -sm layer -ngl 80 -ts 1,1 -p "Your prompt"
• Speed: ~18-22 t/s (vs ~25-30 t/s on a single 5090 with 32GB)

Example: 70B FP16 on Dual 3090 (needs 140GB)

• Won't fit — even dual 3090 only has 48GB. You'd need 3+ GPUs or quantization.

Building a Dual 3090/4090 System

If you've decided multi-GPU is right for you, here's what you need for a proper dual-GPU AI workstation:

Power Supply: 1200W Minimum

• Dual RTX 3090: 2 × 350W = 700W for GPUs alone. Add 200W for CPU/RAM/peripherals. Minimum 1000W; 1200W recommended.
• Dual RTX 4090: 2 × 450W = 900W for GPUs. Minimum 1200W; 1500W strongly recommended.
• Use a high-quality PSU from Corsair, Seasonic, or EVGA. Don't skimp on the PSU — a failure can destroy your entire system.

Motherboard: Proper PCIe Lane Configuration

• Both GPUs should run at PCIe x8 minimum. Look for X570, X670, or Z790 boards with two reinforced x16 slots.
• Consumer CPUs (Ryzen 9, Core i9) typically provide 20-24 PCIe lanes. Two GPUs at x8/x8 is the standard split.
• Threadripper or Xeon workstations provide more lanes (x16/x16), but cost significantly more.
• Ensure the GPUs are physically spaced — they need at least 1 slot gap for cooling.

Cooling: Blower vs Open-Air

This is critical for multi-GPU builds:

Cost Analysis: Dual 3090 vs Single 5090

When Dual 3090 Wins:

• You need 48GB VRAM for large models (70B FP16 partially, or multiple models)
• Training is a primary workload — NVLink + data parallelism shines
• You're on a tight budget — $1,200 total vs $1,999
• You already have one 3090 — adding a second is cost-effective

When Single 5090 Wins:

• Inference is your primary workload — faster per-token generation
• Simplicity matters — single card, standard PSU, no multi-GPU complexity
• Power efficiency matters — 575W vs 700W+ system draw
• 32GB is sufficient for your models

When Multi-GPU Wins vs Single Big GPU

Multi-GPU Wins When:

• Total VRAM exceeds any single card: If you need 48GB+, two 3090s are your cheapest path.
• Training is primary: Data parallelism scales near-linearly for training workloads.
• Batch throughput matters: Serving multiple inference requests simultaneously benefits from independent GPUs.
• Embarrassingly parallel workloads: Whisper batch processing, image generation, data processing.
• Budget constraints: Gradual scaling — add a second GPU later when needed.

Single Big GPU Wins When:

• Inference latency matters: Single-GPU inference is always faster for individual requests (no inter-GPU communication overhead).
• Simplicity valued: One GPU, one driver, no cooling headaches, no power supply challenges.
• Model fits in one card: If 32GB (5090) or 24GB (4090) is sufficient, don't complicate things.
• Power efficiency: Single GPU systems are more power-efficient per token generated.
• Portability: SFF builds, LAN events, mobile workstations.

Software Support for Multi-GPU

The Verdict

Multi-GPU is not a universal upgrade — it's a strategic choice for specific workloads. If you're training models, multi-GPU provides near-linear scaling and is almost always worth it. If you're doing inference, think carefully: a single larger GPU will almost always provide better latency and simpler operation. The multi-GPU path makes sense when you need VRAM capacity that no single card can provide, or when your workload is embarrassingly parallel.

For most practitioners in 2026, the pragmatic path is: buy the biggest single GPU you can afford. Only go multi-GPU when you've hit a VRAM wall that a single card can't overcome, or when training throughput is your primary bottleneck.