Can I run a 70B parameter model on a single GPU?

Not on a single consumer GPU at Q4_K_M. The weights alone are about 43GB, plus a few GB for context, which is more than the 32GB on an RTX 5090. A 48GB card or two 24GB cards can hold it with a modest context. Otherwise part of the model runs from system RAM, which is much slower. Unified-memory systems with 64GB or more can also load it, at lower bandwidth than a high-end GPU.

Is Q4 quantization good enough for local LLMs?

For most home use, Q4_K_M is a reasonable default. It stores about 4.9 bits per weight, roughly 70% less memory than FP16. Quality loss grows as the bit width drops, so step up to Q5_K_M or Q6_K when the model fits with room to spare, and test the tasks you actually care about.

Does context length affect VRAM usage?

Yes. The KV cache grows in a straight line with context length. For Qwen3-32B it is about 1.1GB at 4K tokens, 2.1GB at 8K and 8.6GB at 32K with a 16-bit cache. Ollama can store the cache at 8 bits, which roughly halves it, when flash attention is enabled.

What happens if my model doesn't fit in VRAM?

llama.cpp and Ollama keep the layers that fit on the GPU and run the rest from system RAM. System RAM has far less bandwidth than VRAM, so even a modest offloaded share can cut generation speed sharply. Run ollama ps to see the CPU and GPU split for a loaded model.

Should I buy two 12 GB GPUs or one 24 GB GPU?

Prefer one 24GB card when the target model needs about 24GB. Runtimes can spread a model across two cards, but the split adds PCIe traffic, power draw, heat and setup work. A second card makes more sense when you already own the first one.

How Much VRAM Do You Need for Local LLMs in 2026?

VRAM is the most important number when you shop for a local LLM GPU. CUDA cores, clock speed and board power only matter once the model fits. If it doesn’t fit, part of it spills into system RAM and generation slows down sharply.

This guide shows how to estimate what a model needs: the weights, the context (KV cache) and a little runtime headroom. It then maps the result onto common VRAM tiers.

The Quick Math

A model’s memory footprint has three parts:

VRAM ≈ weights + KV cache + runtime buffers

The weights depend on the parameter count and the quantization format. llama.cpp publishes the average bits per weight for each format, measured on Llama 3.1 8B:

Format	Bits per weight	Bytes per parameter
FP16	16.0	2.00
Q8_0	8.50	1.06
Q6_K	6.56	0.82
Q5_K_M	5.70	0.71
Q4_K_M	4.89	0.61

An 8B model therefore needs about 16GB for its weights at FP16 and about 4.9GB at Q4_K_M, roughly 70% less. Exact file sizes vary a little by model, so check the actual download before you buy hardware.

Weight Sizes by Model Size and Quantization

These are weights only, in GB, calculated from the bits-per-weight figures above. Add the KV cache and runtime buffers on top.

Parameters	Q4_K_M	Q5_K_M	Q6_K	Q8_0	FP16
8B	4.9	5.7	6.6	8.5	16
14B	8.6	10.0	11.5	14.9	28
32B	19.6	22.8	26.3	34.0	64
70B	42.8	49.9	57.4	74.4	140
120B	73.4	85.6	98.4	127.5	240

Mixture-of-experts models need memory for all of their parameters, even though only some of them are active for each token. Size them by total parameters, not the active count.

Context Length: The KV Cache

Every token in the context window stores keys and values for every layer. With a 16-bit cache:

KV cache bytes = 2 × layers × KV heads × head dimension × 2 bytes × tokens

The three models below all use 8 KV heads with a head dimension of 128. They differ in layer count, which is published in each model’s configuration file:

Context	Qwen3-8B (36 layers)	Qwen3-32B (64 layers)	Qwen2.5-72B (80 layers)
4K tokens	0.6 GB	1.1 GB	1.3 GB
8K tokens	1.2 GB	2.1 GB	2.7 GB
32K tokens	4.8 GB	8.6 GB	10.7 GB
128K tokens (if supported)	19.3 GB	34.4 GB	43.0 GB

Two settings change these numbers. Ollama’s default context is 4,096 tokens, and you can raise it with OLLAMA_CONTEXT_LENGTH or the num_ctx parameter. Ollama can also store the cache at 8 bits (OLLAMA_KV_CACHE_TYPE=q8_0, with flash attention enabled), which roughly halves it. Parallel requests multiply the context memory, so serving several users at once needs more headroom.

What Fits at Each VRAM Tier

The table assumes about an 8K context with a 16-bit cache, plus roughly 1GB for runtime buffers. Allow more if the card also drives your display.

VRAM	Fits comfortably	Does not fit well
8 GB	8B at Q4_K_M	8B at Q8_0, anything 14B
12 GB	8B at Q8_0, 14B at Q4_K_M	14B at Q8_0, 24B and up
16 GB	14B at up to Q6_K, 8B with a 32K context	24B at Q4_K_M with useful context, 32B
24 GB	32B at Q4_K_M, 14B at Q8_0 with a longer context	32B at Q8_0, 70B
32 GB	32B at Q5_K_M or Q6_K, or Q4_K_M with a 32K context	70B at Q4_K_M
48 GB	70B at Q4_K_M with a 4K to 8K context, 32B at Q8_0	70B at Q8_0, 120B

8 GB: Small Models Only

Current 8GB cards include the RTX 5060 and the 8GB versions of the RTX 5060 Ti and RX 9060 XT. They run 8B models at Q4_K_M with a short context. That is enough to experiment, but you will outgrow it quickly.

12 GB: The Practical Floor

The Intel Arc B580 has 12GB of GDDR6 at 456 GB/s, and a used RTX 3060 12GB offers the same capacity with CUDA. 12GB runs 8B models at Q8_0 and 14B models at Q4_K_M with a modest context.

16 GB: New Cards for 14B Models

The RTX 5060 Ti 16GB (448 GB/s) is the default new NVIDIA option, and the RTX 5070 Ti adds bandwidth (896 GB/s) without adding capacity. 16GB handles 14B models at up to Q6_K, or smaller models with long contexts. A 24B model at Q4_K_M needs about 14.7GB for weights alone, which leaves too little room for context.

24 GB: The Home Lab Standard

The used RTX 3090 (936 GB/s) is the most common way to reach 24GB. The RTX 4090 (1,008 GB/s) and the RX 7900 XTX are previous-generation 24GB options. 24GB runs 32B models at Q4_K_M with about an 8K context, or 14B at Q8_0 with room to spare. It does not hold a 70B model at Q4_K_M.

For the two most common 24GB NVIDIA cards side by side, see RTX 3090 vs 4090 for LLMs. For AMD’s 24GB and 32GB options, see NVIDIA vs AMD for LLMs.

32 GB: Maximum Single-Card Consumer Capacity

The RTX 5090 pairs 32GB of GDDR7 with 1,792 GB/s. It runs 32B models at Q5_K_M or Q6_K, or at Q4_K_M with a 32K context. A 70B model at Q4_K_M still does not fit. About a third of it would run from system RAM. AMD’s Radeon AI PRO R9700 is a 32GB RDNA 4 workstation card on the ROCm support list.

48 GB and Up: Multi-GPU and Workstation Cards

Two 24GB cards or a 48GB workstation card can hold a 70B model at Q4_K_M with a 4K to 8K context. Ollama spreads a model across all available GPUs when it doesn’t fit on one. A 120B dense model needs about 73GB even at Q4_K_M.

Quantization Quality

Lower bit widths save memory at some cost in output quality, and the loss grows as the bit width falls:

Q8_0 is close to FP16 for most uses and needs about half the memory.
Q6_K and Q5_K_M sit in between. They are a good upgrade when a model fits at Q4_K_M with room to spare.
Q4_K_M is the usual default when VRAM is tight.
Below 4 bits the savings are large, but quality drops faster, so test before relying on it.

Start with Q4_K_M. If the model fits with headroom, try Q5_K_M or Q6_K and compare the results on your own prompts.

When the Model Doesn’t Fit

llama.cpp and Ollama support hybrid CPU and GPU inference. Layers that fit stay in VRAM, and the rest run from system RAM. It works, but it is slow, because generating each token has to read every layer’s weights, and system RAM is much slower than VRAM.

Dual-channel DDR5-5600 has a theoretical peak of about 90 GB/s, a fifth of an RTX 5060 Ti’s 448 GB/s. In a simple bandwidth-bound estimate, a 32B model at Q4_K_M (19.6GB) split as 12GB on a 5060 Ti and 7.6GB in DDR5 takes about two and a half times as long per token as it would if every layer were on the card. Real results depend on the CPU, backend and settings, but the direction is always the same: small offloaded shares cost a lot of speed.

Run ollama ps after loading a model. “100% GPU” means everything is on the card, while a split such as “48%/52% CPU/GPU” means part of the model is running from system RAM.

Unified Memory Systems

Apple’s Mac mini with M5 Pro offers up to 64GB of unified memory at 307 GB/s, and the M6 model goes up to 32GB. A 64GB configuration can hold a 32B model at Q8_0 (about 34GB), which no 24GB card can. Its bandwidth is about a third of an RTX 3090’s, though, so models that fit both run slower on the Mac. See Mac mini vs NVIDIA GPU for local LLMs for the full comparison. The best mini PCs for local AI guide covers AMD Ryzen AI Max+ systems with up to 128GB of shared memory.

The Decision Framework

Find the largest model you want to run, and read its weight size for your preferred quantization from the table above or from the actual download.
Add the KV cache for the context length you need.
Add about 1GB for runtime buffers, and more if the card also drives a display or serves parallel requests.
Buy the smallest card that covers the total with some headroom. Extra VRAM beyond your workload adds cost, power and heat without making the model run faster.

For card-by-card recommendations, see the best GPUs for local LLMs. For lower-cost options, see the best budget GPU for local AI. For Ollama settings that change memory use, see the best GPU for Ollama.