Guide

How much VRAM do you need to run an LLM?

Estimate LLM VRAM from weights, KV cache and overhead: per-precision sizes for 8B to 671B models, GQA explained, and a worked 70B INT4 32k-context example.

By Vi Nguyen · Published September 26, 2026 · Prices on this page are live · Editorial policy

The first question anyone asks before renting a GPU for a language model is whether the model will fit. Get it wrong in one direction and the job dies with an out-of-memory error ten minutes after the weights finish downloading. Get it wrong in the other and you pay for 80 GB of HBM to serve a model that would have run on a 24 GB consumer card. The good news is that VRAM requirements are not mysterious. They come from four terms you can estimate on the back of an envelope: the weights, the KV cache, activations and scratch buffers, and the runtime's own overhead. This guide walks through each one, gives you a table of common model sizes, and works a full example for a 70B model at 32k context.

Weights: parameters times bytes per parameter

The weights are the part everyone knows about, and they are the easiest to compute. Multiply the parameter count in billions by the bytes each parameter takes, and you have gigabytes:

• FP16 / BF16: 2 bytes per parameter. This is how most open-weights models are published.
• FP8 / INT8: 1 byte per parameter. FP8 is native on Hopper (H100, H200) and newer; INT8 works almost everywhere.
• INT4 (GPTQ, AWQ, GGUF Q4 variants): roughly 0.5 bytes per parameter. In practice 4-bit formats store a scale (and sometimes a zero point) for every group of 32 to 128 weights, so real files land closer to 4.5 to 5 bits per weight. Budget 0.55 to 0.6 bytes per parameter if you want a conservative number.

So an 8B model is about 16 GB at FP16, 8 GB at INT8 and 4 to 5 GB at INT4. A 70B model is 140 GB, 70 GB and 35 to 40 GB. That arithmetic alone rules out a lot of configurations: no single 80 GB card holds a 70B model at FP16, no matter what else you do.

Mixture-of-experts models deserve a warning here. A model like DeepSeek V3 activates only a fraction of its parameters for each token, which makes it fast, but every expert still has to sit in memory because any token can route to any expert. For VRAM planning, use the total parameter count, not the active count. Our guide to GPUs for MoE models goes deeper on that.

The KV cache: the term people forget

During generation, a transformer stores the key and value vectors for every previous token in every layer so it does not have to recompute them. That store is the KV cache, and it grows linearly with context length and with the number of sequences you serve at once. For a short chat it is small. For long documents or a busy server it can rival the weights.

The per-token size is:

KV bytes per token = 2 × layers × kv_heads × head_dim × bytes per element

The 2 is one key plus one value. Layers, kv_heads and head_dim come straight from the model's config file (num_hidden_layers, num_key_value_heads, and hidden_size divided by num_attention_heads). Bytes per element is 2 for an FP16 cache and 1 if your engine stores the cache in FP8.

Why grouped-query attention matters

Older models used full multi-head attention, where every attention head had its own key and value projection, so kv_heads equaled the number of query heads. Grouped-query attention (GQA) lets several query heads share one key/value head. Llama 3 70B, for example, has 64 query heads but only 8 KV heads, so its cache is one eighth the size it would be under full multi-head attention. This is the single biggest reason modern models can offer 128k contexts without needing absurd amounts of memory.

Take the published Llama 3 70B configuration: 80 layers, 8 KV heads, head dimension 128. At FP16:

2 × 80 × 8 × 128 × 2 bytes = 327,680 bytes, or about 0.33 MB per token.

If the same model used full multi-head attention with 64 KV heads, that would be 2.6 MB per token, and a single 32k-token conversation would need about 86 GB of cache on its own. With GQA it needs about 10.7 GB. For comparison, Llama 3 8B (32 layers, 8 KV heads, head dimension 128) needs about 0.13 MB per token, so an 8k context costs roughly 1 GB.

Context length, batch size and overhead

Total KV cache is the per-token size multiplied by every token in flight across every sequence:

KV total = per-token KV × context length × concurrent sequences

This is where serving differs from tinkering. One user at 4k context on a 70B model needs about 1.3 GB of cache. Sixteen concurrent users at 8k each need about 43 GB. Engines like vLLM and SGLang allocate the cache in pages and only fill what is actually used, so you rarely hit the theoretical maximum, but they also pre-reserve a large fraction of free VRAM for the cache at startup. That is why a server that "only needs" 40 GB for weights will happily grab 72 GB of an 80 GB card: the extra is cache capacity, and more cache means more concurrent requests.

On top of weights and cache, budget for:

• Activations and scratch buffers. Intermediate tensors during the forward pass. Small for decoding, larger during prefill of a long prompt, which is why engines cap how many prompt tokens they process per step.
• CUDA context and libraries. The driver context, cuBLAS workspaces and compiled kernels typically take several hundred megabytes to a couple of gigabytes per GPU.
• CUDA graphs and fragmentation. Captured graphs and allocator fragmentation eat a bit more.

A 10 to 20 percent margin on top of weights plus cache covers all of that in most setups. That is the same rule our VRAM fit calculator uses, so if you would rather not do this by hand, plug the model and context in there.

Model size by precision, and which GPU class fits

The table below gives approximate weight sizes only. Treat the "fits" column as a starting point for moderate context (a few thousand tokens, a handful of users). Long contexts or heavy concurrency push you one tier up.

A few notes on the rows. GPT-OSS 120B is itself an MoE and ships with its expert weights already in a 4-bit format, which is why it is commonly run on a single 80 GB card. DeepSeek V3 and R1 were released with FP8 weights, so FP8 is their native size, not a quantization. And at the 405B and 671B scale you are in multi-GPU territory regardless of precision, which brings interconnect into the decision: 8-way NVLink nodes of H100 or H200 exist precisely for this.

Worked example: a 70B model at INT4 with 32k context

Suppose you want to run a Llama 3.3 70B class model, quantized to 4-bit AWQ, for a single analyst who pastes long documents and needs a 32,768-token context. Step by step:

1. Weights. 70B × 0.5 bytes = 35 GB nominal. With group scales, call it 38 GB.
2. KV cache per token. 2 × 80 layers × 8 KV heads × 128 head_dim × 2 bytes = 327,680 bytes (FP16 cache).
3. KV cache for the full context. 327,680 × 32,768 tokens = 10.7 GB for one sequence.
4. Subtotal. 38 + 10.7 = 48.7 GB.
5. Overhead. 15 percent of 48.7 is about 7.3 GB, bringing the total to roughly 56 GB.

What that means in hardware terms:

• A single 48 GB card (RTX A6000, L40S) does not fit the full 32k at an FP16 cache. Switch the cache to FP8 and it drops to 5.4 GB, for a total near 50 GB. Still over. Cut the context to about 16k with an FP8 cache and you get roughly 38 + 2.7 plus overhead, around 47 GB: it works, but with almost no headroom.
• Two 24 GB cards (2× RTX 4090) give 48 GB split across two devices, with the same problem, plus per-GPU overhead on each card. Workable at 8k to 16k context.
• Two 32 GB cards (2× RTX 5090) give 64 GB and fit the full 32k with a little room.
• A single 80 GB card (A100 80GB, H100) fits comfortably with about 24 GB left over, which the engine will use to serve two or three more 32k conversations at once.

The same model at FP16 would need 140 GB of weights before any cache, which puts it on two 80 GB cards or a single 192 GB MI300X. For most inference work the quality difference between a good 4-bit or 8-bit quantization and FP16 is small enough that paying for twice the memory is hard to justify, but test on your own prompts before committing.

Memory tiers and what they cost to rent right now

GPU memory comes in a handful of standard sizes, and each tier maps to a class of model. The table is live, pulled from current listings across the providers we track.

GPU	VRAM	Cheapest now	Where	Median across providers	Providers
Nvidia GeForce RTX 4090	24GB	$0.30/hr	io.net	$0.38/hr	7
Nvidia GeForce RTX 5090	32GB	$0.37/hr	Vast.ai	$0.69/hr	6
Nvidia RTX A6000	48GB	$0.32/hr	Nosana	$0.42/hr	6
Nvidia L40S	48GB	$0.38/hr	Lium	$0.79/hr	5
Nvidia A100 80GB PCIe	80GB	$1.79/hr	Thunder Compute	$2.07/hr	2
Nvidia H100	80GB	$2.81/hr	Fluence	$3.20/hr	2
Nvidia H200	141GB	$2.63/hr	Vast.ai	$3.00/hr	4
AMD MI300X	192GB	$0.50/hr	RunPod	$2.86/hr	2

Live data: median hourly price per GPU, collected in the last 24 hours. How we collect prices.

Two spec differences matter beyond capacity. First, memory bandwidth sets the ceiling on single-stream decode speed, because each generated token reads every active weight once. The RTX 4090 has about 1 TB/s of GDDR6X, the RTX 5090 about 1.8 TB/s of GDDR7, the A6000 and L40S sit in the 0.77 to 0.86 TB/s range, while the H100 SXM (3.35 TB/s), H200 (4.8 TB/s) and MI300X (5.3 TB/s) use HBM. A rough upper bound on decode speed is bandwidth divided by the bytes of weights read per token: 38 GB of 70B INT4 weights on an H100 SXM gives a ceiling near 88 tokens per second for one stream, and on an L40S about 23. Real engines land below those ceilings; treat them as estimates, not measurements. Second, datacenter parts have NVLink for multi-GPU work, while consumer cards talk over PCIe, which matters once you split a model across devices.

A quick decision framework

• If your model is 8B to 14B and context is modest, a 24 GB card at FP16 or INT8 is enough. Look at the RTX 4090 and our list of GPUs under $1/hr for inference.
• If it is 32B, INT4 on 24 GB works for short contexts, but a 32 GB RTX 5090 or a 48 GB card buys real headroom. See RTX 5090 vs RTX 4090 for local LLMs.
• If it is 70B and you serve one or two users at modest context, 48 GB at INT4 is the budget floor. For long context or several users, go to 80 GB. Compare the 48 GB options in A6000 vs L40S vs RTX 6000 Ada.
• If it is 120B or larger, start from 80 to 96 GB for 4-bit models and move to 141 or 192 GB cards, or multi-GPU nodes, above that. The H100 vs H200 vs B200 comparison covers the top end.
• If you need long context for many users, size for the KV cache first. It is the term that scales with your traffic; the weights do not.

Whichever tier you land on, check the per-provider prices on each GPU page before renting. The spread between the cheapest marketplace listing and a managed cloud can be several times, and our methodology page explains how those numbers are collected.

FAQ