Hướng dẫn

Fine-tuning an LLM with LoRA and QLoRA: GPU memory and cost.

Memory math for full fine-tuning, LoRA and QLoRA, which rented GPU fits 7B to 70B QLoRA jobs, and a step-by-step training time and cost estimate.

Tác giả Vi Nguyen · Đăng ngày 20 tháng 10, 2026 · Giá trên trang này là giá trực tiếp · Chính sách biên tập

Most people who ask "what GPU do I need to fine-tune a model?" are really asking two questions at once. The first is about memory: will the job fit on the card at all, or will it die with an out-of-memory error ten seconds after launch? The second is about money: once it fits, how many hours will it run, and what does that come to on a rented GPU? This guide answers both, in that order, because the memory question decides which cards are even on the table. The short version: parameter-efficient methods like LoRA and QLoRA have moved fine-tuning of 7B to 70B models from multi-GPU cluster territory onto single rented cards, and the bill for a typical instruction-tuning run is usually measured in tens of dollars, not thousands.

Full fine-tuning, LoRA and QLoRA: what actually changes

A full fine-tune updates every weight in the model. That means you have to hold the weights, a gradient for every weight, and the optimizer's bookkeeping for every weight, all in GPU memory at the same time. It gives you the most freedom to change the model's behavior, and it is the most expensive way to do it by a wide margin.

LoRA (low-rank adaptation) freezes the original weights and trains a pair of small matrices alongside selected layers, usually the attention projections and often the MLP projections too. The frozen base still has to sit in memory so the forward and backward passes can run through it, but it needs no gradients and no optimizer state. Only the adapters, typically well under 1% of the parameter count, are trained.

QLoRA goes one step further: the frozen base is stored in 4-bit (the NF4 format from the original QLoRA paper) and dequantized on the fly during each pass, while the adapters train in 16-bit. The base model's footprint drops to roughly a quarter of its BF16 size. The original paper's headline result was fine-tuning a 65B model on a single 48 GB GPU, which is the reason this technique took over hobbyist and startup fine-tuning almost overnight.

For most instruction-tuning, style adaptation, and domain-specific tasks, LoRA and QLoRA get close to full fine-tuning quality. Full fine-tuning still earns its cost when you are teaching a model substantial new knowledge, continuing pretraining on a large corpus, or changing its behavior deeply enough that a low-rank update isn't expressive enough. If you are not sure which camp you are in, start with QLoRA: it is the cheapest experiment, and it tells you quickly whether the data is any good.

The memory math, step by step

Here is where the bytes go. The numbers assume standard mixed-precision training with the Adam optimizer, which is what nearly every fine-tuning script defaults to.

Full fine-tune: about 16 bytes per parameter

• BF16 weights: 2 bytes per parameter
• BF16 gradients: 2 bytes per parameter
• FP32 master copy of the weights: 4 bytes per parameter
• Adam first moment (momentum): 4 bytes per parameter
• Adam second moment (variance): 4 bytes per parameter

That totals 16 bytes per parameter before a single activation is stored. For a 7B model: 7 × 16 = 112 GB. That is already more than an 80 GB H100 holds, which is why full fine-tuning of even small models means multiple GPUs with sharding (DeepSpeed ZeRO or PyTorch FSDP) or aggressive optimizer tricks such as 8-bit Adam. A 70B model needs about 1,120 GB for weights and optimizer state alone, which is a multi-node job.

LoRA: the base model plus a small adapter budget

With the base frozen in BF16, the fixed cost is 2 bytes per parameter: 14 GB for a 7B model, 26 GB for 13B. The adapter itself is small. Take a Llama-style 7B model (32 layers, hidden size 4,096, MLP size 11,008) with rank 16 on all seven linear projections:

1. Each of the four attention projections is 4,096 × 4,096. A rank-16 adapter adds 16 × (4,096 + 4,096) = 131,072 parameters, so the four together add about 524,000.
2. Each of the three MLP projections is 4,096 × 11,008. The adapter adds 16 × (4,096 + 11,008) = 241,664 parameters, so the three add about 725,000.
3. Per layer that is roughly 1.25 million; across 32 layers, about 40 million trainable parameters, around 0.6% of the model.
4. At 16 bytes per trainable parameter, the adapter's weights, gradients and optimizer state cost about 0.64 GB.

The adapter is a rounding error. The base model and the activations are what fill the card.

QLoRA: about half a byte per parameter for the base

NF4 stores each weight in 4 bits, plus a small overhead for the per-block scaling constants (QLoRA's "double quantization" squeezes that overhead down further). Call it slightly over 0.5 bytes per parameter. That gives rough base footprints of:

These are the fixed costs. On top of them you need activations, which scale with sequence length × micro-batch size × hidden size × number of layers. Gradient checkpointing cuts activation memory sharply by recomputing activations during the backward pass instead of storing them, at the price of roughly one extra forward pass per step. Add the CUDA context and framework overhead (budget a gigabyte or two), and leave 10–20% headroom, because memory fragmentation and the occasional long sample will find any slack you didn't leave. The VRAM fit calculator does the weights-plus-overhead part of this for inference; for training, add the adapter and activation budget described here.

Which GPU fits which QLoRA job

Put the table above against real cards and a clear ladder falls out. Assume gradient checkpointing on, a micro-batch of 1 to 4, and sequence lengths in the 2,048 to 4,096 token range, which covers most instruction-tuning datasets.

• 7B QLoRA: comfortable on any 24 GB card, such as the RTX 4090 or RTX 3090, and workable on 16 GB cards with shorter sequences. A 7B model in plain LoRA (BF16 base, 14 GB) also fits on 24 GB if you keep the batch small.
• 13B QLoRA: a good fit on 24 GB. The base takes about 7 GB and leaves room for real sequence lengths.
• 32B QLoRA: this is where 24 GB gets tight. A ~17 GB base leaves only a few gigabytes for activations, so you are limited to short sequences and a micro-batch of 1. The 32 GB RTX 5090 is a much more comfortable single-card option, and a 48 GB card like the RTX A6000 or L40S removes the constraint.
• 70B QLoRA: a ~37 GB base means 48 GB is the practical floor. It works at modest sequence lengths, which is the scenario the QLoRA paper demonstrated. For long-context training or a larger micro-batch, an 80 GB card such as the A100 80GB or H100 is the comfortable choice.

Splitting a QLoRA job across two consumer cards is possible, but it adds complexity and, on cards without NVLink, communication overhead. If a single larger card is available, it is usually the simpler and faster path. See RTX 4090 vs H100: cost per FLOP for why multi-4090 setups scale poorly.

GPU	VRAM	Rẻ nhất hiện tại	Ở đâu	Trung vị giữa các nhà cung cấp	Nhà cung cấp
Nvidia GeForce RTX 4090	24GB	$0.30/hr	io.net	$0.46/hr	13
Nvidia GeForce RTX 5090	32GB	$0.45/hr	Nosana	$0.70/hr	11
Nvidia RTX A6000	48GB	$0.32/hr	Nosana	$0.54/hr	13
Nvidia L40S	48GB	$0.48/hr	Lium	$1.06/hr	11
Nvidia A100 80GB PCIe	80GB	$1.09/hr	Thunder Compute	$1.65/hr	10
Nvidia H100	80GB	$3.20/hr	Thunder Compute	$3.37/hr	3

Dữ liệu trực tiếp: giá trung vị theo giờ cho mỗi GPU, thu thập trong 24 giờ qua. Cách chúng tôi thu thập giá.

Estimating training time

Memory tells you what fits. Time tells you what it costs. The honest answer to "how long will this take?" is that it depends on your framework, sequence packing, kernel choices and the specific card, so treat what follows as an estimate for budgeting, not a promise. The method is simple:

training time ≈ (dataset tokens × epochs) ÷ throughput in tokens per second

You know the first two numbers exactly. Throughput is the unknown. The best source is a short measured run: launch the job, let it run 50 steps, and read the tokens per second from the logs. Before you have that, you can derive a rough figure from the hardware's compute:

1. FLOPs per token. A forward pass costs about 2 × parameters FLOPs per token. In LoRA the backward pass still has to push gradients back through the frozen layers (about another 2 × parameters) but skips computing weight gradients for them. Gradient checkpointing adds one more forward pass. That lands around 6 × parameters per token, so a 7B model needs about 42 GFLOPs per training token.
2. Usable compute. Take the card's dense BF16 tensor throughput and multiply by a utilization factor. Real fine-tuning jobs rarely sustain more than a fraction of peak, and QLoRA's on-the-fly dequantization costs extra. We use a hypothetical 25% here.
3. Divide. The RTX 4090's spec-sheet BF16 tensor rate with FP32 accumulation, which is what mixed-precision training uses, is about 165 TFLOPS dense. At 25%, that's about 41 TFLOPS usable, and 41 × 10¹² ÷ 42 × 10⁹ ≈ 980 tokens per second.

This is a derived estimate, not a benchmark. Your real number could be higher with sequence packing and fused kernels, or lower with short, padded samples. Replace it with your own measurement as soon as you have one.

A worked cost example

Say you have an instruction dataset of 20 million tokens and want three epochs of QLoRA on a 7B model.

1. Total tokens: 20M × 3 = 60 million.
2. Time at the estimated ~980 tokens/s on an RTX 4090: 60,000,000 ÷ 980 ≈ 61,000 seconds, or about 17 hours.
3. On an H100, the SXM spec sheet lists about 989 TFLOPS of dense BF16. At the same hypothetical 25% utilization that's roughly 247 TFLOPS usable, or about 5,900 tokens/s, and the run takes about 10,200 seconds, or 2.8 hours. (PCIe H100s have lower rated throughput, so expect a longer run there.)

Now plug in live prices. The cheapest RTX 4090 listing right now is $0.30/hr on io.net, so 17 hours comes to about $5.10. The cheapest H100 is $3.20/hr on Thunder Compute, so 2.8 hours comes to about $8.96. Because the estimated H100 run is about six times shorter (17 ÷ 2.8 ≈ 6.1), the H100 is cheaper per run whenever its hourly price is less than about six times the 4090's, and more expensive when it isn't. The bigger difference is that you get your result the same afternoon instead of tomorrow.

A few things push the real bill up. You'll spend some GPU time on setup, downloading weights and failed first attempts, so budget 20–30% on top of the pure training estimate. Evaluation runs between epochs cost time too. And if you rent on a host-supplied marketplace like Vast.ai or Clore.ai, reliability varies by host, so save checkpoints to storage you control. Losing 14 hours of a 17-hour run to a host going offline doubles your cost. The hidden costs of renting GPUs covers storage, egress and idle time in more detail.

Choosing a card: a decision framework

Put the memory table and the time estimate together, and the choice usually comes down to three questions.

• Does the job fit on a 24 GB card with room to spare? (7B or 13B QLoRA, 7B LoRA.) If so, rent the cheapest reliable 24 GB card, usually an RTX 4090 or RTX 3090. Nothing else beats them on cost per run at this size, and the job is small enough that a restart isn't a disaster.
• Is it 32B, or 13B with long sequences? Step up to 32 GB (RTX 5090) or 48 GB (RTX A6000, L40S). The 48 GB cards also cover 70B QLoRA at short context. Our 48 GB comparison goes into the trade-offs.
• Is it 70B with real context lengths, or is turnaround time what matters? Rent an 80 GB A100 or H100. You pay more per hour, but a job that runs in three hours instead of seventeen lets you iterate on your data several times in a day, and data quality is where most fine-tuning gains come from.
• Are you doing a full fine-tune? Then you are shopping for multi-GPU nodes with NVLink, which means managed providers like Lambda or RunPod's Secure Cloud rather than single marketplace hosts. Budget 16 bytes per parameter across the node plus activations, and confirm that the provider offers the interconnect before you book.

One more rule of thumb: run your first experiment on the cheapest card that fits, with a small slice of the data. Most fine-tuning problems (bad formatting, wrong chat template, a learning rate that is too high) show up in the first few hundred steps, and there's no reason to discover them at H100 prices. Once the recipe works, move to the card that gets you the full run in an acceptable time.

After training: serving the adapter

A LoRA adapter is small, typically tens to a few hundred megabytes, so you can keep many variants and swap them over one base model. For deployment you can either merge the adapter into the base weights and serve the result like any other model, or keep it separate and use an inference server that supports loading adapters at runtime. Note that if you trained with QLoRA, you'll usually merge into the full-precision base rather than the 4-bit one, then quantize again for serving if needed. Serving costs follow the usual inference rules: see how much VRAM you need to run LLMs, and use the self-host vs API calculator to check whether hosting your fine-tuned model beats paying per token for a general-purpose one.

Câu hỏi thường gặp