Cách tính
Training a transformer costs about 6 floating-point operations per parameter per training token: two for the forward pass and four for the backward pass. A 7B model trained on 140B tokens therefore needs roughly 6 × 7×10⁹ × 140×10⁹ ≈ 5.9×10²¹ FLOPs. That number is fixed by the model and the data; the GPU only decides how fast you get through it.
Each GPU's speed here is its spec-sheet dense BF16 tensor throughput multiplied by MFU (model FLOPs utilization) — the share of peak a real training run sustains. Well-tuned large runs on H100-class clusters report roughly 35–50%; small jobs, consumer cards and multi-node setups over slow networks land lower, often 20–30%. Change the MFU slider to see how much the answer depends on it — usually more than on which GPU you pick.
Memory sets the minimum number of GPUs. Mixed-precision training with the Adam optimizer holds about 16 bytes per parameter (weights, gradients, FP32 master weights and two optimizer moments) plus activations, which frameworks like FSDP or DeepSpeed ZeRO can shard across GPUs. More GPUs than the minimum doesn't change the GPU-hours or the cost in this model — it shortens the wall-clock time — though in practice scaling efficiency drops as you add GPUs, especially without NVLink/InfiniBand.
The 20 tokens per parameter figure is the Chinchilla compute-optimal ratio; many modern small models are deliberately trained far beyond it because inference is cheaper for a smaller, longer-trained model. For fine-tuning an existing model instead of training from scratch, use the fine-tuning cost calculator — it's orders of magnitude cheaper.