计算方法
Generating a token means reading every active weight from GPU memory once, so single-request speed is set by memory bandwidth: tokens per second ≈ bandwidth ÷ bytes of active weights, and we assume real kernels reach about 70% of the spec. Serving several requests at once reuses each weight read across the whole batch, so throughput grows with concurrency — until the GPU runs out of arithmetic, which we cap at 40% of its dense BF16 peak (2 FLOPs per active parameter per token).
Memory is different: a mixture-of-experts model must keep all of its experts loaded, so it's sized by total parameters, plus the KV cache for your context length and concurrency. When one card isn't enough we use the fewest GPUs that fit (up to one 8-GPU node) and price the whole node.
Cost per million tokens = hourly price ÷ (tokens per second × 3,600) × 1,000,000. The comparison column uses the cheapest hosted API's output-token price for the same model. Self-hosting only reaches these numbers while the GPU is busy: at 20% utilization, multiply the cost by five. The self-host vs API tool and the breakeven guide work through utilization in detail.