Self-hosting vs API: how to find your breakeven.
Compute the cost per million tokens of a self-hosted GPU, see how batch size and utilization move it, and find the breakeven point against hosted LLM APIs.
By Vi Nguyen · Published · Prices on this page are live · Editorial policy
Every team that runs open-weights models eventually asks whether it would be cheaper to host them. The honest answer is that it depends on one number more than any other: how busy you can keep the GPU. A rented GPU costs the same per hour whether it serves ten thousand tokens a second or none, while an API charges only for what you use. Self-hosting wins when a GPU is kept saturated, and loses, often badly, when it is not. This guide shows how to compute the cost per million tokens of a self-hosted GPU, how batch size and utilization move that number, which costs never show up on the hourly price, and when an API is simply the better deal. If you want to skip to your own numbers, the self-host vs API calculator does the math interactively.
The formula: cost per million tokens
An API quotes a price per million tokens. To compare a GPU against it, convert the GPU's hourly price into the same unit:
Cost per 1M tokens = hourly price ÷ (tokens per second × 3,600) × 1,000,000
The denominator is how many tokens the GPU produces in an hour. Divide the hourly price by it and you get the cost per token; multiply by a million for the unit APIs use.
A worked example with round hypothetical numbers: a GPU rented at $2.00 per hour that sustains 1,000 tokens per second of aggregate output.
- Tokens per hour: 1,000 × 3,600 = 3,600,000.
- Cost per token: $2.00 ÷ 3,600,000 = $0.000000556.
- Cost per million tokens: $0.000000556 × 1,000,000 = $0.56.
Now run the same GPU serving a single user at 100 tokens per second. Tokens per hour fall to 360,000 and the cost per million rises tenfold, to $5.56. Same card, same hourly bill, ten times the unit cost. Everything else in this article is a variation on that observation.
With a live price: the cheapest H100 listing right now is $3.20/hr on Thunder Compute. If that card sustained a hypothetical 1,000 tokens per second, it would cost $0.89 per million tokens; at 100 tokens per second, $8.89.
Throughput depends on batch size
The tokens-per-second figure is not a property of the GPU alone. It depends heavily on how many requests the server processes at once. During generation, each step reads every model weight from memory to produce one token per active sequence. At batch size 1 the GPU spends most of its time moving weights and very little doing math, so the speed is bounded by memory bandwidth. At batch size 32, the same weight read produces 32 tokens, and aggregate throughput climbs almost linearly until the GPU runs out of compute or KV cache space.
You can estimate the bandwidth-bound ceiling: step time ≈ (bytes of weights + bytes of KV cache read) ÷ memory bandwidth, and aggregate tokens per second ≈ batch size ÷ step time. For an 8B model at FP16 (about 16 GB of weights, about 0.13 MB of KV cache per token) on an H100 SXM with 3.35 TB/s, with each sequence holding 2,048 tokens of context:
| Concurrent sequences | Bytes read per step (GB) | Step time (ms) | Ceiling, aggregate tokens/sec |
|---|---|---|---|
| 1 | 16.3 | 4.9 | ~200 |
| 8 | 18.1 | 5.4 | ~1,500 |
| 32 | 24.6 | 7.3 | ~4,400 |
| 64 | 33.2 | 9.9 | ~6,500 |
These are estimates from a simple bandwidth model, not measurements. Real engines such as vLLM, SGLang and TensorRT-LLM typically reach a fraction of these ceilings, and prompt processing (prefill), scheduling and sampling take their share of each step. The point is the shape, not the exact values: going from 1 to 32 concurrent sequences multiplies throughput by an order of magnitude on the same hardware, and so divides your cost per token by roughly the same factor. That is why API providers, who pool traffic from thousands of customers into large batches, can charge so little.
Utilization is the variable that decides it
Batch size tells you what a GPU can do while it is busy. Utilization tells you how much of the time it is busy. The cost formula becomes:
Effective cost per 1M tokens = hourly price ÷ (peak tokens per second × utilization × 3,600) × 1,000,000
Take the earlier hypothetical: $2.00 per hour and 1,000 tokens per second at full load, or $0.56 per million. If traffic keeps the GPU at that level only 25 percent of the time, averaged over the hours you pay for, the effective cost is $0.56 ÷ 0.25 = $2.22 per million. At 10 percent it is $5.56. A GPU that sits idle overnight and on weekends is rarely above 30 percent utilization for a business-hours tool, even if it is flat out at 2 p.m.
This gives you a clean way to find the breakeven. Set the effective cost equal to the API price and solve for utilization:
Breakeven utilization = hourly price ÷ (peak tokens per second × 3,600) × 1,000,000 ÷ API price per 1M tokens
Example: with the $0.56 full-load cost above against an API at a hypothetical $0.40 per million, breakeven utilization is 0.56 ÷ 0.40 = 140 percent. That is impossible, so the API wins at any traffic level. Against a hypothetical $2.00 per million API, breakeven is 28 percent: if you can keep the GPU busy more than about a quarter of the paid hours, self-hosting is cheaper. Compare live API prices for your model on the API pricing tool before plugging in a number.
Two consequences follow. First, open-weights models with many competing API providers tend to have low token prices, which makes the breakeven hard to reach; for something like Llama 3.3 70B you need serious, steady volume. Second, the cheaper the GPU per hour, the lower the utilization you need, which is why the price spread between marketplaces and managed clouds matters so much. The live table below shows that spread for common inference cards.
| GPU | VRAM | Cheapest now | Where | Median across providers | Providers |
|---|---|---|---|---|---|
| Nvidia GeForce RTX 4090 | 24GB | $0.30/hr | io.net | $0.54/hr | 9 |
| Nvidia L40S | 48GB | $0.55/hr | Novita AI | $0.97/hr | 11 |
| Nvidia A100 80GB PCIe | 80GB | $1.09/hr | Thunder Compute | $1.48/hr | 7 |
| Nvidia H100 | 80GB | $3.20/hr | Thunder Compute | $3.20/hr | 1 |
| Nvidia H200 | 141GB | $3.95/hr | Vast.ai | $3.97/hr | 3 |
Live data: median hourly price per GPU, collected in the last 24 hours. How we collect prices.
The hidden costs of self-hosting
The hourly price is the visible part of the bill. The rest tends to be larger than teams expect.
- Engineering time. Someone has to pick an inference engine, choose a quantization, tune batch limits and context sizes, set up health checks, autoscaling and logging, and keep all of it current as models and engines release new versions. Even a few days a month of an engineer's time can outweigh a modest GPU bill. Price that time honestly.
- Idle capacity. You pay for the GPU between requests, overnight, and during the long tail of low-traffic hours. To serve peak load with acceptable latency you usually provision above average demand, which pushes utilization down further.
- Cold starts. Scaling to zero sounds like a fix for idle cost, but loading tens of gigabytes of weights onto a fresh instance can take minutes, not seconds. For interactive traffic that means either keeping an instance warm (paying for idle) or accepting slow first responses.
- Redundancy. One GPU is a single point of failure. Production usually wants at least two instances, which doubles the floor cost before you serve a single token. Marketplace hosts with variable reliability, such as those on Vast.ai, make redundancy more important, not less.
- Egress, storage and setup time. Model downloads, persistent volumes, and billed time while an instance boots all add up. Our hidden costs of renting GPUs article itemizes these.
A practical way to include them: add your monthly engineering and redundancy cost to the GPU bill before dividing by tokens. If a $1,500-a-month GPU needs $1,500 a month of engineering attention, your real cost per token is double what the formula above says.
When APIs win, and when self-hosting does
Use this as a first-pass decision framework:
- If your traffic is bursty, low, or unpredictable, use an API. You will not reach breakeven utilization, and the API absorbs your peaks for free.
- If you use a popular open model that many providers host, start with an API. Competition keeps token prices low, and you can switch providers without changing code much.
- If you need a closed frontier model, the question is moot: there is no self-hosted option.
- If you have sustained, high-volume traffic that keeps a GPU above the breakeven utilization for most paid hours, self-hosting can be meaningfully cheaper, especially on marketplace pricing.
- If you run a fine-tuned or custom model that no provider hosts, self-hosting is the default. Some providers offer LoRA adapter hosting, which may be worth checking first.
- If data cannot leave your environment, self-host regardless of the cost math, and budget for it.
- If the workload is offline batch processing, such as embedding a corpus or summarizing an archive, self-hosting on cheap or interruptible GPUs is often the best deal, because you control the batch size and keep the GPU at close to 100 percent utilization for the length of the job.
That last case is worth dwelling on. Batch jobs are where self-hosting most reliably beats APIs, because they remove the utilization problem entirely. You rent the GPU, feed it as fast as it can go, and release it when the job finishes. See spot vs on-demand rental for how interruptible pricing changes the numbers.
A worked breakeven, start to finish
Suppose you serve a 70B model at 4-bit on one 80 GB card rented at a hypothetical $2.50 per hour, running all month (730 hours, $1,825). Suppose a well-configured engine reaches an estimated 600 tokens per second of aggregate output at the concurrency your traffic supports, and you have one engineer spending a hypothetical $1,000 a month of time on it.
- Full-load capacity: 600 × 3,600 × 730 = about 1.58 billion tokens a month.
- Total monthly cost: $1,825 + $1,000 = $2,825.
- At full load: $2,825 ÷ 1,577M tokens = about $1.79 per million.
- At 30 percent utilization (473M tokens): $2,825 ÷ 473 = about $5.97 per million.
Compare that with current hosted prices for the same class of model. If the API costs less than the effective self-hosted number at your realistic utilization, the API wins, and for a popular 70B model at today's API prices it usually does. Our cheapest way to run a 70B model runs this exact comparison against live prices, and the self-host vs API tool lets you plug in your own throughput, utilization and GPU price.