Cheapest way to run a 70B model: API, rented GPU, or your own.
Hosted API, rented GPU or owned hardware for a 70B LLM? Cost math for occasional, internal-tool and production workloads, with live prices and breakeven.
By Vi Nguyen · Published · Prices on this page are live · Editorial policy
A 70B open-weights model like Llama 3.3 70B Instruct is the point where "just run it yourself" stops being obvious. It is too big for one consumer card at full precision, small enough that a single datacenter GPU handles it once quantized, and popular enough that a dozen API providers sell it by the token. There are three ways to pay for it: per token through a hosted API, per hour on a rented GPU, or up front for hardware you own. Which is cheapest depends almost entirely on how many tokens you push through it and how evenly they arrive. This article puts numbers on all three for three realistic workloads, using live prices where prices exist.
What a 70B model actually needs
Weights are roughly parameters times bytes per parameter: about 140 GB at FP16, 70 GB at FP8 or INT8, and 35 to 40 GB at 4-bit. Add the KV cache, which for Llama 3 70B is about 0.33 MB per token at FP16 (80 layers × 8 KV heads × 128 head dimension × 2 for key and value × 2 bytes), and 10 to 20 percent runtime overhead. Our VRAM guide works through the full calculation.
That gives three practical hardware shapes for self-hosting:
- A single 48 GB card at INT4, such as the RTX A6000 or L40S. Fits with a short to medium context and one or two concurrent users.
- Two 24 GB cards at INT4, such as a pair of RTX 4090s or RTX 3090s, splitting the model with tensor parallelism over PCIe. Same capacity as one 48 GB card, often cheaper per hour, a bit more fiddly.
- A single 80 GB card at INT4 or FP8, such as the A100 80GB or H100. Room for long contexts and real concurrency, which is what production serving needs.
Three workloads to price against
Cost comparisons without a workload are meaningless, so here are three profiles. The token counts are round illustrative figures; swap in your own.
- Occasional use. A developer or small team poking at the model a few times a day. Call it 2 million tokens a month, spread unpredictably.
- Steady internal tool. A document assistant or support-drafting tool used during business hours: about 10 hours a day, 22 working days a month (220 hours), 50 million tokens a month.
- Production traffic. A customer-facing feature running around the clock (730 hours a month) at 500 million tokens a month, with latency expectations.
For token-priced APIs we assume three input tokens for every output token, which is typical of chat and retrieval workloads where prompts carry context. If your ratio differs, the math below is easy to redo.
Option 1: a hosted API
Many inference providers serve Llama 3.3 70B at a per-token price, and competition has pushed that price low. Here is the current spread:
| Provider | Input $/1M tokens | Output $/1M tokens |
|---|---|---|
| OpenRouter | $0.10 | $0.32 |
Refreshed daily from provider catalogs. All meta-llama/Llama-3.3-70B-Instruct options.
The monthly API cost formula is simple:
API cost = (input tokens ÷ 1M × input price) + (output tokens ÷ 1M × output price)
At the cheapest listing right now (OpenRouter, $0.10 in and $0.32 out per million), the 3:1 blend works out to $0.16 per million tokens. That puts our three workloads at about:
- Occasional (2M tokens): $0.31 a month.
- Internal tool (50M tokens): $8 a month.
- Production (500M tokens): $78 a month.
The striking thing is how small the first two numbers are. The API bills nothing when idle, handles scaling, and needs no engineering beyond an HTTP client. The trade-offs are rate limits, a shared queue you do not control, and sending your prompts to a third party. Compare providers side by side on our API pricing tool.
Option 2: renting a GPU by the hour
Renting flips the cost structure. You pay for hours, not tokens, so the cost is fixed whether the GPU is busy or idle. Here are the cards that fit a 70B model at 4-bit, at current prices:
| GPU | VRAM | Cheapest now | Where | Median across providers | Providers |
|---|---|---|---|---|---|
| Nvidia RTX A6000 | 48GB | $0.32/hr | Nosana | $0.42/hr | 6 |
| Nvidia L40S | 48GB | $0.38/hr | Lium | $0.79/hr | 5 |
| Nvidia A100 80GB PCIe | 80GB | $1.79/hr | Thunder Compute | $2.07/hr | 2 |
| Nvidia H100 | 80GB | $2.81/hr | Fluence | $3.20/hr | 2 |
| Nvidia GeForce RTX 4090 | 24GB | $0.30/hr | io.net | $0.38/hr | 7 |
| Nvidia GeForce RTX 3090 | 24GB | $0.12/hr | Vast.ai | $0.22/hr | 6 |
Live data: median hourly price per GPU, collected in the last 24 hours. How we collect prices.
The formula here is rental cost = hourly price × GPUs × hours running. For the internal tool we assume you start the instance for business hours only; for production it runs all month.
- One RTX A6000 at $0.32/hr: about $70 for 220 business hours, or $232 for a full 730-hour month.
- Two RTX 4090s at $0.30/hr each: about $132 for 220 hours, or $438 for a full month.
- One H100 at $2.81/hr: about $618 for 220 hours, or $2,051 for a full month.
Those figures use the cheapest fresh listing, which on marketplaces like Vast.ai or Clore.ai comes from host-supplied machines with variable reliability. Managed clouds like Lambda or RunPod's Secure Cloud cost more for the same card. Our marketplaces vs managed clouds piece covers when the premium is worth it.
Can the rented GPU actually handle the traffic? A rough ceiling on single-stream decode speed is memory bandwidth divided by bytes of weights read per token. With about 38 GB of 4-bit weights, an H100 SXM (3.35 TB/s) tops out near 88 tokens per second per stream, an A100 80GB PCIe (about 1.9 TB/s) near 50, and an L40S (about 0.86 TB/s) near 23. These are theoretical estimates, and real engines land below them. Batching changes the picture: because the weights are read once per step for the whole batch, a server handling 16 to 32 concurrent requests can deliver many times the single-stream figure in aggregate. For the production profile, 500 million tokens a month averages about 190 tokens per second around the clock, with peaks higher. That is within reach of one well-batched 80 GB card and a stretch for a 48 GB one, whose limited KV cache room caps concurrency.
Option 3: buying the hardware
Owning hardware turns an hourly cost into a capital cost plus electricity. The prices of GPUs move a lot, so we will not quote one; instead, assume a hypothetical $5,000 for a used workstation with two 24 GB cards, which is the cheapest owned setup that runs a 70B model at 4-bit.
The monthly cost of ownership is:
Owned cost per month = purchase price ÷ amortization months + power (kW × hours × $/kWh)
Amortized over 36 months, $5,000 is about $139 a month. If the box draws 0.8 kW under load and runs all month at a hypothetical $0.15/kWh, power adds 0.8 × 730 × 0.15 = $88. Call it $227 a month for a machine that is always on, before your time, cooling, or the chance a card dies.
Breakeven against renting the same setup is:
Breakeven months = purchase price ÷ (monthly rental cost − monthly power cost)
Renting two RTX 4090s all month at today's cheapest listing costs about $438. Using that number, the hypothetical $5,000 box pays for itself in about 14.3 months of continuous use. At half-time use the rental bill halves but the purchase price does not, so the breakeven roughly doubles.
Buying only wins when utilization is high and sustained, and when you would otherwise rent the same shape of hardware for years. Against an API, it rarely wins at all for a 70B model: your owned box cannot touch the per-token price of a provider running large batches on high-bandwidth datacenter cards.
Putting the three side by side
- Occasional use: use the API. A few dollars a month beats any GPU, rented or owned. Even an hour-a-day rental costs more than the entire API bill, and the cold start of loading 40 GB of weights makes on-demand rental awkward for sporadic requests.
- Steady internal tool: usually the API, sometimes a rented 48 GB card. At 50 million tokens the API bill is typically still smaller than 220 hours of GPU rental. Renting wins when you have privacy or data-residency constraints, need a fine-tuned variant no provider hosts, or when your token volume is far higher than our profile.
- Production traffic: compare an 80 GB rental against the API on real throughput. This is the only profile where self-hosting can win on price, and only if you keep the GPU busy with batched requests. Run the numbers with our self-host vs API calculator and read the breakeven guide before committing.
- Buying: only for sustained, near-continuous load where you already have the space, power and appetite for hardware maintenance.
One more lever deserves mention: interruptible capacity. Spot or interruptible listings can cut rental prices considerably for batch jobs that tolerate restarts, such as offline summarization of a document archive. They are a poor fit for anything user-facing. See spot vs on-demand GPU rental for the trade-offs, and the hidden costs of renting GPUs for the line items that do not appear on the hourly price.