Best Cloud GPU for LLM Inference
Rent the cheapest card whose memory holds your model and its context: a 24 GB RTX 4090 at $0.35/hr serves 7B to 13B models at 4-bit, a 48 GB L40S at $0.96/hr holds a 4-bit 70B for one user at moderate context, and an 80 GB A100 ($1.09/hr) or H100 ($1.99/hr) leaves room for long context and many users. Among cards that fit, more memory bandwidth means faster tokens.
In today's data: Lium, RunPod and Spheron rent all four picks, so one account covers the move from the RTX 4090 to the H100: on Spheron that is $0.58 to $2.75 an hour on-demand.
RTX 4090 for 7B to 13B models
Serves 4-bit 7B to 13B models with fast per-token latency thanks to high consumer memory bandwidth. Best tokens-per-dollar for single-model, single-user or low-concurrency endpoints.
L40S for one hosted model
48GB fits larger models or a bigger KV cache for concurrency, with datacenter reliability for a hosted endpoint. A balanced choice when a 24GB card is too tight but you do not need 80GB.
A100 for long context and many users
The 80 GB A100 holds a 4-bit 70B with room for long context and many users; at 8-bit a 70B needs about 86 GB with its cache, more than one A100. Strong memory bandwidth at a lower rate than the H100.
H100 for the most tokens per second
Highest memory bandwidth here plus FP8, which lifts throughput on large models and heavy concurrency. Pick it when tokens per second per dollar, not just per hour, is what you are optimizing.
Worth knowing
- Fitting the model is step one: a 7B in 4-bit needs about 4 to 6GB, a 70B in 4-bit about 40GB, before the KV cache.
- The KV cache grows with context length and batch size and can rival the weights in memory. Long context or high concurrency needs more VRAM than the weights alone suggest.
- Decode speed is bandwidth-bound, so tokens per second scales with memory bandwidth. Prefer a higher-bandwidth card over a higher-FLOP one for generation.
- For bursty traffic, per-second billing and scale-to-zero matter more than the sticker rate; idle GPUs are the real cost.
FAQ
In 4-bit quantization a 70B model needs roughly 35 to 40GB for weights plus the KV cache, about 44GB in all for one user at a 4,096-token context, so a 48GB card such as the L40S holds it. An 80GB A100 at $1.09/hr or H100 at $1.99/hr leaves room for long context and concurrent users. At 8-bit it needs about 86GB, and at 16-bit more than two 80GB cards.
Token generation reads the full model from memory for every token, so decode speed is limited by how fast the GPU moves data, not how fast it computes. A card with higher memory bandwidth serves more tokens per second even at similar FLOPs.
Providers counts every provider listing an offer that meets the pick's memory, in any billing class. 7 days is the average price move of the same on-demand configurations listed a week ago and today, not the change in the price shown. Prices render from today's verified snapshot, not from when this guide was written. Full table on the homepage; break-even math in the calculator.