Best Cloud GPU for LLM Inference
LLM inference is bandwidth-bound and memory-gated, so the right card is the cheapest one whose VRAM holds your quantized model plus the KV cache. A 24GB RTX 4090 at $0.69/hr serves quantized 7B to 13B models well; an 80GB A100 at $0.89/hr or H100 at $1.99/hr is the floor for 70B. Match memory bandwidth to your throughput target, since tokens per second track bandwidth more than raw FLOPs.
The picks, with live prices
| Pick | GPU | VRAM | On-demand from | Where | |
|---|---|---|---|---|---|
| value pick | RTX 4090 | 24 GB | $0.69 | RunPod secure cloud | Rent → |
| endpoint pick | L40S | 48 GB | $0.88 | Massed Compute on-demand | Rent → |
| scale pick | A100 | 80 GB | $0.89 | Jarvislabs on-demand | Rent → |
| performance pick | H100 | 94 GB | $1.99 | Voltage Park on-demand | Rent → |
RTX 4090 value pick
Serves 4-bit 7B to 13B models with fast per-token latency thanks to high consumer memory bandwidth. Best tokens-per-dollar for single-model, single-user or low-concurrency endpoints.
L40S endpoint pick
48GB fits larger models or a bigger KV cache for concurrency, with datacenter reliability for a hosted endpoint. A balanced choice when a 24GB card is too tight but you do not need 80GB.
A100 scale pick
The 80GB A100 is the practical floor for a 70B model in reasonable precision and for high-concurrency serving where the KV cache grows with batch size. Strong bandwidth at a lower rate than H100.
H100 performance pick
Highest memory bandwidth here plus FP8, which lifts throughput on large models and heavy concurrency. Pick it when tokens per second per dollar, not just per hour, is what you are optimizing.
Worth knowing
- Fitting the model is step one: a 7B in 4-bit needs about 4 to 6GB, a 70B in 4-bit about 40GB, before the KV cache.
- The KV cache grows with context length and batch size and can rival the weights in memory. Long context or high concurrency needs more VRAM than the weights alone suggest.
- Decode speed is bandwidth-bound, so tokens per second scales with memory bandwidth. Prefer a higher-bandwidth card over a higher-FLOP one for generation.
- For bursty traffic, per-second billing and scale-to-zero matter more than the sticker rate; idle GPUs are the real cost.
FAQ
In 4-bit quantization a 70B model needs roughly 40GB for weights plus the KV cache, so an 80GB A100 or H100 is the single-card floor. Higher precision or long context pushes you toward multiple GPUs.
Token generation reads the full model from memory for every token, so decode speed is limited by how fast the GPU moves data, not how fast it computes. A card with higher memory bandwidth serves more tokens per second even at similar FLOPs.
Prices render from today's verified snapshot, not from when this guide was written. Full table on the homepage; break-even math in the calculator.