What GPU do I need for a 70B model?
A 70B model needs roughly 140GB of VRAM at 16-bit precision, so plan on two 80GB cards like an A100 80GB or H100, or a single card with enough memory such as an H200 or MI300X. With 4-bit quantization it can fit on a single 48GB card for inference, with some quality tradeoff. The cheapest verified A100 in our index is $0.89/hr.
| GPU | VRAM | $/hr | Where | |
|---|---|---|---|---|
| A100 | 80 GB | $0.89 | Jarvislabs on-demand | Rent → |
| RTX PRO 6000 | 96 GB | $1.10 | Google Cloud on-demand | Rent → |
| H100 | 94 GB | $1.75 | Lium secure cloud | Rent → |
| GH200 | 96 GB | $2.29 | Lambda on-demand | Rent → |
| MI300X | 192 GB | $2.39 | RunPod secure cloud | Rent → |
| H200 | 143 GB | $2.60 | GMI Cloud on-demand | Rent → |
| B200 | 180 GB | $3.49 | Prime Intellect on-demand | Rent → |
| MI325X | 256 GB | $3.80 | DigitalOcean on-demand | Rent → |
| B300 | 288 GB | $4.99 | Prime Intellect on-demand | Rent → |
| MI350X | 288 GB | $5.49 | RunPod secure cloud | Rent → |
The memory math starts at about 2GB per billion parameters at 16-bit, which puts a 70B model near 140GB before you add the key-value cache for context. That is why full-precision serving usually spans two 80GB GPUs connected with NVLink so they can share the model efficiently.
Quantization changes the picture. At 8-bit the model drops to around 70GB, and at 4-bit closer to 40GB, which brings single-card inference on a 48GB or larger GPU into reach. The tradeoff is a modest hit to output quality that many applications tolerate well.
Fine-tuning is more demanding than inference because you also store optimizer state and gradients. A full fine-tune of a 70B model wants a multi-GPU node, while parameter-efficient methods like LoRA can run on far less. The table below lists 80GB-class cards suited to 70B work.
Related questions
- What is the cheapest 80GB GPU to rent?
- Is 24GB of VRAM enough for running an LLM?
- How does multi-GPU node pricing work?
Numbers on this page come from today's verified snapshot. Full table on the homepage; method in the methodology.