Is 24GB of VRAM enough for running an LLM?
For inference, 24GB is enough to run models up to roughly 13B parameters comfortably, and up to around 30B when you use 4-bit quantization. It is not enough to fully fine-tune large models, and it will not hold a 70B model at usable precision without heavy quantization or splitting across cards. For most single-user inference and light fine-tuning, a 24GB card is a practical, cheap starting point.
| GPU | VRAM | $/hr | Where | |
|---|---|---|---|---|
| RTX A4000 | 16 GB | $0.10 | TensorDock on-demand | Rent → |
| Tesla V100 | 16 GB | $0.17 | DataCrunch on-demand | Rent → |
| RTX 3090 | 24 GB | $0.20 | TensorDock on-demand | Rent → |
| RTX A5000 | 24 GB | $0.27 | RunPod secure cloud | Rent → |
| Tesla T4 | 16 GB | $0.35 | Google Cloud on-demand | Rent → |
| A30 | 24 GB | $0.35 | Massed Compute on-demand | Rent → |
| RTX A6000 | 48 GB | $0.35 | Thunder Compute on-demand | Rent → |
| RTX 4090 | 24 GB | $0.35 | TensorDock on-demand | Rent → |
| L4 | 24 GB | $0.44 | Jarvislabs on-demand | Rent → |
| A40 | 48 GB | $0.44 | RunPod secure cloud | Rent → |
The rough rule is that a model needs about 2GB of VRAM per billion parameters at 16-bit precision, plus headroom for the key-value cache that grows with your context length. So a 7B model fits easily, a 13B model fits, and a quantized 30B model can fit if you accept some quality loss. Long prompts and large batch sizes eat into that budget quickly.
Related questions
- What GPU do I need for a 70B model?
- What is the cheapest GPU for Stable Diffusion?
- What is the cheapest 24GB GPU to rent?
Numbers on this page come from today's verified snapshot. Full table on the homepage; method in the methodology.