What GPU do I need to run Llama 405B?
Llama 3.1 405B needs roughly 810GB of VRAM at 16-bit, so there is no single GPU that runs it at full precision. You need a multi-GPU node, for example eight 80GB H100 or A100 cards, or a smaller cluster of 140GB-class cards like the H200 or MI300X. With 4-bit quantization the footprint drops to around 230GB, which still means several high-memory cards working together. The table below lists the highest-memory GPUs we track for building such a node.
| GPU | VRAM | $/hr | Where | |
|---|---|---|---|---|
| MI300X | 192 GB | $2.19 | RunPod secure cloud | Rent → |
| B200 | 180 GB | $3.50 | Vultr on-demand | Rent → |
| H200 | 143 GB | $3.62 | Massed Compute on-demand | Rent → |
The memory math starts at about 2GB per billion parameters at 16-bit, which puts 405B near 810GB before you add the key-value cache that grows with context length. That is firmly multi-GPU territory. The common full-precision setup is an eight-card 80GB node connected with NVLink so the GPUs can share the model efficiently, and providers rent these as a single instance.
Quantization brings the number down. At 8-bit the model needs roughly 405GB, and at 4-bit closer to 230GB, so a node of four to six 80GB cards, or two to three MI300X at 192GB, can serve it with some quality tradeoff. Fewer, larger cards mean less interconnect overhead, which can matter for latency.
Because 405B is bandwidth and interconnect sensitive, the quality of the multi-GPU fabric matters as much as raw VRAM. The table below lists the largest-memory cards in our index, sorted by price, as building blocks for a node. For a full multi-card instance, see how node pricing works.
Related questions
- What GPU do I need for a 70B model?
- How does multi-GPU node pricing work?
- How much does it cost to train an LLM?
Numbers on this page come from today's verified snapshot. Full table on the homepage; method in the methodology.