Best GPU for AI: Training vs Inference, at Live Rental Prices

Three defaults cover most AI work. For training, rent an H100, from $1.99/hr on-demand today. For inference, rent an A100: the family starts at $0.89/hr on-demand across its 40GB and 80GB variants, undercuts the H100 either way, and the 80GB card holds most open-weight models (per-variant prices are live on the A100 page). On a budget, rent an RTX 4090, from $0.35/hr, the cheapest serious card for anything that fits in 24GB. Every price on this page renders from today's snapshot of the live index, so the numbers you see are current, not the day this guide was written.

Those defaults are starting points. The rest of this guide is how to move off them deliberately: the three axes that actually decide the choice, a workload table, and the honest case for consumer cards.

How to choose: the three axes that matter

1. VRAM versus model size. The first cut is brutal and simple: if the model does not fit in VRAM, the card is useless to you at any price. Weights stored at 16-bit precision take two bytes per parameter, so a 7B model needs roughly 14GB before activations and cache, and a 70B model needs roughly 140GB, which no single mainstream card except the H200's 141GB covers unquantized. Quantization changes the math: at 4-bit, that 70B model shrinks to around 35GB of weights and fits on a 48GB card. Browse cards by capacity at 24GB, 48GB, and 80GB, each with live prices per gigabyte.

2. Throughput versus cost. The cheapest card per hour is not the cheapest card per job. A newer card that costs three times as much per hour but finishes a training run four times faster wins on the only number that matters, total spend per completed job. This is why H100s dominate training despite premium rates, and why inference flips the logic: a request that a 4090 serves fast enough does not get better by finishing faster on an H100, so the cheap card wins. Price the job, not the hour.

3. Interconnect, once you need more than one GPU. A single card is a price question. Multiple cards are a bandwidth question: gradient sync in training hammers the links between GPUs, which is where SXM cards with NVLink separate from PCIe variants of the same chip, and where consumer cards drop out entirely. If your model or batch does not fit on one GPU, read the multi-GPU cluster pricing guide before renting anything.

Workload table

Recommendations below come from our own spec and price data: the cheapest card whose VRAM and tier actually fit the job. Each links to a dedicated page with live per-provider prices, so this table stays a map, not a duplicate.

Workload	Recommended GPU	Why
LLM training	H100 / H200	80GB+ per card, NVLink for multi-GPU scaling, best cost per completed run
LLM fine-tuning (LoRA/QLoRA)	A100 80GB, or RTX 4090 for ≤13B	LoRA needs far less VRAM than full training; small jobs fit consumer cards
LLM inference, large models	A100 / H100	80GB holds big open-weight models; throughput per dollar beats splitting across small cards
LLM inference, ≤13B quantized	RTX 4090 or L4	24GB is plenty at 4-bit or 8-bit; both rent for a fraction of datacenter rates
Stable Diffusion / image generation	RTX 4090	SDXL fits comfortably in 24GB; raw single-card speed at the lowest price
Video generation	L40S / H100	Video models are VRAM-hungry; 48GB is the practical floor, 80GB is comfortable
Computer vision	L4 / RTX 4090	Detection and classification models are small; cheap 24GB cards clear them easily

Rendering work and notebook experimentation have their own pages too: 3D rendering and Jupyter notebooks.

When consumer cards make sense

The RTX 4090 is the best price-performance card on the index for anything that fits in 24GB, from $0.35/hr on-demand and less on community tiers. The RTX 5090 raises that to 32GB with a newer architecture. For fine-tuning small models, serving quantized models, image generation, and rendering, they are hard to argue with.

Two honest caveats. First, consumer cards mostly reach the cloud through marketplace hosts, RunPod Community Cloud, Vast.ai, and SaladCloud among them, rather than big datacenter fleets. That capacity is often interruptible and varies host to host in bandwidth and reliability, which is exactly why it is cheap. Read the secure vs community trade-off before betting a deadline on it. Second, consumer cards are single-card tools: no fast interconnect means multi-GPU training on 4090s spends its savings on synchronization overhead. Fit the job on one card or move up a tier.

FAQ

Do I need an H100, or will an A100 do?

Both carry 80GB in their top configurations, so fit is usually a tie. The H100 is the newer, faster chip and wins whenever the job is long enough that finishing sooner beats the lower hourly rate, which in practice means training. For inference and fine-tuning, the A100 is the better default: the family starts at $0.89/hr on-demand against $1.99/hr for the H100, and that gap is your margin. Compare live rates side by side on the H100 and A100 pages.

How much VRAM do I need for a 70B model?

At 16-bit precision, 70B parameters are roughly 140GB of weights alone, which means multiple 80GB cards or a single H200. Quantized to 4-bit, the same model is around 35GB of weights and runs on a 48GB card like the L40S, with quality loss that most serving workloads tolerate. The VRAM guide pages list every card at each capacity with live prices.

Is a rented RTX 4090 reliable enough for real work?

For batch work that checkpoints, yes, and the price makes it the rational choice. For a production endpoint, prefer on-demand capacity on a datacenter card, or at least the secure tier of a marketplace provider. The failure mode is not the silicon, it is the interruptibility and host variance of community capacity.

Should I train on spot instances?

If your training loop checkpoints and resumes automatically, spot capacity is the single biggest cost lever available. If it does not, one interruption erases the discount. The full trade-off, including where spot pricing sits relative to on-demand and serverless, is in the spot vs on-demand vs serverless guide.

What about AMD?

The MI300X offers 192GB on a single card, more than double an H100, which makes it genuinely interesting for large-model inference if your stack runs on ROCm. Software maturity is the real cost: if your pipeline is CUDA-native, budget porting time before counting the savings.