GPUs · NVIDIA
Can the RTX 5090 run local LLMs?
With 32 GB of VRAM (about 28.8 GB usable) and 1792 GB/s of memory bandwidth, the RTX 5090 runs 34 of 39 popular open models fully on the GPU at 8K context. The largest is Qwen3.6 35B-A3B (MoE) at Q5_K_M, at roughly 200 tokens per second.
Best picks for the RTX 5090
Every model on the RTX 5090
Results at 8K context, picking the highest quality quantization that fits. "Offload" means part of the model runs from system RAM (we assume 32 GB), which works but is much slower.
| Model | Status | Quant | Memory | Speed |
|---|---|---|---|---|
| Qwen3.6 35B-A3B (MoE) | Runs well | Q5_K_M | 28 GB | ~200 tok/s |
| Qwen3 32B | Runs well | Q5_K_M | 28 GB | ~50 tok/s |
| DeepSeek-R1 Distill 32B | Runs well | Q5_K_M | 28 GB | ~50 tok/s |
| Qwen2.5-Coder 32B | Runs well | Q5_K_M | 28 GB | ~50 tok/s |
| Gemma 4 31B | Runs well | Q5_K_M | 29 GB | ~53 tok/s |
| Qwen3 30B-A3B (MoE) | Runs well | Q6_K | 28 GB | ~170 tok/s |
| Qwen3-Coder 30B-A3B | Runs well | Q6_K | 28 GB | ~170 tok/s |
| Qwen3.6 27B | Runs well | Q6_K | 26 GB | ~52 tok/s |
| Gemma 3 27B | Runs well | Q5_K_M | 26 GB | ~60 tok/s |
| Gemma 4 26B-A4B (MoE) | Runs well | Q6_K | 25 GB | ~150 tok/s |
| Mistral Small 3 24B | Runs well | Q6_K | 24 GB | ~59 tok/s |
| Devstral Small 24B | Runs well | Q6_K | 23 GB | ~59 tok/s |
| Codestral 22B | Runs well | Q8_0 | 28 GB | ~49 tok/s |
| gpt-oss 20B | Runs well | Q8_0 | 25 GB | ~130 tok/s |
| Qwen3 14B | Runs well | Q8_0 | 19 GB | ~74 tok/s |
| DeepSeek-R1 Distill 14B | Runs well | Q8_0 | 19 GB | ~74 tok/s |
| Phi-4 14B | Runs well | Q8_0 | 18 GB | ~75 tok/s |
| Gemma 3 12B | Runs well | Q8_0 | 17 GB | ~90 tok/s |
| Mistral Nemo 12B | Runs well | Q8_0 | 16 GB | ~90 tok/s |
| Gemma 4 12B | Runs well | Q8_0 | 17 GB | ~92 tok/s |
| Qwen3.5 9B | Runs well | Q8_0 | 12 GB | ~120 tok/s |
| Qwen3 8B | Runs well | Q8_0 | 11 GB | ~130 tok/s |
| Llama 3.1 8B | Runs well | Q8_0 | 10 GB | ~140 tok/s |
| DeepSeek-R1 Distill 8B | Runs well | Q8_0 | 10 GB | ~140 tok/s |
| Qwen2.5-Coder 7B | Runs well | Q8_0 | 9.8 GB | ~140 tok/s |
| Mistral 7B v0.3 | Runs well | Q8_0 | 9.4 GB | ~150 tok/s |
| Qwen3.5 4B | Runs well | Q8_0 | 6.4 GB | ~230 tok/s |
| Gemma 3 4B | Runs well | Q8_0 | 6.9 GB | ~260 tok/s |
| Qwen3 4B | Runs well | Q8_0 | 5.8 GB | ~270 tok/s |
| Phi-4 Mini 3.8B | Runs well | Q8_0 | 5.2 GB | ~290 tok/s |
| Llama 3.2 3B | Runs well | Q8_0 | 4.6 GB | ~340 tok/s |
| Llama 3.2 1B | Runs well | Q8_0 | 1.6 GB | ~920 tok/s |
| Gemma 3 1B | Runs well | Q8_0 | 1.4 GB | ~1100 tok/s |
| Qwen3 0.6B | Runs well | Q8_0 | 0.9 GB | ~1830 tok/s |
| Llama 3.3 70B | Offload | Q3_K_M | 40 GB | ~4.0 tok/s |
| DeepSeek-R1 Distill 70B | Offload | Q3_K_M | 40 GB | ~4.0 tok/s |
| Llama 4 Scout 109B-A17B | Won’t fit | — | 73 GB | — |
| gpt-oss 120B | Won’t fit | — | 77 GB | — |
| Qwen3 235B-A22B (MoE) | Won’t fit | — | 155 GB | — |
What this means in practice
The RTX 5090 is comfortable with models up to about 30B parameters. Llama 3.3 70B is the most useful step up beyond that: it needs about 40 GB, so part of it runs from system RAM and speed drops to around 4.0 tokens per second. Models from Llama 4 Scout 109B-A17B upward need 73 GB or more and are out of reach. NVIDIA cards have the broadest support: Ollama, LM Studio and llama.cpp all use CUDA out of the box.
Compare the RTX 5090
Questions
What is the biggest model the RTX 5090 can run?
At 8K context the largest model that fits entirely in its 32 GB of VRAM is Qwen3.6 35B-A3B (MoE) at Q5_K_M, at roughly 200 tokens per second.
Can the RTX 5090 run 70B models like Llama 3.3 70B?
Only with heavy CPU offload. It needs about 40 GB at Q4, far more than 32 GB, so speed drops to a few tokens per second. A 48 GB setup or a Mac with 64 GB or more is the practical route.
What is the best model for coding on the RTX 5090?
Qwen3.6 35B-A3B (MoE) at Q5_K_M fits fully and runs at about 200 tokens per second. Pair it with an editor extension like Continue pointed at Ollama.
Estimates based on memory size and bandwidth; real speed varies with drivers, software and context length. See how we calculate VRAM.