Guides · Basics · 6 min read
Context length explained: how much memory does a longer conversation use?
Every local model lists a context length: 32K, 128K, sometimes more. It is how much text the model can keep in mind at once, including your messages, its replies and any documents you paste in. Longer context sounds strictly better, but it costs memory, and on a home GPU that memory often decides whether a model fits at all. This guide shows how much it really takes.
What is context, in tokens?
Models read text as tokens, small chunks of words. In English one token is roughly three quarters of a word, so:
- 4K tokens is about 3,000 words: a long chat or a short article.
- 8K tokens is about 6,000 words: a typical working session.
- 32K tokens is about 24,000 words: a long report or a medium-sized code file set.
- 128K tokens is about 96,000 words: a short book.
Why context uses memory: the KV cache
For every token in the conversation, the model stores a small set of numbers called keys and values, so it does not have to re-read everything for each new word. This store is the KV cache. It grows in a straight line with context length: double the context, double the cache. It sits in VRAM next to the model weights, so it competes for the same space.
How much memory, model by model
KV cache size depends on the model's design, not just its size. Estimates from our checker, with a standard 16-bit cache:
| Model | Weights at Q4 | 8K context | 32K context | 128K context |
|---|---|---|---|---|
| Llama 3.1 8B | 5.2 GB | 1.0 GB | 4.1 GB | 16.4 GB |
| Qwen3 8B | 5.3 GB | 1.1 GB | 4.5 GB | 18 GB |
| Qwen3.5 9B | 6.1 GB | 0.3 GB | 1.1 GB | 4.4 GB |
| Qwen3 14B | 9.6 GB | 1.3 GB | 5.1 GB | 20.5 GB |
| gpt-oss 20B | 13.6 GB | 0.4 GB | 1.5 GB | 6.1 GB |
| Gemma 3 27B | 17.8 GB | 4.0 GB | 15.9 GB | 63.5 GB |
| Qwen3.6 27B | 17.8 GB | 0.5 GB | 2.1 GB | 8.5 GB |
| Llama 3.3 70B | 45.8 GB | 2.6 GB | 10.3 GB | 41 GB |
Two things jump out. First, at 128K context the cache can be bigger than the model itself. Second, newer models with hybrid or sliding-window attention, such as Qwen3.5 9B, Qwen3.6 27B and gpt-oss 20B, use a fraction of the cache of older designs. If you work with long documents, they are the models to pick.
A worked example: Qwen3 14B on a 16 GB card
A 16 GB card gives a model about 14.4 GB. At Q4_K_M, Qwen3 14B needs:
- 10.9 GB at 8K context: fits easily.
- 12.2 GB at 16K: still fits.
- 14.7 GB at 32K: just over the limit, so part of it spills into system RAM and slows down.
Same model, same card: context alone decides whether it runs at full speed. Try it yourself by changing the context setting in the checker.
How to fit more context
- Only set what you need. Ollama and LM Studio let you choose the context length. If you never paste long documents, 8K is plenty and leaves room for a bigger model.
- Pick a model with a small cache. See the table above: Qwen3.5, Qwen3.6 and gpt-oss models hold long context cheaply.
- Quantize the cache. llama.cpp, LM Studio and recent Ollama versions can store the KV cache at 8-bit, roughly halving its size with little quality loss. Look for a "KV cache quantization" or "flash attention" setting.
- Drop the weights one step. Going from Q5 to Q4 frees a few GB for context on a 14B model.
- Start new chats. A fresh conversation resets the cache. Long chats about many topics waste context on old turns.
Does more context make answers worse?
A little, sometimes. Most models handle the start and end of a long context best and can miss details in the middle. A model advertised at 128K is often most reliable well below that. For important work, give the model only the documents it needs rather than everything at once.
Want the full memory formula, including weights and overhead? Read how much VRAM you need.