Which open models handle very long documents? Context windows on paper, what research found, and what long input costs in memory.
Short answer. Several open models accept very long inputs on paper: Llama 3.3 70B and Gemma 3 27B take 128K tokens, the gpt-oss models 131,072, Qwen3.8-27B 262,144 natively, and Qwen3-32B 32,768 (131,072 with a YaRN setting). Accepting a long input is not the same as using it well. The RULER benchmark found that only about half of 17 models claiming 32K or more kept satisfactory performance at 32K, and another study found accuracy drops when the relevant passage sits in the middle. Memory is the other limit: 131,072 tokens costs about 43 GB of cache on a 70B model.
01
Context windows of the models in our library
Cache = 2 × layers × key/value heads × head size × bytes per token, from each model’s published configuration (NVIDIA’s formula). Ollama’s own default context is only 4,096 tokens, so a long window is something you must switch on, with the memory it needs. Qwen’s card also warns that frameworks apply YaRN scaling statically, which can affect performance on shorter texts, and advises enabling it only when you need the length.
Model
Context window (maker’s figure)
KV cache for one conversation (FP16, estimate)
Qwen3-32B
32,768 natively; 131,072 with a YaRN setting
34.4 GB at 131,072
Qwen3-30B-A3B
32,768 natively; 131,072 with a YaRN setting
12.9 GB at 131,072
Llama 3.3 70B Instruct
128k (131,072)
42.9 GB at 131,072
Gemma 3 27B
128K
66.6 GB at 131,072, counting every layer (upper bound)
gpt-oss-20b and gpt-oss-120b
131,072 positions in the configuration
6.4 GB and 9.7 GB, counting every layer (upper bounds)
Qwen3.8-27B
262,144 natively; extensible to about 1,000,000
17.2 GB at 262,144, plus about 0.15 GB of fixed state
02
Long input is not long understanding
The RULER benchmark extended the usual needle-in-a-haystack test to more demanding tasks and evaluated 17 long-context models. Despite near-perfect scores on the simple test, almost all models lost accuracy as the context grew, and only about half of those claiming 32K or more kept satisfactory performance at 32K. An earlier study, Lost in the Middle, found that models tend to use information at the beginning or end of a long input better than information in the middle.
Both studies are from 2023 and 2024 and used earlier models than today’s. Treat them as a reason to test, not as a measurement of the newest releases.
03
Alternatives to pasting the whole document
Retrieval: split the document into passages, find the ones that match the question, and give the model only those, with the source shown. This is what ANVE’s document question answering does.
Staged summaries: summarise sections, then summarise the summaries, keeping a page reference for each fact.
Targeted questions: ask for the clause, table or number you need instead of a general summary.
04
A checklist before relying on long context
Set the context length explicitly and budget the cache memory for every simultaneous conversation.
Count tokens with the model’s own tokenizer. OpenAI’s rule of thumb is about 0.75 English words per token; other languages and scripts usually need more tokens per word.
Test with questions whose answers sit at the start, middle and end of a document you know well.
Ask the model to quote the sentence it relied on, then check the quote.
05
Questions people ask
How many pages is 131,072 tokens?
Roughly 98,000 English words if a token averages three-quarters of a word, which is OpenAI’s rule of thumb for English. Hindi and other scripts usually use more tokens per word, so count with the model’s tokenizer.
Does a 1,000,000-token context mean I can load all our files?
No. Qwen3.8-27B’s cache alone would be about 66 GB at 1,000,000 tokens by this method, and accuracy over such lengths needs testing. Retrieval over your documents is usually the better design.
Why did my local model forget the start of my document?
Ollama’s default context is 4,096 tokens, and a conversation longer than the context window cannot all be seen by the model. Raise the context setting and check the memory it adds.
06
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
