Chunking Cutting documents into passages a search can find and a model can read.
Short answer. Chunking is the step that splits documents into passages, called chunks, before they are embedded and indexed for retrieval. Chunks that are too small lose context; chunks that are too large bury the answer and waste the model’s context window. In NVIDIA’s tests across five datasets, token chunks of 128 and 2,048 tokens generally underperformed medium sizes, factoid questions favoured 256 to 512 tokens, and page-level chunking had the best average accuracy at 0.648. The best chunking follows the document’s own structure: clauses in a contract, line items in an invoice, speaker turns in minutes.
01
How it works
The indexing pipeline is: read the file, extract its text, split it into chunks, embed each chunk, and store the vectors with the original text and metadata. A search later returns chunks, and those chunks are what the language model reads and what a person sees as the source passage. So chunk boundaries decide both what can be found and what can be quoted.
Common strategies are fixed-size chunks with overlap, splitting at paragraph or sentence boundaries, splitting by structure such as headings, pages and clauses, and splitting where the meaning shifts. Overlap repeats the end of one chunk at the start of the next, so a sentence cut by a boundary still appears whole somewhere.
02
What NVIDIA tested
NVIDIA compared fixed token chunks of 128, 256, 512, 1,024 and 2,048 tokens with 15% overlap, page-level chunks and section-level chunks, on five datasets that included financial reports and business documents.
Result
Value
Highest average accuracy across datasets
Page-level chunking, 0.648, with the lowest variation (standard deviation 0.107)
Extreme token sizes
128 and 2,048 tokens generally underperformed medium sizes
Best token size, FinanceBench
1,024 tokens (0.579)
Best token size, Earnings
512 tokens (0.681)
Best on KG-RAG
Page-level chunking (0.520)
By question type
Factoid questions did best at 256 to 512 tokens; complex analytical questions did better with larger chunks or whole pages
03
Chunking office documents
Document
Chunk on
Why
Contract
Numbered clause or section, with its heading
A clause must stay with its conditions, and the heading carries context
Invoice or GST bill
The whole bill, or line-item groups with the header repeated
Totals and tax rates depend on the header
Meeting transcript
Topic blocks, with speaker names kept
Action items refer to who said what
Policy or SOP
Heading plus its paragraphs
The heading says what the rules apply to
Scanned PDF
Page, after text recognition
Layout is lost when the text is flattened
Spreadsheet
A block of rows with the header row repeated
A row means nothing without its column names
04
Settings to test
Size: start between 256 and 1,024 tokens and measure, since the best size depends on the content and the questions.
Overlap: NVIDIA used 15%. Try it, and also try none.
Context: put the document title and section heading at the top of every chunk.
Metadata: keep file, page, section, date and access group with each chunk, so results can be cited and filtered.
Evaluation: use real questions and check whether the right chunk appears in the top five.
Re-index after any change to the strategy.
05
Why it matters for an office
Chunking sets the quality of the source passage shown beside an answer. A chunk that starts mid-sentence makes a poor citation, and a chunk that cuts a clause from its exception can mislead. It also controls how much text goes into the context window: even with a long window, the Lost in the Middle study found models use information in the middle of long inputs less well, so a few precise chunks beat many loose ones. ANVE’s document question answering shows the source passage with each answer.
06
Questions people ask
What is a good chunk size?
There is no single best size. NVIDIA found medium sizes beat the extremes, with 256 to 512 tokens good for factual questions and larger chunks or pages better for analytical ones. Test on your own documents.
Why overlap chunks?
Overlap stops a sentence or fact being split across a boundary and lost. NVIDIA used 15%. Too much overlap wastes storage and returns near-duplicate results.
Should all chunks be the same size?
Not necessarily. Structure-based chunks, such as clauses, vary in length but keep meaning together. Set a maximum so none exceeds the embedding model’s longest input.
Does chunking still matter with a long-context model?
Yes. Long windows cost memory and time, and accuracy can fall for facts in the middle of long inputs, so retrieving a few well-chosen chunks is still better.
07
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
