RAG or fine-tuning: which does an office need? Retrieval for facts, fine-tuning for habits. Most offices start with retrieval.
Short answer. Most offices need retrieval-augmented generation (RAG) first. RAG fetches the relevant passages from your documents at question time and has the model answer from them, so answers can cite a source, follow edits to your files, and drop a document when you delete it. Fine-tuning changes the model itself and suits habits such as house style, output formats and narrow classification, not facts. The research agrees: one study found RAG consistently outperformed unsupervised fine-tuning for adding knowledge, and another found that models learn new facts slowly through fine-tuning and then hallucinate more as they do. Fine-tuning can be cheap with LoRA, but it is a project, not a setting.
01
How each one works
RAG keeps knowledge outside the model. The 2020 paper that introduced the term combined a pre-trained language model with a dense vector index of Wikipedia, searched by a neural retriever, and reported more specific, diverse and factual language than the parametric-only baseline it was compared with. In an office the index holds your policies, files and past work.
Fine-tuning continues training on your examples so the model’s weights change. LoRA, a parameter-efficient method, freezes the base weights and trains small added matrices; its authors report cutting trainable parameters by 10,000 times and GPU memory by 3 times for GPT-3 175B compared with full fine-tuning.
02
Which one fits which need
Need
RAG
Fine-tuning
Answers from documents that change every week
Re-index the changed files
Needs another training run to learn the change
Show where each answer came from
Yes: the retrieved passage is the source
No: what was learned sits in the weights
Remove a withdrawn policy or a former client’s file
Delete it from the index
Hard: facts cannot be removed from the weights file by file
Staff see only what they are allowed to see
Filter retrieval by the user’s access
The model cannot enforce per-user access to what it has learned
House style, tone and a fixed output format
Prompts and templates may be enough
A good fit
A small model for one narrow task, such as classifying or extracting fields
Helps as context
A good fit
New facts the model never saw
A good fit
Weak: new facts are learned slowly and can raise hallucination
Effort to start
An index and a retrieval pipeline
Curated examples, training runs and evaluation
03
What the research says
Ovadia and colleagues compared unsupervised fine-tuning with RAG for injecting knowledge: RAG consistently outperformed it, both for knowledge the model had seen in training and for entirely new knowledge.
Gekhman and colleagues found that models struggle to acquire new factual knowledge through fine-tuning, that examples introducing new knowledge are learned significantly slower, and that as they are learned they linearly increase the model’s tendency to hallucinate.
RAG is not a cure for errors. In Stanford’s 2024 study, commercial legal research tools built on retrieval hallucinated between 17% and 33% of the time, although vendors had described RAG as eliminating or avoiding hallucinations.
04
A sensible order for an office
Start with clear prompts and templates for the tasks people repeat.
Add RAG over approved documents, with the source passage shown beside each answer.
Test on real questions from your own files, including questions whose answer is not in the documents, where the right reply is “not in the documents”.
Consider fine-tuning, for example with LoRA, only if the tests show a gap in style, format or a narrow task.
Re-test after every model change.
05
Where ANVE fits
ANVE answers questions over the office’s own documents and shows the source passage, and its private GPTs are assistants built on the office’s own material. Full document drafting with design templates is not on ANVE yet. ANVE is pre-release, and the software runs today on a development machine.
06
Questions people ask
What is RAG in one sentence?
Retrieval-augmented generation looks up the relevant passages in your documents first and gives them to the model to answer from. The 2020 paper that introduced the term was accepted at NeurIPS.
Does RAG stop hallucinations?
No. Stanford’s 2024 study of commercial legal research tools that use RAG found they hallucinated between 17% and 33% of the time, so citations still need checking.
Is fine-tuning expensive?
Less than it was: the LoRA paper reports cutting trainable parameters by 10,000 times and GPU memory by 3 times for GPT-3 175B. The harder cost is preparing good examples and testing the result.
Can an office do both?
Yes. One option is RAG for facts with a lightly fine-tuned model for the output format. Start with RAG and add fine-tuning only if testing shows a gap.
07
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
