LoRA A small add-on file that changes how a big model behaves.
Short answer. LoRA (low-rank adaptation) is a way to fine-tune a model by freezing its original weights and training a pair of small matrices that are added to selected layers. Only those small matrices are trained and saved, so the result is an adapter of a few tens of megabytes rather than a full copy of the model. In the original paper, LoRA cut the trainable parameters of GPT-3 175B by 10,000 times and the GPU memory needed by 3 times, with no extra delay at inference. For a Llama 3.1 8B model, a rank-16 adapter on the attention layers trains about 13.6 million numbers, 0.17% of the model.
01
How it works
A layer’s weights form a large matrix, W. Instead of changing W, LoRA learns a correction built from two thin matrices, B and A, whose product has the same shape as W but is limited to a low rank r. The trainable count falls from d × k to r × (d + k). For a 4096 × 4096 matrix that is 16.8 million numbers against 131,072 at rank 16, or 0.78%. The original weights stay frozen.
After training, the correction can be merged into W, so the adapted model runs at exactly the speed of the original; Hugging Face’s PEFT documentation notes that LoRA adds no inference latency because adapter weights can be merged. The adapter can also be kept separate and switched on or off.
02
Adapter sizes, worked example
The table applies LoRA to a Llama 3.1 8B model, using the dimensions in Meta’s paper: 32 layers, model width 4096, a feed-forward width of 14336, and 8 key and value heads of 128 numbers each, which is a width of 1024. Adapter files are stored at 16-bit, 2 bytes per number. Counts are trainable numbers, computed from r × (inputs + outputs) for each adapted matrix.
Setting
Trainable numbers
Share of 8 billion
Adapter file at 16-bit
Rank 8, attention layers only
6.8 million
0.085%
about 14 MB
Rank 16, attention layers only
13.6 million
0.17%
about 27 MB
Rank 16, attention and feed-forward layers
41.9 million
0.52%
about 84 MB
Rank 64, attention and feed-forward layers
167.8 million
2.1%
about 336 MB
03
QLoRA and serving many adapters
QLoRA keeps the frozen base model in 4-bit form while training the adapter. Its authors fine-tuned a 65B model on a single 48 GB GPU.
One base model can carry many adapters. vLLM serves several LoRA adapters alongside the base model in parallel, selected by model name in each request, and llama.cpp’s server loads adapters with a --lora option and accepts a scale per request.
vLLM’s documentation warns that loading adapters while the server is running should not be used in production unless the environment is isolated and fully trusted.
Adapters must match the exact base model they were trained on. Keep the base model’s name and checksum with each adapter.
04
Why it matters for an office
LoRA is what makes fine-tuning realistic for an office: it needs far less memory than full fine-tuning, and the QLoRA authors trained a 65B model on one 48 GB GPU. An adapter is a small file that can be versioned, reviewed and deleted, which suits an audit trail better than a re-trained full model. A department could have its own adapter on a shared base model. The cautions from fine-tuning still apply. An adapter is shared by everyone who can use it, so it must not contain material that only some staff are allowed to see.
05
Questions people ask
Is LoRA as good as full fine-tuning?
The paper reports LoRA performing on par with or better than full fine-tuning on RoBERTa, DeBERTa, GPT-2 and GPT-3, with far fewer trainable parameters. Results depend on the task, rank and data, so test on yours.
What does rank mean in LoRA?
It is the width of the two thin matrices. Higher rank means more trainable numbers, a larger adapter and more capacity to learn, at higher cost. The table above compares ranks 8 to 64.
Can I swap LoRA adapters at run time?
Some servers can. vLLM can serve several adapters on one base model, and llama.cpp’s server can load adapters at start-up and apply a scale per request. Check that the base model and adapter match.
What is the difference between LoRA and QLoRA?
QLoRA is LoRA training on a base model compressed to 4-bit, which lowers the memory needed for training. The adapter itself is still small.
06
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
