Is inference the same as running a model?

Yes. Running a trained model to produce output is inference. Training and fine-tuning change the weights; inference does not.

Does using an AI model train it?

Not the model itself. Its weights stay fixed while it answers. A cloud provider may use your inputs to train later models if its terms allow, which is a separate decision from inference.

What hardware does inference need?

Mostly memory: enough to hold the weights and the KV cache, and enough bandwidth to read them quickly. Arithmetic speed matters most for reading long prompts.

What is an inference engine?

The software that loads a model and runs it efficiently, such as llama.cpp, Ollama or vLLM. Different engines suit one user on a laptop or many users on a server.

ANVE: the private AI office, on the record. See what that means

ANVE

Private AI glossary · updated 2026-10-16

Inference
The everyday work of an AI model: answering, as opposed to learning.

Short answer. Inference is the stage where a trained AI model is used to produce an answer from new input, as opposed to training, where the model learns from data. Every question put to an assistant is one inference request. Training is done once by the model’s maker at great cost: Meta reports 30.84 million H100 GPU-hours for Llama 3.1 405B alone. Inference is what an office runs every day, and its memory and speed needs, not training cost, decide what hardware is enough. A request has two phases: prefill, which reads the whole prompt at once, and decode, which writes the answer one token at a time.

Training and inference compared

Aspect	Training	Inference
What happens	The model’s weights are adjusted on huge amounts of text	The finished weights are read to answer a request
Who does it	The model’s maker	Whoever runs the model: a cloud provider or your own machine
Do the weights change?	Yes	No
Scale	Llama 3.1 8B: 1.46 million H100 GPU-hours; 70B: 7.0 million; 405B: 30.84 million	One machine can serve many requests
How often	Once per model version	Every question, all day

The two phases of a request

In prefill, the model processes the whole prompt in one parallel pass and builds the KV cache. NVIDIA describes this phase as compute-bound: it keeps the processor busy. It decides the time to first token. In decode, the model generates the answer one token at a time, each step reading the weights and the cache from memory. NVIDIA describes this phase as memory-bound, since the speed at which data moves from memory, not the arithmetic, dominates. It decides tokens per second.

Databricks gives an example: a 7B model that takes 14 ms per token is moving 14 GB of weights every 14 ms, which is 1 TB per second of memory traffic. That is why memory bandwidth, more than raw compute, sets the speed of a single user’s answer.

What decides speed and cost

Model size and precision: more bytes to read for each token means fewer tokens per second.
Memory bandwidth: the rate at which the machine can stream the weights.
Prompt and answer length: long prompts lengthen prefill, long answers lengthen decode.
Number of users: serving several people at once shares the same hardware.
The engine: llama.cpp, Ollama and vLLM schedule the same model very differently.

Why it matters for an office

Where inference runs is the privacy question. With a cloud service, every prompt travels to the provider’s servers for inference. With a model on your own machine, it does not. Inference is also the recurring cost: cloud services charge per token on every request, while local hardware has a fixed cost and a limit on how many people it can serve at once. Because the weights do not change during inference, using a model does not teach it your documents; whether your text is logged or reused depends on the software around it.

Where ANVE stands

ANVE is designed to run inference on a device in the office. In the development build, its model gateway logs every model call with the user, the app and the model, and records whether the call ran locally or in the cloud. Cloud models are used only if an administrator adds a provider, and their answers are labelled; a policy that blocks confidential data from cloud models is not on ANVE yet. The security page has the details.

Questions people ask

Is inference the same as running a model?: Yes. Running a trained model to produce output is inference. Training and fine-tuning change the weights; inference does not.
Does using an AI model train it?: Not the model itself. Its weights stay fixed while it answers. A cloud provider may use your inputs to train later models if its terms allow, which is a separate decision from inference.
What hardware does inference need?: Mostly memory: enough to hold the weights and the KV cache, and enough bandwidth to read them quickly. Arithmetic speed matters most for reading long prompts.
What is an inference engine?: The software that loads a model and runs it efficiently, such as llama.cpp, Ollama or vLLM. Different engines suit one user on a laptop or many users on a server.

What is not real yet

No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.

Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.

InferenceThe everyday work of an AI model: answering, as opposed to learning.