System prompt The standing instructions a model reads before it reads your question.
Short answer. A system prompt is a block of instructions placed at the start of every conversation with a language model, before the user’s message, to set its role, tone, rules and format: for example, “Answer only from the documents provided; if the answer is not there, say so.” To the model it is simply more text at the start of the context window, marked with a system role. It costs tokens on every request, and it is not a security control. OWASP’s guidance is that a system prompt is neither a secret nor a security control, because prompt injection can alter it and the model can be made to reveal it.
01
How it works
Chat models do not see a conversation as a list of messages. Their software turns the list into one sequence of tokens, with control tokens that mark who is speaking. Hugging Face’s documentation describes the system role as directives on how the model should act, usually placed at the beginning of the chat. Different models use different control tokens, which is why the right chat template matters.
You set a system prompt in the software around the model. In Ollama, a Modelfile has a SYSTEM instruction that fills the system slot in the model’s template. Not every model wants one: DeepSeek advises against a system prompt for its R1 distilled models and says all instructions should go in the user prompt.
02
What belongs in one
Role and audience: who the assistant is for, such as the finance team of a small firm.
Scope: what to answer from. “Use only the documents provided; if the answer is not in them, say so” is the single most useful sentence in an office assistant.
Format: language (Hindi, English or both), length, headings, or a fixed JSON layout.
Evidence: ask for a short quote from the source with each claim.
Uncertainty: say what to do when the question is unclear or the sources disagree.
Keep it short. Every word is read again with every request, so a system prompt that grows to pages slows each answer.
03
What it costs
A system prompt uses context window and memory on every conversation. The table uses 1.3 tokens per English word and the KV cache cost of Qwen3-32B, 262 KB per token at 16-bit. A server that reuses the cache for a shared prefix, such as vLLM with automatic prefix caching, avoids recomputing it and can share one copy between conversations. Without that, each active conversation pays the full amount.
System prompt length
Tokens
KV cache per conversation
200 words (a short brief)
about 260
about 68 MB
1,000 words (a detailed policy)
about 1,300
about 341 MB
3,000 words (rules plus examples)
about 3,900
about 1.0 GB
04
What it cannot do
Keep a secret. OWASP lists system prompt leakage as a risk in its own right. Never put passwords, keys or private data in one.
Enforce permissions. OWASP advises that controls such as privilege separation and authorisation checks must not be delegated to the model. A rule in a prompt is a request, not a lock.
Resist injected text. Instructions hidden in a document the model reads can compete with it. See prompt injection.
Behave the same on every model. A prompt tuned for one model can read differently to another, so retest after any model change.
05
Why it matters for an office
Use the system prompt for behaviour: scope, tone, language and format. Enforce access in software, by deciding which documents a user’s search can reach. Treat prompts as controlled documents: keep them in version control, review changes, and record which version produced an answer. ANVE’s private GPTs are assistants built on the office’s own material, and its model gateway logs every model call with the user, the app and the model. Roles today are administrator and user; teams, finer roles and single sign-on are not on ANVE yet, as its security page states.
06
Questions people ask
Can users see the system prompt?
Not normally, but a model can sometimes be tricked into repeating it. Assume anyone with access to the assistant could read it, and keep secrets out.
How long should a system prompt be?
As short as works. Start with role, scope, format and the rule about missing answers. Add a rule only when testing shows a failure that the rule fixes.
Is a system prompt the same as a custom GPT?
A custom or private assistant is usually a system prompt combined with documents and sometimes tools. The system prompt is the part that sets its behaviour.
Do all models support a system prompt?
Most chat models do, through their chat template. Some makers recommend putting instructions in the user message instead, as DeepSeek does for its R1 distilled models.
07
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
