Can I change a model’s tokenizer?

Not in practice. The model’s learned numbers are tied to its vocabulary. Swapping the tokenizer would require training the model again.

What is a special token?

A reserved entry in the vocabulary that marks structure rather than text, such as the start of a message, the end of a turn or the speaker’s role. Chat templates are built from them.

Why do ChatGPT and Llama count the same text differently?

They use different tokenizers with different vocabularies, so the same sentence is split at different points and gets a different count.

What is the difference between a tokenizer and an embedding model?

A tokenizer only cuts text into pieces and numbers them. An embedding model is a neural network that turns text into a vector that captures meaning.

ANVE: the private AI office, on the record. See what that means

ANVE

Private AI glossary · updated 2026-10-07

Tokenizer
The step that turns text into numbers before a model sees it.

Short answer. A tokenizer is the component that splits text into tokens and maps each token to a number from a fixed vocabulary before a language model reads it, then does the reverse for the reply. Most modern tokenizers use byte-pair encoding: they start from single characters or bytes and repeatedly merge the most frequent pairs until the vocabulary reaches a target size. Llama 3 uses a vocabulary of 128K tokens, which raised compression on English from 3.17 to 3.94 characters per token. Every model family has its own tokenizer, so token counts do not carry over between models, and a model must always be used with the tokenizer it was trained with.

How it works

Byte-pair encoding, or BPE, learns its vocabulary from a large text sample. It begins with single characters, finds the pair that occurs together most often, merges it into one new token, and repeats. Hugging Face’s documentation shows the idea with the words hug, pug, pun and bun: the pair u and g is the most frequent, so ug becomes a token, then un, and so on. Byte-level BPE starts from the 256 possible byte values, so any text can be encoded and no “unknown” token is needed.

SentencePiece is a library that applies BPE or a related method, Unigram, directly to raw text, spaces included, which suits languages such as Chinese and Japanese that do not separate words with spaces. At run time the tokenizer is just a lookup: text in, a list of numbers out. The model never sees the letters.

Vocabularies compared

Vocabulary size is a design choice. A larger vocabulary means fewer tokens per text, so faster generation and more text per context window, but a larger table inside the model. The Hindi column shows the effect on one 21-word sentence; the English version of the same message is 25 tokens in all three.

Tokenizer	Vocabulary size	Hindi sentence (tokens)	Notes
GPT-2	50,257	Not tested	Byte-level BPE, 256 byte tokens plus 50,000 merges plus one end token
OpenAI cl100k_base	100,277	118	An earlier OpenAI encoding, in tiktoken
OpenAI o200k_base	200,019	38	Used with GPT-4o, in tiktoken
Qwen3	151,669	107	Counted on the Qwen3-32B tokenizer
Llama 3	128K	Not tested	Meta’s Table 3 gives 128,000 for all three sizes

Why mismatches cause trouble

A model trained with one tokenizer produces nonsense if fed another’s numbers. Always load the tokenizer that ships with the model.
Token counts differ across families, so a prompt that fits one model’s window may not fit another’s. Recount when you switch models.
Chat models also expect a template of special tokens that mark who is speaking. Hugging Face’s documentation warns that with the wrong control tokens, models perform drastically worse.
Fairness: researchers found the same text can need up to 15 times as many tokens in some languages as in others, which raises cost and latency and shrinks usable context for those languages.

What to check

Count tokens with the model’s own tokenizer, on your real documents: Hindi, Hinglish, tables, invoices and codes, not just English prose.
If most of your text is not English, compare models partly on tokens per document, since a more efficient tokenizer makes the same hardware go further.
Use the chat template the model’s card specifies. Ollama stores a prompt template with each model, and llama.cpp’s server reads the template from the model file by default.

Questions people ask

Can I change a model’s tokenizer?: Not in practice. The model’s learned numbers are tied to its vocabulary. Swapping the tokenizer would require training the model again.
What is a special token?: A reserved entry in the vocabulary that marks structure rather than text, such as the start of a message, the end of a turn or the speaker’s role. Chat templates are built from them.
Why do ChatGPT and Llama count the same text differently?: They use different tokenizers with different vocabularies, so the same sentence is split at different points and gets a different count.
What is the difference between a tokenizer and an embedding model?: A tokenizer only cuts text into pieces and numbers them. An embedding model is a neural network that turns text into a vector that captures meaning.

What is not real yet

No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.

Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.

TokenizerThe step that turns text into numbers before a model sees it.