ANVE: the private AI office, on the record. See what that means

ANVE

Private AI glossary · updated 2026-10-07

Tokens
The pieces of text a model reads and writes, and why Hindi uses more of them.

Short answer. A token is the unit of text an AI model reads and writes: usually a word, part of a word, a number or a punctuation mark. A tokenizer cuts text into tokens before the model sees it, and the model produces its reply one token at a time. In English, one token averages about three-quarters of a word, so 1,000 words are roughly 1,300 tokens. Other scripts cost more: in our count, a Hindi sentence in Devanagari took 38 to 118 tokens depending on the tokenizer, against 25 for its English version. Tokens set how much fits in the context window, how long a reply takes, and what cloud services charge.

01

How text becomes tokens

The sentence “Please send the minutes of the meeting to all partners by Friday.” became 13 tokens with OpenAI’s o200k_base tokenizer: Please, send, the, minutes, of, the, meeting, to, all, partners, by, Friday, and the full stop. Common words are one token each. Rarer words split into pieces: “unbelievably” became three tokens, un, bel and ievably. Numbers, codes and names split into short fragments.

On English prose, we counted 1.30 to 1.35 tokens per word, using two OpenAI tokenizers and Qwen3’s on the abstracts of the Transformer paper and the Llama 3 paper listed below (335 words). That is about 5 characters per token. Meta reports 3.94 characters per token for the Llama 3 tokenizer on a sample of English. A safe working range is 4 to 5 characters, or about 1.3 tokens, per English word.

02

The same message in three languages

We counted tokens for one meeting-minutes sentence in English, Hindi and Hinglish, using three tokenizers. Words are counted by spaces. The Hindi and Hinglish versions say the same thing as the English one.

  • English: The partners agreed to review the lease agreement on Monday, and Neha will send the revised draft to the client by Friday.
  • Hindi: भागीदारों ने सोमवार को पट्टा समझौते की समीक्षा करने पर सहमति जताई, और नेहा शुक्रवार तक संशोधित मसौदा ग्राहक को भेजेंगी।
  • Hinglish: Partners ne Monday ko lease agreement review karne par sahmati di, aur Neha Friday tak revised draft client ko bhejengi.
Version of the sentenceWordsOpenAI o200k_baseOpenAI cl100k_baseQwen3
English2225 tokens25 tokens25 tokens
Hindi (Devanagari)2138 tokens118 tokens107 tokens
Hinglish (Hindi in Latin letters)2027 tokens32 tokens32 tokens
03

Numbers and codes cost more than prose

  • The dummy GST number 29ABCDE1234F1Z5 is 15 characters and took 8 tokens on both OpenAI tokenizers and 11 on Qwen3.
  • The amount ₹1,25,000 took 6 tokens on o200k_base, because digits and commas split apart.
  • The line “Invoice No. INV/2026-27/00123 dated 07/10/2026 for ₹1,25,000 plus 18% GST” has 10 words and took 32 tokens on both OpenAI tokenizers and 46 on Qwen3, which is 3.2 to 4.6 tokens per word.
  • Researchers found that the same text translated into different languages can differ in tokenized length by up to 15 times, and the gap persists even in tokenizers built for many languages.
04

Why it matters for an office

Tokens are the unit behind four practical limits. Capacity: the context window is counted in tokens, so Hindi text fills it faster than English. Speed: the model writes one token per step, so a page of Hindi takes longer to write on a tokenizer that needs four to five times as many tokens, as two of the three we tested did. Cost: cloud services bill per token, and researchers note that token-length gaps between languages mean unequal cost of access. Quality: the same researchers link these gaps to concerns about output quality across languages, so test with your real documents.

Meeting minutes in ANVE work from transcripts in Hindi, English and Hinglish, so token counts for mixed-language text matter when sizing a model and its context.

05

Questions people ask

How many tokens is a page of text?
About 650 tokens for a 500-word page of English prose, using 1.3 tokens per word. A page of invoices, tables or codes can need two or three times as many.
Are tokens the same as words?
No. A token can be a whole word, part of a word, a digit or a symbol. Short common words are usually one token, and long or rare words are several.
Why do Hindi and other Indian scripts use more tokens?
Most tokenizers learn their vocabulary from text that is mostly English and other Latin-script languages, so a Devanagari word may be cut into many small pieces. Newer tokenizers with larger vocabularies narrow the gap: in our sentence, o200k_base used 38 tokens against 118 for the older cl100k_base.
Do all models count tokens the same way?
No. Each model family has its own tokenizer, so the same text gives different counts. Count with the tokenizer of the model you will use.
06

What is not real yet

No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.

Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.