Rate limit A cap on how fast a user, key or program may call a service.
Short answer. A rate limit caps how many requests, or how many tokens, a user, key or application may send in a given period. When the cap is hit the server answers HTTP 429 Too Many Requests, and may add a Retry-After header saying how long to wait. Cloud AI providers measure limits in requests per minute and tokens per minute, and Anthropic uses a token bucket, so capacity refills continuously. On an office AI server a rate limit does a second job: it stops one person, or one runaway script, from using all of a shared GPU. OWASP recommends rate limits and user quotas to prevent unbounded consumption.
01
What gets limited
Providers count more than one thing at once, and whichever limit is reached first applies. OpenAI’s own example: a limit of 20 requests per minute is used up by 20 tiny requests, even though a limit of 150,000 tokens per minute has barely been touched.
Limit
What it counts
Example from the providers
Requests per minute (RPM)
Calls, however small
OpenAI lists RPM and daily requests (RPD)
Tokens per minute (TPM)
Tokens in and out
OpenAI lists TPM and daily tokens (TPD); Anthropic counts input and output tokens separately (ITPM, OTPM)
Monthly spend cap
Money spent
Anthropic pauses API use at a tier’s spend cap until the next month starts
02
What happens at the limit
The server returns status 429. RFC 6585 defines it as the user having sent too many requests in a given amount of time, and says the response may include a Retry-After header telling the client how long to wait. The RFC leaves counting to the server: per resource, across the whole server, or per user identified by credentials or a cookie. OpenAI and Anthropic both return 429 with a retry hint, and OpenAI also reports remaining requests and tokens in response headers.
Good client code waits for the hinted time, then retries with growing delays instead of hammering the server. Anthropic adds that a sharp rise in usage can trigger acceleration limits, so traffic should be ramped up gradually.
03
Why an office should set its own
A private server has no provider standing in front of it. If every request is accepted, three problems appear.
One heavy user or script slows everyone down on a shared GPU.
A runaway agent loop sends request after request. OWASP’s excessive-agency guidance suggests rate limiting to reduce the number of undesirable actions in a given period, which gives monitoring time to catch them.
A leaked key can be used without limit. OWASP’s unbounded-consumption entry recommends rate limits, user quotas, timeouts and throttling.
04
Choosing limits
Start from observed use: log a normal week, then set the limit comfortably above the busiest honest user.
Limit per person or per key, not only per server, so that one user cannot use another’s share.
Cap tokens as well as requests, because one huge document costs far more than a short question.
Give agents and scripts lower limits than people, plus a cap on steps per request.
Tell staff what happens at the limit: a clear message and a wait time, not a silent failure.
Review the limits after a change of hardware or model.
05
Where ANVE stands
ANVE’s agent loop allows up to seven tool steps per request, which caps the work behind one request, and every model call is logged with the user, the app, the model and whether it ran locally or in the cloud, so heavy use is visible. Limits per key depend on the governed API, which gives each user or agent a key and is not on ANVE yet. How limits per user or key will work is a question to settle in writing before a paid pilot.
06
Questions people ask
What does error 429 mean?
Too Many Requests: you have sent more requests or tokens than the limit allows in the current period. Wait for the time in the Retry-After header, if there is one, and then retry.
Is a rate limit the same as a quota?
Close, but not the same. A rate limit controls speed over a short period, such as a minute, while a quota or spend cap controls the total over a longer period, such as a month. Anthropic’s documentation treats the two as separate controls.
Do local models have rate limits?
Only if the server, or a gateway in front of it, is set up to enforce them. A bare model runtime may accept requests until the hardware is saturated, so an office should add limits.
Can I raise my limit with a cloud provider?
Often, yes. OpenAI describes usage tiers that upgrade automatically as total credit purchases grow, and Anthropic places organisations in tiers based on usage history and account standing.
07
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
