Rate Limiting

Rule: without a limit, one client's mistake becomes everyone's outage. A rate limit is not hostility toward consumers — it is the contract that lets you keep serving all of them.

Part of the API set: contracts & versioning · resource modeling · list endpoints · async operations · webhook provider · rate limiting (this file)

The anti-pattern
Why development hides it
Capacity is finite in more ways than CPU
The algorithms
What to key on
Not all endpoints cost the same
Responding to a limited client
Distributed counting
Rate limiting is not load shedding
Review checklist

The anti-pattern

@app.get("/orders")            # no limit of any kind
def list_orders(): ...

The client everyone hoped for arrives, integrates enthusiastically, and polls this endpoint in a tight loop from twenty workers. Their bug consumes your entire database capacity, and every other customer is now down — including the ones who did nothing wrong.

"Just pay and resources are unlimited."

That belief is what an unlimited API teaches consumers. There is no such thing: capacity is finite, and the only question is whether it is allocated deliberately or taken by whoever loops fastest.

Why development hides it

One client, generated by you, calling at human pace. There is no second tenant to be starved, no retry loop, no misconfigured cron, and no scraper.

The failure is a fairness problem, and fairness cannot exist with one consumer. It appears the first time a consumer's behaviour is not what you imagined — which is always, eventually.

Capacity is finite in more ways than CPU

Rate limiting protects several resources, and only the first is obvious:

Compute and database — the direct cost.
Cost — per-request spend on third-party APIs, LLM tokens, egress, storage. Here a runaway client produces a bill rather than an outage, which is often worse because nothing alerts.
Downstream quota — your provider limits you. One tenant burning your shared quota takes the capability away from every tenant.
Operational capacity — humans. Every incident consumes attention that is not elastic.

So limits belong on anything scarce, not only on anything slow.

The algorithms

Algorithm	Behaviour	Use when
Fixed window	Count per calendar window	Simplest; allows a 2× burst across a boundary
Sliding window	Count over a rolling interval	Smooths the boundary burst; more state
Token bucket	Tokens refill at a rate; a request spends one	Best default — allows bursts up to bucket size, sustained rate bounded
Leaky bucket	Queue drains at a fixed rate	Smooth output to a fragile downstream
Concurrency limit	N in flight, not N per second	Long or variable requests, where duration matters more than count

Two notes worth carrying. Fixed windows have a real boundary flaw: a client can send its full allowance at 10:59:59 and again at 11:00:00, delivering 2× the limit in a moment. And a concurrency limit is often what you actually want — for expensive endpoints, three simultaneous heavy reports matters more than requests-per-minute, and it maps directly onto pool capacity (see connection-pools-and-latency).

Token bucket is the right default because bursts are legitimate: a client syncing after downtime should be allowed to catch up, then be held to the sustained rate.

What to key on

The key determines who is isolated from whom, and getting it wrong makes the limit useless:

API key / tenant — the correct default for a business API. Each customer gets their own budget.
User id — for user-facing APIs where one tenant has many users.
IP — only for unauthenticated traffic. It is a poor identity: NAT puts thousands of users behind one address, and attackers rotate addresses freely.
Endpoint + key — so a cheap endpoint's traffic cannot exhaust the budget for an expensive one.

Tiered limits are the honest model for a multi-tenant API: free, standard, and enterprise get different budgets, stated in the contract. That converts "we throttled you" from an argument into a plan.

And protect the pre-authentication surface separately. Login, signup, password reset, and token endpoints need stricter limits keyed on IP and account, because they are abuse targets and the attacker has no API key to limit.

Not all endpoints cost the same

A flat requests-per-minute limit treats a keyed lookup and a five-year report as equivalent. They are not.

Cost-weighted limiting charges a request what it costs — a lookup costs 1, a filtered list costs 10, a report costs 500. This is how you avoid the two failure modes of a flat limit: either it is loose enough for the expensive endpoint to hurt you, or tight enough to make cheap calls unnecessarily painful.

Where you can, put the expensive shapes behind an async operation instead and rate-limit submissions. That gives a bounded queue rather than a rejected request, which is a better answer for both sides.

Responding to a limited client

Rejecting is not enough; the client needs to know what to do:

429 Too Many Requests
Retry-After: 30
RateLimit-Limit: 1000
RateLimit-Remaining: 0
RateLimit-Reset: 1756640430

{ "error": { "code": "rate_limit_exceeded",
             "message": "Limit of 1000 requests/hour exceeded.",
             "retry_after": 30 } }

429, not 503. 503 says I am broken; 429 says you are over budget. Consumers, and their circuit breakers, treat those differently — and a 503 will trip a circuit breaker that should not be tripping.
Always send Retry-After. Without it, a well-meaning client retries immediately and makes it worse.
Expose remaining budget on successful responses, so clients can self-throttle before being rejected. This is the difference between a limit clients cooperate with and one they collide with.
Never let a 429 be silent. If a client is being limited persistently, tell someone — the same reasoning as failing webhook deliveries.

Distributed counting

With N instances, a per-instance counter gives N× the intended limit. Options:

Shared counter in Redis — accurate, and adds a network hop plus a dependency to every request. Use atomic increments with expiry; a GET-then-SET is a race, per transaction-isolation.
Local counters with limit/N — no dependency, and unfair under uneven load balancing.
Local with periodic sync — approximate, cheap, and usually good enough.
At the edge — API gateway, CDN, or service mesh. Rejects before consuming your capacity at all, which is the strongest form.

The essential design decision: fail open or fail closed when the limiter's store is unavailable. Failing closed turns a Redis blip into a total outage; failing open leaves you unprotected during exactly the incident where protection matters. For most APIs, fail open with a conservative local fallback and alert loudly.

Rate limiting is not load shedding

Two related tools, often conflated:

Rate limiting is fairness, applied per client, mostly independent of current load. A tenant over budget is rejected even on an idle system.
Load shedding is survival, applied when the system is overloaded, regardless of whose budget remains. It rejects work you cannot complete in time.

You need both. Rate limits alone will not save you from legitimate aggregate traffic; shedding alone lets one tenant starve the others. And shed before the queue is deep — work whose client has already timed out is capacity spent on nothing, which is the same waste as caching's stampede.

Review checklist

Synthesized from TechCraft's API Design Patterns series, part P12 (collection). The framing of rate limiting as a production contract, the "pay and resources are unlimited" misconception, and operational capacity as a limited resource are TechCraft's; the algorithm comparison, response design, and checklist are this repository's.

Contents​

The anti-pattern​

Why development hides it​

Capacity is finite in more ways than CPU​

The algorithms​

What to key on​

Not all endpoints cost the same​

Responding to a limited client​

Distributed counting​

Rate limiting is not load shedding​

Review checklist​

Contents