Rate Limiting AI APIs: Preventing Abuse Without Blocking Legitimate Users

Erwin Hermanto

5 min read

·

1 day ago

--

Listen

Share

Three weeks ago, one of our internal tools hit the OpenAI rate limit mid-demo. In front of a client. The culprit wasn’t abuse — it was a single power user running a batch job that happened to coincide with a traffic spike from everyone else. We fixed the demo embarrassment that afternoon with a blunt fix: a flat request cap per API key. Two days later, support tickets started coming in from our heaviest (and most valuable) users, all saying some version of “why am I getting throttled, I’m not doing anything weird.”

Press enter or click to view image in full size

That’s when it became clear a flat request limit was the wrong tool for the job. Here’s what we replaced it with, and why.

Why request-count limits fail for AI APIs

Most rate limiters were built for a world where every request costs roughly the same — a GET to /users/42 and a GET to /users/99 are interchangeable in terms of server load. LLM calls aren't like that. A one-line completion and a 4,000-token RAG query hitting the same endpoint can differ in cost by 50x. If you cap requests-per-minute, you're effectively letting someone drain your budget with a handful of huge prompts while punishing someone else running lots of small, cheap ones.

The fix is to stop counting requests and start counting tokens.

Token-based limiting

The idea is simple: instead of “100 requests per minute,” you track “50,000 tokens per minute,” counting both the prompt and the expected completion. Here’s a minimal implementation in Go using a sliding window backed by Redis:

package main


import (
	"context"
	"fmt"
	"time"

	"github.com/redis/go-redis/v9"
)

type TokenLimiter struct {
	rdb        *redis.Client
	windowSize time.Duration
}

func NewTokenLimiter(rdb *redis.Client) *TokenLimiter {
	return &TokenLimiter{rdb: rdb, windowSize: time.Minute}
}

// Allow checks whether `tokens` more usage fits within the user's
// budget for the current sliding window. It returns the remaining
// budget so callers can surface it in response headers.
func (tl *TokenLimiter) Allow(ctx context.Context, userID string, tokens, budget int) (bool, int, error) {
	key := fmt.Sprintf("ratelimit:tokens:%s", userID)
	now := time.Now().UnixMilli()
	windowStart := now - tl.windowSize.Milliseconds()

	pipe := tl.rdb.TxPipeline()
	pipe.ZRemRangeByScore(ctx, key, "0", fmt.Sprintf("%d", windowStart))
	sumCmd := pipe.ZRangeWithScores(ctx, key, 0, -1)
	_, err := pipe.Exec(ctx)
	if err != nil {
		return false, 0, err
	}

	used := 0
	for _, z := range sumCmd.Val() {
		used += int(z.Score)
	}

	if used+tokens > budget {
		return false, budget - used, nil
	}

	member := fmt.Sprintf("%d:%d", now, tokens)
	tl.rdb.ZAdd(ctx, key, redis.Z{Score: float64(tokens), Member: member})
	tl.rdb.Expire(ctx, key, tl.windowSize*2)

	return true, budget - used - tokens, nil
}

The sliding window (rather than a fixed per-minute bucket) matters more than it looks. A fixed window lets a user burn their entire budget at 59 seconds and again at 0 seconds — effectively doubling their real rate at the boundary. The sorted-set approach above avoids that by expiring entries continuously.

One detail we got wrong the first time: don’t wait until you have the actual completion token count to charge the user. Estimate the completion length upfront (we use max_tokens as a worst-case estimate, then reconcile after the response), or someone can queue up requests faster than you can deduct their usage.

User tiers

Token budgets alone don’t solve the “which users matter more” problem. A free-tier user and an enterprise customer shouldn’t share the same ceiling, and a single global limit either starves your best customers or leaves the door open for free accounts to be farmed. We settled on a simple tier table that gets looked up per request:

from dataclasses import dataclass
from enum import Enum

class Tier(Enum):
    FREE = "free"
    PRO = "pro"
    ENTERPRISE = "enterprise"

@dataclass
class TierLimits:
    tokens_per_minute: int
    burst_allowance: float  # multiplier for short bursts
    priority: int           # lower = served first under load

TIER_CONFIG = {
    Tier.FREE:       TierLimits(tokens_per_minute=10_000,  burst_allowance=1.0, priority=3),
    Tier.PRO:        TierLimits(tokens_per_minute=100_000, burst_allowance=1.5, priority=2),
    Tier.ENTERPRISE: TierLimits(tokens_per_minute=1_000_000, burst_allowance=2.0, priority=1),
}

def resolve_budget(tier: Tier, requested_tokens: int) -> int:
    cfg = TIER_CONFIG[tier]
    return int(cfg.tokens_per_minute * cfg.burst_allowance)

The burst_allowance field earns its keep more than I expected. Legitimate usage is bursty — someone pastes a long document and fires off five follow-up questions in ten seconds, then goes quiet for a minute. A limiter with zero burst tolerance treats that pattern identically to a script hammering your endpoint. Giving paid tiers more burst headroom than free tiers also turned out to be a decent abuse signal on its own: scripted abuse tends to sustain load rather than burst and back off.

Graceful degradation instead of hard 429s

The part that actually fixed our support tickets wasn’t the token math — it was what we did before returning a 429. A hard rejection is the right move against a clear attacker, but for a legitimate user who’s simply closer to their limit than usual, a flat-out block feels broken. We now degrade in stages:

1. Queue and delay — if a user is near their limit, hold the request briefly (200–800ms, randomized) instead of rejecting it outright.
2. Downgrade the model — route to a cheaper, faster model for non-critical requests once a user crosses 80% of budget, rather than cutting them off entirely.
3. Reject with a clear retry-after — only once budget is fully exhausted, and always with a Retry-After header so clients can back off correctly instead of hammering the endpoint again immediately.

func handleRequest(limiter *TokenLimiter, userID string, tier Tier, tokens int) (int, map[string]string) {
	budget := resolve_budget(tier, tokens)
	allowed, remaining, _ := limiter.Allow(context.Background(), userID, tokens, budget)

	headers := map[string]string{
		"X-RateLimit-Remaining": fmt.Sprintf("%d", remaining),
	}

	if allowed {
		return 200, headers
	}

	usageRatio := float64(budget-remaining) / float64(budget)
	if usageRatio < 0.95 {
		headers["X-RateLimit-Degraded"] = "model-downgraded"
		return 200, headers // proceed, but caller routes to a cheaper model
	}

	headers["Retry-After"] = "30"
	return 429, headers
}

This single change — queueing and downgrading before rejecting — cut our 429 rate by more than half without changing the actual token ceilings at all. Most of what users were experiencing as “being blocked” was really just a brief spike that a 300ms delay would have smoothed over.

What abuse actually looked like, once we could see it

After switching to token-based limits, the usage graph made the real abusers obvious in a way request counts never did — a handful of accounts sustaining near-constant usage at the very top of their budget, hour after hour, with none of the natural burst-and-idle rhythm everyone else showed.

Tokens/min (sampled over 6 hours)
Legit user   ▇▇░░░▇▇▇░░░░▇▇░░░░▇▇▇░░░░▇▇░░░▇▇▇░░
Abusive acct ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇
             0h    1h    2h    3h    4h    5h    6h

That shape — flat-lined at the ceiling — became our actual abuse signal, not the raw volume. We now flag accounts sustaining >90% of budget for more than 15 consecutive minutes for review, separate from the rate limiter itself.

What I’d tell myself three weeks ago

Don’t start with request counts for an LLM API — you’ll just end up migrating to token-based limits later anyway, and in the meantime you’ll throttle the wrong people. Build tiers in from day one even if you only have two of them. And treat the 429 as a last resort, not a first response — most of what looks like abuse is just load you haven’t smoothed out yet.