rate limiting

Rate limiting is a security and resource management technique that restricts the number of requests a user, IP address, or API key can make to an AI service within a given time window. It is essential for preventing abuse, managing costs, and maintaining service availability.

Why Rate Limiting Matters for AI

• Prevent Model Extraction: Attackers query the model thousands of times to build a surrogate copy. Rate limits make this impractically slow.
• Cost Control: LLM inference is expensive — unrestricted access can lead to massive, unexpected bills from API abuse or automated scripts.
• Denial of Service: Without limits, a single user can monopolize GPU resources, degrading service for everyone.
• Adversarial Probing: Rate limits slow down automated jailbreaking attempts, red-teaming scripts, and prompt injection exploration.

Common Rate Limiting Strategies

• Fixed Window: Allow N requests per time window (e.g., 100 requests per minute). Simple but allows bursts at window boundaries.
• Sliding Window: Smooth the window to prevent boundary bursts. More complex but fairer.
• Token Bucket: Tokens accumulate over time; each request costs a token. Allows short bursts while enforcing average rate.
• Token-Based Limits: For LLMs, limit by tokens per minute (TPM) rather than requests, since a long prompt consumes far more resources than a short one.
• Tiered Limits: Different limits for different plan levels (free tier: 10 RPM, paid: 1000 RPM, enterprise: custom).

Implementation Considerations

• Identification: Rate limit by API key, user account, IP address, or some combination.
• Response Codes: Return HTTP 429 (Too Many Requests) with a `Retry-After` header indicating when the client can try again.
• Graceful Degradation: Consider returning cached or lower-quality responses instead of hard rejections.
• Monitoring: Track rate limit hits to identify abuse patterns and adjust limits.

Rate limiting is a standard practice across all LLM API providers (OpenAI, Anthropic, Google) and is critical for both security and business sustainability.