...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
On this page.
| Table of Contents |
|---|
Introduction
The benefits of using an API rate limiter :
Prevent resource starvation caused by Denial of Service (DoS) attack, either intentional or unintentional, by blocking the excess calls.
Reduce cost by limiting servers and allocating more resources to high priority APIs. Important when we use paid third party APIs.
Prevent servers from being overloaded by filtering out excess requests.
Part 1 - Understand the problem and establish design scope
Type of rate limiter ? client-side or server-side ? in a middleware ? | Server-Side or Middleware |
|---|---|
Rules of throttle ? | Different sets of throttle rules (IP, user ID, etc.) |
Scale of the system ? Small, Medium, big company | Big company |
In a distributed environment ? | Yes. |
Implemented as a separate service or in application code ? | No matter. |
Inform user who are throttled? | Yes. |
Additional requirements to consider
limit excessive requests
low latency - not slow down http response time
limit the use of memory as possible
can be shared across servers and processes
exception handling - show clear exceptions to users
high fault tolerance - if rate limiter goes offline, it does not affect the system
Part 2 - High level design
Client-side implementation is not the right option because client is not a reliable place : client requests can be forged by malicious actors.
Where to put the rate limiter ?
...
Choice of implementation is a middleware : API Gateway is a fully managed service supporting rate limiting, SSL termination, authentication, IP whitelisting, etc.
How To choose the right implementation ?
Middleware vs Server-Side (API Servers) ? Use a guideline :
Evaluate the current technology stack;
Identify the rate limiting algorithm - on server-side, we have full control of algo.
If Microservice Architecture and including an API gateway (performing tasks like authentication, IP whitelisting, etc.)
...
Regarding the algo, we need a counter to keep track of how many requests are sent from the same user, IP Address, etc.
Part 3 - Design deep dive
To deep dive, we need to manage rules and other components (MQ, Cache, …). See example rules in the page - References & Glossary for rate limiter.
...
If the request is not rate limited, it is forwarded to API servers.
If there are too many request according to the rules (from cache), the rate limiter return 429 error code to the user and the request is dropped or forwarded to a queue.
Part 4 - Pros & Cons
Pros
Help to control the resources services consume, impose limits on the rate;
Appropriate for large-scale repetitive automated tasks (batch processing,…)
Cons
Demanding to implement in a distributed environment;
Synchronize the rate limiters in a distributed environment.