For: operators of sites whose search or similar feature calls an external API. This article generalises something that actually happened on a site this site's operator runs. It names no site, no external service, and gives no exact limits or dates. It does not cover attack procedures.
What happened
On a site this site's operator runs, the search feature called an external API on every search. Within a few days search traffic rose to about 20 times normal. The external API had a concurrency cap, and the vast majority of searches (over 90%) returned 503.
The first assumption was that a per-IP limit (N per minute) would contain it. That was wrong.
Search traffic rose about 20 times in a few days
The external API's concurrency cap was hit and nearly all searches returned 503. At this point it just looked like more errors.
First assumption: a per-IP limit will contain it (wrong)
Counting per minute per IP would catch a few IPs making a very large number of calls. This was not that kind of traffic.
What measuring showed
One day's roughly 60,000 searches came from roughly 60,000 distinct IPs, about one request per IP. The pattern looked like a residential-proxy network. The user agent was an ordinary browser, the Referer was the site itself, and the queries were real product names. Price collection was the likely purpose.
The external API's daily quota ran out early in the day
The bot's searches used up the quota early, and normal visitors could not search for most of the day.
Why it did not look like the real problem
The quota was protected, so the external API looked healthy and the site saw only more errors. Protecting the quota and letting users use the feature turned out to be different things.
What was expected: a few IPs making many calls
Hits the per-IP limit → stopped ✓
What happened: about 60,000 IPs, one request each
No IP reaches the limit → only the external API's quota drains
= while the counting unit is the IP, this shape is invisible
A protected upstream quota does not mean users can use the feature
Whatever protects an external API's quota or concurrency exists to keep that API from breaking. While it works, the upstream shows no errors. The cost lands on the side that lost the quota: your own visitors.
That the protection was working is what made the problem hard to see. What to watch is the success rate real visitors get when they search.
How it was stopped
The design that did not stop it
Any search request from anyone calls the external API. The only limit is a per-IP count. Distributed traffic keeps each IP's count low, so it never triggers the limit.
The design that did
The external API is called only for searches carrying a signed token (cookie) issued by the site's own page. Without a token, return a light page or 403. On top of that, hourly caps globally and per token.
- Signed token
- A token issued by the site's own page became the condition for calling the external API. Without it, a light page or 403
- Protecting aggregates
- Untokened searches were excluded from the popular-searches ranking, which the bot's queries had inflated
- Hourly caps
- Both a global cap and a per-token cap
Right after release, 503s went to zero. About 100,000 untokened requests a day now get a 403, and normal searches go through.
Lessons
- A per-IP limit does not work against a distributed bot. One request per IP stops nothing, however low the limit.
- A protected upstream quota is not the same as users being able to use the feature. Protection can hide the problem.
- Aggregates such as rankings get polluted. Along with protecting the external API, restrict what feeds them.
What to do on your side
The order below is this site's own advice, not a statement from the cited sources.
Check a site-issued token before any expensive step
Before a step that consumes cost or quota, such as an external API call, check for a signed token issued by your own page. Give the token an expiry and verify it on the server. If it is missing, do not proceed; return a light page or 403.
Keep untokened traffic out of aggregates
Build popular searches, query statistics and recommendations only from tokened searches, so a bot's queries do not leak into what your site displays.
Set global and per-token caps
Use an hourly global cap and a per-token cap rather than a per-IP count. The global cap is the last backstop that stops the external API's quota being used up.
Add result caching and queueing
Serve repeated queries from a cache without calling the external API again, and queue requests when busy. When one measure is bypassed, the quota is less likely to vanish at once.
Watch the success rate visitors see
Record the share of real visitors whose searches succeed, not only the external API's error rate, and alert when it drops. This is the "able to notice something is wrong" state from the organisational security baseline.
A token is not authentication; it raises the attacker's cost
A token is not authentication. What a token gives you is a stop to traffic that simply hammers the search URL; it is not a complete defence.
That is why it is combined with caching, queueing and per-token caps. In this site's view, per-token caps do not help if a bot fetches a fresh token every time, so always pair them with a global cap and caching. This site's view, as in rate limiting and abuse control, is to make abuse not worth the cost rather than to drive its success rate to zero.
Sources (primary)
- OWASP API Security Top 10 2023, "API4:2023 Unrestricted Resource Consumption" — api-security.owasp.org (limiting how often a client can interact with the API in a timeframe; configuring spending limits for external services)
- OWASP Automated Threats, "OAT-011 Scraping" — owasp.github.io (defines scraping as collecting content or data for use elsewhere, such as price-fixing; data needing no authentication can be targeted)
Read next
- Design: Rate limiting and abuse control: designing limits that stop brute force and runaway costs
- Prerequisite: X-Forwarded-For spoofing and trusted proxies (the basis for counting by IP)
- Operations: The organisational security baseline (how to notice something is wrong)