That peak is around 400 req/s, right? I would expect the usual solution to be to just rate limit each user to some reasonable limit, and then 429 if theyre over.
I feel like 400 req/s should be absorbed, especially since my 5€/month VPS can handle about 2x that, sustained. I might be missing something here, but that just doesnt seem like a peak big enough to warrant degrading user experience.
Sadly I'm often categorized by websites as "probably a bot haha get fucked", and its lost sites hundreds or thousands of $$$ worth of revenue over the years, just from me alone.
The service is unauthenticated so there aren't any "users" to rate limit on. You could try to guess based on IP but it's trivial to get access to massive amounts of IPs (there are proxy/scraping services that do this for you)
That's 400 req/s on a distributed cloud environment. 400 requests per second on a $5 VPS is practically free, 400 requests per second on Azure Lambda (or whatever they call it) can bankrupt you.
That $5 VPS will also be null routed within days with the kinds of DDOS traffic sites like HIBP get.
Where is your 5 euro/month VPS hosted, and does it use HTTPS?
I did a non-scientific test last year, in which I ran a basic Apache instance that returned a 204 response (No Content) on $5 instance and HTTPS alone made it drop around 500 requests/second (once again, no other processing happening other than returning headers for a 204 response)[1]. My understanding at the time is that in general on lower priced virtualization options you don't get VMs which provide access to CPU instructions that speed up encryption/decryption.
any reasonably multithreaded http server with nginx infront should easily serve 500+ requests per second, on something like a 5€/month hetzner VPS (x86, you get 2x the cores if you choose arm but thats a new offer so ill pretend it doesnt exist).
Only then will it cause significant delay. If your service drops connections at 500+ requests per second, something is off - You can configure quite the large accept backlog, like a few thousand, and you should only get an additional few ms delay per additional request.
If you're bottlenecked anywhere, it shouldnt drop connections until its like very very bad.
Thats assuming you need to access a DB or do other checks -- static content servers should start struggling on a 2-4 core system at around 10k requests per second, but that doesn't really matter for this.
Can you share your Hetzner VPS address, do you do your own TLS termination and can I test your endpoint with Apache benchmark? I ask because theoretical numbers, in my opinion, are thrown around to often, and when I hear thousands of responses/second on budget VPS's I want to see real life applications that actually handle that load. In my experience the difference between Apache and nginx is negligible, and if your service that runs on a $5 (or euro) a month VPS handles thousands of requests/second I'm going to be more than impressed when I see those numbers myself through a test.
I was actually going to test that myself, for you, with a $5 Linode. However, I spent about an hour fighting with siege/drill/hey and finally gave up. siege crashes and locks up all the time, as well as sometimes getting stuck for a minute or two between requests. hey reported 2 req/second for https, which I don't find believable either...
Sure but there's still TBs of data to check? Ideally it uses a data structure like a tree or hash that's efficient for searching through that amount of data (which might just be an index in a relational DB).
That's like saying every S3 request searches though exabytes of data. No, each request only access the data it needs. Yes, the dataset is large but "each request" only reads a few megabytes at most.
I feel like 400 req/s should be absorbed, especially since my 5€/month VPS can handle about 2x that, sustained. I might be missing something here, but that just doesnt seem like a peak big enough to warrant degrading user experience.
Sadly I'm often categorized by websites as "probably a bot haha get fucked", and its lost sites hundreds or thousands of $$$ worth of revenue over the years, just from me alone.