> I'm a bit confused. Is anyone's website currently being DDOS'd by scrapers for LLMs or is this a hypothetical?
It is very real and the reason why Anubis has been created in the first place. It is not plain hostility towards LLMs, it is *first and foremost* a DDoS protection against their scrapers.
I've set up a few honeypot servers. Right now OpenAI alone accounts for 4 hours of compute for one of the honeypots in a span of 24 hours. It's not hypothetical.
25k+ hits/minute here. And that's just the scrapers that doesn't just identify themselves as a browsers.
Not sure why you believe massive repeated scraping isn't a problem. It's not like there is just one single actor out there, and ignoring robits.txt seems to be the norm nowadays.
> I'm a bit confused. Is anyone's website currently being DDOS'd by scrapers for LLMs or is this a hypothetical?
Yes, there are sites being DDoSed by scrapers for LLMs.
> If you can't handle the traffic of one request per page per week or month, I think there are bigger problems to solve.
This isn't about one request per week or per month. There were reports from many sites that they're being hit by scrapers that request from many different IP addresses, one request each.
If you can't handle the traffic of one request per page per week or month, I think there are bigger problems to solve.