Hacker Timesnew | past | comments | ask | show | jobs | submitlogin

Crawler, AI or not, cannot afford to pay per visit. The entire model of crawling works because the incremental cost of each crawl is so low. Even fractions of a penny would be prohibitive.


As it should be. Bots are still user agents much like browsers


If you're paying per token for AI, you can also pay a smaller amount to use the Web.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: