Hacker Timesnew | past | comments | ask | show | jobs | submitlogin

Are you indicating that everything in the IA was added via a person clicking on “archive this page”?

I don’t think that is correct. A lot of it was added via automated methods?



Not necessarily via a button but all of the pages were submitted to them by 3rd parties. So, they don't seem to crawl to discover new URIs eagerly, see https://hackertimes.com/item?id=29643506

For example, I host my own ArchiveBox at home (you get fulltext search as a bonus) and it is configured to submit every URL I save to IA: https://imgur.com/a/Yhnxo1W IA considers that to be manual submission not subject to robots.txt rules.

https://archivebox.io




Consider applying for YC's Summer 2026 batch! Applications are open till May 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: