Hacker Timesnew | past | comments | ask | show | jobs | submitlogin

Semrush misbehaved so badly for such a long time with various levels of incompetence that I have special notes in my robots.txt files going back at least 8 years and eventually got someone in legal to pay attention. I see zero value to me or any potential users of mine in letting an 'SEO' firm lumpishly trample all over my site barging out real visitors. And some of Semrush's competitors were just as bad.

The current round of AI nonsense also very poor. Again had to send legal notes to investor relations and PR depts in at least one well-known case, as well as all the technical measures, to restore some sort of decorum.



My experience with them has been the same. On one of my employer's websites, the top three bots were Google, Bytedance and Semrush. It is a small website for a niche audience, not even in English, and changes very infrequently (like once or twice a quarter). That did not stop these three bots from hammering the site, every second


We were getting hammered on nofollow links. Someone made a change to stop putting the nofollow in and using JavaScript to traverse on interaction, but that person left sometime after the toggle went in and we discovered it had never been turned on much later, when we were rearranging our ingress load balancer and needed to tweak how we handled bot throttling. That was at least 10% of the requests during peak traffic and extended the duration of the crawls by a lot. So dumb.




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: