HN 提问：是谁在运行滥用的爬虫？
8 分•作者: jonatron•3 个月前
就我个人而言，我曾不得不将应用程序容量增加两倍，数据库容量增加一倍，仅仅是为了一个爬虫。我记得 Cloudflare 曾经发现 Perplexity 是“坏人”之一（https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives），但还有其他人发现谁还在运行大规模且行为不端的爬虫吗？
查看原文
Personally I once had to triple app capacity and double database capacity just for just one scraper. I remember Cloudflare once figured out that Perplexity was one of the bad guys ( https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/ ) but has anyone else figured out who else is running large scale and poorly behaved scrapers?