HackerNews中文版

我最近一直在研究爬虫架构。我发现最有用的两个资料是博客文章“2025年，在短短24小时内爬取十亿网页”和Mercator论文（“Mercator：一个可扩展的、可扩展的Web爬虫”）。这两篇文章，以及我遇到的其他大多数资料，都侧重于爬取广阔的开放网络，而不是针对特定的一组域名。对于产品价格来说，我们关注的是后者。例如，Mercator指出了DNS解析是一个主要的瓶颈，但当你只访问几百个域名时，这实际上并不是一个问题。另一个差距是，这两者都假定是静态HTML。对于我们的用例，我们需要一个无头浏览器，并且我们还需要处理Cloudflare和类似的防机器人系统。特别是对于产品价格，许多网站发布价格订阅，这简化了事情，但也有很多网站没有这样做，并且要获得良好的覆盖范围仍然需要抓取。我们目前的系统每天大约处理5亿个页面，我们正在寻求提高其性能。这里有人在这个领域有经验吗，或者知道关于使用无头浏览器扩展定向（而不是广泛）爬虫的文章/博客文章吗？欢迎提供任何建议。

查看原文

I've been reading up on crawler architecture. The two most useful sources I've found are the blog post "Crawling a billion web pages in just over 24 hours, in 2025" and the Mercator paper ("Mercator: A Scalable, Extensible Web Crawler").<p>Both of these, and most other material I've come across, focus on crawling the broad open web rather than a targeted set of domains. For product prices it's the latter. Mercator calls out DNS resolution as a major bottleneck, for example, but when you're only hitting a few hundred domains that isn't really a concern.<p>The other gap is that both assume static HTML. For our use case we need a headless browser, and we also have to deal with Cloudflare and similar anti-bot systems.<p>For product prices specifically, a lot of sites publish price feeds which simplifies things, but plenty don't, and getting good coverage still requires scraping. Our current system does about 500M pages/day and we're looking to improve its performance.<p>Does anyone here have experience in this space, or know of articles/blog posts on scaling targeted (rather than broad) crawlers with headless browsers? Any pointers appreciated.

Ask HN: 如何将定向