HackerNews中文版HackerNews CN
GitHubTwitter

Show HN: WhiskeySour – 速度提升 10 倍的 BeautifulSoup 替代品

3 分•作者: ayas_behera•3 个月前
问题<p>我使用BeautifulSoup已经有一段时间了。它在Python抓取中以易用性著称，但当处理大规模数据集时，它几乎总是成为性能瓶颈。<p>在Python中解析复杂或庞大的HTML树，通常会面临高内存分配成本以及树遍历过程中Python对象模型的开销。在我的生产抓取工作负载中，解析器消耗的CPU周期甚至超过了网络I/O。Lxml速度很快，但在处理大型文档时也会占用大量内存，并且可能在处理格式不规范的HTML时出现问题。<p>解决方案<p>我希望保持BS4的API兼容性，这让它非常出色，但同时消除减慢高容量管道速度的开销。它还使用了html5ever，这就是我构建WhiskeySour的原因。是的……我*用直觉写了整个东西*。<p>WhiskeySour是一个即插即用的替代品。你应该能够用“from whiskeysour import WhiskeySour”替换“bs4 import BeautifulSoup”，并立即看到速度提升。以前需要30多分钟的工作流程现在可能只需要不到5分钟。<p>我在这里分享了该库的详细架构： <a href="https:&#x2F;&#x2F;the-pro.github.io&#x2F;whiskeySour&#x2F;architecture&#x2F;" rel="nofollow">https:&#x2F;&#x2F;the-pro.github.io&#x2F;whiskeySour&#x2F;architecture&#x2F;</a><p>这是与bs4和html.parser的基准测试报告：<a href="https:&#x2F;&#x2F;the-pro.github.io&#x2F;whiskeySour&#x2F;bench-report&#x2F;" rel="nofollow">https:&#x2F;&#x2F;the-pro.github.io&#x2F;whiskeySour&#x2F;bench-report&#x2F;</a><p>这是该仓库的链接：<a href="https:&#x2F;&#x2F;github.com&#x2F;the-pro&#x2F;WhiskeySour" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;the-pro&#x2F;WhiskeySour</a><p>我分享这个的原因<p>我正在寻求社区在两个方面的反馈：<p>1. 边缘情况：如果你有BS4能很好处理的特别混乱或格式不规范的HTML，我很想知道WhiskeySour是否会遇到任何回归问题。<p>2. 基准测试：如果你正在运行高容量解析器，我很乐意你能在你自己的数据集上运行一个测试并分享结果。
查看原文
The Problem<p>I’ve been using BeautifulSoup for sometime. It’s the standard for ease-of-use in Python scraping, but it almost always becomes the performance bottleneck when processing large-scale datasets.<p>Parsing complex or massive HTML trees in Python typically suffers from high memory allocation costs and the overhead of the Python object model during tree traversal. In my production scraping workloads, the parser was consuming more CPU cycles than the network I&#x2F;O. Lxml is fast but again uses up a lot of memory when processing large documents and has can cause trouble with malformed HTML.<p>The Solution<p>I wanted to keep the API compatibility that makes BS4 great, but eliminates the overhead that slows down high-volume pipelines. It also uses html5ever which That’s why I built WhiskeySour. And yes… I *vibe coded the whole thing*.<p>WhiskeySour is a drop-in replacement. You should be able to swap from &quot;bs4 import BeautifulSoup&quot; with &quot;from whiskeysour import WhiskeySour&quot; and see immediate speedups. Your workflows that used to take more than 30 mins might take less than 5 mins now.<p>I have shared the detailed architecture of the library here: <a href="https:&#x2F;&#x2F;the-pro.github.io&#x2F;whiskeySour&#x2F;architecture&#x2F;" rel="nofollow">https:&#x2F;&#x2F;the-pro.github.io&#x2F;whiskeySour&#x2F;architecture&#x2F;</a><p>Here is the benchmark report against bs4 with html.parser: <a href="https:&#x2F;&#x2F;the-pro.github.io&#x2F;whiskeySour&#x2F;bench-report&#x2F;" rel="nofollow">https:&#x2F;&#x2F;the-pro.github.io&#x2F;whiskeySour&#x2F;bench-report&#x2F;</a><p>Here is the link to the repo: <a href="https:&#x2F;&#x2F;github.com&#x2F;the-pro&#x2F;WhiskeySour" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;the-pro&#x2F;WhiskeySour</a><p>Why I’m sharing this<p>I’m looking for feedback from the community on two fronts:<p>1. Edge cases: If you have particularly messy or malformed HTML that BS4 handles well, I’d love to know if WhiskeySour encounters any regressions.<p>2. Benchmarks: If you are running high-volume parsers, I’d appreciate it if you could run a test on your own datasets and share the results.