问 HN：挖掘科学论文
1 分•作者: davidbjaffe•3 个月前
人们在使用 LLM 从科学论文中挖掘信息方面的经验是什么？
我自己的经验：我首先尝试从 3730 篇临床试验论文中提取抗药抗体 (ADA) 发生率，这些论文都收录在 PubMed 中。我从 PDF 文件开始。Claude Opus 4.7 使用我们编写的规则文档分析了每个 PDF。运行所有论文大约花了一周时间，因为我不断达到会话限制；总费用约为 25 美元（美元）。我们从 909 篇论文中获得了实际发生率。其余的论文大多是未提供发生率或不符合我们标准的情况，包括仅同时使用一种药物。
我阅读了其中 30 篇论文，并重新阅读了那些我从 Claude 处得到不同答案的论文，得出的结论是它出错了一次，而我出错了三次。
所以这有效，但不是完全方便：会话限制意味着我无法启动它然后离开。或者我不知道如何实现这种能力。此外，我很好奇本地模型会如何表现。
为此，我在我的 Mac M5 Max（128 GB 内存）上尝试了 llama 3.3 70B。我使用了 Ollama、Q4_K_M、128 k 上下文，在 pdftotext -layout 之后输入了大约 80 k 个 token。
一篇论文花了 18 分钟；该模型无法确定 ADA 发生率，而该发生率在论文中是明确存在的。一篇论文不是一个合适的基准，但进行适当的测试太慢了。显然，这里速度问题的一部分是 Claude 可以访问服务器群，而我只在一台 Mac 上运行。这是某人在进行本地计算时将面临的实际问题的一部分。
对于一次回答一篇论文或同时使用多篇论文，这种类型的问题的最新进展是什么？我很乐意听到成功的案例！
查看原文
What are peoples' experiences with using LLMs to mine information from scientific papers?<p>My own experience: I first attempted to extract the anti-drug antibody (ADA) rate from each of 3730 clinical-trial papers, all indexed in PubMed. I started from PDFs. Claude Opus 4.7 analyzed each PDF using a written rules doc that we had formulated. Running all the papers took about a week because I kept hitting session limits; the total cost was ~$25 (USD). We got actual rates from 909 papers. The rest were mostly cases where the rate was not present or did not meet our criteria, including administering only one drug at a time.<p>I read thirty of the papers and re-read those where I got a different answer from Claude, concluding that it had erred one time and I had erred three times.<p>So this works, but is not totally convenient: session limits mean that I can't start it up and walk away. Or I don't know how to engineer this capability. In addition I was curious how local models would perform.<p>To that end I tried llama 3.3 70B on my Mac M5 Max (128 GB mem). I used Ollama, Q4_K_M, 128 k context, ~80 k input tokens after pdftotext -layout.<p>One paper took 18 minutes; the model was unable to determine the ADA rate, whereas it is clearly in the paper. One paper is not a proper benchmark but it's too slow to do a proper test. Clearly part of the speed issue here is that Claude has access to a server farm, whereas I'm running on just one Mac. This is part of the practical problem that someone would face with local computation.<p>What is the state of the art on this type of problem, for answering questions one paper at a time or using many papers at once? I'd love to hear success stories!