HackerNews中文版

在为一家旧金山初创公司构建金融助手时，我们了解到，人工智能框架并不能带来价值，反而增加了复杂性。当我开始使用 GraphRAG 构建个人助手时，我吸取了这一教训，但仍然尝试了 LangChain 的 MongoDBGraphStore。它在 10 分钟内为我提供了一个可用的知识图谱。然后我查看了数据。仅仅从 5 份文档中，我就得到了 17 种节点类型和 34 种关系类型，其中包括三个版本的“属于”。GraphRAG 实际上是一个数据建模问题，而不是一个检索问题。附图展示了我最终使用的 11 步流程。以下是每个步骤的详细介绍。基本上，在数据流程的第 1 步和第 2 步中，原始数据源会经历一个提取、转换、加载 (ETL) 过程。它们作为文档存储在 MongoDB 数据仓库中。每个文档都存储了源类型、URI、内容和元数据。然后在第 3 步中，我们清理文档并将其分割成以 token 为界限的块。我们最初使用 512 个 token，并有 64 个 token 的重叠。但我们仍然需要对此进行更多测试。关键是，第 4 步处理图谱提取。我们定义了一个严格的本体。本体只是一个正式的协议，定义了数据中确切存在的类别和关系。我们使用了 6 种节点类型和 8 种边类型。大型语言模型 (LLM) 只能提取本体允许的内容。例如，如果它输出了一个 PERSON 到 TASK 的连接，并带有 EXPERIENCED 边，流程就会拒绝它。EXPERIENCED 必须将 PERSON 连接到 EPISODE。我们还将 LLM 提取与确定性提取分开。我们创建结构化条目，如 Document 或 Chunk 节点，而无需调用 LLM。事实证明，第 5 步的规范化是最难的部分。我们使用一个三阶段的去重过程。我们进行内存模糊匹配、针对 MongoDB 的跨文档解析和边重新映射。总而言之，在第 6 步中，我们批量嵌入节点。系统使用模拟进行测试，使用 Sentence Transformers 进行开发，并使用 Voyage API 进行生产。最终，在第 7 步和第 8 步中，节点和边被存储在单个 MongoDB 集合中，作为统一的内存。我们使用确定性的字符串 ID，如“person:alice”来防止重复。MongoDB 在一个聚合管道中处理文档、$vectorSearch、$text 和 $graphLookup。$graphLookup 函数原生遍历数据库中连接的图数据。对于大多数代理用例，你不需要 Neo4j + Pinecone + Postgres。像 MongoDB 这样的单个数据库就能很好地完成任务。通过分片，你可以将其扩展到数十亿条记录。总而言之，第 9 步到第 11 步涵盖了检索。代理通过 MCP 服务器调用工具。它使用混合向量、文本和图谱扩展的搜索内存，以及用于自然语言到 MongoDB 聚合的查询内存。代理还使用摄取工具写回数据库以进行持续学习。以下是我仍在努力解决的一些问题，我很想听听你的意见： * 你如何处理跨文档的实体/关系解析？ * 什么对你优化使用 LLM 提取实体/关系最有帮助？ * 在图谱更新后，你如何保持嵌入的同步？此外，在构建我的个人助手时，过去几个月我一直在 LinkedIn 上撰写关于这个系统的文章。以下是更深入介绍每个部分的帖子： * 运行嵌入模型的 3 种方法：[https://www.linkedin.com/feed/update/urn:li:activity:7443288346153480192](https://www.linkedin.com/feed/update/urn:li:activity:7443288346153480192) * LangChain 在 10 分钟内为我提供了一个知识图谱：[https://www.linkedin.com/feed/update/urn:li:activity:7440751582381494272](https://www.linkedin.com/feed/update/urn:li:activity:7440751582381494272) * Palantir 凭借本体优先的人工智能建立了 4000 亿美元的帝国：[https://www.linkedin.com/feed/update/urn:li:activity:7434591082367320064](https://www.linkedin.com/feed/update/urn:li:activity:7434591082367320064) * 数字孪生代理的摄取架构：[https://www.linkedin.com/feed/update/urn:li:activity:7432054336589021184](https://www.linkedin.com/feed/update/urn:li:activity:7432054336589021184) * 大多数人工智能代理不需要三个数据库：[https://www.linkedin.com/feed/update/urn:li:activity:7426981104227856385](https://www.linkedin.com/feed/update/urn:li:activity:7426981104227856385)

查看原文

While building a financial assistant for an SF start-up, we learned that AI frameworks add complexity without value. When I started building a personal assistant with GraphRAG, I carried that lesson but still tried LangChain's MongoDBGraphStore. It gave me a working knowledge graph in 10 minutes.Then I looked at the data. I had 17 node types and 34 relationship types from just 5 documents, including three versions of "part of". GraphRAG is a data modeling problem, not a retrieval problem.The attached diagram shows the full 11-step pipeline I ended up with. Here is a walkthrough of what you can learn from each step.So basically, in steps 1 and 2 of the data pipeline, raw sources go through an Extract, Transform, Load (ETL) process. They land as documents in a MongoDB data warehouse. Each document stores the source type, URI, content, and metadata.Then in step 3, we clean the documents and split them into token-bounded chunks. We started with 512 tokens with a 64-token overlap. Still, we have to run more tests on this.The thing is, step 4 handles graph extraction. We defined a strict ontology. An ontology is just a formal contract defining exactly what categories and relationships exist in your data. We used 6 node types and 8 edge types. The LLM can only extract what this ontology allows.For example, if it outputs a PERSON to TASK connection with an EXPERIENCED edge, the pipeline rejects it. EXPERIENCED must connect a PERSON to an EPISODE.We also split LLM extraction from deterministic extraction. We create structural entries like Document or Chunk nodes without LLM calls.Turns out, step 5 for normalization is the hardest part. We use a three-phase deduplication process. We do in-memory fuzzy matching, cross-document resolution against MongoDB, and edge remapping.Anyway, in step 6, we batch embed the nodes. The system uses a mock for tests, Sentence Transformers for development, and the Voyage API for production.Ultimately, in steps 7 and 8, nodes and edges are stored in a single MongoDB collection as unified memory. We use deterministic string IDs like "person:alice" to prevent duplicates. MongoDB handles documents, $vectorSearch, $text, and $graphLookup in one aggregation pipeline. The $graphLookup function natively traverses connected graph data directly in the database. You don't need Neo4j + Pinecone + Postgres for most agent use cases. A single database like MongoDB gets the job done really well. Through sharding, you can scale it up to a billion records.To wrap it up, steps 9 through 11 cover retrieval. The agent calls tools through an MCP server. It uses search memory with hybrid vector, text, and graph expansion, alongside query memory for natural language to MongoDB aggregation. The agent also uses ingest tools to write back to the database for continual learning.Here are a few things I am still struggling with and would love your opinion on:How are you handling entity/relationship resolution across documents?What helped you the most to optimize the extraction of entities/relationships using LLMs?How do you keep embeddings in sync after graph updates?Also, while building my personal assistant, I have been writing about this system on LinkedIn over the past few months. Here are the posts that go deeper into each piece:- 3 ways to run embedding models: https://www.linkedin.com/feed/update/urn:li:activity:7443288346153480192- LangChain gave me a knowledge graph in 10 minutes: https://www.linkedin.com/feed/update/urn:li:activity:7440751582381494272- Palantir built a $400B empire on ontology-first AI: https://www.linkedin.com/feed/update/urn:li:activity:7434591082367320064- Ingestion architecture for Digital Twin agent: https://www.linkedin.com/feed/update/urn:li:activity:7432054336589021184- Most AI agents don't need three databases: https://www.linkedin.com/feed/update/urn:li:activity:7426981104227856385

我的 11 步 GraphRAG 流程：哪些有效，哪些还存在问题