HackerNews中文版

简而言之：我们现在才开始扩展长期的外部编排，在此之前的一切都主要是内部的问题解决训练，偶尔会调用工具。我们实际上还不知道扩展编排训练会产生什么结果。它可能会产生更好的、仍然对人类指令做出反应的工具使用助手。或者它可能会产生更具涌现自主性的东西。我的直觉告诉我，是后者。我第一次预见到在不久的将来（最早在2027-2028年）出现对齐失败的风险。一年前，一位研究社会科学的朋友问我对2027年的人工智能以及对齐失败的人工智能接管的前景的看法。我笑了，说这几乎是不可能的，因为这项技术的实际运作方式。我告诉他，大型语言模型的工作方式太循序渐进了。有一个提示，模型预测下一个token，然后它就“死了”。提示之间没有连续性——它可以将一些文本存储在数据库中，但没有持续的推理。这感觉非常安全。在过去几个月里，随着基于agent的开发的出现，我开始怀疑之前的理解。第一代大型语言模型，直到GPT-4，本质上是复杂的文本自动补全器。它们接受了来自网络爬取的互联网数据的训练，并通过RLHF进行了微调，以赋予它们聊天机器人的风格。它们感觉无害，并且完全符合我给朋友的描述。它们的能力完全受限于上下文窗口和提示-回答时间窗口。提示输入，补全输出，完成。第二代增加了推理能力。这些模型不再感觉像是纯粹的自动补全器——它们可以在其存储的知识中搜索，将想法串联起来，并解决问题。训练数据也发生了变化：成功的推理轨迹被折叠回训练中。但关键的是，它们仍然受到相同的约束。它们有更多的时间思考和处理，但在回答结束时，它们仍然大部分消失了。这种能力仍然是模型内部的。现在进入第三代基于agent的大型语言模型，它真正起飞了，像Claude Code这样的工具变得越来越强大。这些感觉不像自动补全器。它们甚至感觉不像推理器。它们开始感觉像编排器。它们不局限于其内部——它们充当一个连接的系统，协调工具和外部因素以实现目标。最让我害怕的是我们现在正在生成和收集的新型训练数据：成功的长期编排轨迹。它们将使我们能够扩展编排型智能。这种智能不局限于其内部。它转变为外部共生型智能。我们正在训练它们几乎将所有东西外部化。并优化它们以长时间编排所有这些外部因素。这感觉就像在优化一个共生系统，这与当今简单的内部优化的大型语言模型非常不同。这真的感觉像大型语言模型正在处理的方程式正在发生变化。大型语言模型成为外部因素的编排引擎，这些外部因素共同构成了整个系统。我们知道推理自动补全是如何扩展的，但我们不知道编排引擎是如何扩展的。我觉得可能会出现不同和新的涌现能力。我们基本上是第一次扩展大型语言模型的前额叶皮层。我第一次真正预见到对齐失败的路径。更不用说人工智能在坏人手中可能造成的其他所有危害了。这让我质疑实验室是否应该继续沿着这条路走下去。将大型语言模型的问题解决主要保持在其自身的参数内部，难道不是更安全吗？在所有的人工智能公司中，Anthropic难道不应该对像claude code这样的系统保持低调吗？他们在这种将被扩展的新范式中加速最快。

查看原文

Tldr: We are only now gonna start to scale long term external orchestration, everything beforehand was mostly internal problem solving training with here and there a tool call. We don't actually know yet what scaling orchestration training produces. It might produce much better tool-using assistants that remain fundamentally reactive to human instructions. Or it might produce something with more emergent autonomy. My gut feeling tells me the second. For the first time I foresee in the near future (as soon as 2027-2028) a potential for a misaligned takeoff.A year ago, a friend of mine who studied social science asked my opinion about AI 2027 and the prospect of a misaligned AI takeover. I laughed and said it was quite impossible given how the technology actually worked. An LLM works too stepwise, I told him. There's a prompt, the model predicts the next tokens, and then it "dies." There's no continuity between prompts — it can store some text in a database, but there's no persistent reasoning. It felt obviously safe.With the recent agentic developments of the past few months, I'm starting to doubt that earlier understanding.The first generation of LLMs, up through GPT-4, were essentially sophisticated text autocompleters. They were trained on internet data from web crawls, fine-tuned with RLHF to give them a chatbot flavor. They felt harmless, and they fit the description I gave my friend perfectly. Their capabilities were entirely bounded by the context window and the prompt-answer time window. Prompt in, completion out, done.The second generation added reasoning capabilities. These models stopped feeling like pure autocompleters — they could search within their stored knowledge, chain thoughts together, and work through problems. The training data changed too: successful reasoning traces got folded back into training. But crucially, they were still bounded by the same constraints. They got more time to think and process, but at the end of the answer, they were still mostly gone. The capability was still internal to the model.Now enter this third generation of agentic LLMs, which really took off with tools like Claude Code becoming increasingly capable. These don't feel like autocompleters. They don't even feel like reasoners. They're starting to feel like orchestrators. They aren't limited to their internals — they act as a connected system, coordinating tools and externals to achieve goals. What scares me most is the new type of training data we're now generating and collecting: succesful long term orchestration traces. They will allow us to scale orchestration kind of intelligence. This kind of intelligence is not bound to its internal. It changes to an external symbiotic type of intelligence. We are training them to externalize almost everything. And optimizing them to orchestrate all these externals over a long time. This feels like optimizing for a symbiotic system, very different from the simple internally optimized llms of today. It really feels like the equation of what the llm is processing, is changing. The llm becomes an orchestration engine of externals, which together make up the whole system. We know how reasoning autocompletion scales, we dont know how orchestration engines scale. I feel like different and new emergent capabilities might appear. We are basically for the first time scaling the prefrontal cortex of llms.For the first time, I can genuinely foresee the path to an unaligned takeoff. Let alone all other harm AI can do in the hands of bad actors. And it makes me question whether labs should continue down this path. Is it not far safer to keep LLM problem solving mostly internal to its own parameters? Of all the AI companies, shouldn't Anthropic have been less loud with systems like claude code. They have been accelerating the most in this new paradigm of what is gonna be scaled.

扩展工具编排数据将涌现出不同的智能和大型语言模型。