ARCHE3-7B – 采用SmartRouter和基础课程训练的稀疏专家混合模型
1 分•作者: OpenSynapseLabs•4 个月前
这是我在 HN 上的第一篇帖子——有点紧张，但很兴奋能分享我一直在构建的东西。
我一直在开发一个 70 亿参数的稀疏专家混合（Mixture-of-Experts，MoE）原型，它实际上可以在消费级硬件上运行。例如，在 Colab T4 上，它在训练期间使用大约 5 GB RAM 和 5 GB VRAM，推理时大约需要 3.5–5 GB。
我花了很多时间在以下几个方面：
路由（SmartRouter）
我尝试以一种实用的方式解决路由崩溃问题。我没有让所有 token 都涌入少数“最受欢迎”的专家，而是结合了几种方法：负载均衡损失、用于保持分布平坦的熵奖励、训练期间的抖动噪声以及可学习的温度。它在保持很大一部分专家活跃方面表现出乎意料地好。我已经开源了路由代码（hive_router.py），如果有人想研究数学原理或将其用于自己的项目，可以参考。
基础课程训练（FCT）
在标准的预训练之前，我让模型通过结构化的推理模式运行——目前有 290 种，涵盖 14 个认知领域。每个模式都遵循严格的序列：观察 → 先验 → 更新 → 涟漪 → 类比 → 行动。
为了让它在我的设置上实际运行，我使用了一些特定的技巧。首先，我使用仅目标损失（屏蔽标签和输入，只计算实际推理负载（如更新或行动）的梯度）。其次，我不得不编写一个自定义的 SparseExpertAdamW，它只为该步骤中实际活跃的专家实例化优化器状态。如果没有这个，20480 个专家的优化器状态会完全压垮我的 RAM。
到目前为止，我已经完成了 14 个领域中的 5 个。一件很酷的事情是：每个新领域的损失都比前一个领域低（例如，系统领域从 2.149 下降到 0.941），因此似乎跨领域的迁移确实发生了。
简而言之，架构如下：
d_model = 2048
10 层（5 个密集核心 + 5 个融合）
20480 个专家（8 个领域 × 2560）
动态 Top-K（2–4）
内存映射权重 + 多巴胺学习 v1
模型已上传到 HuggingFace：https://huggingface.co/OpenSynapseLabs/arche3-7b
我将基准测试和图表放在了 GitHub 上：https://github.com/OpenSynapseLabs/arche3-benchmarks
局限性（坦白说）：
我还没有运行标准的基准测试（MMLU、GSM8K、HumanEval），只完成了 5/14 个 FCT 领域，数据集仍然很小，需要适当的扩展。此外，这目前是一个单人项目。我确实使用了 Gemini 和 Claude 来加速部分实现，但架构和核心想法是我自己的。
我非常感谢任何反馈，特别是如果你对 MoE 模型中的路由、课程预训练或进一步扩展（正在考虑下一个 350 亿参数）感兴趣。
我的主要目标是构建能够增强人类思维的系统，而不是取代它。如果这听起来像你想要尝试或贡献的东西，请随时通过 opensynapselabs@proton.me 联系我。我很乐意分享更多细节和私有仓库。
感谢阅读！
查看原文
This is my first post on HN — a bit nervous, but excited to share what I've been building.<p>I’ve been working on a 7B sparse Mixture-of-Experts prototype that can actually run on consumer hardware. For example, on a Colab T4 it uses around 5 GB RAM and 5 GB VRAM during training, and roughly 3.5–5 GB for inference.<p>A couple of things I spent a lot of time on:<p>Routing (SmartRouter)
I tried to tackle routing collapse in a practical way. Instead of letting all tokens dump into a few "favorite" experts, I combined a few things: load balancing loss, an entropy bonus to keep the distribution flat, jitter noise during training, and a learnable temperature. It works surprisingly well at keeping a good portion of experts active. I’ve open-sourced the router code (hive_router.py) if anyone wants to look at the math or grab it for their project.<p>Foundation Curriculum Training (FCT)
Before standard pretraining, I run the model through structured reasoning patterns — currently 290 of them across 14 cognitive domains. Each pattern follows a strict sequence: OBSERVE → PRIOR → UPDATE → RIPPLE → ANALOGY → ACT.<p>To make this actually run on my setup, I'm doing a couple of specific tricks. First, I use a Target-Only Loss (masking out the tags and inputs and only calculating gradients on the actual reasoning payloads like UPDATE or ACT). Second, I had to write a custom SparseExpertAdamW that only instantiates optimizer states for the experts that are actually active on that step. Without this, the optimizer states for 20,480 experts would have absolutely crushed my RAM.<p>So far I’ve completed 5 out of 14 domains. One cool thing: every new domain starts with a lower loss than the previous one (for example, the Systems domain went from 2.149 down to 0.941), so it seems like the cross-domain transfer is actually happening.<p>The architecture in short:<p>d_model = 2048<p>10 layers (5 Dense Core + 5 Fusion)<p>20,480 experts (8 domains × 2560)<p>Dynamic Top-K (2–4)<p>memory-mapped weights + Dopamine Learning v1<p>Model is up on HuggingFace: https://huggingface.co/OpenSynapseLabs/arche3-7b
And I put the benchmarks & graphs on GitHub: https://github.com/OpenSynapseLabs/arche3-benchmarks<p>Limitations (to be honest):
I haven’t run standard benchmarks yet (MMLU, GSM8K, HumanEval), only 5/14 FCT domains are done, and the dataset is still small and needs proper scaling. Plus, this is a solo project so far. I did use Gemini and Claude to speed up parts of the implementation, but the architecture and core ideas are my own.<p>I’d really appreciate any feedback, especially if you’re into routing in MoE models, curriculum pretraining, or scaling this further (thinking about 35B next).<p>My main goal is to build systems that amplify human thinking, not replace it. If that sounds like something you'd want to mess around with or contribute to, feel free to reach out at opensynapselabs@proton.me. I'm happy to share more details and the private repo.<p>Thanks for reading!