展示 HN：通过打破 DDR4 时序规则，在 DRAM 中运行 PrismML 的 Bonsai
11 分•作者: pcdeni•2 个月前
PrismML 的 1 位/三元 Bonsai 模型所引发的兴奋，正让业界密切关注智能手机巨头（尤其是苹果）如何在边缘设备上实现大型语言模型（LLM）。
将 AI 迁移到设备端是一项明智且必要的战略。它确保了用户隐私的绝对安全，符合欧盟法规的要求，从根本上将经济重心从昂贵的云端推理转移开，并为用户寻求真正的 AI 芯片而推动一次重大的硬件升级超周期铺平了道路。
要创建一个智能的设备端“语义路由器”，模型需要达到 270 亿参数以上的规模。在手机上实现这一点需要极端的量化，例如 PrismML 的三元权重。
然而，软件界常常忽略一个
查看原文
The excitement surrounding PrismML’s 1-bit/ternary Bonsai models has the industry closely watching how smartphone giants, particularly Apple, will implement LLMs on edge devices.<p>Moving AI on-device is a brilliant and necessary strategy. It ensures absolute user privacy in alignment with EU regulations, fundamentally shifts the economics away from costly cloud inference, and paves the way for a significant hardware upgrade supercycle as users seek true AI-capable silicon.<p>To create a smart on-device "Semantic Router," models need to reach the 27B+ parameter scale. Achieving this on a phone requires extreme quantization, such as PrismML’s ternary weights.<p>However, a critical hardware reality often overlooked by the software world is that fitting the weights in RAM is not equivalent to moving them. Running a 27B ternary model on standard LPDDR encounters a significant memory bandwidth limitation. Transferring gigabytes of data across the SoC bus for each token generation can lead to thermal throttling of the NPU and excessive battery drain.<p>This raises an important question: why are we still transferring data to the compute? Why not execute AI inference natively within the memory?<p>Frustrated with academic PIM simulations that overlook bare-metal physics, I developed CaSA, an architecture that performs ternary LLM inference directly inside COTS DRAM through charge-sharing, completely bypassing the memory bus.<p>Software quantization is a great initial step, and CaSA provides the physical hardware substrate needed to complete the bridge: <a href="https://github.com/pcdeni/CaSA" rel="nofollow">https://github.com/pcdeni/CaSA</a>