本地 SLM 作为云 API 调用的压缩层
1 分•作者: asong56•3 个月前
不久前发现了 caveman 项目 (https://github.com/JuliusBrussee/caveman)，并开始有点痴迷。它生成的输出很密集，但不能直接展示给用户。所以我一直在尝试将它隐藏在中间——本地 SLM 模型压缩输入，云端模型以 caveman 风格进行推理，本地 SLM 模型再将其扩展回来。用户永远看不到压缩的部分。<p>目前通过 Candle 运行 Phi-3 模型进行压缩，速度足够快。云端调用的时间也更短。还没真正统计过 token 数量。<p>扩展步骤是问题所在。将 caveman 的输出还原成可读文本比压缩输入更难，而且本地模型在这里更容易出错。不确定这是否是提示词的问题，还是这种规模的模型本身就存在上限。<p>此外，也不确定这种方法在低 API 调用量的情况下是否有意义。增加的复杂性可能不值得。
查看原文
Found caveman a while back (https://github.com/JuliusBrussee/caveman) and got kind of obsessed with it. Dense outputs, but you can't show them to a real user. So I've been trying to hide that in the middle -- local SLM compresses the input, cloud model reasons in caveman-style, local SLM expands it back. User never sees the compressed parts.<p>Running Phi-3 via Candle for compression. Fast enough. Cloud calls are shorter. Haven't done real token counting yet.<p>The expansion step is the problem. Re-hydrating caveman output into readable text is harder than compressing the input and the local model makes more mistakes there. Not sure if that's a prompting issue or just a ceiling for a model this size.<p>Also not sure this makes sense at low API volumes. The added complexity might not be worth it.