将一个 900KB 的 Transformer 模型过拟合，以将一个 100MB 的 CSV 文件压缩到 7MB
2 分•作者: spidy__•4 个月前
我构建了一个实验，使用过拟合的 Transformer 和算术编码来压缩单个文件。
我没有训练模型进行泛化，而是训练了一个 900KB 的 Transformer 来记忆单个文件并预测下一个字节。这些预测
查看原文
I built an experiment that uses an overfitted transformer and arithmetic coding to compress individual files.<p>Instead of training the model to generalize, I train a 900KB transformer to memorize a single file and predict the next byte. Those predictions are fed into an arithmetic coder to produce the compressed output.<p>On a 100MB NYC taxi CSV, it compresses to about 7MB (~0.5 bits/byte). On a 100MB slice of enwik9, it compresses to about 21MB (~1.68 bits/byte).<p>It's pretty slow right now (roughly 20–30 minutes of training and 45 minutes each for compression and decompression on my AMD 7800XT).<p>Checkout the repo - https://github.com/samyak112/pym-particles