MicroVault:一个用大约1000行代码实现的简单分布式Blob存储

1作者: afpereira7 个月前
我厌倦了阅读分布式存储系统成千上万页的文档。我只想可靠地在几台服务器上存储文件。<p>所以我构建了 MicroVault - 一个内容可寻址的 Blob 存储,你真的可以理解它。<p>*它的功能:* - 通过复制在 N 个卷服务器上存储 Blob - 使用 nginx 进行服务(读取零开销) - 使用 SQLite 进行元数据管理 - 使用一致性哈希进行分发 - 简单的 HTTP API<p>*它的不足:* - 自定义网络协议 - 分布式共识 - 内置身份验证/加密(使用标准工具) - 任何复杂的东西<p>*架构:* - 主服务器:协调写入,在 SQLite 中存储元数据 - 卷服务器:nginx + 用于 PUT/DELETE 的小型封装器(约 100 行) - 存储:磁盘上的常规文件,按哈希前缀组织 - 工具:重建、重新平衡、验证、压缩<p>*为什么选择 nginx?* 因为它已经被优化了几十年,用于提供静态文件服务。对于读取(大多数操作),它纯粹是 nginx,零开销。<p>*灵感来自:* George Hotz 的 minikeyvalue - 证明分布式存储不必复杂。<p>整个代码库足够小,可以在一个晚上审核完毕。你可以在一个周末将其 fork 并进行定制。<p>GitHub:https://github.com/afonp/microvault<p>使用 Go、SQLite、nginx 构建,并对复杂性保持健康的不尊重。
查看原文
I got tired of reading thousands of pages of documentation for distributed storage systems. All I wanted was to store files across a few servers reliably.<p>So I built MicroVault - a content-addressable blob store that you can actually understand.<p>*What it does:* - Stores blobs across N volume servers with replication - Uses nginx for serving (zero overhead on reads) - SQLite for metadata - Consistent hashing for distribution - Simple HTTP API<p>*What it doesn&#x27;t do:* - Custom network protocols - Distributed consensus - Built-in auth&#x2F;encryption (use standard tools) - Anything complex<p>*The architecture:* - Master server: coordinates writes, stores metadata in SQLite - Volume servers: nginx + tiny wrapper (~100 lines) for PUT&#x2F;DELETE - Storage: regular files on disk, organized by hash prefix - Tools: rebuild, rebalance, verify, compact<p>*Why nginx?* Because it&#x27;s been optimized for serving static files for decades. For reads (the majority of operations), it&#x27;s pure nginx with zero overhead.<p>*Inspired by:* George Hotz&#x27;s minikeyvalue - proof that distributed storage doesn&#x27;t need to be complex.<p>The entire codebase is small enough to audit in an evening. You could fork it and make it your own in a weekend.<p>GitHub: https:&#x2F;&#x2F;github.com&#x2F;afonp&#x2F;microvault<p>Built with Go, SQLite, nginx, and a healthy disrespect for complexity.