Ask HN: Supabase PG 升级擦除了生产数据库，PITR 备份失败
1 分•作者: supai•5 个月前
我们目前在 Supabase 上遭遇了全面生产中断和严重的数据丢失，并且无法从他们的支持团队那里得到回应。我们希望他们团队的成员能在这里看到这条消息。<p>故障时间线：<p>1. 我们对实例执行了 Postgres 版本升级。
2. 由于未知原因，这次升级触发了磁盘大小的意外降级。
3. 我们运行了标准的 REINDEX 命令：
REINDEX DATABASE postgres;
由于步骤 2 中的错误导致磁盘空间严重受限，磁盘完全耗尽了空间。
4. 这种空间耗尽事件导致整个数据库被清空。
5. 我们立即尝试了时间点恢复 (PITR)，但恢复过程在 Supabase 端失败。<p>我们的项目现在完全无法访问。<p>我们有一个紧急支持工单 (#SU-342355)，发布在 GitHub 讨论区，并在 X 平台上进行了联系，但没有收到任何人工回复。<p>如果 @kiwicopple、@antwilson 或任何 Supabase 基础设施工程师正在阅读此内容：请不要删除底层的 AWS EBS 卷。我们需要工程师手动挂载该卷并提取 WAL 或原始数据页面，然后再覆盖这些数据块。<p>欢迎社区提供关于如何进一步升级此事件的建议。
查看原文
We are currently experiencing a total production outage and severe data loss on Supabase, and we cannot get a response from support. We are hoping someone from their team sees this here.<p>The Timeline of Failure:<p>1. We performed a Postgres version upgrade on our instance.
2. For an unknown reason, this upgrade triggered an unexpected downgrade of our disk size.
3. We ran a standard REINDEX:
REINDEX DATABASE postgres;
Because the disk space was severely limited by the bug in step 2, the disk ran out of space entirely.
4. This out-of-space event caused the entire database to wipe.
5. We immediately attempted a Point-in-Time Recovery (PITR), but the restore process is failing on Supabase's end.<p>Our project is now completely inaccessible.<p>We have an open critical support ticket (#SU-342355), posted on GitHub discussions, and reached out on X, but have received zero response from a human.<p>If @kiwicopple, @antwilson, or any Supabase infra engineers are reading this: please do not delete the underlying AWS EBS volume. We need an engineer to manually mount the volume and extract the WAL or raw data pages before the blocks are overwritten.<p>Any advice from the community on escalating this further is appreciated.