SHOW HN: Cloudflare Workers 的用量熔断器

8作者: ethan_zhao5 个月前
我运营着 3mins.news (https://3mins.news),这是一个完全基于 Cloudflare Workers 构建的 AI 新闻聚合器。后端有 10 多个 cron 触发器,每隔几分钟运行一次——RSS 获取、文章聚类、LLM 调用、电子邮件发送。 问题在于:Workers 付费计划有严格的月度限制(1000 万次请求、100 万次 KV 写入、100 万次队列操作等)。它没有内置的“达到限制时暂停”功能——CF 会直接开始收取超额费用。KV 写入的费用是超过上限后每百万次 5 美元,因此重试循环的 bug 可能会迅速变得昂贵。 AWS 有预算警报,但这些都是被动通知——等你读到电子邮件时,损害已经造成了。我想要主动的、应用级别的自我保护。 所以我构建了一个面向内部的断路器——它不是为了防止下游故障(Hystrix 模式),而是监控我自己的资源消耗,并在达到上限之前优雅地降级。 关键设计决策: * **按资源设置阈值:** Workers 请求(超额费用为每百万次 0.30 美元)仅在达到 80% 时发出警告。KV 写入(超额费用为每百万次 5 美元)可以在达到 90% 时触发断路器。并非所有资源都同样危险,因此有些配置为仅警告(trip=null)。 * **滞后效应:** 在 90% 时触发,在 85% 时恢复。这 5% 的差距可以防止振荡——如果没有它,系统会在每次检查周期都在触发和恢复之间来回切换。 * **监控失败时的安全措施:** 如果 CF 使用情况 API 宕机,则保持最后已知状态,而不是假设“一切正常”。监控中断不应掩盖使用量激增。 * **警报去重:** 按资源、按月。如果没有它,一旦某个资源达到 80%,你会在本月剩余时间内收到大约 8,600 封相同的电子邮件。 实现方式:每 5 分钟,并行查询 CF 的 GraphQL API(请求、CPU、KV、队列)+ 可观察性遥测 API(日志/跟踪),评估 8 个资源维度,将状态缓存到 KV。在检查之间,它是一个简单的 KV 读取——基本上是免费的。 当断路器触发时,所有计划任务都会被跳过。cron 触发器仍然会启动(你无法阻止它),但它做的第一件事是检查断路器,如果已触发则退出。 它已经在生产环境中运行了两周。在本月初捕获了 KV 读取量在 82% 时的峰值——收到一封警告邮件,进行了调查,修复了根本原因,从未达到触发阈值。 该模式应适用于任何按量计费的无服务器平台(Lambda、Vercel、Supabase)或任何有预算上限的 API(OpenAI、Twilio)。核心思想是:将你自己的资源预算视为一个健康信号,就像你对待下游服务的错误率一样。 如果大家有兴趣,我很乐意分享代码细节。 完整的实现代码和测试文章:https://yingjiezhao.com/en/articles/Usage-Circuit-Breaker-for-Cloudflare-Workers
查看原文
I run 3mins.news (https:&#x2F;&#x2F;3mins.news), an AI news aggregator built entirely on Cloudflare Workers. The backend has 10+ cron triggers running every few minutes — RSS fetching, article clustering, LLM calls, email delivery.<p>The problem: Workers Paid Plan has hard monthly limits (10M requests, 1M KV writes, 1M queue ops, etc.). There&#x27;s no built-in &quot;pause when you hit the limit&quot; — CF just starts billing overages. KV writes cost $5&#x2F;M over the cap, so a retry loop bug can get expensive fast.<p>AWS has Budget Alerts, but those are passive notifications — by the time you read the email, the damage is done. I wanted active, application-level self-protection.<p>So I built a circuit breaker that faces inward — instead of protecting against downstream failures (the Hystrix pattern), it monitors my own resource consumption and gracefully degrades before hitting the ceiling.<p>Key design decisions:<p>- Per-resource thresholds: Workers Requests ($0.30&#x2F;M overage) only warns at 80%. KV Writes ($5&#x2F;M overage) can trip the breaker at 90%. Not all resources are equally dangerous, so some are configured as warn-only (trip=null).<p>- Hysteresis: Trips at 90%, recovers at 85%. The 5% gap prevents oscillation — without it the system flaps between tripped and recovered every check cycle.<p>- Fail-safe on monitoring failure: If the CF usage API is down, maintain last known state rather than assuming &quot;everything is fine.&quot; A monitoring outage shouldn&#x27;t mask a usage spike.<p>- Alert dedup: Per-resource, per-month. Without it you&#x27;d get ~8,600 identical emails for the rest of the month once a resource hits 80%.<p>Implementation: every 5 minutes, queries CF&#x27;s GraphQL API (requests, CPU, KV, queues) + Observability Telemetry API (logs&#x2F;traces) in parallel, evaluates 8 resource dimensions, caches state to KV. Between checks it&#x27;s a single KV read — essentially free.<p>When tripped, all scheduled tasks are skipped. The cron trigger still fires (you can&#x27;t stop that), but the first thing it does is check the breaker and bail out if tripped.<p>It&#x27;s been running in production for two weeks. Caught a KV reads spike at 82% early in the month — got one warning email, investigated, fixed the root cause, never hit the trip threshold.<p>The pattern should apply to any metered serverless platform (Lambda, Vercel, Supabase) or any API with budget ceilings (OpenAI, Twilio). The core idea: treat your own resource budget as a health signal, just like you&#x27;d treat a downstream service&#x27;s error rate.<p>Happy to share code details if there&#x27;s interest.<p>Full writeup with implementation code and tests: https:&#x2F;&#x2F;yingjiezhao.com&#x2F;en&#x2F;articles&#x2F;Usage-Circuit-Breaker-for-Cloudflare-Workers