Show HN: 在阈值被突破前检测 API 性能下降
2 分•作者: AnchorFlow•3 个月前
在构建支付编排系统时，我遇到了一个问题：
大多数监控工具会在达到阈值时发出警报（例如，P95 > 1000ms）。但实际上，系统在达到这些限制之前往往就会开始退化——尤其是在突发流量的情况下。
因此，我尝试在 FastAPI 应用内部直接检测阈值被突破之前的性能下降。
我构建了一个小的中间件，它可以：
* 跟踪每个路由模板（例如 /users/{id}）的 P95 延迟
* 从最近的流量中动态学习基线
* 使用变化率检测峰值（不仅仅是静态阈值）
* 计算一个 0–100 的健康评分，并显示趋势方向（改善 / 稳定 / 恶化）
* 将事件存储在 Redis Streams 中，用于重放和调试
一个有趣的结果是：
在合成负载测试中（在 60 秒内，延迟从约 200ms 逐渐增加到约 1200ms，P95 警告阈值为 1000ms），变化率检测始终在静态阈值警报之前就发现了性能下降。虽然窗口很小，但通常足以在超过警报阈值之前注意到系统压力。
设计约束：
* 对请求路径的开销接近于零（异步，即发即弃的写入）
* 如果 Redis 不可用，必须静默失败
* 不需要外部监控堆栈（在应用内运行）
使用示例：
```python
pythonapp.add_middleware(RequestMetricsMiddleware, alert_engine=engine)
```
背景：
这是我正在构建的更大系统的一部分，该系统将云服务与移动货币 API（EcoCash 等）集成，其中部分失败和延迟峰值很常见。
还处于早期阶段——尚未在实际生产流量下进行测试。
很好奇其他人是如何在 FastAPI 或类似系统中处理早期性能下降检测的。
代码库：[https://github.com/Tandem-Media/fastapi-alertengine](https://github.com/Tandem-Media/fastapi-alertengine)
PyPI：[https://pypi.org/project/fastapi-alertengine/](https://pypi.org/project/fastapi-alertengine/)
查看原文
While building a payment orchestration system, I ran into a problem:
Most monitoring tools alert when a threshold is already breached (e.g. P95 > 1000ms). But in practice, systems often degrade before hitting those limits — especially under bursty traffic.
So I experimented with detecting degradation before thresholds are crossed, directly inside a FastAPI app.
I built a small middleware that:<p>Tracks P95 latency per route template (e.g. /users/{id})
Learns a baseline dynamically from recent traffic
Detects spikes using rate-of-change (not just static thresholds)
Computes a 0–100 health score with trend direction (improving / stable / degrading)
Stores events in Redis Streams for replay and debugging<p>One interesting result:
In synthetic load tests (gradual latency ramp from ~200ms to ~1200ms over 60 seconds, with a P95 warning threshold at 1000ms), rate-of-change detection consistently surfaced degradation slightly before static threshold alerts. The window is small, but it was often enough to notice system stress before crossing alert thresholds.<p>Design constraints:<p>Near-zero overhead on the request path (async, fire-and-forget writes)
Must fail silently if Redis is unavailable
No external monitoring stack required (runs in-app)<p>Example usage:
pythonapp.add_middleware(RequestMetricsMiddleware, alert_engine=engine)<p>Context:
This is part of a larger system I'm building that integrates cloud services with mobile money APIs (EcoCash, etc.), where partial failures and latency spikes are common.
Still early — hasn't been tested under real production traffic yet.<p>Curious how others are handling early degradation detection in FastAPI or similar systems.
Repo: <a href="https://github.com/Tandem-Media/fastapi-alertengine" rel="nofollow">https://github.com/Tandem-Media/fastapi-alertengine</a>
PyPI: <a href="https://pypi.org/project/fastapi-alertengine/" rel="nofollow">https://pypi.org/project/fastapi-alertengine/</a>