Ask HN：你如何防止生产环境中的 AI 智能体失控？
1 分•作者: techbuilder4242•6 个月前
大家好！
似乎出现了一个持续的趋势（也是我的直觉），即各公司正从聊天机器人转向能够真正执行操作的AI智能体——调用API、修改数据库、进行购买等。
我很感兴趣：如果你们在生产环境中运行这些智能体，除了防范提示词注入之外，是如何处理安全层的？
问题：
- 你们的智能体是如何阻止执行意外操作（删除记录、未经授权的交易）的？
- 你们是否真的遇到过智能体失控，导致资金或数据损失的情况？
- 现有的工具（IAM策略、审批流程、监控）足够了吗，还是存在差距？
想弄清楚这是否是一个值得解决的真正问题，或者现有的方法是否运行良好。
查看原文
Hi all!<p>There seems to be an ongoing trend (and my gut feeling) of companies moving from chatbots to AI agents that can actually execute actions—calling APIs, modifying databases, making purchases, etc.
I'm curious: if you're running these in production, how are you handling the security layer beyond prompt injection defenses?<p>Questions:<p>- What stops your agent from executing unintended actions (deleting records, unauthorized transactions)?
- Have you actually encountered a situation where an agent went rogue, and you lost money or data?
- Are current tools (IAM policies, approval workflows, monitoring) enough, or is there a gap?<p>Trying to figure out if this is a real problem worth solving or if existing approaches are working fine.