HN 读者求助：Amazonbot 正在恶意抓取我的网站，并且无视 robots.txt
4 分•作者: pera•3 天前
年初时，我决定在我的一处个人网站上搭建一个爬虫和大型语言模型（LLM）蜜罐，其中包含一个假的 Git 仓库，仓库里的代码包含假的 HTTP 端点。这个仓库的地址隐藏在一个公开页面的注释里。
大约三周前，来自 Amazon Searchbot 的 IP 地址试图请求一个包含在 shell 脚本里的假端点。
我的 robots.txt 文件明确包含了 Amazonbot。
说实话，我对这是亚马逊的行为感到惊讶。这种行为是否合法？
查看原文
At the beginning of the year I decided to set up a scraping and LLM honeypot on one of my personal websites which included a fake git repo with code containing fake HTTP endpoints. The address to this repo was hidden in a public page inside a comment.<p>About three weeks ago IP addresses from Amazon Searchbot attempted to make requests to the fake endpoints included inside a shell script.<p>My robots.txt explicitly includes Amazonbot.<p>I am honestly surprised that this is coming from Amazon. Is this kind of behavior legal?