开源浏览器内日志清理工具，需要反馈
2 分•作者: andriimb•大约 1 个月前
刚刚发布了一个与框架无关的日志清理器，可在浏览器和 Node.js 中运行。它旨在清理日志文件，然后再将其发送到第三方平台进行处理、分析或故障排除。用例？考虑到 HuggingFace 的泄露事件，你会将任何用户名、密码、个人身份信息 (PII)、配置等信息提供给 AI 吗？
该清理器会审查敏感信息、标识符和 PII，并用稳定的 HMAC 令牌替换每个值，以降低意外泄露日志中包含的敏感或机密信息的风险。
该库已集成到 LogTotal 中，并直接在浏览器中运行，允许在日志离开用户环境之前在本地进行清理。
如果您需要完全受控和隔离的数据清理环境，可以在您自己的基础设施中部署和运行此库。该清理器也可以在隔离环境中安装和使用，确保敏感日志数据保留在您完全控制的环境中。
该库没有运行时依赖项。相同的编译规则可在浏览器标签页和 Node.js CLI 中运行。安装说明请参见 git。
令牌是 ruleId || 0x00 || original 的 HMAC-SHA-256，截断为 16 个十六进制字符。相同的密钥 + 相同的值 + 相同的规则 ⇒ 相同的令牌。不同的密钥会产生不同的令牌。生成的密钥使用十六进制编码；粘贴的密钥默认为 utf8。
我们如何知道要清理什么？这是基于 SOC Prime 数据集中超过 100 万条检测规则。
它能 100% 解决日志清理问题吗？不能，存在一些边缘情况，因此我在此征求您的反馈。
该工具是 Apache 2.0 开源许可，并将保持开源。
我花了 20 多年时间处理 SIEM、日志管理、数据湖以及现在的数据管道，令人惊讶的是竟然没有这样的工具。所以我们做了一个，并希望得到任何反馈！
这是朝着更大规模的社区免费服务 LogTotal迈出的第一步。
开源浏览器内日志清理器，需要反馈
https://github.com/socprime/logtotal-sanitizer
查看原文
Just released a framework-agnostic log sanitizer working in browsers and Node.js. It's designed to sanitize log files before they are sent to third-party platforms for processing, analysis, or troubleshooting. Use Case? thing HuggingFace breach, would you feed any AI your usernames, passwords, PII, configs etc?<p>The sanitizer redacts secrets, identifiers, and PII and replaces each value with a stable HMAC token to reduce the risk of accidentally exposing sensitive or confidential information contained in logs.<p>This library is already integrated into LogTotal and runs directly in the browser, allowing logs to be sanitized locally before they leave the user's environment.<p>If you require a fully controlled and isolated data sanitization environment, you can deploy and run this library within your own infrastructure. The sanitizer can also be installed and used in air-gapped environments, ensuring that sensitive log data remains within an environment you fully control.<p>The library has no runtime dependencies. The same compiled rules run in a browser tab and in a Node.js CLI. Complete instructions how to install it are on git.<p>Tokens are HMAC-SHA-256 of ruleId || 0x00 || original, truncated to 16 hex chars. Same key + same value + same rule ⇒ same token. A different key produces different tokens. Generated keys use encoding hex; pasted keys default to utf8.<p>How do we know what to sanitise? This is based on SOC Prime dataset of over 1 million detection rules.<p>Does it solve log sanitation 100%? No, there are edge cases, and hence I ask for your feedback here.<p>Tool is open source under Apache 2.0. and will remain so.<p>I've spent over 20 years dealing with SIEM, log management, data lakes and now data pipelines and was amazed there is no such tool out there. So we made one and would appreciate any feedback!<p>This is a first step towards a larger community free service, logtotal.<p>open source in-browser log sanitizer, feedback required<p>https://github.com/socprime/logtotal-sanitizer