Tip: your logging stack is a secrets store. Keep keys out of debug logs, and keep observability backends patched and off the internet
- Redact at the source: wrap key, token and seed types so they can't be stringified, and add a scrubbing processor (e.g. in the OpenTelemetry Collector) that drops fields and patterns that look like secrets before export.
- Treat a log-level change as a change: raising production to debug should be time-boxed and reviewed, and debug statements touching sensitive operations shouldn't ship.
- Scan what you already stored: run a secret scanner over log indexes and traces; if a secret is found, rotate it and purge the data rather than waiting for retention to expire.
- Observability UIs aren't internal by default: put SigNoz, Grafana, Kibana, Jaeger and similar behind SSO/VPN, alert when the allow rule or WAF rule protecting them changes, and check from outside that they aren't reachable.
- Patch them like production: monitoring stacks set up by a consultant and left on an old version are a pattern; give them an owner and an update schedule.
Sources: Proxima: Security Incident - Post-Mortem & Full Reimbursement of Affected Users (X article by co-founder Theo, Nov 14, 2026) · OpenTelemetry: Handling sensitive data