---
title: Tip: your logging stack is a secrets store. Keep keys out of debug logs, and keep observability backends patched and off the internet | ctxsec
url: https://ctxsec.com/items/tip-secrets-in-debug-logs-observability-backends-private-patched
description: "Proxima lost 5,324 SOL because a debug log wrote private keys, a production log-level change turned it on, and the logs sat 15 days in an outdated, publicly reachable SigNoz instance. Each of those is a common, fixable condition."
status: "guidance based on Proxima's Nov 14, 2026 post-mortem"
published: 2026-11-14
---

> Proxima lost 5,324 SOL because a debug log wrote private keys, a production log-level change turned it on, and the logs sat 15 days in an outdated, publicly reachable SigNoz instance. Each of those is a common, fixable condition.

[← Tips & Guides](/tips) · [markdown](/items/tip-secrets-in-debug-logs-observability-backends-private-patched.md)

guidance based on Proxima's Nov 14, 2026 post-mortem

2026-11-14 7:35 PM ET

### Tip: your logging stack is a secrets store. Keep keys out of debug logs, and keep observability backends patched and off the internet

- **Redact at the source:** wrap key, token and seed types so they can't be stringified, and add a scrubbing processor (e.g. in the OpenTelemetry Collector) that drops fields and patterns that look like secrets before export.
- **Treat a log-level change as a change:** raising production to debug should be time-boxed and reviewed, and debug statements touching sensitive operations shouldn't ship.
- **Scan what you already stored:** run a secret scanner over log indexes and traces; if a secret is found, rotate it and purge the data rather than waiting for retention to expire.
- **Observability UIs aren't internal by default:** put SigNoz, Grafana, Kibana, Jaeger and similar behind SSO/VPN, alert when the allow rule or WAF rule protecting them changes, and check from outside that they aren't reachable.
- **Patch them like production:** monitoring stacks set up by a consultant and left on an old version are a pattern; give them an owner and an update schedule.

Sources: [Proxima: Security Incident - Post-Mortem & Full Reimbursement of Affected Users (X article by co-founder Theo, Nov 14, 2026)](https://x.com/0xTheo__/status/2121728996631269792) · [OpenTelemetry: Handling sensitive data](https://opentelemetry.io/docs/security/handling-sensitive-data/)
