How Production Secrets End Up in Your Logs, and How to Actually Stop It

This starts from an actual incident: tenant configuration secrets showed up in production logs. The post traces exactly how they got there, and the path is mundane enough that most Python codebases have it. Something raised ValueError(f"Failed processing config for {config}"), the config object was a dataclass or Pydantic model whose auto-generated __repr__ serializes every field, logger.exception() faithfully recorded the traceback with that message in it, and a recent feature had routed failures through a handler that happened to have the sensitive object in scope.

The core argument: the leak can't happen if the object never serializes its secrets. So the primary defense is at the model. Write explicit __repr__ and __str__ that leave secrets out, use Pydantic's SecretStr for individual fields so they render as asterisks, or mark fields Field(repr=False). The backup layer is a logging filter that redacts known sensitive keys, but the post is upfront about its limit: it only catches record.args when it's a dict with %-style formatting, and f-strings sail right past it because the message is already assembled. For structured logging, use structlog processors instead. Then add tests that assert secrets don't appear in repr, str, or exception messages, and map every place your logs end up (aggregators, SIEMs, archives) before you need to know.

Key takeaways

• __repr__ is a contract, not a debugging convenience: anything may serialize it, including your logger; control it on anything holding a credential
• A redaction filter is a safety net, not the fix: it misses f-string messages entirely, which is how most leaks look; fix it at the object, keep the filter as defense in depth
• Do the incident prep before the incident: know your log destinations, per-system deletion options, compliance timelines; rotate exposed secrets deliberately, not in a panic

Who this is for

Backend and platform engineers working on systems that handle credentials, tokens, or per-tenant secrets, plus the people responsible for what happens after a leak. Examples are Python (dataclasses, Pydantic, the logging module, structlog), but the failure mode and the fix generalize.

The full write-up, with the code for each defense, is on DEV.to.