How To Turn On Debug Logging For One User In Production

A customer files a ticket. Something is wrong with their invoices. You can't reproduce it — it works on staging, it works for you, it works for the other four thousand accounts.

You know exactly which log lines would tell you what's happening. They're logger.LogDebug calls that have been sitting in that code path for two years. You cannot turn them on, because turning them on turns them on for everyone, and at production volume that's a five-figure logging bill and a haystack you'll never find the needle in.

This is one of the most common problems in operating software, and it is remarkably badly served. Let's look at why, and at what actually works.

The clearest statement of the problem

In 2017 someone opened Serilog #1078, Add a way to override minimal log level for particular execution context:

In some situations it's desirable to have a way to enable deeper logging for some failing request without affecting the whole system and thus logging huge amount of logs from system under load and without a need to change config or implement functionality to change log level system-wide.

That's the request, precisely stated, and framed around multi-tenant apps.

It's been open since 2017. You can watch the same shape recur everywhere:

• Can logger level set per API request? — Elixir Forum, 918 views. Answer: not supported. There's an OTP PR that would enable it, "stalled in discussion." The suggested workaround is to raise global verbosity and filter manually, which the responder immediately flagged as a performance risk.
• Per-Request Debug Logging via API Parameter — wants "debug": true per call, because running the whole proxy in debug is "impractical."
• Per Request Minimum Log Level — wants the level set from an HTTP header so support can debug a live call without paying for firehose aggregation.
• Multitenant LoggingLevelSwitch, How to dynamically change Serilog's minimum log level for a tenant, Modifying Rails logging level per request.

Every single one gets told the same thing.

Why "raise the level and filter downstream" is the wrong answer

It's the universal advice, and it does work in the sense that you end up with the logs. The problem is where the filter sits.

Set your app to DEBUG and filter in Datadog, and here's what happens to a debug line for a request that isn't the one you care about:

1. The log call is enabled, so the arguments get evaluated. That innocent logger.LogDebug("state: {@Order}", order) now serializes an object graph.
2. The message gets formatted.
3. It gets written — a syscall, or a channel send plus a background flush.
4. It gets shipped over the network to your aggregator.
5. It gets ingested and billed, because you pay on ingest, not on query.
6. Then it gets filtered.

You paid for all six steps to keep 0.01% of the output. On a service doing a few thousand requests a second, steps 1 and 2 alone are enough to move your p99 — and it's a self-inflicted latency regression that arrives during an incident, which is the worst possible time.

The filter is in the wrong place. It should be at step 0, at the call site, before anything is evaluated.

There's a second-order problem too. Verbose logging changes system behavior — slower, more I/O, more GC pressure — so the moment you enable it you're no longer debugging the system that had the bug. Heisenbugs love this.

Sampling doesn't rescue you either

The natural next thought is "we sample traces, isn't that the same thing?" It isn't, and a GitLab engineer laid out exactly why in Gitaly #4808: their sampling rate is 0.1%, so when a specific user or project reports a problem, there is no trace data for them. The proposal was a feature flag scoped to a user or project that forces collection for that actor.

Random sampling gives you a statistically representative view. Debugging a customer complaint is the opposite job — you need one specific non-random actor. The same tension shows up in the OpenTelemetry spec discussion Overriding decisions of the built-in trace samplers, where the counter-argument is a good one: "I believe observability will be compromised if the application can control what data is collected and what is not." Both sides are right, which is why it's still a discussion.

What people build instead

Nobody accepts this, so everyone builds something. The patterns are consistent.

1. A header that raises the level for one request

The oldest and most direct approach. INNOQ wrote it up in 2015 as Per request debugging with Log4j 2 filters: an x-debug-enabled header goes into ThreadContext, and a DynamicThresholdFilter promotes that one request to TRACE, writing to a separate file. Juliano Mohr's version uses X-Log-Level. There's even a whole Magento module for it.

It's elegant, and it has a problem we'll come back to.

2. Targeting on tenant via MDC

Duda's engineering team published Multi-Tenant Debugging with Spring Boot, motivated by exactly the cost argument — "enabling debug per package or even in a specific class can produce lots of unnecessary I/O." Their answer is a Logback TurboFilter reading a tenant ID out of MDC.

TurboFilter is the right hook because it runs before the logging event is constructed. Step 0, not step 6.

3. Ring buffer, flush on error

The sideways answer, from Will Sargent's Triggering Diagnostic Logging on Exception: keep DEBUG and TRACE in a bounded in-memory buffer, discard continuously, and only write it out when something throws. His framing of why you want it:

None of that tells you what the code was doing before it got to the exception.

You get the debug context for failures and pay nothing for successes. It doesn't help with "this customer's invoices are subtly wrong but nothing throws," but for crashes it's excellent, and it composes fine with everything else here.

4. Buy it

Some vendors ship this as a product feature, which is decent evidence that the problem is real:

• Salesforce Trace Flags — per-user debug logs as a platform primitive, and they've had it for years. You set a trace flag on a specific user, with an expiry, and get debug logs for that user's activity only.
• Curity — opens with "How can a developer debug production environments for certain integrations without compromising security?" and answers with a Log4j2 script filter scoped to specific OAuth client IDs.
• Lightrun — "add trigger conditions for users, flags, or specific code paths."

Will Sargent's Targeted Diagnostic Logging in Production is still the best general writeup, from 2019, wiring LaunchDarkly flags into Logback turbo filters. His motivating example is the one everybody recognizes: the bug that only happens in Europe, on Firefox.

The security half nobody writes about

Pattern 1 — the request header — has a problem that most of the writeups skip, and it's worth being direct about.

A client-controlled log level is a trust boundary.

If X-Log-Level: DEBUG is honored from any caller, you've built two vulnerabilities at once:

An economic denial-of-service. Anyone who can send that header can multiply your ingest bill. No auth bypass, no exploit — just a header on a loop.

Information disclosure into a lower-trust store. This is the serious one. Debug logs contain things info logs don't: full request bodies, query parameters, deserialized objects, sometimes tokens. Your log aggregator is usually a broader access surface than your production database — more people can read it, it's retained differently, and it's often replicated to a third party. Flipping one user to DEBUG can quietly move that user's sensitive data into a system with weaker controls. Teams find this out the hard way, usually while looking for something else.

There's a privacy dimension too. If you're subject to GDPR, "we log everything this named individual does, at a different retention tier, because support asked" is a purpose-limitation and retention question, not just an ops decision. It's very defensible when it's scoped and time-bounded. It's much less defensible when someone turned it on eight months ago and forgot.

Three things follow, and I'd treat them as requirements rather than nice-to-haves:

1. Target server-side, not client-side. You decide whose logs are verbose, based on identity you've already authenticated. Don't let the caller pick. The Magento module gets this right — you mint a time-limited key and the header only works with that key, so the client can't self-select.
2. Time-bound it. An expiry, or at minimum a review. Google's Stackdriver Debugger logpoints auto-expired after 24 hours; that's the right instinct.
3. Audit it. Turning on debug for a named customer is an action against that customer's data. It should be attributable to a person, with a timestamp.

The general shape of the answer

Strip away the implementations and every working solution does the same thing:

Move the level decision to the point where request identity is known, and make it evaluate a rule instead of reading a constant.

That's why a LoggingLevelSwitch, a slog.LevelVar, or a zap.AtomicLevel can't do it. They're all one number for the whole process — that's their entire design, and it's why they're fast. Per-user targeting means the level is a function of the request, and a function needs an argument.

Practically, you need three pieces:

1. Rules that live outside the code — otherwise "debug for this customer" is a deploy, and you're back where you started.
2. Request context at the call site — user, tenant, whatever you target on.
3. The check before evaluation, so you don't pay for messages you discard.

Doing it with Quonfig

This is what we built Quonfig for, so here's the honest version: it's the three pieces above, with the rules stored in git and pushed to every instance over SSE.

.NET (docs) — attach context per request, then ask at the call site:

app.UseQuonfigContext((http, ctx) =>
{
    if (http.User.FindFirst(ClaimTypes.Email)?.Value is { } email)
        ctx["user"] = new ContextProperties { ["email"] = email };
 
    if (http.User.FindFirst("tenant_id")?.Value is { } tenant)
        ctx["tenant"] = new ContextProperties { ["id"] = tenant };
});

public sealed class InvoiceService(IBoundQuonfig quonfig, ILogger<InvoiceService> logger)
{
    public void Compute(Invoice invoice)
    {
        if (quonfig.ShouldLog("Acme.Billing", Quonfig.Sdk.LogLevel.Debug))
        {
            logger.LogDebug("Computing {Id}: {@Lines}", invoice.Id, invoice.Lines);
        }
    }
}

IBoundQuonfig is scoped, so it already carries this request's context.

Go (docs) — attach a ContextSet to the context.Context and use the Quonfig handler:

cs := quonfig.NewContextSet().
    WithNamedContextValues("user", map[string]interface{}{"email": user.Email}).
    WithNamedContextValues("tenant", map[string]interface{}{"id": tenant.ID})
 
ctx := quonfig.ContextWithContextSet(r.Context(), cs)
 
logger.DebugContext(ctx, "computing invoice total", "amount", total)

Use DebugContext, not Debug — the non-context methods have nowhere to put the targeting data, so your rules will silently never match. And this needs NewQuonfigHandler, not NewQuonfigLeveler: slog.Leveler.Level() takes no arguments, so a leveler structurally cannot make a per-request decision.

Ruby (docs) — same idea, with the context passed to the check:

if client.should_log?(logger_path: 'Acme::Billing',
                      desired_level: :debug,
                      contexts: { user:   { email: current_user.email },
                                  tenant: { id: current_tenant.id } })
  logger.debug { "computing invoice total: #{total}" }
end

If you're on SemanticLogger, client.semantic_logger_filter(config_key:) wires the same check in as a filter so you don't hand-write the guard on every call — it feeds SemanticLogger's log.name in as the logger path automatically.

One thing to internalize across all three SDKs: the logger path is not part of the config key. There's one config — your LoggerKey — and the path arrives as a context property called quonfig-sdk-logging.key. Paths are passed through verbatim (no case or separator normalization), and there is no automatic parent walk. Acme::Billing set to DEBUG does not implicitly cover Acme::Billing::Invoicing; you express that with a "starts with" rule, ordered above the broader one.

That's a different model from log4j's implicit hierarchy, and it's the thing most likely to confuse you on day one. It's also what makes per-user targeting possible at all: once the logger path is just another targeting dimension, it sits alongside user.email and tenant.id in the same rule engine.

The same integration exists in every server SDK — each page has a Dynamic Log Levels section with the wiring for that ecosystem: JavaScript · Node · Python · Java · Go · .NET · Ruby · Swift

Then the rule itself:

Rules evaluate top to bottom. One named user at DEBUG, two tenants at DEBUG, enterprise plans at INFO, everyone else stays at WARN.

Reading it aloud is the whole point: dana@northwind.example gets DEBUG, the northwind and globex tenants get DEBUG, everyone else stays at WARN. That's the sentence you wanted when the ticket came in, and it takes effect in seconds without a deploy.

Honest limits

Things I'd want to know before adopting any of this, including ours:

How much this touches your code depends on the language. In most cases it's a filter installed once at startup, not a change to your log calls:

• Go — QuonfigHandler wraps your handler, and Enabled(ctx, level) receives the request context. Targeting works through ordinary logger.DebugContext(ctx, …) calls. No guards anywhere.
• .NET — AddQuonfigFilter gates every ILogger category, and the Serilog level-switch provider does the same for Serilog. Both are startup wiring, no call-site changes.
• Ruby — semantic_logger_filter / stdlib_formatter slot into the logger you already have.

The exception is per-request targeting in .NET and Ruby. ILogger.IsEnabled(LogLevel) takes no context argument — the same structural gap as slog.Leveler — so a filter installed at startup can only see the category, never who's asking. Per-user rules there need an explicit ShouldLog(...) against the request-scoped client, on the specific paths you care about. Go doesn't have this problem because slog threads a context.Context all the way down.

Separately, and unrelated to targeting: guarding expensive log arguments (if (logger.IsEnabled(...)), or Ruby's block form) is ordinary logging hygiene. Any filter runs inside the log call, so it can't save you from serializing an object graph you passed as an argument. That's true with or without any of this.

You're adding a dependency to your logging path. Ask what happens when it's unreachable. (Quonfig evaluates against a locally cached config and fails open — no rule found means log it — so a network partition can't silence your logs. Verify the equivalent for whatever you pick, including a hand-rolled one.)

It doesn't help if the log lines aren't there. Targeting turns on logging that exists. If the interesting code path has no debug statements, you still need a deploy — which is the gap Datadog's Dynamic Instrumentation and Lightrun address, and it's a different product category.

Targeted logging is a privileged action against a named person's data. Scope it, expire it, audit it. See above.

Where to start

If you take one thing away: the reason this is hard isn't that log levels are badly designed. It's that a log level is one number and you're asking a question about one request.

Once you see it that way the options get clearer. Ring buffers if your failures throw. A TurboFilter or equivalent if you're on the JVM and want to build it yourself. Config-driven rules if you want it to work across services without a deploy each time.

What I'd avoid is the default advice. "Raise the level and filter downstream" is what every one of those threads gets told, and it's the one answer that puts the filter in the most expensive place possible — after you've already paid for everything.

Also: Change log levels at runtime in Go · Change log levels at runtime in .NET