Change Log Levels At Runtime In Go

There's a question on Stack Overflow called Change log level of Go lang slog in runtime. It's from 2023, it has 23,705 views, and it has two answers.

Twenty-three thousand people have hit that page. That's the whole story of dynamic logging in Go: everybody needs it, almost nobody finds a complete answer, and most people arrive after they've already built their logger the wrong way.

TL;DR — For one process you can reach: slog.LevelVar or zap's AtomicLevel, wired in when the logger is built — you can't retrofit it. For a fleet: stop pushing levels at processes and put them in config that every instance reads. That's what Quonfig does — log levels as config, changed in a UI, live on every instance in seconds. And for debug on one user without debug on everyone: that's its own post.

The mistake almost everyone makes first

Here's the logger setup in roughly every Go service:

logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
    Level: slog.LevelInfo,
}))

Now production is misbehaving and you want debug output. So you go looking for logger.SetLevel().

It doesn't exist. And that isn't an oversight — it's the consequence of a decision you already made, three weeks ago, on that line. slog.HandlerOptions.Level is a slog.Leveler:

type Leveler interface {
	Level() Level
}

slog.Level satisfies that interface all by itself, because a Level trivially returns itself. So when you passed slog.LevelInfo, you passed a value. It got copied into the handler. There is nothing left to mutate.

This is the single most common shape of every Go log-level question. Zap package in Go prints all logs even I set log level as Debug level (9,427 views, one answer) is the same discovery in zap's clothing. How to dynamically change log level in uber/zap logger, Is it possible to update the log level of a zap logger at runtime?, Updating level a logger without creating a new pointer — all the same person, at the same moment, in a different repo.

The fix has to happen at construction time. You can't retrofit it, which is exactly why so many people find out too late.

slog: use a LevelVar

The standard library ships the answer. slog.LevelVar is a Leveler backed by an atomic that you keep a handle to:

var logLevel = new(slog.LevelVar) // defaults to LevelInfo
 
func init() {
    logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
        Level: logLevel, // a pointer — the handler asks it on every record
    }))
    slog.SetDefault(logger)
}

Now this works, from anywhere, at any time, with no logger rebuild:

logLevel.Set(slog.LevelDebug)

The handler calls Level() on every single record, so the change is instant, and LevelVar is atomic, so it's safe to set from an HTTP handler while requests are in flight. If you want the log package's output to follow along, Go 1.22 added slog.SetLogLoggerLevel.

zap: use an AtomicLevel

zap has had this longer than slog has existed, and the same trap applies — you have to opt in at construction.

atom := zap.NewAtomicLevelAt(zap.InfoLevel)
 
cfg := zap.NewProductionConfig()
cfg.Level = atom
logger, _ := cfg.Build()
 
// later, from anywhere:
atom.SetLevel(zap.DebugLevel)

The trap people hit: logger.WithOptions() returns a clone. That's the root of zap #591 — you change a child logger's level and nothing appears to happen. AtomicLevel sidesteps this because every clone shares the same level pointer. zap even ships an HTTP handler for free:

// GET  /loglevel -> {"level":"info"}
// PUT  /loglevel -d '{"level":"debug"}'
http.Handle("/loglevel", atom)

That's the complete answer — for one process

Honestly. If you run one instance and you can reach it, LevelVar plus an HTTP endpoint is forty lines with no dependencies, and you should stop reading and go implement it.

Here's where it stops being complete:

Your PUT /loglevel reached exactly one pod out of twelve, and the request you're trying to debug went to one of the other eleven. When the autoscaler adds a thirteenth, it boots at INFO no matter what you've curled. You can script the fan-out — but now you've built a mutating production endpoint that needs auth and a line item in the next security review, and nothing ever turns it back off. Every real incident ends with DEBUG left on over a weekend, discovered on next month's observability bill.

Stop pushing levels — read them from config

The fix for all of it is one move: stop pushing a level at each process, and have every process read its level from config. Change the config once; every instance converges — including the ones that don't exist yet.

This is what we built Quonfig for, and we're opinionated about the shape:

• One log-level config per service. log-level.checkout is the checkout service's level. It lives alongside your feature flags, not in a YAML file per pod.
• A default per environment. DEBUG in dev, INFO in staging, WARN in prod — set once, visible in one place.
• A change is data, not a deploy. Flipping prod to DEBUG is an edit in the UI: audited, attributable, and live on every instance in seconds over SSE. Turning it back off is the same edit — no endpoint to secure, no fan-out script, no pod that missed the memo.

One config per Go service, a level per environment. This list is the entire operational surface — there's nothing to SSH into.

In code, it drops into the exact slot your LevelVar occupied:

client, err := quonfig.NewClient(
    quonfig.WithSdkKey(os.Getenv("QUONFIG_BACKEND_SDK_KEY")),
    quonfig.WithLoggerKey("log-level.checkout"),
)
if err != nil {
    log.Fatal(err)
}
defer client.Close()
 
logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
    Level: quonfig.NewQuonfigLeveler(client, "checkout"), // a Leveler, like LevelVar
}))

QuonfigLeveler implements slog.Leveler, so it's asked on every record — same contract as LevelVar, except the answer comes from the config instead of a local variable. (There's also a NewQuonfigHandler for rules that depend on the request — more on that below.)

Want to start free? It's a folder of JSON

The same SDK (github.com/quonfig/sdk-go, MIT-licensed) can read the config from a directory instead of an API — quonfig.WithDataDir("./quonfig-workspace") instead of WithSdkKey, no account needed. Your levels become files in your repo:

// quonfig-workspace/log-levels/log-level.checkout.json
{
  "key": "log-level.checkout",
  "type": "log_level",
  "valueType": "log_level",
  "default": {
    "rules": [
      { "criteria": [{ "operator": "ALWAYS_TRUE" }], "value": { "type": "log_level", "value": "WARN" } }
    ]
  },
  "environments": []
}

The files ship inside your image or ConfigMap, so a level change is a commit and a deploy — reviewable, diffable, and when the deploy lands, that's it: every instance, new pods included, is logging at the new level. Don't undersell that. It fixes the construction-time trap and the drifting fleet, and for plenty of teams a clean deploy is the whole answer.

If you also need realtime — a level change that lands mid-incident, without a deploy — then the directory has to update on running instances. It's just git, so you can build that yourself: sync the repo onto your instances (a git-sync sidecar works) and add quonfig.WithDataDirAutoReload(true) so the SDK hot-reloads when the files change. Or use the service, which is essentially just SSE streaming of a datadir: same file format, same code, swap WithDataDir for WithSdkKey. Your choice.

Debug for one user, not everyone

There's one thing the PUT /loglevel endpoint can never do, no matter how well you script it: at production traffic, DEBUG for everyone is unaffordable, and the request you care about is one in ten thousand. Because a Quonfig log level is a rule list evaluated per record — with request context — "dana@northwind.example gets DEBUG, everyone else stays at WARN" is one rule in the UI. That's the difference between debugging in production and not being able to afford to.

Wiring the request context through slog — and why it requires Handler.Enabled(ctx, …) rather than a Leveler — is its own post: How to turn on debug logging for one user in production.

Which answer do you need?

• One instance, ten minutes of debug output. slog.LevelVar plus an HTTP endpoint. Don't over-engineer it.
• A fleet. Move the levels into config the instances read. JSON files and WithDataDir are free and give you a clean deploy-to-change workflow — and if you want realtime too, git-sync plus WithDataDirAutoReload gets you there on your own. The service is for when you'd rather not run that loop yourself: a UI, an audit trail, and changes streaming to every instance in seconds.
• Debug for one customer. Rules on request context — read this next.

Whichever you pick: set it up before you need it. Every question linked in this post is someone who needed the level changed right now and discovered the decision was made when the logger was built. slog.LevelVar costs you one line today and saves you a deploy during an incident.

Related: How to turn on debug logging for one user in production · Change log levels at runtime in .NET