From Working to Production-Ready: Engineering grlog, a Zero-Dependency Go Logger

grlog is a structured logging library for Go I built with one deliberate constraint: zero third-party dependencies. The whole thing — typed fields, JSON formatting, file rotation, gzip compression, async pipelines — is standard library. Version 0.0.1 worked, in the way hobby code works: it logged messages, the tests passed, the demo was convincing. Then I started treating it like production infrastructure, and it fell apart in a very educational way.

A logger is the worst possible place for subtle bugs. It sits on the hottest path of every request, it's the thing you trust when everything else is on fire, and its failure modes are invisible — a lost log line doesn't crash anything, it just quietly isn't there when you need it during an incident. This post is the story of the v0.1.0 overhaul, in four acts: the concurrency bugs, the durability model, the allocation hunt, and the log/slog adapter. All code below is the actual source.

Act one: the bug hunt

The first bug was a panic with a stack trace ending in 'close of closed channel'. The setup: derive a request-scoped logger with WithContext, defer Close on it (good hygiene!), and also defer Close on the parent (also good hygiene!). In v0.0.1, deriving a logger copied the struct — so parent and child each held the same shutdown channel and each believed they owned it. Two Closes, one channel, one panic. The same ownership confusion had a quieter sibling: RemoveSink on a derived logger mutated the shared sinks slice in place, so siblings suddenly saw ghost or duplicated sinks.

Both bugs had the same root cause — multiple handles believing they owned shared state — so they got one fix: split the logger into a core and views. All mutable state lives exactly once in an unexported loggerCore; the Logger your code holds is a cheap view carrying only a core pointer and its own permanent fields.

// loggerCore holds all state shared by the Logger views derived from one
// NewLogger call. WithContext and With return lightweight views onto the same
// core, so sinks, the async queue, the worker goroutine, and the closed flag
// exist exactly once per logger.
type loggerCore struct {
    level     atomic.Int32   // Minimum log level (atomic for lock-free reads)
    sinks     []LogSink      // List of output sinks
    sinksMu   sync.RWMutex   // Protects sinks slice modification
    queue     chan LogEntry  // Buffer for async logging
    closeChan chan struct{}  // Signal channel for shutdown
    wg        sync.WaitGroup // Wait group for async worker
    closed    atomic.Bool    // Whether logger is closed (atomic)
    // ...
}

// Logger is a cheap view onto shared state.
type Logger struct {
    core          *loggerCore // Shared state (sinks, queue, worker, level)
    contextFields []Field     // Fields added to every entry; immutable per view
}

func (l *Logger) With(fields ...Field) *Logger {
    merged := make([]Field, 0, len(l.contextFields)+len(fields))
    merged = append(merged, l.contextFields...)
    merged = append(merged, fields...)
    return &Logger{core: l.core, contextFields: merged}
}

With the state in one place, teardown becomes an atomic ownership transition: whichever view calls Close first wins a CompareAndSwap, and everyone else gets a no-op. But there's a second, sneakier problem hiding in async shutdown. A goroutine can pass the closed-flag check inside log(), get descheduled, and enqueue its entry after the worker has already exited — an entry stranded in the queue forever. The fix is one humble line: after waiting for the worker, drain the queue once more.

func (l *Logger) Close() error {
    c := l.core
    if !c.closed.CompareAndSwap(false, true) {
        return nil // Already closed — safe from any view, any goroutine
    }

    // Stop async worker
    if c.async {
        close(c.closeChan)
        c.wg.Wait()
        // Catch entries enqueued in the race window between a concurrent
        // log()'s closed-check and the CAS above.
        c.drainQueue()
    }

    // ... close all sinks, aggregating errors with errors.Join
}

The sink-corruption bug got the standard treatment for read-heavy shared collections: copy-on-write. RemoveSink builds a new slice instead of shifting elements in the shared backing array, so an in-flight writeSinks holding the old slice keeps iterating safely over the old snapshot. Cheap for a list that changes a handful of times per process lifetime, and it eliminates the whole class of torn-read bugs.

Each of these fixes is pinned by a regression test named after its bug — TestClose_ParentAndChildView_NoPanic, TestRemoveSink_OnChildView_AffectsCoreConsistently, TestClose_AsyncDrain_NoLostEntries — in a dedicated file where the changelog's Fixed section and the test list mirror each other one-to-one. On top sit 23 race-detector tests, ending in TestRace_FullConcurrencyStorm, which does exactly what it sounds like: every operation the API offers, from many goroutines, simultaneously, under -race in CI.

Act two: never lose a log

Async logging is a queue, and every queue eventually fills. What happens then is not an implementation detail — it's the API's most important promise, and v0.0.1 made it silently (queue full? write synchronously, hope nobody notices the latency). v0.1.0 makes the caller choose, because the right answer genuinely depends on the service:

// dispatch routes a finished entry to the sinks, honoring async settings.
func (c *loggerCore) dispatch(entry LogEntry) {
    if !c.async || c.queue == nil {
        c.writeSinks(entry)
        return
    }

    switch c.overflow {
    case OverflowBlock:
        select {
        case c.queue <- entry:
        case <-c.closeChan:
            // Shutting down; write directly rather than blocking forever.
            c.writeSinks(entry)
        }
    case OverflowDrop:
        select {
        case c.queue <- entry:
        default:
            c.dropped.Add(1) // observable via Stats().DroppedEntries
        }
    default: // OverflowSyncFallback
        select {
        case c.queue <- entry:
        default:
            // Queue full - log synchronously to avoid loss
            c.writeSinks(entry)
        }
    }
}

SyncFallback never loses an entry but lets the caller pay for a write under load. Block preserves strict ordering at the cost of caller latency — note the closeChan case, without which a blocked caller would deadlock at shutdown. Drop never blocks and never lies about it: every discarded entry increments a counter you can scrape into metrics. The policy you'd pick for a payment service and the one you'd pick for a game server are different, and that's the point.

File rotation had two durability bugs of its own. First: backup filenames had one-second granularity, so two rotations inside the same second silently overwrote the first backup — data loss by filename collision. Second, and nastier: if the rotation's rename or reopen failed, the sink was left holding a closed file handle, and every write for the rest of the process lifetime failed. A logger that can be permanently disabled by one transient filesystem error is a logger you can't trust.

// nextBackupPath returns a backup filename that does not exist yet.
// Nanosecond timestamps plus a sequence suffix make collisions impossible
// even when rotations happen within the same instant.
func (s *FileSink) nextBackupPath() string {
    stamp := time.Now().Format("20060102_150405.000000000")
    path := filepath.Join(s.baseDir, fmt.Sprintf("%s_%s.log", s.baseFile, stamp))
    for seq := 1; ; seq++ {
        if _, err := os.Stat(path); os.IsNotExist(err) {
            return path
        }
        path = filepath.Join(s.baseDir, fmt.Sprintf("%s_%s.%d.log", s.baseFile, stamp, seq))
    }
}

// In rotate(): on any failure, reopen the original file so logging
// continues; rotation is retried on a later write.
if err := os.Rename(currentPath, backupPath); err != nil {
    reopenErr := s.reopen()
    return errors.Join(fmt.Errorf("failed to rename file: %w", err), reopenErr)
}

The same self-healing posture runs through Write itself: if rotation fails, the entry that triggered it is still written to the current file — the rotation error is reported, but no log line dies for it. And background gzip compression of rotated backups writes to a .gz.tmp file and renames only on success, so a reader listing the directory never sees a half-written archive. If any step fails, the uncompressed original is kept. Boring, paranoid, correct.

Act three: zero allocations where it counts

v0.0.1 formatted JSON the obvious way: build a map[string]interface{}, hand it to json.Marshal. That's 15 allocations and 1.4 microseconds per entry — per log line, on the hottest path in the process. The v0.1.0 numbers, from the same benchmarks on the same machine:

Three changes did almost all of the work. First, fields are typed structs instead of maps: String("user_id", id) is a plain struct literal carrying a type tag, so constructing it allocates nothing and formatting it is a switch statement instead of reflection. Second, formatting became an append: formatters implement AppendFormat(dst []byte, entry) []byte and write into a buffer the sink borrows from a sync.Pool. Third, the JSON encoder is hand-rolled — strconv.AppendInt instead of boxing into interface{}, a byte-wise string escaper instead of json.Marshal.

// bufPool recycles formatting buffers across log writes.
var bufPool = sync.Pool{
    New: func() interface{} {
        b := make([]byte, 0, 1024)
        return &b
    },
}

func putBuffer(b *[]byte) {
    // Don't hoard unusually large buffers; let them be collected.
    if cap(*b) > 1<<16 {
        return
    }
    *b = (*b)[:0]
    bufPool.Put(b)
}

func (s *WriterSink) Write(entry LogEntry) error {
    buf := getBuffer()
    defer putBuffer(buf)
    *buf = formatEntry(s.formatter, *buf, entry)

    s.mu.Lock()
    defer s.mu.Unlock()
    _, err := s.writer.Write(*buf)
    return err
}

The putBuffer guard is a detail worth stealing: one pathological 10 MB log line would otherwise park a 10 MB buffer in the pool forever. Buffers that grew beyond 64 KB are simply not returned — the pool stays small, the outlier gets collected.

Hand-rolling JSON is the part that should make you nervous — it made me nervous. The escape hatches are where the correctness lives: strings that aren't valid UTF-8 are delegated to encoding/json (which sanitizes them) instead of being emitted raw; NaN and Inf floats — not representable in JSON — are emitted as quoted strings; and a field value that fails json.Marshal degrades to its fmt %v form rather than discarding the whole entry. Every one of those cases is a regression test, because every one is a way to emit a line your log pipeline can't parse — or to lose the line entirely.

And the cheapest log line is the one below the level threshold. The trick that makes a filtered Debug call cost 2.7 nanoseconds is subtle: log() receives fields as a variadic slice, and as long as that slice is never stored, Go's escape analysis keeps it on the caller's stack. The merge into a heap slice happens only after the level check passes:

func (l *Logger) log(level LogLevel, msg string, fields ...Field) {
    c := l.core
    if level < LogLevel(c.level.Load()) {
        return // ~2.7ns and zero allocations: fields never escaped
    }
    // ...
    // Merge view fields with call-site fields. The variadic slice is copied
    // rather than stored, so it never escapes: filtered-out calls above stay
    // completely allocation-free.
    if n := len(l.contextFields) + len(fields); n > 0 {
        merged := make([]Field, 0, n)
        merged = append(merged, l.contextFields...)
        merged = append(merged, fields...)
        entry.Fields = merged
    }
    c.dispatch(entry)
}

None of these numbers live only in the README. Several benchmarks assert allocation counts — a change that accidentally puts an allocation back on the hot path doesn't ship a slower library, it fails CI. Performance claims you don't enforce are performance claims you used to have.

Act four: meeting the ecosystem where it is

Since Go 1.21, the standard library has its own structured logging front-end: log/slog. Asking people to rip out slog calls to adopt your logger is a losing pitch — and unnecessary, because slog was explicitly designed to let backends plug in. grlog ships a slog.Handler, so one line at startup reroutes every slog call in your codebase (and in your dependencies) through grlog's sinks, formatters, rotation, and async pipeline.

Implementing the interface is four methods; implementing it correctly is the actual work. slog has real semantics to honor: WithGroup must prefix the keys of every attribute added after it, groups nest recursively, WithAttrs must pre-resolve attributes at derivation time, and caller information arrives as a raw program counter on the record. grlog flattens groups into dotted keys and converts each attribute to its typed field by Kind:

// appendSlogAttr converts one slog.Attr (recursively for groups) into fields.
func appendSlogAttr(fields []Field, prefix string, a slog.Attr) []Field {
    if a.Equal(slog.Attr{}) {
        return fields
    }
    v := a.Value.Resolve()
    key := prefix + a.Key

    switch v.Kind() {
    case slog.KindGroup:
        groupPrefix := prefix
        if a.Key != "" {
            groupPrefix = key + "."
        }
        for _, ga := range v.Group() {
            fields = appendSlogAttr(fields, groupPrefix, ga)
        }
        return fields
    case slog.KindString:
        return append(fields, String(key, v.String()))
    case slog.KindInt64:
        return append(fields, Int64(key, v.Int64()))
    case slog.KindDuration:
        return append(fields, Duration(key, v.Duration()))
    // ... Uint64, Float64, Bool, Time, and Any as the fallback
    }
}

So slog.Info("login", slog.Group("user", slog.String("id", "u1"))) comes out the other side as a typed field user.id — grouped structure preserved, flat enough for any formatter. Caller info is recovered from the record's PC via runtime.CallersFrames and formatted to match grlog's own file:line:function layout, so log lines look identical whichever front-end produced them. The handler also respects the logger's level, closed state, and sampler — it's a full citizen of the pipeline, not a shim.

The lesson generalizes beyond logging: when the platform defines an interface, implementing it is a growth strategy. Nobody has to choose grlog to use grlog — they set it as the slog default and their existing code doesn't know anything changed.

What production-ready actually meant

The gap between v0.0.1 and v0.1.0 wasn't features — it was the same code held to a different standard. The lessons that carried over into everything I've built since:

• Shared state needs exactly one owner. Every concurrency bug in the hunt — the double-close panic, the corrupted sink slice, the stranded queue entries — was multiple handles believing they owned the same resource. The core-and-views split fixed all three at once.
• A fixed bug isn't fixed until its reproduction is a named, permanent test. grlog's regression file mirrors the changelog's Fixed section one-to-one; reintroduce a bug and a test named after it fails.
• Make failure policies explicit API. The overflow policy could have stayed a hidden default; forcing the choice between never-lose, strict-order, and never-block made the tradeoff the caller's — documented, observable in Stats(), honest.
• Enforce performance, don't report it. Benchmarks that assert 0 allocs/op turn 'blazing fast' from marketing into a CI gate.
• A logging failure must never disable future logging. Self-healing rotation, rate-limited error reporting, atomic compression — the logger has to be the most boring, most reliable component in the process.

One last detail I enjoy: open grlog.go and line 1 reads // File: grlog.go. Those headers are stamped and kept fresh by bark, the file-header CLI I wrote about previously — my tools maintaining my tools. grlog is MIT-licensed and on GitHub; it's v0.1.0, so read the honest caveats in the FAQ before betting a fleet on it — but every claim in this post is a test or a benchmark you can run yourself.