[{"content":"","date":null,"permalink":"https://robinsiep.com/blog/posts/","section":"Posts","summary":"","title":"Posts"},{"content":"","date":null,"permalink":"https://robinsiep.com/blog/","section":"Robin Siep","summary":"","title":"Robin Siep"},{"content":" In my previous post I illustrated why teams struggle to produce useful Go errors and how to address this. This post will show you how to make sure your logs surface enough context in production while staying readable in your console. What to log #The first thing to figure out is what to log, which falls into roughly three categories:\nFirst there are errors. These should always be logged at the ERROR level, unless you have a good reason not to.\nBe sure to include the stack trace (using the approach from the previous post) and the relevant conditions under which the error occurred. This might get logged like this:\n{\u0026#34;@t\u0026#34;: \u0026#34;2026-03-12T03:44:57.8532799Z\u0026#34;, \u0026#34;@mt\u0026#34;: \u0026#34;Failed to get user: connection refused\u0026#34;, \u0026#34;@x\u0026#34;: \u0026#34;stacktrace..\u0026#34;, \u0026#34;@l\u0026#34;: \u0026#34;error\u0026#34;} Next are business events and important state transitions. These are generally logged at the INFO level. Again, include relevant context, most importantly who performed the action (for example, by including the user ID):\n{\u0026#34;@t\u0026#34;: \u0026#34;2026-03-12T03:44:57.8532799Z\u0026#34;, \u0026#34;@mt\u0026#34;: \u0026#34;Processing order\u0026#34;, \u0026#34;orderID\u0026#34;: 7453025715642961920, \u0026#34;userID\u0026#34;: 2047126549831876608, \u0026#34;@l\u0026#34;: \u0026#34;informational\u0026#34;} The last category is communication with other systems: your database, third-party services, and client applications. Levels here vary by importance. Inbound HTTP requests and outbound third-party calls are usually worth INFO, other communication may be too noisy for anything but DEBUG:\n{\u0026#34;@t\u0026#34;: \u0026#34;2026-03-12T03:44:57.8532799Z\u0026#34;, \u0026#34;@mt\u0026#34;: \u0026#34;POST /api/orders returned 201 in 142ms\u0026#34;, \u0026#34;method\u0026#34;: \u0026#34;POST\u0026#34;, \u0026#34;path\u0026#34;: \u0026#34;/api/orders\u0026#34;, \u0026#34;status\u0026#34;: 201, \u0026#34;duration_ms\u0026#34;: 142, \u0026#34;@l\u0026#34;: \u0026#34;debug\u0026#34;} Keeping these three categories in mind, there are a few things I see teams getting wrong when deciding what to log:\nLogging sensitive details. Examples include application secrets, user credentials, and PII. Running the production environment at the wrong log level. INFO is generally the right default. Prematurely optimising log volume. There\u0026rsquo;s a balance to be struck here, but logging too little hurts more for the vast majority of teams. Logging formats #When you\u0026rsquo;ve figured out what to log, the next step is to decide your log format(s). We\u0026rsquo;ll consider two formats: structured logs and plain text logs.\nStructured logs, like the examples in the previous section, are essential in production. As your application grows, you\u0026rsquo;ll need to filter and query your logs to reason about them — something that\u0026rsquo;s vastly easier with structured data.\nLocally, you want plain text logs like the ones shown in the screenshot below. They\u0026rsquo;re harder to query but easier to read compared to to structured logs. Colour and formatting lets you scan a running stream while you work, and a bit of pizzazz makes the dev loop more pleasant.\nAn example of debug logs in plain text format. Minimally styled, but with attribute support. One thing most people miss on this topic is to keep plain text logs running in production, alongside the structured ones. When your log ingestion pipeline breaks (and it will), the plain text logs are what you\u0026rsquo;ll have to fall back on and may even help you fix the broken pipeline.\nImplementation using slog #Below I\u0026rsquo;ll walk through how to implement a production-ready logging setup with slog following best practices. The patterns here apply regardless of your logging library; slog is just the lowest-friction choice because it\u0026rsquo;s in the standard library. Alternatives like zap work the same way, with broadly similar APIs and reportedly better performance.\nI\u0026rsquo;ll assume some familiarity with slog\u0026rsquo;s basics (but the standard library docs are a good primer if you need one).\nslog provides two handler types out of the box: JSONHandler for structured logs and TextHandler for plain text logs. They\u0026rsquo;re minimal: they won\u0026rsquo;t pick up the stack traces attached to our errors, and the text handler doesn\u0026rsquo;t provide any highlighting. We\u0026rsquo;ll be making our own versions that do provide these features.\nStructured logs using the SeqHandler #Our custom structured log handler, SeqHandler, will take a slog.Record and send it to Seq, a self-hostable log aggregation and search platform.\nThe full handler is available on gist but the most interesting parts can be found below. It shows how it converts a slog.Record into a CLEF event, and how it handles errors by extracting the stack trace into the @x field that Seq displays as exception data, and (where possible) attaching the response body of HTTP errors. Reminder: be careful with the latter from a security perspective.\nfunc (handler *SeqHandler) recordToSeqEvent(record slog.Record) (map[string]any, error) { seqLevel, err := slogLevelToSeqLevel(record.Level) if err != nil { return map[string]any{}, err } event := map[string]any{ \u0026#34;@t\u0026#34;: record.Time.Format(time.RFC3339Nano), \u0026#34;@l\u0026#34;: seqLevel, \u0026#34;@m\u0026#34;: record.Message, } eventAttrs := map[string]any{} record.Attrs(func(attr slog.Attr) bool { addAttr(eventAttrs, attr) return true }) // .. return event, nil } func addAttr(event map[string]any, attr slog.Attr) { if attr.Key == \u0026#34;err\u0026#34; { if err, ok := attr.Value.Any().(error); ok { addErrorAttrs(event, err) return } } event[attr.Key] = attr.Value.Any() } func addErrorAttrs(event map[string]any, err error) { event[\u0026#34;@x\u0026#34;] = fmt.Sprintf(\u0026#34;%+v\u0026#34;, err) httpError, ok := errors.Cause(err).(HTTPError) if ok { if responseBody, err := httpError.ReadBody(); err == nil { event[\u0026#34;respBody\u0026#34;] = responseBody } } } Possible improvements to this handler can be adding support for traces and spans1, and batching records before sending them to Seq to reduce network overhead.\nHuman readable logs using the text handler #Our custom text handler, TextHandler, wraps any given slog.Handler. I like to pair it with charmbracelet/log, which logs to your terminal in a colourful, human-readable format by default (including support for attributes) and allows for extensive customisation:\nfunc newConsoleHandler(level Level) slog.Handler { charmlogger := charmlog.NewWithOptions(os.Stdout, charmlog.Options{ ReportTimestamp: true, Level: level.Charm, }) return NewTextHandler(charmlogger) } The full text handler is available on gist. It does two main things, recursively, for each slog.Attr:\nReplaces tabs in attribute strings with four spaces, for cleaner formatting (especially when used together with charmbracelet/log). Adds the stack trace of any error instances, including the response body of HTTP errors where possible. As with the SeqHandler, this can be further improved by adding support for tracing and spans1.\nFallback to plain text in production #Earlier in the post, I argued for keeping plain text logs running in production alongside structured ones by not only running both handlers in parallel, but also falling back to text when the structured pipeline breaks, and logging the breakage itself. A small wrapper around the SeqHandler does the job:\ntype seqHandlerWithFallback struct { *seqHandler fallbackHandler slog.Handler } func (handler *seqHandlerWithFallback) Handle(ctx context.Context, record slog.Record) error { err := handler.seqHandler.Handle(ctx, record) if err != nil { seqErrRecord := slog.NewRecord(record.Time, slog.LevelError, \u0026#34;Failed to post event to Seq\u0026#34;, 1) seqErrRecord.Add(\u0026#34;err\u0026#34;, err) _ = handler.fallbackHandler.Handle(ctx, seqErrRecord) fallbackErr := handler.fallbackHandler.Handle(ctx, record) if fallbackErr != nil { return errors.Join(err, fallbackErr) } } return nil } func (handler *seqHandlerWithFallback) WithAttrs(attrs []slog.Attr) slog.Handler { return \u0026amp;seqHandlerWithFallback{ seqHandler: handler.seqHandler.WithAttrs(attrs), fallbackHandler: handler.fallbackHandler.WithAttrs(attrs), } } func (handler *seqHandlerWithFallback) WithGroup(name string) slog.Handler { return \u0026amp;seqHandlerWithFallback{ seqHandler: handler.seqHandler.WithGroup(name), fallbackHandler: handler.fallbackHandler.WithGroup(name), } } On failure, the wrapper writes two records to the fallback handler: an error record explaining that Seq is unreachable, and the original record that would otherwise have been lost if you\u0026rsquo;re running no other handlers.\nWiring it together #As discussed previously, running multiple log handlers in parallel can be beneficial to provide redundancy. slog provides the slog.NewMultiHandler function, which takes one or more handlers, for exactly this purpose. Log formats can be specified together with their levels in your application\u0026rsquo;s configuration. You can then proceed to loop through this mapping and create a new handler for each entry:\n// New creates a new slog.Logger with the given level and format. func New(formatsWithLevels map[Format]Level) (*slog.Logger, error) { var handlers []slog.Handler for format, level := range formatsWithLevels { handler, err := newHandler(format, level) if err != nil { return nil, err } handlers = append(handlers, handler) } multiHandler := slog.NewMultiHandler(handlers...) return slog.New(multiHandler), nil } A proper Enabled implementation on each handler ensures it only fires for its configured level.\nWrapping up #Errors in Go are useless without their context. Your logging implementation exists to surface that context. Get both right and your application becomes much easier to maintain.\nThis is part 2 of a series. Part 1 covers Go\u0026rsquo;s error model and why most teams get it wrong.\nTraces and spans are usually next to implement, after logs, when improving the observability of your system. OpenTelemetry has a good primer on this topic.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","date":"13 April 2026","permalink":"https://robinsiep.com/blog/posts/go-logs/","section":"Posts","summary":"","title":"Telling the story right: efficient logging in Go"},{"content":" Logs are the primary way to understand what your system is doing, and error logs most of all. But your logs can only surface what your errors carry. Due to a fundamental difference in how Go treats errors compared to most other ecosystems, the ones I see from clients\u0026rsquo; Go applications consistently carry less than they should. The case for logical traces #Unlike many other languages, errors in Go do not contain stack traces. Instead, errors are typically wrapped with context as they propagate up through the system.\nfunc greetUser(id int64) error { user, err := getUser(id) if err != nil { return fmt.Errorf(\u0026#34;unable to get user: %w\u0026#34;, err) } fmt.Printf(\u0026#34;Hi %s\u0026#34;, user.name) return nil } The error produced by the example above might look like this:\nunable to get user: failed to query users table: connection refused Compared to the stack trace that might get produced by a Java application with the same failure:\njava.sql.SQLException: Connection refused: connect at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129) at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:825) at com.mysql.cj.jdbc.ConnectionImpl.\u0026lt;init\u0026gt;(ConnectionImpl.java:448) at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241) at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198) at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677) at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:189) at com.example.db.ConnectionPool.acquire(ConnectionPool.java:87) at com.example.repository.UserRepository.findById(UserRepository.java:42) at com.example.service.UserService.getUser(UserService.java:28) at com.example.handler.GreetingHandler.greetUser(GreetingHandler.java:19) at com.example.handler.GreetingHandler$$FastClassBySpringCGLIB$$abc123.invoke(\u0026lt;generated\u0026gt;) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ... 47 more Our Go example essentially builds a logical trace. When maintained well, it has two advantages over a traditional stack trace: it\u0026rsquo;s human-readable and it\u0026rsquo;s complete even across channels, goroutines, and other handoffs that a stack trace might lose.\nGaps in your story #Maintaining this context, however, takes constant discipline from your team. Any lapse may make it difficult, if not impossible, to trace errors that surface. Take the error from the previous section: connection refused is all that\u0026rsquo;d be logged if the context had not been added.\nAnd, like comments, the context may become outdated and misleading as your application develops. A wrap message written two years ago may no longer capture enough detail to figure out why the underlying error occurred.\nIn practice, most teams I work with fail to keep up with this maintenance burden (or even realise it exists). As a result, the errors their software emits in production do not contain sufficient information to resolve their cause or sometimes even determine what went wrong.\nTraditional stack traces do not suffer from this. They do not need to be kept up to date, and for the thread where the error occurred, they\u0026rsquo;re guaranteed to be complete and exact.\nThe right approach for most teams #Logical traces and stack traces serve the same purpose: attaching context to errors so they can be understood later. Logical traces give you richer context but only when there are no gaps, making them expensive. Stack traces, on the other hand, give you cheaper context automatically and reliably.\nAlthough stack traces are not added to errors in Go by default, they\u0026rsquo;re quite simple to add yourself. This allows us to choose between the two alternatives. However, the question isn\u0026rsquo;t actually which to use, it\u0026rsquo;s how much of the expensive, context-rich kind your team can sustain. This leads me to advocate a middle-ground approach for most teams:\nBy adding stack traces whenever your application creates an error or first encounters one made by a third-party you ensure all errors in your Go application contain a stack trace with the least amount of work and cognitive load. Additional context can be added throughout when desired by the programmer, but is no longer crucial. The stack trace will always be there to fall back on.\nReturning to our earlier example: a bare connection refused error is perfectly debuggable if it carries a stack trace pointing through getUser and greetUser. Developers can still add context like failed to query users table: connection refused, but this approach doesn\u0026rsquo;t create a burden to do so just to produce useful errors.\nThe concept is simple enough to not require reliance on any third-party library. That said, pkg/errors1 by Dave Cheney remains a well-documented implementation to achieve just this if you\u0026rsquo;d rather not roll your own.\nNext steps #Getting your errors right is half the job. The other half is surfacing that context in your logs. Read my next post to find out more.\nThis package is in maintenance mode since some of its features have been added to the standard library. However, it can be considered feature-complete and remains a great choice. If you prefer something actively maintained, cockroachdb/errors follows similar patterns.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","date":"4 April 2026","permalink":"https://robinsiep.com/blog/posts/go-errors/","section":"Posts","summary":"","title":"Go errors are a story, most teams lose the plot"},{"content":"","date":null,"permalink":"https://robinsiep.com/blog/categories/","section":"Categories","summary":"","title":"Categories"},{"content":"","date":null,"permalink":"https://robinsiep.com/blog/tags/","section":"Tags","summary":"","title":"Tags"}]