Better integration with spring structured logging

[kamilhark]

Since Spring Boot 3.4, there is a way to configure logging directly in spring in a way the output is a json document.
https://spring.io/blog/2024/08/23/structured-logging-in-spring-boot-3-4

When using default sink and writer, the entire logbook json is serialized to a string.
That string (message) is later used in structured logging to populate the message, the final log look like that:

{
  "@timestamp": "2025-06-17T13:57:03.956984242Z",
  "log": {
    "level": "TRACE",
    "logger": "org.zalando.logbook.Logbook"
  },
  "message": "\"duration\": \"267\",\n\"protocol\": \"HTTP\\/1.1\",\n\"status\": \"200\",\"type\": \"response\"",
  "ecs": {
    "version": "8.11"
  },
  ...
}

That means the content of the message will not be recognized as a json in tools like ElasticSearch not will not be indexed correctly.

The solution my team implemented is not to put the whole json as string into message, but to use MDC to populate individual fields from what logbook collected (a map) and keep the message as human readable string (which I think is how it's meant to be)

class StructuredLogbookSink(private val formatter: JsonHttpLogFormatter, private val writer: HttpLogWriter) : Sink {

    override fun write(precorrelation: Precorrelation, request: HttpRequest){
        val msg = "Request: ${request.method} ${request.requestUri}"
        val requestLogMap = formatter.prepare(precorrelation, request)

        withMDC(requestLogMap) {
            writer.write(precorrelation, msg)
        }
    }

    override fun write(correlation: Correlation, request: HttpRequest, response: HttpResponse){
        val msg = "Response: ${request.method} ${request.requestUri} - Status: ${response.status}"
        val responseLogMap = formatter.prepare(correlation, response)

        withMDC(responseLogMap) {
            writer.write(correlation, msg)
        }
    }

    private fun withMDC(map: Map<String, Any>, block: () -> Unit) {
        map.forEach { (key, value) ->
            MDC.put("logbook.$key", value.toString())
        }
        block()
        map.keys.forEach { key ->
            MDC.remove("logbook.$key")
        }
    }
}

Using that sink will result in log like that:

{
  "@timestamp": "2025-06-17T13:57:03.956984242Z",
  "log": {
    "level": "TRACE",
    "logger": "org.zalando.logbook.Logbook"
  },
  "message": "Response: GET http:\/\/localhost:8080\/api\/actuator\/health - Status: 200",
  "logbook": {
    "duration": "267",
    "protocol": "HTTP\/1.1",
    "status": "200",
    "origin": "local",
    "type": "response"
  },
  "ecs": {
    "version": "8.11"
  }
} 

What I don't like about it is that it uses JsonHttpLogFormatter to get the map of log properties even if the json is not used in the end, it's just the method that extracts properties from request/response is there.
Second it's a custom code that we will need to maintain and possible this is something more people will be interested about.

Do you think such a feature could be implemented directly in logbook / spring boot starter?

[kasmarian]

Seems like a useful feature to have in Logbook. Would you be interested in contributing?

[mikehaertl]

As this is also about integration with ElasticSearch (via ECS) I think the log output should be a bit different.

If I understand correctly the whole point of ECS is that it defines standard fields that should be used as much as possible::
https://www.elastic.co/docs/reference/ecs/ecs-field-reference

For example the HTTP related data the fields are defined here:
https://www.elastic.co/docs/reference/ecs/ecs-http

So instead of the "logbook" property I think it should rather look something like this:

"http": {
  "version": "1.1",
  "request": {
    "method": "GET",
  },
  "response": {
    "status_code": 200,
  }
}

[janar-rahumeel]

Hi here, something is cooking from my side, how I can contribute?

[mikehaertl]

Maybe as an inspiration: this is how we solved it. It's probably far from complete but it serves our purposes. We've simply used logbook together with logback-ecs-encoder.

@Log4j2
@Configuration
@RequiredArgsConstructor
class LogbookConfig {

    private final ObjectMapper objectMapper = new ObjectMapper();
    private final AuthenticationService auth;

    @Bean
    HttpLogFormatter httpLogFormatter() {
        return new CustomHttpLogFormatter();
    }

    @Bean
    HttpLogWriter httpLogWriter() {
        return new CustomHttpLogWriter();
    }

    class CustomHttpLogFormatter implements HttpLogFormatter {

        @Override
        public String format(Precorrelation precorrelation, HttpRequest request) throws IOException {

            putMdc(MdcEcsField.USER_ID, getUserId());

            putMdc(MdcEcsField.SOURCE_ADDRESS, request.getRemote());

            putMdc(MdcEcsField.HTTP_REQUEST_ID, precorrelation.getId());
            putMdc(MdcEcsField.HTTP_REQUEST_METHOD, request.getMethod());
            putMdc(MdcEcsField.HTTP_VERSION, request.getProtocolVersion());

            // Added as JSON string - requires logstash config to convert to real object
            putMdc(MdcEcsField.HTTP_REQUEST_HEADERS, objectMapper.writeValueAsString(request.getHeaders()));

            final URI uri = URI.create(RequestURI.reconstruct(request));
            putMdc(MdcEcsField.URL_FULL, uri.toString());
            putMdc(MdcEcsField.URL_DOMAIN, uri.getHost());
            putMdc(MdcEcsField.URL_PATH, uri.getPath());
            putMdc(MdcEcsField.URL_PORT, Integer.toString(uri.getPort()));
            putMdc(MdcEcsField.URL_QUERY, uri.getQuery());
            putMdc(MdcEcsField.URL_SCHEME, uri.getScheme());

            // "GET /some/url?some=param"
            final String path = uri.getQuery() == null
                ? uri.getPath()
                : uri.getPath() + "?" + uri.getQuery();

            return request.getMethod() + " " + path;
        }

        @Override
        public String format(Correlation correlation, HttpResponse response) throws IOException {

            putMdc(MdcEcsField.HTTP_RESPONSE_STATUS_CODE, Integer.toString(response.getStatus()));
            putMdc(MdcEcsField.HTTP_VERSION, response.getProtocolVersion());

            // Added as JSON string - requires logstash config to convert to real object
            putMdc(MdcEcsField.HTTP_RESPONSE_HEADERS, objectMapper.writeValueAsString(response.getHeaders()));

            putMdc(MdcEcsField.EVENT_DURATION, Long.toString(correlation.getDuration().toNanos()));
            putMdc(MdcEcsField.CUSTOM_DURATION, Long.toString(correlation.getDuration().toMillis()));

            final String path = getMdc(MdcEcsField.URL_QUERY) == null
                ? getMdc(MdcEcsField.URL_PATH)
                : getMdc(MdcEcsField.URL_PATH) + "?" + getMdc(MdcEcsField.URL_QUERY);

            // "HTTP/1.1 200 - GET /some/url?some=param"
            return "%s %d - %s %s".formatted(
                response.getProtocolVersion(),
                response.getStatus(),
                getMdc(MdcEcsField.HTTP_REQUEST_METHOD),
                path
            );
        }

        private void putMdc(MdcEcsField field, String value) {
            MDC.put(field.toString(), value);
        }

        private String getMdc(MdcEcsField field) {
            return MDC.get(field.toString());
        }

        private String getUserId() {
            UserDto user = auth.getUser();
            if (user != null && user.id() != null) {
                return user.id().toString();
            }
            return "anonymous";
        }

    }

    class CustomHttpLogWriter implements HttpLogWriter {

        @Override
        public void write(@NotNull Precorrelation precorrelation, @NotNull String request) {
            log.info(request);

            // Remove some MDC fields to not pollute other log entries too much
            // but keep those used in the response's format() above and also
            // the HTTP_REQUEST_ID. This allows to filter all log messages for a request.
            removeMdc(MdcEcsField.SOURCE_ADDRESS);
            removeMdc(MdcEcsField.HTTP_REQUEST_HEADERS);
            removeMdc(MdcEcsField.URL_FULL);
            removeMdc(MdcEcsField.URL_DOMAIN);
            removeMdc(MdcEcsField.URL_PORT);
            removeMdc(MdcEcsField.URL_SCHEME);
        }

        @Override
        public void write(@NotNull Correlation correlation, @NotNull String response) {
            log.info(response);

            MDC.clear();
        }

        private void removeMdc(MdcEcsField field) {
            MDC.remove(field.toString());
        }
    }
}

/**
 * ECS fields that can be added to MDC.
 *
 * They will be picked up by the `logback-ecs-encoder` and added to the JSON log output.
 * @see https://www.elastic.co/docs/reference/ecs/ecs-field-reference
 */
@Getter
@RequiredArgsConstructor
public enum MdcEcsField {
    ECS_VERSION("ecs.version"),

    USER_ID("user.id"),

    SOURCE_ADDRESS("source.address"),

    HTTP_VERSION("http.version"),

    HTTP_REQUEST_ID("http.request.id"),

    HTTP_REQUEST_METHOD("http.request.method"),

    HTTP_RESPONSE_STATUS_CODE("http.response.status_code"),

    // Semi-official, see https://github.com/elastic/ecs/issues/232
    HTTP_REQUEST_HEADERS("http.request.headers"),

    // Semi-official, see https://github.com/elastic/ecs/issues/232
    HTTP_RESPONSE_HEADERS("http.response.headers"),

    URL_FULL("url.full"),

    URL_DOMAIN("url.domain"),

    URL_PATH("url.path"),

    URL_PORT("url.port"),

    URL_QUERY("url.query"),

    URL_SCHEME("url.scheme"),

    EVENT_DURATION("event.duration"),

    /**
     * Duration in ms
     */
    CUSTOM_DURATION("custom.duration"),

    ;

    private final String name;

    public String toString() {
        return name;
    }
}