Dev json logs

[krezreb]

Description:

Modifies the LineMessageFormatter to support outputting of log messages as json (one json object per line). This has a number of benefits for observability:

• No more multiline logs
• Logs can now be parsed out of the box without having to write specific parsers
• End-Users' ability to poison logs is limited as all logs are properly escaped

Also adds support for customers to add custom json processing to each line (useful for example for adding APM trace and span ids in for Datadog Unified service tagging users)

Review

• Functional review done
• Potential edge cases thought about (behavior of the code with strange input, with strange internal state or possible interactions with other Matomo subsystems)
• Usability review done (is anything maybe unclear or think about anything that would cause people to reach out to support)
• Security review done
• Wording review done
• Code review done
• Tests were added if useful/possible
• Reviewed for breaking changes
• Developer changelog updated if needed
• Documentation added if needed
• Existing documentation updated if needed

Depends on plugin PR matomo-org/plugin-LogViewer#99

To test json_logs

1. Checkout the dev-json-logs branch on Matomo core
2. Modify config/config.ini.php set enable_json = true (see example below)

[log]
log_writers[] = file
log_level = DEBUG
enable_json = true
logger_file_path = tmp/logs/matomo.log

3. Tail the the log file and you should see the logs as json
4. Check the logviewer plugin index.php?module=LogViewer logs should appear correctly in the interface

To test exclude_pattens

1. Modify config/config.ini.php set one or more exclude_patterns, separated by pipes(see example below)

[log]
log_writers[] = file
log_level = DEBUG
exclude_patterns =  "Piwik\Plugin\Manager|Matomo is not set up"
enable_json = true
logger_file_path = tmp/logs/matomo.log

2. Tail the the log file and you should see the logs as json, EXCEPT FOR logs matching the exclude patterns

To test custom_function_file

1. Modify config/config.ini.php specify custom_function_file (see example below)

[log]
log_writers[] = file
log_level = DEBUG
custom_function_file = /var/www/html/config/custom.php
enable_json = true
logger_file_path = tmp/logs/matomo.log

2. Example custom.php

<?php

function userHandler(array $array) : ?array
{
        $array["foo"] = "bar";
        return $array;
};
# return a string matching the name of the above function
return 'userHandler';

2. Tail the the log file and you should see the logs as json with the above custom logic run, in the above case a key "foo" will be set to the value "bar"

[patrickli]

@krezreb please fix the PHPCS style check first. Referece: https://innocraft.atlassian.net/wiki/spaces/DE/pages/2395570181/Create+a+Github+action+to+check+if+code+formatting+is+as+per+PSR+standard

[patrickli]

@michalkleiner can you re-review this?

[michalkleiner]

Works as expected, tested the standard logging, the exclude parameters and the custom function mechanism.

@sgiehl can you think of any edge cases here?

[michalkleiner]

I wonder whether it would be possible to keep the severity, date, tag and request ID alongside the message itself for easier filtering in the LogViewer.. at the moment it obviously all goes into the JSON string and then the filtering is not working.

[patrickli]

No issues with the PR itself, but just don't name any field tag.

[krezreb]

I wonder whether it would be possible to keep the severity, date, tag and request ID alongside the message itself for easier filtering in the LogViewer.. at the moment it obviously all goes into the JSON string and then the filtering is not working.

Um in my testing this screen worked with json logging enabled. Could you share your ini settings please?

Tweak information in the LogViewer so that the fields are still filled with relevant information

[krezreb]

I wonder whether it would be possible to keep the severity, date, tag and request ID alongside the message itself for easier filtering in the LogViewer.. at the moment it obviously all goes into the JSON string and then the filtering is not working.

Um in my testing this screen worked with json logging enabled. Could you share your ini settings please?

OK so identified the problem, there's dependent changes in the LogViewer sub repo, you also need to checkout the dev-json-logs branch

[michalkleiner]

Requires a submodule update first, but otherwise this tested ok in the Log viewer area in Matomo as well as in the file output.

[sgiehl]

Introducing JSON logging sounds like a nice thing in general. But we maybe should do that in a more clean way and use more of the features we already have or monolog and/or DI allows us to.

[sgiehl]

This will actually also return true if enable_json = 0 is set in the config. Might be unexpected.

[sgiehl]

Is there a specific reason why this is handled with a custom include file rather than using a LogProcessor class and configure using that LogProcessor by pre/appending it to log.processors di config?

[sgiehl]

Have you checked if that class is ALWAYS instantiated only once per request? If that happens a second time include_once will return true, instead of the files content.

[sgiehl]

I might have preferred having a dedicated JsonMessageFormatter class (maybe inheriting from this class) and decide which class to use within the DI config. That might be more clean than having one class for all that.

[sgiehl]

Only setting the constructor parameters here actually only allows using such new "features" for file logging. Any reason why that shouldn't be available for e.g. database or screen logging?