Skip to content

Parse epoch time from log line #683

Open

Description

@exortech

Is your feature request related to a problem? Please describe.
Currently our log files contain a timestamp in epoch time for each line. There is no option for parsing epoch time using the timestamp_format property (https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html#CloudWatch-Agent-Configuration-File-Logssection). As a result, the CloudWatch Agent is using the time of parsing as the timestamp which could be quite different from the logged timestamp. This means needing to rely on custom sorting in CloudWatch Insights and makes formatting the timestamp more challenging.

Describe the solution you'd like
A mechanism to specify that the agent should use the timestamp in the log files (via timestamp_format or other means).

We are using structured JSON logs (as hopefully many are these days), so parsing out the timestamp could be done using a JSON parser instead of timestamp format regexes.

Even just a default where if the (JSON) structured log contained a property called "timestamp" or "time" that would be used as the logged timestamp. Using the parsing time is as the default timestamp is just really imprecise.

Describe alternatives you've considered
The alternative would be to generate an ISO8601 formatted timestamp in our log files and then parse it using the CloudWatch Agent - but this feels really suboptimal as the CloudWatch Agent is just converting it back into a timestamp after parsing. It would be preferable to be able to parse and use the epoch time directly from the log file entry.

The other option is to bypass the CloudWatch Agent altogether and post LogEvents directly to CloudWatch, but this feels like overkill.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions