Add deltatocumulativeprocessor for AMP destinations

[dricross]

Description of the issue

Currently, the CloudWatch Agent uses the prometheusremotewrite exporter to send metrics to Amazon Managed Prometheus. It does not support delta metrics so the metrics are dropped:

Non-cumulative monotonic, histogram, and summary OTLP metrics are dropped by this exporter.

This issue was discovered while updating the AMP integration test: aws/amazon-cloudwatch-agent-test#465 (comment)

Description of changes

To remediate, the agent will use deltatocumulativeprocessor in AMP destination pipelines to convert delta metrics to cumulative metrics before being processed by the exporter.

The processor has two settings:

• max_stale: how long until a series not receiving new samples is removed
• max_streams: upper limit of streams to track. new streams exceeding this limit will be dropped

The max_stale setting defaults to 5m which may not be ideal for all users. If users emit hourly delta metrics, for example, then the ingested datapoints will always start a new stream. The cumulativetodeltaprocessor is very similar processor which has a similar setting. By default, the processor uses infinite staleness so metric streams will never be dropped. Ideally, we'd configure the deltatocumulativeprocessor in the same manner the be consistent, but infinite staleness is not an option for the deltatocumulativeprocessor. Instead, we'll follow AWS's PutMetricData API convention of supporting metrics up two weeks:

You can specify time stamps that are as much as two weeks before the current date, and as much as 2 hours after the current day and time.

The max_streams setting defaults to 64-bit integer maximum (9223372036854775807) which is effectively infinite. This is consistent with the cumulativetodeltaprocessor which uses an unbounded map to keep track of metrics.

See related integration test update: aws/amazon-cloudwatch-agent-test#494

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Ran the AMP integration test with agent updates and then locate delta metrics in AMG:

Updated AMP integration test to check for delta counter and delta histogram metrics now that they are supported.

Requirements

Before commit the code, please do the following steps.

1. Run make fmt and make fmt-sh
2. Run make lint