Component(s)

receiver/prometheus

Describe the issue you're reporting

Problem Statement : The current default behaviour of [receiver/prometheus] is to set reset points by using the first scrape time and subsequently observing value resets. This default behaviour almost definitely causes incorrectly injected reset points since it's entirely stateful to the collector process and not the scraped targets themselves. This can in theory cause wildly inaccurate rate/increase calculations in any otel backends (or oss-prom) if STs are used a part of rate calculations.

Can we either 1) remove the default initial point adjustment behaviour or 2) move the receiver.prometheusreceiver.RemoveStartTimeAdjustment feature flag to StageBeta?

I'd prefer 2) as much of the receiver's built-in adjustment logic has also been ported over to processor/metricstartttime which exposes explicit configuration ergonomics to the user on how adjustments should be made. I've also some ideas on improvements to the metricsstarttime processor (link/write-up tbd).

appendix

Note: promreceiver uses the following methods to infer resets (by setting ST) for each metricGroup, in order of priority:

• Negotiate scrape exposition format to one of: OpenMetricsText or PrometheusProto
  -- requires the EnableCreatedTimestampZeroIngestion feature flag to be ON
  -- both have limited default support
  -- OM scrape incurs a cpu regression

• Use the metric family’s _created gauge value to set CTs for each metricGroup in the metricFamily.
  -- Requires the UseCreatedMetric feature flag
  -- Requires scrape target to emit a _created gauge for each label set/family

• Use the scrape target’s process_start_time_seconds gauge to estimate CT
  -- Requires the RemoveStartTimeAdjustment feature flag to be OFF
  -- Requires the use_start_time_metric config to be ON
  -- Requires the scrape target to emit process_start_time_seconds gauge (or a custom gauge)

• Use the collector’s own process start time
  -- Requires the UseCollectorStartTime feature flag to be ON and RemoveStartTimeAdjustment feature flag to be OFF
  -- Requires the use_start_time_metric config to be ON

• Use the first scrape time of metric
  -- RemoveStartTimeAdjustment feature flag to be OFF

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}