[service] Internal counter metrics now have a .filtered suffix instead _filtered for Prometheus

[marcinsiennicki95]

Component(s)

service Telemetry

What happened?

Describe the bug

Hi after upgrade from version 0.118 to version 0.120 we found out that name of metrics has changed

Steps to reproduce

Upgrade from 0.118 to 0.120

What did you expect to see?

Names should be the same after the upgrade

What did you see instead?

It looks like all metrics with suffix filtered has been modified. Last separator changed from _ to .
`

The metric names for version 0.120 can be found at this link. It is unclear why they are currently displayed with underscores (_) and what the correct naming convention should be

Also we don't see these metrics in official metrics
https://opentelemetry.io/docs/collector/internal-telemetry/

Collector version

0.120

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

telemetry:
  logs:
    level: {{ $.Values.global.logLevel }}
    encoding: json
  metrics:
    level: {{ $.Values.global.metricLevel }}
    readers:
      - pull:
          exporter:
            prometheus:
              host: 0.0.0.0
              port: 8888
              without_type_suffix: true
              without_units: true
              without_scope_info: true
{{- end }}

Log output

Additional context

No response

[dmathieu]

This is documented in the changelog.
https://github.com/open-telemetry/opentelemetry-collector/blob/main/CHANGELOG.md

Users who do not customize the Prometheus reader should not be impacted. The change to update the internal telemetry to use otel-go config can cause unexpected behaviour
for end users. This change is caused by the default values in config being different from what the Collector has used in previous versions. The
following changes can occur when users configure their service::telemetry::metrics::readers:

• the metric name will append a _total suffix if without_type_suffix is not configured. Set without_type_suffix to true to disable this.
• units will be appended to metric name if without_units is not configured. Set without_units to true to disable this.
• a target_info metric will be emitted if without_scope_info is not configured. Set without_scope_info to true to disable this.

[marcinsiennicki95]

I added the proposed fix code, which resolved the problem with _total, but it did not resolve the issue of renaming the metric _filtered to .filtered

without_type_suffix: true
without_units: true
without_scope_info: true

This change is not reflected in the changelog, and I would like to confirm the final state to determine whether it is a bug or the intended behavior. I need to update the dashboard if is desired state

What is the final metric name

0.120.1

otelcol_processor_filter_datapoints.filtered
otelcol_processor_filter_logs.filtered
otelcol_processor_filter_spans.filtered

or
0.118

otelcol_processor_filter_datapoints_filtered
otelcol_processor_filter_logs_filtered
otelcol_processor_filter_spans_filtered

[marcinsiennicki95]

Should I move it to the contrib repo if it is directly related to metrics produced by components from that repo?

[songy23]

@marcinsiennicki95 the underscore to dot issue is due to prometheus receiver, see https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CHANGELOG.md#v01200.

receiver/prometheus: Prometheus receiver now uses scrapers in Prometheus 3.0. (#36873) There are a number of breaking changes in Prometheus 3.0. Learn more about those changes and migration guide on https://prometheus.io/docs/prometheus/latest/migration/.

TLDR is you can add:

global:
  metric_name_validation_scheme: legacy

to your prometheus receiver, then the metric names should revert to previous formats.

And yes this should be moved to -contrib since it is a breaking change in prometheus receiver.

[marcinsiennicki95]

Thx