RFC - Pipeline Component Telemetry

[djaglowski]

This PR adds a RFC for normalized telemetry across all pipeline components. See #11343

edit by @mx-psi:

• Announced on #otel-collector-dev on 2024-10-23: https://cloud-native.slack.com/archives/C07CCCMRXBK/p1729705290741179
• Announced on the Collector SIG meeting from 2024-10-30

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.53%. Comparing base (68f0264) to head (3211be7).
Report is 112 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #11406      +/-   ##
==========================================
- Coverage   92.15%   91.53%   -0.63%     
==========================================
  Files         432      441       +9     
  Lines       20291    23922    +3631     
==========================================
+ Hits        18700    21896    +3196     
- Misses       1228     1650     +422     
- Partials      363      376      +13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[codeboten]

Thanks for opening this as a RFC @djaglowski!

[djaglowski]

Based on some offline feedback, I've broadened the scope of the RFC, while simultaneously clarifying that it is intended to evolve as we identify additional standards.

[jaronoff97]

a few questions, I really like this proposal overall :)

[jpkrohling]

I think this needs normative language. Setting the otel.signal for the OTLP receiver for its bootstrap operations are certainly misleading to people operating the collector who are unaware of the inner working of this specific component (ie, that it's a singleton).

So:

attributes MUST be omitted if the corresponding component instances

[djaglowski]

This is a tricky topic and I'm not sure we can be so strict. It certainly makes sense to me that e.g. logs generated while initializing the singleton should not be attributed to one signal or pipeline. However, we can still attribute metrics to a particular signal (e.g. if otlp receiver emits 10 logs and 20 metrics, do you want a count of "30 items" or "10 logs" and "20 metrics". Maybe this is a good argument for splitting the proposed metrics by signal type, e.g. produced_metrics, produced_logs, etc. This would allow those metrics to share the same set of attributes with other signals produced by the instance.

[jpkrohling]

I know this states "pipeline component telemetry", and that extensions aren't technically part of a pipeline, but it feels wrong to leave them out: otel.component.kind and .id could definitely apply to them as well.

[djaglowski]

The scope of this effort has been increased a lot already. Can we leave extensions for another proposal? Personally I don't feel I have enough expertise with extensions to author such details.

[jpkrohling]

How costly? I remember talking to someone about this in the past, and they mentioned that it's not that expensive, given that it just delegates to what already exists in protobuf:

opentelemetry-collector/pdata/ptrace/pb.go

Line 22 in 9907ba5

return pb.Size()

It would be nice to have benchmarks to have data backing this (or the other) claim. I would definitely see as very useful to have a histogram of item/batch sizes and having it as optional means that people might only find out about it when they'd benefit from having historical data in the first place.

[djaglowski]

Suggested change

	These are high cost to compute, so by default they should be disabled (and not calculated).
	These may be high cost to compute, so by default they should be disabled (and not calculated). This default setting may change in the future if it is demonstrated that the cost is generally acceptable.

How's this wording?

[jpkrohling]

This is clearly a tracing case for me :-) The rule of thumb to me is: is the information related to a particular transaction? Then it should go into a span.

[djaglowski]

Makes sense if we agree that we should capture a span for the consume function.

[jpkrohling]

Context is passed down, isn't it? We can definitely instrument the ingress part of the component, and ask components to add span links if they are messing with the context. This way, the trace for a pipeline with a batch processor would end at the batching processor, but a new trace with a span link would be created pointing to the originating batch request.

[djaglowski]

I see, so instead of closing the span when data comes out, we close it when the consume func returns?

I think the duration of the span would be meaningful only for some synchronous processors, and could be meaningful for syncronous connectors (e.g. if they create and link spans to represent the work associated with the incoming data). But what about asynchronous components? Do we accept that the span is just measuring a quick handoff to the internal state of the component? Is this going to be misleading to users?

[jpkrohling]

Some of my comments might have been discussed before, in which case, feel free to ignore me and just mark the items as resolved.

[dmitryax]

OTel SemConv guidelines require metric attributes to be namespaces so they can be placed in a global registry along with resource attributes. I don't think we can use otel for collector. It probably should be otelcol. Or otel.collector instead. However, otel. seems like kind of a restricted namespace. Not sure if we can use it or not. I think we need to run this by the otel semantic conventions SIG group before merging the RFC.

[mx-psi]

@dmitryax if you think contacting the semantic convention SIG is a requirement for this RFC to be merged, please request changes on the PR so that we follow the process

[djaglowski]

I'm fine with changing the attribute names but delaying this functionality further for an entirely separate naming process seems unnecessary. Can we really not choose our own attributes names?

[mx-psi]

This sentence from the doc linked by @dmitryax makes me hesitant about keeping the current approach:

Any additions to the otel.* namespace MUST be approved as part of OpenTelemetry specification.

[djaglowski]

This sentence from the doc linked by @dmitryax makes me hesitant about keeping the current approach:

Any additions to the otel.* namespace MUST be approved as part of OpenTelemetry specification.

I agree we can't just use the otel namespace. Can't we choose another on our own though? I like the otelcol suggestion.

[mx-psi]

otelcol works for me, it's also the one we use for metrics so it makes sense to be consistent here

[tigrannajaryan]

@mx-psi asked me to chime in to help interpret the spec.

You are reading it correctly. If you want to use otel. prefix it needs to go through semconv group.

What can you use instead? If you read the preceding paragraph the closest applicable rules seems to be this:

The name is specific to your company and may be possibly used outside the company as well. To avoid clashes with names introduced by other companies (in a distributed system that uses applications from multiple vendors) it is recommended to prefix the new name by your company's reverse domain name, e.g. com.acme.shopname.

If we consider "OpenTelemetry" the "company" then we should use "io.opentelemetry." as the org-specific prefix, e.g. "io.opentelemetry.collector." for the Collector.

However, I am not sure I like such a long prefix. "otelcol" or "otel.collector" looks much nicer to me.

I cannot find spec recommendations about when a new top-level namespace (like "otelcol") is allowed to be used. I would argue the spec should have a rule like this and the rule should be that it needs to go through semconv, just like for anything under "otel." namespace.

My recommendation is this:

• Decide which prefix you want to use: "otelcol." or "otel.collector." appear to be your top choices.
• Submit a request to get this added to semconv. We should be able to convince semconv group to accept it and I do not anticipate difficulties here.

You don't need to be blocked or wait until acceptance by semconv group, there is a ton of work that needs to happen anyway and replacing metric names should be trivial anytime before you declare them stable.

One additional thing that I think is worth thinking about: is this Collector-only telemetry or it can be applicable to SDK processors/exporters? Collector's telemetry is going to be much richer and needs to be more expressive, so I am not sure if it is beneficial to model Collector and SDK the same way. However, if we end up deciding that indeed we want to model them the same way, with SDK's use cases being virtually a subset of Collector's use cases, then "otel." prefix seems to be the most natural choice.

[djaglowski]

Thanks @tigrannajaryan. I'll open an issue to add otelcol as a top level namespace in the semantic conventions.

One additional thing that I think is worth thinking about: is this Collector-only telemetry or it can be applicable to SDK processors/exporters? Collector's telemetry is going to be much richer and needs to be more expressive, so I am not sure if it is beneficial to model Collector and SDK the same way. However, if we end up deciding that indeed we want to model them the same way, with SDK's use cases being virtually a subset of Collector's use cases, then "otel." prefix seems to be the most natural choice.

This is intended to be collector-only telemetry. It is attributed to the collector's instances of components, which are governed by a model that is not intended to be aligned with other parts of the project.

[djaglowski]

I've opened open-telemetry/semantic-conventions#1555 to propose otelcol as a new top-level namespace.