Dynamic sampler decision based on instrument

[goldyliang]

I am looking for the way to build a dynamic sampler which can decide sampling (sending to exporter) the current span or not based on a condition. The condition can not be judged at the entrance of the span but has to be judged while a certain method is hit based on the arguments of that method.

Our use case is to only sample “slow” requests. The condition of “slow” depends on the duration of the request and also how much data has been transferred during it which is available as a method argument at the entrance of a certain method.

I am thinking an instrument module as extension instead of the sampler would work, but can the sampling flag be altered at the point of method instruments?

[goldyliang]

Or can I create an extension for exporter which inherits an existing one, but add the logic of go/no go based on inspection on the span's data?

[trask]

hi @goldyliang! The agent and SDK use "head-based" sampling, making their decision at the start of each span, when the span duration is not available yet. You may want to look into tail-sampling provided by the OpenTelemetry Collector. Or alternatively you could possibly implement some kind of tail-sampling in the SpanProcessor pipeline for your exporter depending on your needs.

[goldyliang]

Thanks for the idea.

Would it be possible to just update the "shouldSample" flag in customized instrument for current span? That would be satisfying.

[trask]

Would it be possible to just update the "shouldSample" flag in customized instrument for current span?

since the sampling decision is made at the start of the span, updating something in the middle of the span will not affect the sampling

[goldyliang]

Isn't the shouldSample flag to indicate whether the collected grace will be stored (sampled) in the collector or not? The agent will still always collect the trace, but just that it will override the decision of exporting the log to sample or not, at some point during the instrumented code.

[trask]

hey @goldyliang, the sampler is called before starting each span. if the sampler returns DROP, then a dummy span is returned from startSpan and no data is captured on it.