Cache instrumentation (internal/cortex/chunk/cache) emits span events with empty name via OTel bridge, causing OTLP export rejection by strict backends #8938

Description

internal/cortex/chunk/cache/instrumented.go (vendored from Cortex) uses
the OpenTracing API to log cache fetch/store activity onto the active span:

// Store
sp := ot.SpanFromContext(ctx)
sp.LogFields(otlog.Int("keys", len(keys)))

// Fetch
sp := ot.SpanFromContext(ctx)
sp.LogFields(otlog.Int("keys requested", len(keys)))
...
sp.LogFields(otlog.Int("keys found", len(found)), otlog.Int("keys missing", len(keys)-len(found)))

https://github.com/thanos-io/thanos/blob/main/internal/cortex/chunk/cache/instrumented.go#L78

Thanos wires OpenTracing call sites like this one through
go.opentelemetry.io/otel/bridge/opentracing
(pkg/tracing/migration/bridge.go), so these LogFields calls end up
producing real OTel span events when tracing is exported via OTLP.

The bridge's LogFields implementation always creates the OTel event with
an empty name:

func (s *bridgeSpan) LogFields(fields ...otlog.Field) {
	s.otelSpan.AddEvent("", trace.WithAttributes(otLogFieldsToOTelAttrs(fields)...))
}

(filed upstream as open-telemetry/opentelemetry-go#8643)

The OTLP spec documents Span.Event.name as "semantically required to be
set to non-empty string"
(https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/trace/v1/trace.proto#L220-L228).
Backends that enforce this — e.g. AWS OpenSearch Ingestion Service /
OpenSearch Data Prepper's otel_trace_source — reject the entire OTLP
export request when any event in the batch has an empty name:

org.opensearch.dataprepper.model.trace.DefaultSpanEvent:
checkArgument(!builder.name.isEmpty(), "name cannot be an empty string");

Because the backend rejects the whole gRPC request on this validation
error, one thanos-query-frontend span carrying a fifocache.fetch /
fifocache.store / results_cache event (any span produced by this cache
instrumentation) causes the OTel collector's export to fail with HTTP 400,
gets marked Permanent/non-retryable, and every span in that batch is
dropped — not just the offending one.

Real-world impact observed

Running thanos-query-frontend with tracing exported via an OTel collector
to AWS OSIS, we saw sustained HTTP 400 export failures
(not retryable error: ... responded with HTTP Status Code 400) and large
dropped_items counts on the trace exporter. Root-caused by deploying an
unmasked OSS Data Prepper instance and capturing the real (otherwise
AWS-masked) error:

Failed to parse request with error 'name cannot be an empty string'.

Every rejected span's events{} blocks had attributes (e.g.
"keys requested", "keys found", "keys missing") but no name field
set — consistent with the LogFields call sites in
internal/cortex/chunk/cache/instrumented.go.

What I expected to happen

Cache instrumentation spans/events export successfully via OTLP to any
spec-compliant backend.

What happened instead

Traces from thanos-query-frontend (and any other Thanos component using
this cache instrumentation with tracing enabled) get silently dropped in
bulk by backends that validate OTLP's non-empty Event.name requirement.

Root cause

Upstream bug in go.opentelemetry.io/otel/bridge/opentracing's
LogFields/LogKV, tracked at
open-telemetry/opentelemetry-go#8643 — the bridge
always emits AddEvent("", ...) regardless of the fields logged, instead of
honoring the OpenTracing "event" field-naming convention.

Possible workaround on the Thanos side

Until the bridge is fixed upstream, consider one of:

• Avoid sp.LogFields(...) in internal/cortex/chunk/cache/instrumented.go
  in favor of sp.LogKV("event", "<name>", ...) (if/once the bridge honors
  the "event" key), or
• Use native OTel span events with explicit names for this instrumentation
  path instead of routing through the OpenTracing bridge, or
• Document this as a known limitation until
  bridge/opentracing: LogFields/LogKV always emit AddEvent with an empty name, violating OTLP's "semantically required" non-empty Event.name open-telemetry/opentelemetry-go#8643 is resolved.

Environment

• thanos-io/thanos (traced via internal/cortex/chunk/cache/instrumented.go)
• go.opentelemetry.io/otel/bridge/opentracing v1.36.0 (per go.mod)
• Export target: AWS OpenSearch Ingestion Service / OpenSearch Data Prepper
  otel_trace_source, but any spec-compliant OTLP consumer would reject
  the same payload.