Consider switching weaver live-check to OTLP-based CI workflow

[cijothomas]

The current weaver integration uses a custom ToWeaverJson serializer + Docker file mount to feed telemetry into weaver. It works, but we're maintaining a custom serializer that's coupled to weaver's internal JSON format.

opentelemetry-rust-contrib recently adopted a simpler pattern: a CI workflow starts weaver as an OTLP receiver (using the upstream composite actions), runs the instrumented app with a real OTLP exporter pointing at weaver, then stops and checks findings. No custom serializers, validates the actual wire format.

Examples:

• tower instrumentation
• actix instrumentation

Unit tests stay as-is (InMemoryExporter + direct assertions). The CI workflow just adds the semconv compliance gate on top.

Might be worth considering for here too. Thoughts?

[martincostello]

I did consider it when I saw the new action, but the thing I think it lacks is the ability to run locally in the inner dev loop as part of a "normal" test. We could maybe use both for depth, but swapping means you're pushing validation right rather than left.

coupled to weaver's internal JSON format.

Is this the case? Isn't it public if it's an explicitly documented output format from the CLI?

[cijothomas]

Fair point. I think both are valid.
The closer-to-dev is totally valid scenario - I was mostly relying/recommending on InMemoryExporters for that purpose.

The otlp + weaver-live is like integration test - treat the instrumentation as a blackbox, use weaver to validate that the instrumentation produces what I advertises to be producing. Not really meant for fast-dev-loop (InMemoryExporter is best suited there IMHO - ultra fast, no deps. but it has no native weaver skills, so the person writing unit test has to manually write all assertions and keep it up-to-date)

[martincostello]

Yeah, because they're move heavyweight than in-memory and assertions, I've intentionally used them sparingly across the codebase just for belt-and-braces additional validation.