Dash0 OpenTelemetry Rust Tracing Reproduction

This repo contains a MRE for reproducing Dash0 connection issues with OpenTelemetry Rust via the Tracing crate over the OTLP protocol using gRPC as transport.

Note: I run OTEL via tracing, because most crates central in the Tokio ecosystem is already instrumented with tracing.

While you should probably run the OTEL Collector instead, it's weird that the traces reach the local Jaeger UI, but the same setup faces Connection Reset errors when trying to send traces to Dash0.

The app uses the tokio runtime to "simulate" a long-running, async process that mimics a web server.

Requirements

You need a Rust toolchain with Cargo installed. The Rust version is pinned by the rust-toolchain.toml file, so Cargo will automatically use the correct toolchain version when you run the app.

Reproduction

First, we'll verify that the tracing exporter works with the Jaeger backend.

# Start Jaeger
docker compose up -d

# Enable very verbose logging at trace level
export RUST_LOG=trace
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"

# Running the app produces a trace. Close the process with Ctrl+C for graceful
# shutdown and to ensure all traces are sent.
cargo run

If all goes well, the traces should be visible in the Jaeger UI at http://localhost:16686.

# Configure Dash0 transport, keeping the same protocol as before
export DASH0_TOKEN="my-token"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingress.eu-west-1.aws.dash0.com:4317"
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer $DASH0_TOKEN,Dash0-Dataset=dev"

# Produces a `Connection reset` error trace when trying to send traces to Dash0.
cargo run

Exact logs from my reproduction following the steps here can be found in the error.log file. If all went well you should have a similar output in the console.