Distributed tracing via OpenTelemetry

[dlorenc]

Overview

Add OpenTelemetry (OTel) distributed tracing so requests can be traced end-to-end across HTTP, DB, policy evaluation, and blob store layers. Split from #104.

Spans to instrument

• Incoming HTTP requests (root span per request, with trace ID propagated to logs)
• DB queries / transactions (child spans for Commit, Merge, Rebase, Purge, etc.)
• OPA policy evaluation
• Blob store operations (Put, Get, Delete)

Implementation

• Add go.opentelemetry.io/otel and OTLP gRPC exporter (go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc)
• Initialize tracer provider in cmd/docstore/main.go; shut down gracefully on exit
• Wrap HTTP handler with otelhttp.NewHandler for automatic request spans
• Add otel.Tracer calls at key DB and policy entry points
• Propagate trace_id into slog log lines (inject from span context into logger)

Configuration

DOCSTORE_OTEL_ENDPOINT=http://collector:4317   # empty = tracing disabled (default)
DOCSTORE_OTEL_SERVICE_NAME=docstore
DOCSTORE_OTEL_SAMPLE_RATE=1.0                  # 1.0 = 100%, reduce in production

Notes

• When DOCSTORE_OTEL_ENDPOINT is unset, the tracer should be a no-op — zero overhead and no dependency on a collector
• trace_id injected into slog output enables log/trace correlation in Cloud Logging / Grafana
• GCP users can point at the Cloud Trace OTLP endpoint directly