feat(contrib/lovoo/goka): add goka integration with APM and DSM support#5029
Draft
anapsix wants to merge 1 commit into
Draft
feat(contrib/lovoo/goka): add goka integration with APM and DSM support#5029anapsix wants to merge 1 commit into
anapsix wants to merge 1 commit into
Conversation
Add a new contrib package instrumenting github.com/lovoo/goka with both Datadog APM distributed tracing and Data Streams Monitoring in a single integration. goka builds its consumer group on top of IBM/sarama but owns the sarama.ConsumerGroupHandler internally, so the contrib/IBM/sarama integration cannot attach. This package instruments through goka's public extension points instead, requiring no changes to goka: - NewTracer + WrapContext (goka.WithContextWrapper) sets the inbound DSM checkpoint and returns a context whose Emit/Loopback propagate the active consume span and a DSM outbound checkpoint into the emitted headers. - WrapCallback runs each input message inside a kafka.consume span, finishing it around the callback (panics recovered and error-tagged) and recording the DSM commit offset on success, honoring ctx.DeferCommit. - No kafka.produce span is created: goka.Context.Emit is asynchronous with no completion handle, so trace continuity is preserved by propagating the consume span through the outbound headers instead. - EmitHeaders covers the standalone goka.Emitter path. - WithService, WithDataStreams, and WithLoopSuffix options; a single header carrier satisfies both the tracer and datastreams TextMap interfaces. Register PackageLovooGoka in instrumentation/packages.go and contribIntegrations in ddtrace/tracer/option.go. Includes unit tests (carrier round-trip, DSM no-op when disabled, span parent/child linkage, error tagging, emit/loopback injection, DeferCommit, context delegation) and an INTEGRATION-gated end-to-end test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
e7c4af0 to
c6dbdd9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds a new
contrib/lovoo/gokaintegration (modulegithub.com/DataDog/dd-trace-go/contrib/lovoo/goka/v2, packagegoka) that instruments thegoka Kafka-streams library with both APM distributed
tracing and Data Streams Monitoring (DSM) in a single integration.
goka builds its consumer group on top of
IBM/saramabut owns thesarama.ConsumerGroupHandlerinternally (it passes the*Processoritself toconsumerGroup.Consume), socontrib/IBM/saramacannot attach. This package instrumentsthrough goka's public extension points instead — no changes to goka are required:
NewTracer(...)+WrapContext(register viagoka.WithContextWrapper): sets the inboundDSM
direction:incheckpoint and returns agoka.ContextwhoseEmit/Loopbackpropagatethe active consume span and a DSM
direction:outcheckpoint into the outbound headers viagoka.WithCtxEmitHeaders.WrapCallback: wraps eachgoka.Inputcallback so every input message runs inside akafka.consumespan, finished when the callback returns and error-tagged onctx.Fail(panics are recovered, tagged, and re-raised so goka still shuts the processor down).
WithContextWrapperhas no completion hook, so the callback is the only place to finish thespan. This is also where the DSM commit offset is recorded — on successful processing, or, if
the handler uses
ctx.DeferCommit, when that deferred commit succeeds.kafka.producespans.goka.Context.Emitis asynchronous and returns no handle, so aproduce span could never be finished on the emit's real completion nor tagged with an async
delivery error. Trace continuity is instead preserved by propagating the consume span through
the emitted message headers, so the downstream consumer continues the trace.
EmitHeaders: covers the standalonegoka.Emitterpath.WithService,WithDataStreams, andWithLoopSuffix(to keepLoopbackDSMcheckpoints aligned when the app changes goka's loop suffix via
goka.SetLoopSuffix).traceranddatastreamsTextMap interfaces.Follows the existing Kafka-family contrib conventions (
IBM/sarama,twmb/franz-go):instrumentation.Loadwith a newPackageLovooGoka,WithService/WithDataStreamsoptions,the base64 DSM carrier, standard messaging tags, dedicated submodule with
replace => ../../.., andmocktracer+ DSM pathway tests. RegistersPackageLovooGokain
instrumentation/packages.goandcontribIntegrationsinddtrace/tracer/option.go.No
orchestrion.yml: unlike sarama/franz-go there is no single constructor call to rewrite(a processor is assembled from a user-defined graph plus options), so the integration is an
explicit one-line opt-in.
Known limitations (inherent to goka's extension points)
kafka.consumespan is finished when the callback returns; because emits areasynchronous, an emit that fails after the callback (and shuts the partition down) is not
reflected on the span.
WrapCallback, so anInputwhose callback is notwrapped reports no committed offset to DSM.
value on produce only when the value is a raw
[]byte/string; codec-encoded values areopaque here and reported as 0.
VisitValuescallbacks, whoseTopic()is thevisit name; with DSM enabled these produce an inbound checkpoint keyed on that name.
Motivation
There is currently no way to get APM traces or DSM checkpoints from a goka processor without
hand-rolling instrumentation. Related discussion: #5028. Opening as a draft pending
maintainer agreement on the approach (per CONTRIBUTING).
Reviewer's Checklist
ok=falsebranches. Includes anINTEGRATION-gated end-to-end test.)make lint(golangci-lint, gocritic disabled for the nested module as inscripts/lint.sh) is clean.go test -race ./...passes in the new module;check-supported-config,check-modules(fix-modules→ no diff),check-format,copyright, andchecklocksall verified locally.make generateproduces no diff (the orchestrion aggregator only imports contribs that ship anorchestrion.yml).go.workentry — flagging for guild review.make fix-modulesproduces no diff.)