Skip to content

Conversation

@Drowze
Copy link

@Drowze Drowze commented Sep 3, 2025

This patch depends on the other two patches that have pull requests open:

I'll open a new PR to integrate this patch as well once the previous patches get reviewed and merged upstream.


What does this PR do?

Adds support for multiple Karafka/Waterdrop configurations, based on the topic name.

Motivation:

Sometimes it makes sense to have distributed_tracing on for some topics but not for others, example scenario:

  • in most topics, consumers process iterate through the messages and process them as usual - distributed_tracing makes sense here
  • in some topics, consumers "pre-process" the messages into smaller batches (likely their own objects, not a Karafka's MessageBatch), so they can be processed more efficiently later - distributed_tracing makes little sense here (and can actually make the spans harder to follow). In this scenario, the application should probably handle the tracing continuation itself (if so desired).

In the above scenario, a configuration could look like:

Datadog.configure do |c|
  c.tracing.instrument :karafka, distributed_tracing: true
  c.tracing.instrument :karafka, describes: "critical_topic", distributed_tracing: false
end

@Drowze Drowze force-pushed the support-karafka-configuration-per-topic-backport branch 2 times, most recently from e6bf519 to e995a61 Compare September 4, 2025 01:32
@Drowze Drowze changed the title Support different karafka/waterdrop configurations per topic backport Support different karafka/waterdrop configurations per topic Sep 4, 2025
@Drowze Drowze force-pushed the support-karafka-configuration-per-topic-backport branch from e995a61 to d1efb1c Compare September 4, 2025 10:00
@Drowze Drowze merged commit bb91a4a into Over-haul:karafka-patches Sep 4, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants