feat: instrument the pub/sub layer with Prometheus histograms - #4548
Open
igor-kupczynski wants to merge 1 commit into
Open
feat: instrument the pub/sub layer with Prometheus histograms#4548igor-kupczynski wants to merge 1 commit into
igor-kupczynski wants to merge 1 commit into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
github-actions
Bot
added
documentation
7 tasks
Contributor
Benchmark resultsCompared against |
igor-kupczynski
force-pushed
the
pubsub-metrics
branch
from
July 28, 2026 13:59
6b68e64 to
6df9e5f
Compare
Adds publisher-side and end-to-end latency histograms to the pub/sub layer, for
every backend, via a decorator:
hatchet_pubsub_publish_duration_seconds{kind, topic_kind, result}
hatchet_pubsub_transit_seconds{kind, topic_kind}
Choosing between pub/sub backends needs numbers, and the kind label makes a
mixed fleet directly comparable. These need to be deployed and collecting before
any shard is switched to a new backend, so that there is a baseline to compare
against rather than only simultaneous data from a fleet that already changed.
Publish duration measures how long Pub blocks the caller, which is not broker
latency: NATS buffers the write and returns immediately while RabbitMQ does a
synchronous channel write, so the two are not comparable that way. Transit
latency needs a wire stamp, so Message grows a published_at field; it is clamped
at zero to absorb clock skew, and the buckets start at 100us so that two fast
brokers don't collapse into a single bucket.
The decorator is wrapped inside the tenant-pubs gate so that gated drops are not
counted as publishes, and it stamps a copy rather than the caller's Message,
which callers also hand to the durable queue. Mixed fleets are safe in both
directions: older engines ignore the unknown JSON field, their unstamped
messages simply skip the transit observation, and omitzero keeps published_at
off durable messages entirely.
igor-kupczynski
force-pushed
the
pubsub-metrics
branch
from
July 28, 2026 14:05
6df9e5f to
c46ee58
Compare
6 tasks
Contributor
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Wraps every backend's
msgqueue.PubSub(the interface #4480 split from the durable queue) in a decorator recording two Prometheus histograms:Backend choice needs numbers;
kindmakes a mixed fleet comparable.Stacked on #4550 (NATS pub/sub backend); review that first — this diff is instrumentation-only. #4550 is opt-in and enables nothing on merge, so this PR is the real gate: a baseline must be collecting before any shard switches to NATS.
Two caveats (also documented):
The transit stamp is a new
published_atenvelope field. Mixed fleets are safe both ways: older engines ignore the unknown JSON field, unstamped messages skip the transit observation, andomitzerokeeps the field off durable messages.Pubstamps a copy — callers reuse the same*Messagefor the durable queue. The decorator sits inside the tenant-pubs gate, soDisableTenantPubsdrops aren't counted as publishes.Type of change
What's Changed
internal/msgqueue/instrumented_pubsub.go— the decoratorpkg/integrations/metrics/prometheus/pubsub.go— histogram definitions and bucketsinternal/msgqueue/msg.go—published_atenvelope fieldpkg/config/loader/loader.go— wires the decorator for every backendfrontend/docs/pages/self-hosting/prometheus-metrics.mdx— docsChecklist
Changes have been:
Testing
Unit tests cover no mutation of the caller's
Message, transit observed only for stamped messages, and theresultlabel on success and error paths. The NATS integration suite also passes.🤖 AI Disclosure
I acknowledge that an LLM was used in the creation of this Pull Request, in accordance with Hatchet's AI_POLICY.md.
Details: Cursor agent implemented the instrumentation, tests, and docs from a reviewed plan.