feat: add NATS pub/sub backend

[igor-kupczynski]

Description

Adds NATS as a pub/sub backend, selectable per deployment via SERVER_MSGQUEUE_PUBSUB_KIND=nats + SERVER_MSGQUEUE_PUBSUB_NATS_URL. NATS implements only the best-effort pub/sub side of the split from #4480; durable queues stay on RabbitMQ/Postgres.

Core NATS, no JetStream, so delivery is at-most-once — which matches what the pub/sub path already promises.

Stacked on #4548 (pub/sub Prometheus instrumentation), so that the metrics used to evaluate this backend are already deployed and have a baseline before NATS is enabled anywhere. Review #4548 first; this PR's diff is NATS-only.

Type of change

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

What's Changed

internal/msgqueue/nats plus config, docs, and a compose service.

• Startup validation. Connect is synchronous and checks the server's advertised max_payload against the 16MiB the durable RabbitMQ path allows. Task stream events routinely exceed the NATS default of 1MiB, and a refused startup is easier to diagnose than an oversized publish failing later under load.
• Fail fast while disconnected. ReconnectBufSize(-1) means publishes error out rather than silently buffering messages that a best-effort subscriber will never wait for.
• Chunking parity with RabbitMQ. Oversized multi-payload messages are split recursively until each chunk fits under max_payload.
• Subject prefix. SERVER_MSGQUEUE_PUBSUB_NATS_SUBJECT_PREFIX (default hatchet.pubsub). The other backends namespace installations at the connection level (a RabbitMQ vhost, a Postgres database); NATS namespaces by subject, so this is the parity knob for installations sharing a NATS server.
• Auth as connect options. Username and password are passed to the client rather than embedded in the URL, so reconnects to cluster peers discovered through gossip authenticate too.

Checklist

Changes have been:

• Tested (unit, integration, or manually with steps specified)
• Linted and formatted
• Documented (where applicable)
• Added to CHANGELOG (where applicable) -- see Keep a Changelog

Testing

docker compose up -d nats
go test -tags integration ./internal/msgqueue/nats/...
go test ./internal/msgqueue/...

The integration suite covers tenant fanout to multiple subscribers, a scheduler-topic roundtrip, an 8MiB payload, oversized-message chunking, a compressed roundtrip, and a custom subject prefix (asserted against a raw NATS subscription on the expected subject).

The NATS server needs max_payload: 16777216; see hack/dev/nats-server.conf, which the compose service mounts.

Notes for reviewers

Rebased onto main after #4480 merged. #4480's review absorbed several refactors this branch originally carried, so they are no longer in the diff:

• The shared gzip decompress extraction — main moved the whole Compressor into msgqueue, going further than this branch did
• Message.CopyWithPayloads — main landed the same shallow-copy fix as Clone(), which the NATS chunking path now uses
• The rabbitmq and postgres pub/sub changes — after rebasing, those files were identical to main

Two things the rebase surfaced, both fixed here:

• A comment in the NATS Sub path claimed the durable RabbitMQ path compresses payloads in place. refactor: separate pub/sub interface from durable task queue #4480 fixed exactly that, so it is now a short note explaining that the decompress branch is defensive parity with rabbitmq/pubsub.go
• The NATS Sub path was not resetting Compressed = false after decompressing, unlike rabbitmq/pubsub.go

One thing worth a reviewer's opinion: hack/dev/nats-server.conf sets max_payload: 16777216 for RabbitMQ parity, and NATS itself warns at startup that payloads over 8MiB "are generally discouraged and could lead to poor performance". Parity is the conservative choice for a drop-in backend, but if we don't actually need >8MiB pub/sub messages it would be better to find that out before rollout.

Related

• Builds on refactor: separate pub/sub interface from durable task queue #4480 (merged) — separated the pub/sub interface from the durable task queue
• Stacked on feat: instrument the pub/sub layer with Prometheus histograms #4548 — pub/sub Prometheus instrumentation

---

🤖 AI Disclosure

• I acknowledge that an LLM was used in the creation of this Pull Request, in accordance with Hatchet's AI_POLICY.md.

• Details: Cursor agent implemented the NATS pub/sub backend, config/docs/compose wiring, and tests from a reviewed plan, and performed the rebase onto main after refactor: separate pub/sub interface from durable task queue #4480 merged.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hatchet-docs	Ready	Preview, Comment	Jul 28, 2026 2:02pm

[github-actions]

⚠️ Optional test failure: The load-online-migrate job failed on this PR. This check is non-mandatory and does not block merging, but may be worth investigating. View logs

[github-actions]

Benchmark results

goos: linux
goarch: amd64
pkg: github.com/hatchet-dev/hatchet/internal/msgqueue
cpu: AMD Ryzen 9 7950X3D 16-Core Processor          
                              │ /tmp/old.txt │            /tmp/new.txt            │
                              │    sec/op    │    sec/op     vs base              │
CompressPayloads_1x10KiB-8       76.88µ ± 1%   79.00µ ±  5%       ~ (p=0.310 n=6)
CompressPayloads_10x10KiB-8      869.2µ ± 3%   915.7µ ±  4%  +5.36% (p=0.009 n=6)
CompressPayloads_10x100KiB-8     10.23m ± 7%   11.06m ±  4%  +8.11% (p=0.015 n=6)
CompressPayloads_Concurrent-8    60.50µ ± 4%   63.43µ ± 29%       ~ (p=0.093 n=6)
geomean                          451.0µ        474.7µ        +5.25%

                              │ /tmp/old.txt │            /tmp/new.txt            │
                              │     B/op     │     B/op      vs base              │
CompressPayloads_1x10KiB-8      10.88Ki ± 2%   10.92Ki ± 1%       ~ (p=0.905 n=6)
CompressPayloads_10x10KiB-8     108.3Ki ± 1%   108.3Ki ± 1%       ~ (p=0.818 n=6)
CompressPayloads_10x100KiB-8    2.920Mi ± 0%   2.920Mi ± 0%       ~ (p=0.485 n=6)
CompressPayloads_Concurrent-8   54.25Ki ± 0%   54.35Ki ± 0%       ~ (p=0.065 n=6)
geomean                         117.6Ki        117.7Ki       +0.15%

                              │ /tmp/old.txt │            /tmp/new.txt            │
                              │  allocs/op   │ allocs/op   vs base                │
CompressPayloads_1x10KiB-8        5.000 ± 0%   5.000 ± 0%       ~ (p=1.000 n=6) ¹
CompressPayloads_10x10KiB-8       32.00 ± 0%   32.00 ± 0%       ~ (p=1.000 n=6) ¹
CompressPayloads_10x100KiB-8      63.00 ± 2%   63.00 ± 0%       ~ (p=1.000 n=6)
CompressPayloads_Concurrent-8     17.00 ± 0%   17.00 ± 0%       ~ (p=1.000 n=6) ¹
geomean                           20.35        20.35       +0.00%
¹ all samples are equal

_{Compared against main (969d558)}

[gitguardian]

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.

---

^{_{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}}

[github-actions]

⚠️ Optional test failure: The load-online-migrate job failed on this PR. This check is non-mandatory and does not block merging, but may be worth investigating. View logs

[github-actions]

⚠️ Optional test failure: The load-online-migrate job failed on this PR. This check is non-mandatory and does not block merging, but may be worth investigating. View logs

[github-actions]

⚠️ Optional test failure: The load-online-migrate job failed on this PR. This check is non-mandatory and does not block merging, but may be worth investigating. View logs

[github-actions]

⚠️ Optional test failure: The load-online-migrate job failed on this PR. This check is non-mandatory and does not block merging, but may be worth investigating. View logs

[github-actions]

⚠️ Optional test failure: The load-online-migrate job failed on this PR. This check is non-mandatory and does not block merging, but may be worth investigating. View logs

[github-actions]

⚠️ Optional test failure: The load-online-migrate job failed on this PR. This check is non-mandatory and does not block merging, but may be worth investigating. View logs

[github-actions]

⚠️ Optional test failure: The load-online-migrate job failed on this PR. This check is non-mandatory and does not block merging, but may be worth investigating. View logs

[github-actions]

⚠️ Optional test failure: The load-deadlock job failed on this PR (optimistic-scheduling=false). This check is non-mandatory and does not block merging, but may be worth investigating. View logs

[igor-kupczynski]

GitHub marked this PR as merged in error — the NATS backend has not landed on main.

While reordering the stack, this PR's base was pointed at pubsub-metrics, and that branch was then updated to include this branch's commit. GitHub saw the head commit become reachable from the base and auto-closed the PR as merged, then deleted the head branch. The mergeCommit it recorded (a4707f7cf) was never on main.

Merged PRs can't be reopened, so the work continues in #4550 — same branch, same change, based on main. The pub/sub instrumentation that was originally bundled in here is now a separate PR, #4548, stacked on top of #4550.

Nothing was lost; only the PR number changed. The review discussion here is still the earlier context.