Skip to content

refactor: use outbox for olap message flush - #4543

Open
abelanger5 wants to merge 4 commits into
mainfrom
belanger/outbox-pt-1
Open

refactor: use outbox for olap message flush#4543
abelanger5 wants to merge 4 commits into
mainfrom
belanger/outbox-pt-1

Conversation

@abelanger5

@abelanger5 abelanger5 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

(also still a WIP, description to come)

Description

Fixes # (issue)

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • Documentation change (pure documentation change)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Refactor (non-breaking changes to code which doesn't change any behaviour)
  • CI (any automation pipeline changes)
  • Chore (changes which are not directly related to any business logic)
  • Test changes (add, refactor, improve or change a test)
  • This change requires a documentation update

What's Changed

  • Add a list of tasks or features here...

Checklist

Changes have been:

  • Tested (unit, integration, or manually with steps specified)
  • Linted and formatted
  • Documented (where applicable)
  • Added to CHANGELOG (where applicable) -- see Keep a Changelog
🤖 AI Disclosure
  • I acknowledge that an LLM was used in the creation of this Pull Request, in accordance with Hatchet's AI_POLICY.md.
  • Details: [e.g. generating tests, writing docs]

@abelanger5
refactor: pub/sub separate from durable task queue
d239557
@abelanger5
address review: topic kind enum, MsgHandler rename, PubTenantMessage …
69c3092 
…helper, loader helper

- Replace Topic's tenantStream bool with a TopicKind enum
  (tenant-stream / scheduler-partition)
- Rename AckHook to MsgHandler: on the durable queue it acts as a
  pre/post-ack hook, on the pub/sub it's the sole handler with no ack
  semantics
- Add msgqueue.PubTenantMessage(ctx, l, mq, ps, queue, msg): single source of
  truth for the stream-consumed message IDs; does the durable send (when a
  queue is given) plus the tenant-stream publish (when the ID is one the
  dispatcher's streams consume), collapsing every producer call site to one
  call. The API cancellation path (handleCancelTasks) now goes through it
  too, restoring exact parity with the old implicit mirror for task-cancelled
- Extract the pub/sub construction in the loader into createPubSubV1
@abelanger5
Merge remote-tracking branch 'origin/main' into belanger/msgqueue-pub…
3c8da09 
…sub-split
@abelanger5
refactor: use outbox for olap message flush
b8c0a51
@vercel

vercel Bot commented Jul 27, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hatchet-docs Ready Ready Preview, Comment Jul 27, 2026 5:57pm

Request Review

@github-actions github-actions Bot added the engine Related to the core Hatchet engine label Jul 27, 2026
@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

⚠️ Optional test failure: The load-online-migrate job failed on this PR. This check is non-mandatory and does not block merging, but may be worth investigating. View logs

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Benchmark results

goos: linux
goarch: amd64
pkg: github.com/hatchet-dev/hatchet/internal/msgqueue/rabbitmq
cpu: AMD Ryzen 9 7950X3D 16-Core Processor          
                              │ /tmp/old.txt │           /tmp/new.txt            │
                              │    sec/op    │   sec/op     vs base              │
CompressPayloads_1x10KiB-8       79.21µ ± 2%   79.84µ ± 5%       ~ (p=0.329 n=6)
CompressPayloads_10x10KiB-8      914.4µ ± 2%   908.7µ ± 2%       ~ (p=0.180 n=6)
CompressPayloads_10x100KiB-8     11.02m ± 0%   11.08m ± 4%       ~ (p=1.000 n=6)
CompressPayloads_Concurrent-8    71.63µ ± 3%   74.96µ ± 5%       ~ (p=0.065 n=6)
geomean                          489.0µ        495.5µ       +1.32%

                              │ /tmp/old.txt │            /tmp/new.txt            │
                              │     B/op     │     B/op      vs base              │
CompressPayloads_1x10KiB-8      10.98Ki ± 1%   10.98Ki ± 1%       ~ (p=0.461 n=6)
CompressPayloads_10x10KiB-8     108.3Ki ± 1%   108.3Ki ± 2%       ~ (p=0.818 n=6)
CompressPayloads_10x100KiB-8    2.920Mi ± 0%   2.920Mi ± 0%  +0.00% (p=0.015 n=6)
CompressPayloads_Concurrent-8   54.18Ki ± 0%   54.31Ki ± 0%       ~ (p=0.394 n=6)
geomean                         117.8Ki        117.9Ki       +0.07%

                              │ /tmp/old.txt │            /tmp/new.txt            │
                              │  allocs/op   │ allocs/op   vs base                │
CompressPayloads_1x10KiB-8        5.000 ± 0%   5.000 ± 0%       ~ (p=1.000 n=6) ¹
CompressPayloads_10x10KiB-8       32.00 ± 0%   32.00 ± 0%       ~ (p=1.000 n=6) ¹
CompressPayloads_10x100KiB-8      63.00 ± 0%   63.00 ± 0%       ~ (p=1.000 n=6) ¹
CompressPayloads_Concurrent-8     17.00 ± 0%   17.00 ± 0%       ~ (p=1.000 n=6) ¹
geomean                           20.35        20.35       +0.00%
¹ all samples are equal

Compared against belanger/msgqueue-pubsub-split (3c8da09)

Base automatically changed from belanger/msgqueue-pubsub-split to main July 28, 2026 09:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

engine Related to the core Hatchet engine

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@abelanger5