world-postgres: poll faster in pg-boss #463
Conversation
🦋 Changeset detectedLatest commit: f9f0a8f The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
No benchmark result files found in benchmark-results |
turns out pg-boss is not using LISTEN/NOTIFY with Postgres, which makes it a polling queue management system. This means we need to tune the polling interval. The minimum is 0.5s, so I set it like so. I believe this shows we need to drop pg-boss altogether. Signed-off-by: Gal Schlezinger <[email protected]>
Signed-off-by: Gal Schlezinger <[email protected]>
Schniz
force-pushed
the
schniz/poll-faster-in-pg-boss
branch
from
137629a to
701fdcf
Compare
|
Since LISTEN/NOTIFY was requested in 2017 but never implemented, might be worth considering Graphile Worker which uses it natively: Graphile Worker benchmarks show it's able to handle ~2.6ms avg latency while processing ~11,800 jobs/sec The public API wouldn't change since it's internal to world-postgres. However, it would be a breaking change for users already running pg-boss in production (different DB schema, existing jobs won't migrate). Would need a major version bump with migration docs. Thoughts? Would be happy to work on this |
|
Just noticed #332 introduces a QueueDriver abstraction that would make adding Graphile Worker much cleaner. Instead of replacing pg-boss entirely, it could be added as an alternative driver |
|
@alex-holovach Good thinking. I'll ping the author of #332 to see if they want to continue on their solution. In the meantime, I'll merge this PR. |
i just used pg boss because it seemed to be more widely used, but indeed a bad decision on my part: @copilot 
|
turns out pg-boss is not using LISTEN/NOTIFY, which makes it a polling
queue management system. This means we need to tune the polling interval. The minimum is 0.5s,
so I set it like so. I believe this shows we need to drop pg-boss altogether.