feat: drain_poll_ms config, DuckDB 1.5, clean DuckDB teardown#35
Merged
feat: drain_poll_ms config, DuckDB 1.5, clean DuckDB teardown#35
Conversation
Add drain_poll_ms to group config (default 200ms) controlling how frequently flush threads check for new changes. Remove the WAL-side dirty_tables tracking and notification — flush threads now self-poll on their own interval, simplifying the architecture. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Explicitly DETACH the DuckLake database before dropping DuckDB connections in FlushWorker and snapshot consumer for clean teardown. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
YuweiXiao
force-pushed
the
worktree-fix-regressions
branch
from
fc8b648 to
9de8989
Compare
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
YuweiXiao
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Flush threads now self-poll for new changes via a configurable
drain_poll_msinterval (default 200ms), replacing the WAL-sidedirty_tablestracking and notification mechanism. This simplifies the WAL consumer path and decouples drain frequency from flush frequency.DuckDB connection teardown is cleaned up: FlushWorker and snapshot consumer now explicitly DETACH the DuckLake database before dropping connections. The
atexit/_exitworkaround for the Linux double-free crash has been removed — the proper fix is a linker-level change in pg_ducklake (relytcloud/pg_ducklake#90).DuckDB crate bumped from 1.4.3 to 1.10500.0 (DuckDB 1.5).
Changes
drain_poll_msconfig key toduckpipe.config/duckpipe.group_confignotify_table()anddirty_tablestracking from WAL consumerFlushWorker::drop()and snapshot consumerDetachOnDropatexit+_exitworkaround (superseded by fix: hide internal C++ symbols on Linux to prevent double-free at exit pg_ducklake#90)duckdbcrate to=1.10500.0group_configregression test resilient to new config keysTest plan
cargo fmt --checkpassesmake installcheck— all regression tests pass🤖 Generated with Claude Code