feat: per-group configuration with global config table#32
Merged
Conversation
YuweiXiao
force-pushed
the
worktree-group-level-config
branch
2 times, most recently
from
6fc4709 to
918171e
Compare
Replace 3 GUCs (flush_interval, flush_batch_threshold, max_queued_changes) and corresponding daemon CLI args with a table-based config system that supports per-group overrides. - Add `duckpipe.global_config` key-value table seeded with defaults - Add `sync_groups.config` JSONB column for per-group overrides - Resolution order: hardcoded defaults ← global_config ← per-group JSONB - New config keys: duckdb_memory_limit, duckdb_threads, flush_interval_ms, flush_batch_threshold, max_queued_changes - FlushWorker now applies SET memory_limit / SET threads from resolved config - SQL API: set_config, get_config, set_group_config, get_group_config - GroupConfig/ResolvedConfig types with serde, validation, resolution logic Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove redundant flush_batch_threshold/flush_interval_ms scalar fields from FlushCoordinator (read from resolved_config directly) - Remove dead _resolved_config param from do_flush_buffer - Add ResolvedConfig::to_group_config() and get_key() helper methods - Use helpers in get_group_config to eliminate manual field matching - Fix set_group_config race with SELECT ... FOR UPDATE - Add audit logging for set_config and set_group_config - Change default flush_interval_ms to 5000, flush_batch_threshold to 50000 - Override fast flush values in auto_start test for CI Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
max_concurrent_flushes is now part of the config table system (global_config + per-group JSONB overrides) instead of being a separate PostgreSQL GUC. This allows different sync groups to have different concurrency limits. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- group_config.out: add max_concurrent_flushes to table listing, JSON outputs, and error messages (7 rows instead of 6) - regression.conf: remove dead duckpipe.flush_interval GUC (now managed via global_config table) - daemon helpers.sh: replace removed --flush-interval/--flush-batch-threshold CLI args with SQL-based duckpipe.set_config() calls - api.rs: add max_concurrent_flushes to hardcoded valid keys error message Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
YuweiXiao
force-pushed
the
worktree-group-level-config
branch
from
6f95a2b to
64b9755
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
duckpipe.global_configkey-value table andsync_groups.configJSONB column for per-group config overridesduckdb_memory_limit,duckdb_threads,flush_interval_ms,flush_batch_threshold,max_queued_changesset_config,get_config,set_group_config,get_group_configwith key/value validationFlushWorkernow appliesSET memory_limit/SET threadsfrom resolved configflush_interval,flush_batch_threshold,max_queued_changes) and 3 daemon CLI args — replaced by config tablesGroupConfig(serde, Option fields) andResolvedConfigtypes induckpipe-coreTest plan
group_configregression test: defaults, set/get global, set/get per-group, resolution, invalid key/value errors, nonexistent group errorsmake installcheck)cargo checkcompiles cleanlycargo fmtapplied🤖 Generated with Claude Code