Skip to content

fix(falkordb): serialize rebalance config handoff - #3269

Draft
weicao wants to merge 1 commit into
fix/falkordb-switchover-timeoutfrom
fix/falkordb-rebalance-config-race
Draft

fix(falkordb): serialize rebalance config handoff#3269
weicao wants to merge 1 commit into
fix/falkordb-switchover-timeoutfrom
fix/falkordb-rebalance-config-race

Conversation

@weicao

@weicao weicao commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

What this fixes

The rebalance config producer and ape-dts consumer are regular containers, so Kubernetes may start the consumer before /tmp/reshard.ini exists. A fresh runtime run observed that ordering once; three bounded retries on the retained cluster succeeded, so this change is scoped to the demonstrated startup race.

The producer cannot be converted to an init container because the KubeBlocks action workload injects the selected Pod metadata and credentials into regular podSpec.containers.

Changes

  • write the generated config to /tmp/reshard.ini.tmp and atomically publish it with mv
  • make the consumer wait up to 120 seconds for a non-empty final config
  • fail with an explicit diagnostic on timeout
  • exec the original /ape-dts process so its exit status remains authoritative
  • add a rendered structural contract test covering container placement/order, shared storage, exact image/commands, atomic publication, bounded timeout, explicit failure, and final exec ordering

Validation

  • focused ShellSpec: 1 example, 0 failures
  • FalkorDB ShellSpec: 44 examples, 0 failures, 4 pre-existing Bash 3.2 skips
  • helm lint addons/falkordb
  • full FalkorDB chart render
  • ShellCheck for the new spec and rendered consumer script
  • independent focused re-review: NO BLOCKER

This PR is intentionally stacked on PR #3265. After #3265 merges, it can be retargeted to main without changing this commit.

Fixes #3268

@weicao
weicao requested review from a team and leon-ape as code owners July 26, 2026 10:02
@codecov-commenter

codecov-commenter commented Jul 26, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 51 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (b8da48a) to head (4a7c7a8).

Files with missing lines Patch % Lines
...-ut-spec/rebalance_config_handoff_contract_spec.sh 0.00% 51 Missing ⚠️
Additional details and impacted files
@@                       Coverage Diff                       @@
##           fix/falkordb-switchover-timeout   #3269   +/-   ##
===============================================================
  Coverage                             0.00%   0.00%           
===============================================================
  Files                                  164     165    +1     
  Lines                                25383   25434   +51     
===============================================================
- Misses                               25383   25434   +51     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@weicao
weicao force-pushed the fix/falkordb-rebalance-config-race branch from 4a6ba53 to 4a7c7a8 Compare July 26, 2026 10:21
@weicao

weicao commented Jul 26, 2026

Copy link
Copy Markdown
Contributor Author

Fresh focused runtime evidence for exact head 4a7c7a81ef3b89bab285aaf49276500ae50128d9:

  • run: pr3269-rebalance-r1-1832
  • natural terminal: runner exit 0, restart 0
  • result: 42 passed, 0 failed, 0 skipped
  • paths exercised: 3 -> 4 shards, custom rebalance, and 4 -> 3 shards
  • each operation reached Succeed; slot maps were clean; key checks were base 80/80 and out 40/40; graph checks were >= 20
  • evidence archive SHA-256: 32f6703a32247fa43556362ec96579d6c6a4616e25e25879ac6e2764d18196e9
  • ordinary cleanup reached NotFound with no force/finalizer edits, and the original OpsDefinition spec was restored

Evidence boundary: this is one scoped focused runtime PASS for the replacement head. It is not a whole-suite or release-readiness claim. A fresh full run has started separately and is not yet terminal.

@leon-ape
leon-ape marked this pull request as draft July 27, 2026 01:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants