Skip to content

fix(mariadb): recover guarded publication after crash - #3281

Open
weicao wants to merge 12 commits into
helen/mariadb-r27-reviewed-15-head-candidatefrom
helen/mariadb-pre-dcs-secondary-defer
Open

fix(mariadb): recover guarded publication after crash#3281
weicao wants to merge 12 commits into
helen/mariadb-r27-reviewed-15-head-candidatefrom
helen/mariadb-pre-dcs-secondary-defer

Conversation

@weicao

@weicao weicao commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Problem

Independent review found a crash boundary where the publication handler could die after deleting the durable guard but before deleting its shared lock or returning a terminal response.

The paired addon then saw a stale lock and an ambiguous transport result. The old recovery performed strongest-fence rollback, but could not reconstruct whether publication had committed and left guard=absent + publication_lock=present. The exact control returned rc=3.

This is an observed code-path failure, not preventive-only hardening.

Solution

The addon now gives each primary-accept transaction an exact durable token and treats the paired syncer's atomic guard→receipt rename as the only commit fact.

  • Before opening any writer plane, it atomically persists the exact token in .primary-write-accept-pending.
  • It clears only a previous lifecycle's committed receipt while holding the caller commit lock.
  • A nominal CLI success without an exact durable receipt is rejected.
  • An ambiguous response is accepted only when the guard is absent and .primary-write-publication-committed contains the same exact token.
  • That exact receipt authorizes idempotent cleanup of the stale publication lock and convergence to committed terminal.
  • Missing, damaged, or mismatched receipt never migrates a result from another transaction. After the bounded live-handler deadline, recovery restores the current exact guard, applies the strongest writer fence, and returns nonzero.
  • Fresh-container lifecycle cleanup removes stale lock owners and the previous committed receipt; same-process live owners remain protected by the existing two-phase lifecycle journal.

The existing guard-first retry remains in place: the runtime enters the authoritative commit/publish transaction before public role lookup and never falls through to the old role-based writer-open path.

Deterministic evidence

The old crash control failed with guard absent + stale lock + no receipt. The new exact-receipt control converges with rc=0, exact token preserved, stale lock removed, and no strongest rollback.

A separate negative control seeds a mismatched receipt. It proves the addon does not accept it, restores the current transaction's exact guard, leaves the stale lock for lifecycle cleanup, applies strongest rollback, and returns rc=3.

Exact identity

  • head: 83524394698ba6c425b26062eb700c5af22e32f2
  • tree: 52ba9bb63c851a92144f32ac7ce85af2fbfa231a
  • direct parent: 17aabaf9dc809c5dbbbafe575acf31ba97773106
  • base: 50ea795d9ca167b2b022a7b3df4108aeb548b802

Paired syncer PR #376 head: c350875d087e7fe50346e637073aa6648916ce90.

Methodology docs PR #1287 head: aead9d14138581dd8db1770716835065ed102e29.

Validation and boundary

Local exact-head validation is green:

  • exact publication controls: 25 examples, 0 failures;
  • primary-write linearization controls: 10 examples, 0 failures;
  • full MariaDB ShellSpec: 864 examples, 0 failures, 9 pre-existing pending examples;
  • Bash parse, ShellCheck error severity, Helm dependency/lint, and git diff --check.

Paired syncer full, race, vet, focused controls, and diff-check are green locally.

GitHub CI is terminal for both exact heads. Addon head 83524394698ba6c425b26062eb700c5af22e32f2 is CLEAN with 7 SUCCESS, 0 pending, 0 red. Paired syncer head c350875d087e7fe50346e637073aa6648916ce90 has 7 executed checks SUCCESS, 0 pending, 0 red, plus the trigger-only check-image job SKIPPED by design; its unchanged retry cleared the sole pre-source install pcregrep environment failure.

The previous R16 scene remains preserved. Test has made zero package/import/create/START/repair/rerun/cleanup mutations for this successor. There is no runtime/product PASS or release-ready claim. A new independent review is required after the Dev-only image identity is frozen; that image is review evidence and must not be reused by Test.

Reviewer focus

Please focus on:

  1. whether only an exact current-transaction receipt can authorize committed recovery;
  2. whether missing/mismatched receipt always restores the exact guard before fail-closed rollback;
  3. whether fresh-lifecycle receipt cleanup can affect any same-process live owner.

Merge/runtime order: paired syncer PR #376 must be built and pinned before this addon transaction enters runtime validation.

@weicao
weicao requested review from a team and leon-ape as code owners July 29, 2026 05:53
@weicao
weicao requested a review from xuriwuyun July 29, 2026 05:53
@codecov-commenter

codecov-commenter commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 528 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (50ea795) to head (8352439).

Files with missing lines Patch % Lines
...ication_primary_write_commit_linearization_spec.sh 0.00% 174 Missing ⚠️
...pts-ut-spec/mariadb_image_entrypoint_retry_spec.sh 0.00% 118 Missing ⚠️
...ut-spec/replication_full_accept_user_fence_spec.sh 0.00% 97 Missing ⚠️
...ec/replication_runtime_role_promotion_race_spec.sh 0.00% 62 Missing ⚠️
...pts-ut-spec/semisync_rejoin_fence_template_spec.sh 0.00% 62 Missing ⚠️
...iadb/scripts-ut-spec/replication_roleprobe_spec.sh 0.00% 12 Missing ⚠️
...ipts-ut-spec/reconfigure_persisted_alpha86_spec.sh 0.00% 1 Missing ⚠️
...ts-ut-spec/replication_entrypoint_readonly_spec.sh 0.00% 1 Missing ⚠️
...plication_merged_semisync_startup_recovery_spec.sh 0.00% 1 Missing ⚠️
Additional details and impacted files
@@                              Coverage Diff                              @@
##           helen/mariadb-r27-reviewed-15-head-candidate   #3281    +/-   ##
=============================================================================
  Coverage                                          0.00%   0.00%            
=============================================================================
  Files                                               159     160     +1     
  Lines                                             24975   25493   +518     
=============================================================================
- Misses                                            24975   25493   +518     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@weicao weicao changed the title fix(mariadb): preserve writable bootstrap before DCS leader fix(mariadb): serialize publication rollback Jul 31, 2026
@weicao weicao changed the title fix(mariadb): serialize publication rollback fix(mariadb): make primary publication lifecycle crash-safe Jul 31, 2026
@weicao weicao changed the title fix(mariadb): make primary publication lifecycle crash-safe fix(mariadb): retry guarded primary accept transaction Jul 31, 2026
@weicao weicao changed the title fix(mariadb): retry guarded primary accept transaction fix(mariadb): recover guarded publication after crash Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants