ZooKeeper dual-active sync can miss watch-triggered reconciliation and delete tombstone safety

[weicao]

Problem

PR #504 fixes ZooKeeper dual-active registry sync risks in ape-dts.

The old implementation relied on scan-style synchronization and did not have enough protection for the production dual-active cases we reviewed:

1. Large path trees make pure periodic scans too expensive and too slow for timely synchronization.
2. Delete handling must not only remove the business znode. It also needs an ordered tombstone shadow so stale data cannot recreate a deleted node.
3. When the ZooKeeper session is disconnected or uncertain, reconciliation must not create false delete tombstones from partial reads.

Test scenario

The constructed scenario is two ZooKeeper 3.9 instances running as source and target, with ape-dts tasks syncing in dual-active style.

Validation covers:

• watch-triggered create/update synchronization;
• delete tombstone shadow with full metadata;
• initial full reconciliation after restart;
• session disconnect detection through a TCP proxy;
• CI and dt-tests compile compatibility on the PR head.

Reproducibility

This is reproducible with the IDC2 ZooKeeper pair and the PR branch used by #504.

Observed endpoints in the test record:

• source: 192.168.10.41:32347
• target: 192.168.10.41:31188

Reproduction steps

1. Start source and target ZooKeeper instances.
2. Start ape-dts ZooKeeper sync task from source to target with the PR feat: add ZooKeeper dual-active registry sync connector #504 branch.
3. Create or update /app/* znodes on the source and verify target receives them through watch-triggered sync.
4. Delete /app/service-c on the source and verify target removes the business znode and writes deleted=true tombstone metadata under /__ape_dts_shadow/app/service-c.
5. Stop the dt-main process, write more data on the source, restart dt-main, and verify initial full reconciliation brings target data back into consistency.
6. Put a TCP proxy between dt-main and source ZooKeeper, stop the proxy long enough for session uncertainty/expiry, and verify logs show session_uncertain protection and skipped periodic reconciliation while the session is uncertain.

Evidence

• PR: feat: add ZooKeeper dual-active registry sync connector #504
• PR head: c1b4ea3a7555d1007ae9a32510507fb6f3274169
• IDC2 runtime evidence in Slock thread: #ape-dts:a19dc224, msgs b36cd66b, a200fc80, 3a8849af
• GitHub visible checks on PR feat: add ZooKeeper dual-active registry sync connector #504: all SUCCESS as of the latest patrol check

Boundary

This issue tracks the PR-level fix and merge evidence. It is not a release-ready claim. Checkpoint recovery was SKIP in this run because the test config used Dummy resumer.