Disagg EstablishDisaggTask keeps MEET_LOCK for ~2s after large txn already committed

[JaySon-Huang]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Environment:

• TiFlash next-gen disaggregated build: ENABLE_NEXT_GEN=ON, ENABLE_NEXT_GEN_COLUMNAR=OFF
• Compute node (CN) reads via StorageDisaggregated::readThroughTiFlashWrite
• Write node (WN) handles EstablishDisaggTask and performs learner read (wait index + resolve locks)
• Storage engine is next-gen / cloud-storage-engine style TiKV (has txn_info! resolve_lock logs)

Observed production-like case (TiFlash logs in investigate.md, plus tidb.log / tikv.log):

1. A large TiDB write enters 2PC ([BIG_TXN], txnStartTS=468120244537524231, keys=43320, size≈5.6MB).
2. An MPP query starts on CN with start_ts=468120244616167462.
3. CN sends EstablishDisaggTask to WN for table scan regions.
4. WN learner read hits a local Put lock on region 91537:
   • lock_version=468120244537524231 (= write txn start_ts)
   • min_commit_ts=468120244603060470 (min_commit_ts < start_ts)
   • txn_size=2166, lock_ttl≈20126
5. WN returns error_locked; CN calls lock_resolver->resolveLocks(...) and retries EstablishDisaggTask.
6. TiKV begins resolve_lock committing transaction for the same start_ts with
   commit_ts=468120244616167447 (commit_ts < query start_ts, same physical ms).
7. TiFlash keeps retrying MEET_LOCK for ~2s (wait_cost=0ms, identical lock fields) until EstablishDisaggTask finally succeeds.

Relevant CN / WN code paths:

// StorageDisaggregatedRemote.cpp — CN resolve then always retry Establish
auto before_expired = cluster->lock_resolver->resolveLocks(
    bo,
    sender_target_mpp_task_id.gather_id.query_id.start_ts,
    locks,
    pushed);

// TODO: Use `pushed` to bypass large txn.
throw Exception(error_msg, ErrorCodes::DISAGG_ESTABLISH_RETRYABLE_ERROR);

// LearnerReadWorker.cpp — WN disagg always throws LockException to CN
if (is_wn_disagg_read && !region_locks.empty())
    throw LockException(std::move(region_locks));

// DAGStorageInterpreter.cpp — no local bypass retry for disagg task
catch (const LockException & e)
{
    if (context.getDAGContext()->is_disaggregated_task)
        throw; // let CN retry
    ...
}

2. What did you expect to see? (Required)

Once CN has resolved the lock status and TiKV has determined the write txn is already committed with commit_ts < query start_ts (i.e. the committed data is visible to this snapshot), the disagg read path should not keep failing EstablishDisaggTask on the same stale local LockCF entry for seconds.

Expected behavior options:

1. After CN resolveLocks confirms the txn is committed / cleanup is in progress, the next EstablishDisaggTask (or WN local retry) should wait for a fresh read index that covers the resolve/commit raft logs, so WN no longer sees the old LockCF entry.
2. Or, once the lock's final status is known, propagate enough information (bypass_lock_ts / resolved txn id, or equivalent) so WN can skip the already-handled lock while secondary resolve is still applying.
3. Avoid a long CN↔WN EstablishDisaggTask retry loop with wait_cost=0ms that re-observes an unchanged local lock.

In short: after “txn already committed + resolve_lock committing secondary”, disagg read should converge on learner apply / correct bypass, not spin on the same MEET_LOCK.

3. What did you see instead (Required)

Timeline (UTC, 2026-08-03)

time	component	event
06:03:37.095	TiDB / client-go	[BIG_TXN] start 2PC: txnStartTS=468120244537524231, keys=43320, puts=43320, size=5678162
~06:03:37.121	Query	MPP start_ts=468120244616167462
06:03:37.153	TiFlash WN	Learner read MEET_LOCK on region 91537; same lock fields as below
06:03:37.154+	TiFlash CN	error_locked → resolveLocks → retry EstablishDisaggTask
06:03:37.199 ~ 06:03:37.220	TiKV (CSE)	512× resolve_lock committing transaction for start_ts=468120244537524231, commit_ts=468120244616167447, request_source=external_Select (and empty)
06:03:37.221 ~ 06:03:38.736	TiFlash WN/CN	Same lock repeatedly returned; every learner read shows wait_cost=0ms and identical lock info
06:03:39.307	TiFlash CN	EstablishDisaggTask finally succeeds (error=false)

Evidence from tidb.log / tikv.log

tidb.log (client-go txnkv/transaction/2pc.go, threshold keys>10000 or size>4MB):

[BIG_TXN] keys=43320 puts=43320 size=5678162 txnStartTS=468120244537524231

→ Confirms the blocking lock belongs to a very large 2PC write (tens of thousands of secondary locks).

tikv.log (next-gen / cloud-storage-engine resolve_lock.rs via txn_info!):

resolve_lock committing transaction
  start_ts=468120244537524231
  commit_ts=468120244616167447
  min_commit_ts=468120244603060470
  use_async_commit=false
  lock_type=Put ttl=20126 for_update_ts=468120244537524231
  request_source=external_Select   # Select-path lock resolver (matches CN resolveLocks)

Important ts relation (same physical ms):

• write commit_ts logical = 23
• query start_ts logical = 38
• ⇒ commit_ts < start_ts ⇒ committed data is visible to this query

So this is not primarily “alive large txn waiting for TTL / pushed bypass”.
It is: txn already committed; secondary locks are being committed by resolve_lock; TiFlash learner still observes the old LockCF and keeps retrying.

Why MVCC still reports the lock on WN (correct locally)

On WN LockCF check (DecodedLockCFValue::getLockInfoPtr):

• lock_version < start_ts
• min_commit_ts < start_ts
• lock type is Put (not skippable PessimisticLock/SharedLock)

Until the resolve/commit raft log is applied on the TiFlash learner (or the lock is explicitly bypassed), WN must treat it as blocking.

Why the retry is slow / inefficient (problem)

1. CN resolveLocks triggers TiKV secondary commit (request_source=external_Select), but EstablishDisaggTask retries do not effectively wait for learner to catch the new resolve/commit index (wait_cost=0ms repeatedly).
2. Disagg WN path always rethrows LockException to CN; no local bypass / short-wait retry like non-disagg MPP/batch-cop (Optimize lock handling in MPP local read #10992 does not apply to is_disaggregated_task).
3. Large txn (43320 keys) means secondary resolve/apply can take noticeable time; during that window CN↔WN spins on the same local lock for ~2s.

Notes on related PRs:

• #10986: invalidate stale read-index cache after local lock is found. Relevant to “resolve already committing on TiKV, but learner reused old read index and did not wait for resolve logs”. Worth verifying whether this branch includes it; it addresses part of this failure mode.
• #10992: local bypass-lock retry for MPP/batch-cop. Explicitly rethrows on is_disaggregated_task, so this CN↔WN EstablishDisaggTask loop is unchanged.
• CN TODO: Use pushed to bypass large txn helps the “txn still alive / not expired” case; for this incident (already committed, commit_ts < start_ts), the missing piece is mainly wait for resolve apply / avoid stale read-index reuse, not pushed TTL wait.

4. What is your TiFlash version? (Required)

tidb: v8.5.4-nextgen.202510.24
tikv: v8.5.4-nextgen.202510.31
tiflash: v8.5.4-nextgen.202510.13

• ENABLE_NEXT_GEN=ON
• ENABLE_NEXT_GEN_COLUMNAR=OFF (reads use readThroughTiFlashWrite, not columnar path)

[JaySon-Huang]

What #10986 helps / does not help (disagg, ENABLE_NEXT_GEN=ON, ENABLE_NEXT_GEN_COLUMNAR=OFF)

Context

On this path, CN calls EstablishDisaggTask → WN does learner read (read-index + wait-index + LockCF check). On MEET_LOCK, WN returns LockException; CN runs resolveLocks on TiKV and retries Establish.
#10986 invalidates the WN read-index history (and query-level cache) when a local LockCF lock is found, so the next retry fetches a fresh read index instead of reusing a stale one.

What #10986 helps

It helps the case:

1. WN waited to read index X and still saw a local lock.
2. CN resolveLocks commits/cleans the lock on the TiKV leader.
3. The resolve/commit raft log is not yet applied on the TiFlash learner (or applied index has not advanced past that log).
4. Without Invalidate read-index cache when local lock is found #10986, the next Establish reuses history read index X → wait_cost≈0 (already applied >= X) → does not wait for the newer resolve log → hits the same LockCF entry again → retry loop.
   With Invalidate read-index cache when local lock is found #10986, the next retry requests a newer read index and waits until the learner has applied the resolve/commit log (assuming apply catches up), which can stop that empty retry loop sooner.
   Impact: lower EstablishDisaggTask / resolveLocks retry amplification and better latency when the lock is already resolved on TiKV but the learner was behind.

What #10986 does not help

1. Applied already past the resolve log
   LockCF is checked against the current in-memory LockCF after wait-index, not a snapshot at index X. If applied already includes the resolve/commit, the lock is gone even with a stale history X. Stale history alone does not make the lock “reappear”.
2. Txn still alive / secondary resolve still in flight
   If TiKV has not finished committing/cleaning secondaries (e.g. large txn with tens of thousands of keys), a fresh read index still sees the lock until LockCF is actually cleared on the learner. Invalidate read-index cache when local lock is found #10986 does not clear locks or speed up TiKV secondary resolve.
3. pushed / bypass for not-yet-expired large txns
   Disagg CN still has TODO: Use pushed to bypass large txn and does not pass bypass_lock_ts into the next Establish. Invalidate read-index cache when local lock is found #10986 does not implement that.
4. Columnar path (ENABLE_NEXT_GEN_COLUMNAR=ON + use_columnar)
   Columnar does not use WN EstablishDisaggTask + TiFlash learner LockCF for this check. Invalidate read-index cache when local lock is found #10986’s stale read-index issue is largely irrelevant there.

Relation to issue #11013

In #11013, TiKV logs showed resolve_lock committing transaction with commit_ts < query start_ts (txn already committed / visible), while TiFlash kept MEET_LOCK with wait_cost=0ms for ~2s. #10986 is relevant to the “resolve on leader, learner not waited to new index” slice of that, but it is not a complete fix for large-txn secondary resolve latency or missing pushed bypass on the disagg path.